Eigen engineers have deployed IT infrastructure, networks, systems and specialist applications for Oil & Gas operators in the Caspian since 2003. Since then, they have been supporting advanced operational technology systems in the region with numerous upgrades and systems and data migration projects, amidst a period of substantial technological change.
As a key element of this delivery, Eigen has supported a large Aspentech installed base of six IP.21 historians, which have been customised to support the digital oil field applications in use since 2006.
When the operator wanted to establish a data lake to enable big data analytics with real time data, the big data vendor involved quickly encountered the challenges involved in comparison with using other data types. Difficulties arose from factors including real time process data being updated continuously for more than 105,000 historian tags in the event of system and network outages, creating data gaps.
Oil & Gas operators have attempted to enter the big data and analytics space for some time, but few have been aware of the difficulties involved when dealing with traditional process historian data streams in a real and complex operational environment.
A fully qualified partner was required for specialist IP.21 deployment work, in order to address the issues that would inevitably arise, and Eigen was chosen.
Using low-cost IT offshoring companies for comparison, Eigen was also contracted based on the projected total cost of the service being lower due to a reduced delivery timescale. The data lake project was set up involving a complex journey developing and testing techniques, and improving the approach for incrementally better results as the implementation progressed.
In aiming to set up a data lake with real-time data from IP.21, the big data supplier approached Eigen to set up OPC DA (Data Access) connections to the historians. However, this created gaps in the data when the real time data was not available, due to system or network outages, because the OPC DA standard only sees live data available in real time.
Typically, outages do not affect the rest of the applications because the IP.21 historians have catch-up / back fill mechanisms that operate once the systems are restored. Gaps in the data created a big problem for the data lake however, as large chunks of valuable data were missing.
In response, Eigen trained and supported the sSupplier analysts to set up SQLPlus scripts to backfill gaps and to later capture all the data. However, this created heavy loads for the historian servers, affecting other operation-critical applications connected to the same data sources.
The decision was taken to set up an Enterprise IP.21 that would consolidate all the data from the Asset’s IP.21 servers, to decouple the data lake operations from the other operation-critical applications connected to the asset-level IP.21s. At this point, Eigen ensured that the set-up of all the different IP.21 servers were harmonised due to the existing customisations: a very careful semi-manual step-by-step process aided by cleverly designed SQLPlus scripts - and far more complex than the typical scenario.
Having completed the main implementation in 3 weeks, Eigen has been assisting the big datasupplier since 2015. The work and support to incorporate more data as new historians and facilities come online remains ongoing. By implementing innovative and novel approaches at the intersection of engineering and IT, and resolving the practical issues that typically arise, Eigen was able to resolve all the technical difficulties in setting up a truly cutting edge real time process data lake to enable true digital transformation.