Stephen Magee of AMT-SYBEX explains how to meet the scalability requirements of smart metering using non-specialist databases and hardware
When we set out on our journey to build an industry-leading meter data management solution back in 2011, the technical challenge seemed almost overwhelming. Neither the database technologies nor the hardware seemed likely to be able to cope with the requirements of capturing, storing and analysing streams of data from several million smart meters.
As a result, most companies in the energy sector were seriously concerned that nationwide smart meter deployments would force them to invest millions of pounds in clusters of enterprise-class servers and storage, just to handle the high-volume, high-velocity metering data.
At AMT-SYBEX, we recognised this as a key opportunity. If we could crack the problem of large-scale meter data management using commodity hardware, we would be able to help our clients avoid potentially enormous costs.
And that’s what we achieved. Working initially with IBM, we successfully launched a new and powerful Meter Data Management System (MDMS) called Affinity Meterflow that could support up to 100 million meters, running on a single 16-core server. Over the following years, we had successfully implemented this solution for a number of our clients.
Fast forward to 2015, and things have changed. The latest generation of hardware is many times more powerful than even the fastest enterprise servers of four years ago. And database technology has advanced too – the newest versions of relational databases such as Oracle and IBM DB2 support features such as parallelisation to boost performance, and real-time compression to minimise total data volumes.
As a result, clients recently began asking us whether it would be possible to run our Affinity Meterflow application on top of a standard relational database, rather than the more proprietary time-series database platform that was originally used to develop the solution.
Again, this posed a technical challenge that – if we could crack it – would provide some significant benefits for our clients.
We recognised that in a number of cases, our clients wanted a solution that could be managed easily by existing database administration teams. Equally, storing the data in a standard relational data-structure, rather than a proprietary time-series format, would make it easier for our clients to integrate the data with other applications, such as reporting and analytics tools.
We used our considerable expertise in developing the highest performing solutions in the market, and worked closely with Oracle, Intel, and one of our clients to design a proof-of-concept for running a ported version of Affinity Meterflow on an Oracle Exadata platform. Exadata is an all-in-one appliance that includes an Oracle relational database together with all the necessary processing, storage and networking technologies, in a single box that clients can plug in and start using immediately. It’s an attractive option, because it eliminates all the complexity of building an infrastructure piece by piece.
The benchmark results were very encouraging – running on 1/8th of a single Exadata appliance, Meterflow was easily able to support a portfolio of up to 10 million smart meters. The target was to be able to load two days’ worth of half-hourly meter readings within four hours – and we far exceeded that. In fact, our system could load 34 days’ data for 10 million meter points in 30 minutes – so it would easily support the data volumes and timeframes that UK energy companies would need. The system also demonstrated impressive response times measured in milliseconds for Meterflow queries for business users.
This is a big step forward for smart meter management – proving that specialised hardware and standard relational databases are now capable of supporting even the largest smart meter deployments. In effect, this will transform smart meter data management from a highly technical, specialist discipline into a business-as-usual activity for the IT department.
The benefits are clear, and the client who worked with us on the proof-of-concept has already commissioned us to work on a production version of the solution. We expect many other energy companies to follow suit, and we look forward to helping them cut hardware costs, simplify IT management and make their meter data more available for business users with Affinity Meterflow on an Oracle Exadata platform.