Projects / Big data platform for water
Big data architectures for latency, freshness and scale
Future-proofing big data storage with modern a data lakehouse architecture.
Keywords
Data engineering
Big data
Compliant clouds
IoT
Water
Operations
Introduction
NSVA manages water and waste water in Northwestern Skåne, Sweden. Apart from the daily operations of maintaining steady water flows, the company supervises the entire system with a large number of IoT sensors with the aim of preventing incidents before they occur and being able to act fast once incidents happen.
Challenge
With data sets from multiple different time eras and geographical locations in different formats relating to multiope
Goal
Develop a data platform that can ingest data in different formats from multiple systems and store and make data available to all kinds of users, including operators, business analysts, data engineers and data scientists.
Solution
A data integration platform handles incoming streams, from text data to IoT device data. It passes data to the data lake where it is stored, partitioned and made available through different interfaces, SQL being one of them. In addition, data is catalogued and made available for multiple use cases, including real time dashboards, big data analytics and off the shelf visualization software.
Apart from ingesting, storing and democratizing big data, we advice in research and innovation for new solutions within large scale water supply systems. Together with NSVA, we stay in a constant dialog with Sweden Water Research and other actors within the industry, making sure our water systems are kept safe, stable and operational.
Results
A scalable data platform ready for the massive incoming data flows of the future, where users can access big data for both real time and batch analytics needs.