Smart infrastructures are becoming increasingly important, especially for IoT and Smart City applications. However, in order for them to work, efficient handling of huge and constantly accumulating amounts of data is required. With the help of advanced open source architectures, the challenge can be mastered, as can be shown using the practical example of the smart city. The prerequisite for this is extensive know-how in order to evaluate the technologies used (best-of-breed) and to link them seamlessly with one another via suitable interfaces.
Since 2018 the global data volume is growing annually by about 27 percent according to IWD 2019. In the year 2927 it will be at 200 zettabytes. This rapid increase is mainly due to the ever-increasing spread of the Internet of Things (IoT) and the „smart“ infrastructures associated with it.
The problem: the data is usually only within one can be used to a very limited extent and thus remain in the well-known silo. However, „smart“ use of data will only be possible if data from a wide variety of sources is intelligently aggregated. In practice, this often proves to be difficult. This is mainly due to the fact that data is generated by numerous and very different actors such as companies, public institutions or individuals, and that in turn various sensor interfaces and protocols are used. This makes an efficient exchange of data very complicated, if not impossible.
In the meantime, however, there are advanced architectural solutions that help smart networking of complex structures can become reality. The focus is on the open source approach, the advantages of which have now been established across the board in IT. Numerous data and IoT projects have shown that innovative software solutions based on open source bring all the key features with them: flexibility and scalability, manageable costs, open standards and thus independence from specific manufacturers.
Flexibility in particular is indispensable – especially in view of the complexity and the very dynamic development of data in most applications. An IoT platform must be able to run stably and securely despite a wide variety of requirements and technological challenges. This is exactly where the special strengths of Open Source lie.
In order to get the diverse data out of their silos and to ensure the necessary scalability and adaptability, the Building a platform requires a software architecture that consists of different open source components. Below we introduce them and how they each fulfill one of five core tasks that are essential for the smooth operation of an IoT platform:
Integration, combination and evaluation of different data streams
So that information from the different sources can be processed in real time, a central data storage point with enormous throughput is required. Kafka has established itself as a so-called distributed commit log. The solution is already in 60% of all Fortune 209 companies and is characterized, among other things, by the fact that they can easily handle enormous data streams with over 1600. Handled messages per second. What also makes the Kafka architecture so valuable for an open source platform is its high level of resiliency and scalability, which is demonstrated, for example, in automatic partition replication and the ease of adding brokers.
The mere existence of data is not sufficient to derive any benefit from them. Accordingly, a component for data integration and subsequent analysis is also required for the platform. The Pentaho Data Integration (PDI) low-code data integration platform, which is responsible for extracting, transforming and loading (ETL) processes, has proven itself as a complement to Kafka. The big advantage: Thanks to the graphical drag-and-drop interface, no programming knowledge is required to create professional ETL routes with the solution. In addition, thanks to the open source architecture, PDI can be integrated into any existing infrastructure and can be expanded as required.
Multitenant API for data clients and suppliers
A secure, open data API is a key requirement an IoT platform. It guarantees that every type of user, but also devices such as sensors, can interact with the platform. The solution that has proven itself for this task is called Kong. It is the most widely used open source API gateway in the world. Kong enables centralized management and multi-tenancy, and provides any number of service endpoint APIs that are available regardless of backend service capabilities. In addition, the gateway also scores with a high security standard. The platform uses an algorithm to identify suspicious accesses at an early stage and responds with various partially or fully automated measures.
Near-Realtime Reporting and Device Management
One of the basic requirements for the practical operation of smart platforms It is to ensure the functionality of the sensors and devices used to collect data. This requires a digital replica of the area of application that provides an overview of the use and condition of the devices. A proven open solution is ThingsBoard. The platform visualizes data and enables business users to easily develop dashboards. With the help of rule chains, ThingsBoard triggers an alarm in real time in the event of problems and forwards information to external bodies. Here, too, thanks to low-code, users often do not need in-depth programming knowledge.
So to speak, the last step in data processing is its preparation, analysis and visualization. The open Pentaho Business Analytics tool offers both operational and analytical evaluations and delivers OLAP reports as well as printable standard reports. The tool is characterized by its high level of user-friendliness and enables, among other things, self-service reporting, the definition of individual key figures and MDX-based queries.
Best Practice Smart City: Manage data streams using open source technologies
One A particularly complex challenge is the efficient use of data in connection with Smart City projects. This can involve the data of individual transport facilities such as train stations or the networking of traffic lights for the flow of traffic, which is already very demanding due to the wealth of information and their mutual dependency is. An even more ambitious goal is to cover an entire city and thus, for example, be able to control a wide variety of traffic flows (car, bike, public transport) effectively and in real time. Not only that there are huge amounts of data – on road traffic, weather and environmental conditions as well as from the administration, there are usually several data sources for each type of data. In traffic, for example, the information comes from parking garages, public transport or car and bike sharing services as well as from sensors under the roadway that record the traffic.
But that’s not all: Almost all of this data is constantly changing. The solution is to combine proven technologies into an IoT platform using a best-of-breed approach. However, the smart city is just one of countless smart use cases. The architecture presented above is, so to speak, a blueprint for all areas in which smart data integration is important in the IoT context, from smart buildings to smart production. This is made possible by a consistent open source approach for each individual component of the architecture. This results in a virtually limitless adaptability with regard to every conceivable type of IoT project.
Author: Stefan Müller, Director Big Data Analytics & IoT, it-novum GmbH
NachrichtenJetRead MoreSicherheit, and, CITY, iot, Nachrichten, open, smart, source, with