our white paper

The Leverege Platform
Technical White Paper
The Internet of Things (IoT) is the next evolution of the Internet; providing the opportunity
to collect and analyze massive amounts of new information. However, simply having access
to more information is not sufficient - businesses need to turn this tidal wave of data into
actionable insights that can be used to improve products and increase operational
efficiency. The Leverege platform is a comprehensive IoT solution that helps cut through
the noise and enables businesses to run smarter and leaner.
This Leverege Platform white paper provides insight into how Leverege ​
Reason​
™, Leverege
Vision​
™, and Leverege Curator​
™ work synergistically to solve large-scale customer data
problems for IoT applications.
The Leverege Platform | Leverege LLC ©2015 | www.leverege.com
About Leverege
Leverege develops software products that enable customers to intelligently manage and
visualize large networks of diverse sensors; providing actionable insights and spotting trends
that lead to better decision making.
Leverege is headquartered in Germantown Maryland, just outside of Washington DC, and
was founded in 2014 by successful technology entrepreneurs with deep experience in
sensor networking and “big data” system architectures. The mission-critical products built by
the Leverege team are used daily by Fortune 500 companies and Governments around the
world and span many industries including defense and aerospace, advertising, and sports
and entertainment.
1
The Leverege Platform | Leverege LLC ©2015 | www.leverege.com
Leverege Architecture Overview
The Leverege Platform is a distributed, scalable system designed to collect and analyze data
streams, enabling customers to manage and visualize large networks of sensors. Several
overarching objectives in the design of platform are:
● Deliver a complete platform that also includes APIs as well as integrated tools for
management, monitoring, and deployment.
● Create a platform that easily scales and adapts to customer needs.
● Build support for a diverse data collection environments that may be custom or
include M2M and internet messaging standards such as MQTT, CoAP, DDS, HTTP/2
and others.
● Provide a robust system capable of running across a wide range of deployment
environments that include on-premise, in a co-location facility, or via a cloud
provider.
● Provide an open architecture that allows customers and partners easy access to the
data as well as the ability to extend the platform.
This architecture allows the platform to support a wide variety of sensor and data collection
use cases, such as:
●
●
●
●
●
●
●
●
Defense and aerospace
Industrial control
Asset tracking and inventory management
Smart city initiatives such as smart parking and resource metering
Vehicle tracking and fleet applications
Logistics and remote monitoring
Infrastructure protection and perimeter security
Smart lighting and energy management
2
The Leverege Platform | Leverege LLC ©2015 | www.leverege.com
System Architecture
The Leverege Platform is a comprehensive visualization and decision support platform for
the Internet of Things (IoT) that is composed of three major component areas: ​
Reason​
™,
Vision​
™, and ​
Curator​
™. The following sections introduce the major component areas and
provide additional details about each set of components.
Leverege ​
Reason​
™
Leverege ​
Reason™ is a “big data” message processing system that supports complex event
processing (CEP), providing the ability to combine data from multiple sources to detect
patterns and quickly identify trends, threats, or important data elements. Reason supports
processing data in real time, allowing a user to take immediate action, and batch processing
for jobs that evaluate data collected in the past. Reason also enables users to create ad-hoc
jobs for answering one-time or infrequent questions. All analysis results are then available
for visualization and interaction in ​
Vision™ and are also accumulated in long term storage.
Data streams can also pass directly to Vision without any intermediary processing.
3
The Leverege Platform | Leverege LLC ©2015 | www.leverege.com
Leverege’s open architecture allows for any information flowing through Reason to be
available in near real time for Vision, utilized for real time batch processing jobs, or simply
available for reporting. It also enables customers to build their own reporting or
visualization tools on top of Reason. Each component in the system is scalable and
distributable, allowing for horizontal scaling as needed. Leverege makes it easy to increase
or decrease the number of running components in any subsystem to respond to changes in
the data volume or processing requirements. This is important as it alleviates issues around
resource bottlenecks caused by overutilization and eliminates resource waste due to
underutilization.
Ingestion and Comprehension
The ingestion and comprehension subsystems are the entry point for all data into the
system. Each data source is collected through data ingestors, which can be scaled to
consume very small or very large data sets. Ingestion and comprehension are split into
separate components to facilitate rapid data ingestion. This architecture ensures that there
is very little delay in capturing the raw data from the source into the Leverege Platform for
processing and interaction. As mentioned previously, the number of ingestors can be
quickly increased or decreased to respond appropriately to changes in source data volume.
Once the data is consumed by the ingestors, it moves through the pipeline to the
comprehension components. Each of these components is designed to understand the
ingested data such that it can route the data appropriately and place it onto the proper data
queue for downstream processing. Once data is in a data queue, it can be consumed by any
number of downstream processes, including real time processing pipelines, batch
4
The Leverege Platform | Leverege LLC ©2015 | www.leverege.com
processing, or directly by Vision without any additional data processing. Data is not limited
to a single data queue or a single processing path - the comprehension components place
data into single or multiple queues depending on the governing business logic and that data
can be consumed by both the real time processing and batch processing subsystems as well
as by Vision directly. Data queues by default are designed to retain the data for a
configurable time period, but they also support the ability to purge the data once it has been
read from the queue. All of the data can also be stored in long term storage from any
location in the data pipeline.
Real Time Processing
The real time processing components enable operational intelligence (OI), allowing
customers to take immediate action against live data feeds. Real time data processing
involves continuous evaluation of data streams, the results of which may be utilized in real
time operational dashboards or tactical displays. The Reason real time processing
subsystem consumes data from data queues and evaluates the data stream in a processing
pipeline. A processing pipeline involves one or more steps, each of which evaluate the data
and generate output or transform the data. This includes performing various activities such
as:
●
●
●
●
Correlating events or data collected from multiple devices
Applying statistical noise reduction filters
Discerning patterns in the data flow
Triggering notifications for geo-fencing, activity monitoring, environmental
monitoring, or other event analysis
Reason is also able to utilize historical data in combination with the real time data streams
to detect patterns in the data on the fly. Each relevant data point in the processing flow may
be stored or made directly available to other components in the Leverege ecosystem.
Batch Processing
Batch processing provides a facility for evaluating very large data sets over an extended
period of time in an automated fashion. This type of processing is typically thought of as
more related to traditional business intelligence (BI) mechanisms. Each of these processes
is broken into jobs that are scheduled to run across a scalable cluster. Every job or job
grouping is assigned a schedule and relative priority, which determine when the job runs
and how many, or what percentage, of resources are allocated to completing the job. Job
priority can be altered on the fly if required to meet a deadline or to free up more resources
for other jobs. All jobs can also be cancelled or paused, if needed.
5
The Leverege Platform | Leverege LLC ©2015 | www.leverege.com
The batch processing engine also supports the creation of ad-hoc jobs for those times when
a customer has a new question to answer but doesn’t require a scheduled job. Each of
these ad-hoc jobs can easily be turned into a scheduled job as well.
Data Storage
The data storage layer is a multi-model datastore used to securely house all forms of
customer data together into a single storage system that is both horizontally and vertically
scalable. It is the heart of the overall Leverege platform as this layer is used as a repository
for all raw and processed data in the system. This architecture allows Leverege to store
petabyte sized data sets composed of many different types of data and make those datum
available for quick retrieval, while replicating the data across a storage cluster to ensure
data availability. Each data type is stored in the most appropriate form as determined by
the type, use, and format of the data.
Some examples of data that can be stored in the data storage layer include:
●
●
●
●
●
Geospatial data that could be used for geo-visualization, geo-fencing, or geo-routing
Time series data that indicates energy use or temperature change over time
Geospatial and relationship / graph data that may be used for network analysis
Sensor messages that can be used to visualize trends or detect threats
Video and images files that can be tagged, correlated, and evaluated via object
detection algorithms
While the list above indicates some of the data storage types and examples of data the
storage layer supports, the real power is that these different datasets and technologies can
be used together to build comprehensive solutions to complex analysis and visualization
problems in an environment that ensures fault tolerance and blazing fast data retrieval.
Leverege ​
Vision​
™
Leverege ​
Vision™ is a portion of the Leverege
platform which customers utilize to gain insight
into their sensor networks.
Vision provides
Command and Control (C2) dashboard functionality
giving customers not only a tactical view into the
current state of their network, but also an interface
to manage and operate the entire system.
Additionally, Vision includes a robust API layer that
powers the presentation layer and can be used to
more deeply integrate with the Leverege platform.
6
The Leverege Platform | Leverege LLC ©2015 | www.leverege.com
Presentation Logic
A user interface should not be simply utilitarian - it should be human-centric, intuitive, and
easy to use. The Vision UI is an extensible web-based framework that allows customers to
quickly understand and make decisions on all of the data that is available. It is designed to
run 24x7 and can support both passive and active operational modes.
Vision allows a customer to visualize their data in a number of ways that include everything
from simple tables and detail boxes, to much more complex geospatial and time series
views, as well as visualization drill downs. The logic used in data presentation is separate
from the data itself, allowing the UI to remain responsive, lightweight, and to easily adapt to
changes in the data. This also allows Vision to support a wide variety of device form factors,
from very large, room-size tactical command centers to individual desktop displays and all
the way down to tablets and smart phones.
Each of the screens are built using a configuration driven approach. Vision utilizes display
and layout factories such that each object can be defined and then tied together into a
complex interface. Display objects can be easily reused from screen to screen and a
model-view-controller approach naturally supports the linking of events and objects among
display elements. This feature enables users to quickly navigate to the information that they
need to find or search for objects of interest across the entire interface without losing
situational awareness.
Service Layer
The Vision service layer serves as the middleware between Vision and Reason. Everything
that appears in the Vision UI passes through this layer, which facilitates the strong
separation of concerns that allows the UI to remain lightweight and unencumbered with
business logic related to serving data.
This layer enables data retrieval and
collaboration technologies, as well as
sending commands back upstream to the
devices. Every action taken in the UI is
governed by a Role Based Access Control
(RBAC) security service, providing the
granular controls customers need to ensure
that each user has the ability to perform
their job while protecting the data from
untrained or unauthorized individuals.
7
The Leverege Platform | Leverege LLC ©2015 | www.leverege.com
The Vision service layer also serves as a web-based API into the Leverege Platform. Every
datapoint or service available to the Vision UI is also available through the Services API.
Most of this functionality is provided through REST services, although endpoints that provide
streaming data feeds leverage technologies such as websockets or server sent events (SSE).
Leverege ​
Curator​
™
Every good system requires a powerful management layer and the Leverege Platform is no
exception. ​
Curator™ is our platform management solution. Just as Reason and Vision are
the C2 platform for IoT, Curator is the control platform for all of the Reason and Vision
components and infrastructure.
Curator allows customers to view the Leverege system logs and inspect the overall system
health in one place. However, it goes several steps beyond simple monitoring and
inspection. Curator allows an administrator to provision new resources, define data
governance rules, change batch job schedules, and plays a central role in registering new
components. It serves as a virtual operating system, resource manager, configuration
manager, and monitoring solution all rolled up into one cohesive set of management tools
with an intuitive UI that places all of that power at an administrator’s fingertips.
8