# Data Routing and Filtering

# Overview

This feature introduces a new optional system module (Edge Router) that allows for routing telemetry messages based on their payload to different local endpoints in addition to the cloud endpoint. It allows the user to configure filter queries that define the routing requirements for any specific telemetry message.

A general overview of how the Edge Router is integrated with other Edge device components appears below.

edge-device-components

# How to Use Data Routing and Filtering

Configure the Edge Router and routing rules in Type Definition.
Configure the Consumer System Module that can consume topics produced by the Edge Router.
Configure the Typed Module that can publish MQTT telemetry messages.
Deploy the Edge Proxy with types created in the previous steps.
Publish MQTT messages from the Typed Module.
Observe how the Edge Router routes messages to the desired Consumer System Module or to the cloud based on routing rules.

# Technical Overview

Connected device sends a telemetry message to the Edge Module.
Message is forwarded from the module to the MQTT broker.
The Edge Router system module receives all types of telemetry messages from all modules.
The Edge Router routes the telemetry message to the correct topic based on the filters specified in the router configuration.
The Consumer System Modules and the Edge Proxy consume all telemetry messages that correspond to their subscribed topic.

Note that the Edge Router is an optional system module and if not present, the Edge will default to its original behavior where all telemetry messages are routed directly to the Edge Proxy which forwards the telemetry messages to the cloud.

# Components

All ABB Ability™ Platform components associated with this feature are defined in this section.

# Edge Proxy

The Edge Proxy no longer forwards telemetry messages to the cloud when the Edge Router is enabled. The Edge Proxy will start the Edge Router if the Edge Router is specified in the Edge configuration (abb.ability.configuration). Once the Edge configuration is updated, all Edge Router related configurations are written to a volume by the Edge Proxy for the Edge Router to read, which will in turn update itself automatically. If the Edge Router system module is removed from the Edge configuration, the Edge Router will be killed and the Edge Proxy will return to its default behavior.

The Edge Proxy is responsible for:

Starting the Edge Router system module, if it is defined in the Edge configuration (abb.ability.configuration).
Parsing the Edge Router configuration from the Edge configuration (abb.ability.configuration), which is then written to a separate JSON configuration file (/var/ability/edge-router/config/configuration.json) and mounted to the Edge Router container.

NOTE

Edge Proxy will not forward to the Cloud any telemetry messages that match the topic pattern modules/+/events/# when the Edge Router is present in the Edge configuration abb.ability.configuration.

Edge Proxy forwards to the cloud all telemetry messages matching the topic pattern cloud/#.

# Edge Router

The Edge Router is a new optional component introduced by this feature. It is a system module that routes all telemetry messages to their desired endpoint based on the filters specified in the Router configuration. The Edge Router builds a routing table based on the Router configuration which contains all filters for all routes specified. Whenever the router configuration is changed, the Edge Router will automatically initiate a routing table update. If no filters are specified in the Router configuration, the router will default to a “catchall” case where all telemetry messages will be forwarded to the cloud (route cloud/all/# with the filter true). In order to compile a filter expression, the Edge Router must perform a syntactical validation of the filter. If the filter syntax is invalid, the filter expression defaults to false and a warning message is created in the Router logs.

The Edge Router is responsible for:

Connecting to the MQTT Broker
Subscribing to the modules/+/events/# topic
Reading the routing configuration that is provided by the Edge Proxy
Updating the routing table when the routing configuration is changed
Publishing the received telemetry messages to hot/#, warm/#, cold/# and/or cloud/# topics based on the filters specified in the routing configuration

Action	When	Then
Connect to MQTT Broker	Application is started	MQTT Broker is connected
		Connection to Broker is lost
Subscribe to `modules/+/events/#` topic	MQTT Broker is connected	Edge Router is subscribed to `modules/+/events/#` topic
Invalidate Configuration	Application is started	Configuration is invalidated and read from disk
Update routing filters	Routing configuration is changed (invalidated)	Routing table is built for routing configuration
Route Message	MQTT message is received	Message is routed to destination topics depending on routing configuration

# Type Definitions (Routing Configuration)

The routing configuration is defined in the Edge configuration (abb.ability.configuration). Because the Edge Router is a system module, its definition must be added under the systemModules section in the Edge configuration. Filters for specific routes can be grouped in order to create a clearer overview of the filter rules. A route consists of a location and a path. The location determines whether the filter is for local routing or for routing to the cloud. A path specifies the storage path under a location.

Following is a sample Type Definition for the Router configuration:

{
  "properties": {
    "systemModules": {
      "edge-router": {
        "image": {
          "description": "Edge router image",
          "isMandatory": true,
          "dataType": "string",
          "value": "abbability.azurecr.io/edge/router:1.0.0"
        },
        "configuration": {
          "temperatureFilters": {
            "edge": {
              "hot": {
                "filter": {
                  "dataType": "string",
                  "value": "variable='temperature' AND value>90"
                }
              },
              "warm": {
                "filter": {
                  "dataType": "string",
                  "value": "variable='temperature' AND value>60"
                }
              },
              "cold": {
                "filter": {
                  "dataType": "string",
                  "value": "variable='temperature' AND value>20 AND value<40"
                }
              }
            }
          },
          "motorFilters": {
            "edge": {
              "cold": {
                "filter": {
                  "dataType": "string",
                  "value": "variable STARTS_WITH 'motor'"
                }
              }
            },
            "cloud": {
              "all": {
                "filter": {
                  "dataType": "string",
                  "value": "variable STARTS_WITH 'motor' AND model IN ['abb.ability.device','custom']"
                }
              }
            }
          }
        }
      }
    }
  }
}

Points to consider when creating filters:

A group (e.g. temperatureFilters or motorFilters in the example above) can have any name as long as it conforms to the JSON standard.
A Location is the location where data is stored. Supported values are:
- Edge, which instructs data to be stored locally to the consumer system modules.
- Cloud, which sends data to the Edge Proxy and eventually sends it to the cloud.
A Path is the data storage destination. Supported values are:
- For local routing (Edge), the paths can be hot, warm, and/or cold.
- Data is therefore sent to the respective topics, hot/#, warm/#, and cold/#, which can be consumed by Consumer System Modules that are subscribed to the respective topic(s).
Note that:
- The path for the location cloud must only be all.
- Data for the cloud is sent to the topic cloud/#.
Not all locations and paths are required to be specified.
The Edge Router will not start if either "image" or "configuration" under "edge-router" is not provided. This will result in default behavior where the Edge Proxy is subscribed to the modules/+/events/# topic and routes all telemetry messages to the cloud.
If no filters are specified in "configuration", the routing table will default to routing all telemetry messages to the cloud/# topic.
Filters are written in query expression language.
Variables in filters are dynamically deduced from the payload. In the example above, the variables are variable, value, and model.
If a filter is syntactically invalid, the filter expression will default to false.

# Typed Module

A business line's custom-made Typed Module can publish telemetry messages to modules/+/events/#.

# Consumer System Module

There is a new set of system modules introduced in this feature that can consume locally routed message topics hot/#, warm/#, and cold/#. These system modules are not part of the Edge and are created by the business lines themselves.

Persistent data transfer

All Consumer System Modules must manually acknowledge all received MQTT messages in order to maintain persistent data transfer.

# Other Feature Details

Following is a list of essential remarks and/or attributes regarding this feature.

The Router module does not validate the telemetry message payload against its corresponding info model; this is done in the Cloud.
A single telemetry message can be routed to multiple storage paths. All routes where a filter evaluates to true will receive the message.
Only system modules should be allowed to subscribe to telemetry message topics. Otherwise, the separation of modules from each other will be weakened, allowing modules to listen to telemetry message topics receiving data from all other modules.
Since the routing rules are stored centrally in the Edge configuration model, effective with platform release 19.09, the model may exceed the maximum size of the object model payload (64k).

Please note that the following functions are NOT managed by the Edge Router.

Data source loss, restart, disconnect etc.
Filtering by type for an object that is referenced in a telemetry message
Routing of data inside the cloud (e.g. cloud/hot/#, cloud/warm/#, or cloud/cold/#)

← Build Custom Arm Kernel Learning Paths →