# Endpoints Performance
This page presents summarized data from the non-functional (performance) testing team to offer some estimated performance guidelines based on a specific environment and use case parameters as outlined in the following sections. Moreover, multiple factors affect the speed and quality of the internet connection, which is beyond ABB's control. Use this reference as a guide, not as a guarantee of performance.
# Test Environments
The data presented below was generated from these two test environments based on a hypothetical business line of expected usage of a solution built on the Ability Platform. Test results shown below do not represent all the possible permutations of solution uses.
# Environment 1
| Stable-02 | |
|---|---|
| Component | Configuration |
| Instance | Ability-Platform-Int-I-Ops-Stable-02 |
| APIM-Instance | Premium (1 Unit) |
| Service bus | Premium |
| Event Hub | 1 Unit - Standard Tier |
| Telemetry cache pods count | 5 |
| Cosmos-IM | Max RU/s: 5k |
| Cosmos-TD | Max RU/s: 5k |
| AzureCacheForRedis | C2 Standard |
| TSI | Sku: S2 | StorageConfigUnit: 1 unit |
| Application Sights-Daily Cap | 5GB (Data sampling at 100%) |
| AKS-Kubernetes-AuthZ | Pod-count:2 (default count with disabled hpa) |
| AKS-Kubernetes-IM | Pod-count:1 (default count with disabled hpa) |
| AKS-Kubernetes-TD | Pod-count:1 (default count with disabled hpa) |
| AKS-Kubernetes-Data-api | Pod-count:2 (default count with disabled hpa) |
| APIM-Region | Developer(1 Unit) |
| AuthZ-Region-AKS | Pod-count:2 (default count with enabled hpa) |
| Principal Manager-ScaleUp | P1V2 |
| Principal Manager-ScaleOut | CustomAutoScale-Instances (2,4,2) |
| AKS-Kubernetes-auditlogging (Region) | Pod-count:2 (default count with enabled hpa) |
# Environment 2
| NFT01-Master | |
|---|---|
| Component | Configuration |
| Instance | Ability-Platform-Int-I-Ops-Stable-02 |
| APIM-Instance | Premium (1 Unit) |
| Service bus | Premium |
| Event Hub | 20 Unit - Standard Tier |
| Cosmos-IM | Max RU/s: 30k |
| Cosmos-TD | Max RU/s: 30k |
| AzureCaheForRedis | C2 Standard |
| TSI | Sku:P1 | StorageConfigUnit:1 unit |
| Application Sights-Daily Cap | 10GB (Data sampling at 100%) |
| AuthZ-Instance-ScaleUp | P1V3 |
| AuthZ-Instance-ScaleOut | CustomAutoScale-Instances (2,2,2) |
| DA-Instance-ScaleUp | P1V3 |
| DA-Instance-ScaleOut | CustomAutoScale-Instances (3,3,3) |
| AKS-Kubernetes-IM | Pod-count:10 (default count with disabled hpa) |
| AKS-Kubernetes-TD | Pod-count:15 (default count with disabled hpa) |
| APIM-Region | Premium (1 Unit) |
| AuthZ-Region-ScaleUp | P1V3 |
| AuthZ-Region-ScaleOut | CustomAutoScale-Instances (2,2,2) |
| Principal Manager-ScaleUp | P1V3 |
| Principal Manager-ScaleOut | CustomAutoScale-Instances (2,2,2) |
| AKS-Kubernetes-auditlogging (Region) | Pod-count:2 (default count with enabled hpa) |
# Tests Description
The following tests were used to measure ingestion behavior under varying loads and durations for each of the endpoints listed in the tables below.
| Test | Description |
|---|---|
| Scale to Fail | Slowly increasing the number of users and using the results to define the red zone. |
| Red Zone Test at 90% | Running test at 90% load. |
| Red Zone Test at 60% | Running test at 60% load. |
| File Upload Test | Uploading 1/10/100MB files via API. |
| Message Latency | Putting data into the system with a device then measure the time until it is available. |
# Endpoints Performance Test Results
The following test results are from the Stable-02 environment. To make the results more valuable, we excluded a couple of peaks, so we calculated the 90th percentile value for the tested endpoints.
# Information Model Endpoints
| Service Endpoint | S2F | RZ90 | RZ60 |
|---|---|---|---|
| POST: /objects | 0,47 | 0,50 | 0,34 |
| PUT: /objects/objectId/models/modelId | 0,21 | 0,17 | 0,19 |
| PATCH: /objects/objectId/models/modelId | 0,56 | 0,59 | 0,41 |
| POST: /objects/objectId/models/modelId/references/out | 1,15 | 1,15 | 0,72 |
| POST: Query | 0,57 | 0,68 | 0,37 |
| PUT: /objects/objectId/models/modelId/extension | 0,92 | 0,98 | 0,52 |
| PUT: /references/referenceId/attributes | 9,24 | 9,04 | 10,97 |
| GET: /objects | 0,25 | 0,24 | 0,23 |
| GET: /objects/objectId/models | 0,20 | 0,19 | 0,12 |
| GET: /objects/objectId/models/modelId | 0,27 | 0,25 | 0,19 |
| GET: /objects/objectId/models/modelId/extension | 0,22 | 0,20 | 0,17 |
| GET: /references/referenceId/ | 9,36 | 9,55 | 11,02 |
| GET: /objects/objectId/models/modelId/references | 1,08 | 1,10 | 0,29 |
| GET: /references/referenceId/attributes | 13,39 | 13,14 | 16,25 |
| GET: /references/referenceId/attributes/attribute | 13,82 | 14,32 | 15,69 |
| DELETE: /references/referenceId/attributes/attribute | 9,35 | 9,46 | 10,77 |
| DELETE: /objects/objectId/models/modelId/references/out/referenceName | 1,10 | 0,95 | 0,33 |
| DELETE: /references/referenceId | 9,21 | 10,13 | 9,98 |
| DELETE: /objects/objectId/models/modelId/extension | 0,76 | 0,81 | 0,45 |
| DELETE: /objects/objectId/models/modelId | 1,18 | 1,18 | 0,76 |
# Type Definition Registry Endpoints
| Service Endpoint | S2F | RZ90 | RZ60 |
|---|---|---|---|
| POST: /modeldefinitions | 0,11 | 0,11 | 0,10 |
| POST: /modelDefinitions/modelId/types | 0,14 | 0,14 | 0,13 |
| GET: /modelDefinitions | 0,12 | 0,12 | 0,12 |
| GET: /modelDefinitions/modelId | 0,09 | 0,08 | 0,08 |
| GET: /modelDefinitions/modelId/types/typeId | 0,09 | 0,09 | 0,09 |
| GET: /modelDefinitions/modelId/types/typeId/versions | 0,09 | 0,09 | 0,08 |
| GET: /modelDefinitions/modelId/types/typeId/versions/version | 0,09 | 0,09 | 0,08 |
| DELETE: /modelDefinitions/modelId/types/typeId | 0,12 | 0,13 | 0,12 |
| DELETE: /modelDefinitions/modelId | 0,10 | 0,11 | 0,09 |
# Data Access Endpoints
| Service Endpoint | S2F | RZ90 | RZ60 |
|---|---|---|---|
| POST: /data/alarms All | 1,21 | 1,17 | 1,08 |
| POST: /data/alarms Count | 1,04 | 0,98 | 0,92 |
| POST: /data/alarms Properties | 1,20 | 1,15 | 1,09 |
| POST: /data/events All | 1,21 | 1,17 | 1,09 |
| POST: /data/events Count | 1,05 | 0,98 | 0,91 |
| POST: /data/events Properties | 1,20 | 1,15 | 1,09 |
| POST: /data/variables All | 1,12 | 1,09 | 1,01 |
| POST: /data/variables Avg | 1,06 | 1,01 | 0,93 |
| POST: /data/variables Count | 1,08 | 0,98 | 0,92 |
| POST: /data/variables First | 1,06 | 0,98 | 0,92 |
| POST: /data/variables Last | 0,12 | 0,14 | 0,12 |
| POST: /data/variables Max | 1,04 | 1,00 | 0,92 |
| POST: /data/variables Min | 1,06 | 0,98 | 0,92 |
| POST: /data/variables Properties | 1,11 | 1,05 | 0,99 |
| POST: /data/variables Sum | 1,06 | 1,00 | 0,92 |
# Latency
Latency is defined as the processing time in seconds required for data that enters a device to become available. Latency is useful for determining whether system resources are adequate to managing data ingress and processing. Measures of latency appear below for hot path, warm path, and last known value. Latency test results are from the NFTP1 test environment.
| Hot Path Tested (s) | Warm Path Tested (s) | Last Known Value Tested (s) |
|---|---|---|
| 0,108 | 31,58 | 0,452 |
The characteristics of each storage sink and its testing are as follow:
- Hot Path
The Hot Path is the fastest route through the Platform. For each variable sent to the environment, only the most recently received value is preserved. This is tested as part of a regular regression test suite in the following configuration by sending a following message:
- 9 variables,
- 1 alarm,
- 1 event,
with 1 device per minute.
It is recommended that variables destined for Hot Path access should be combined into fewer messages rather than packing each variable into a separate message.
- Last Known Value
Once data has been received by the Platform, a variable's last known value can be retrieved via API call. This is tested as part of a regular regression test suite in the following configuration by sending:
- 9 variables,
- 1 alarm,
- 1 event,
with 1 device per minute.
- Warm Path
After historical data is stored, it can be retrieved through a REST API. This is tested as part of a regular regression test suite in the following configuration by sending:
- 9 variables,
- 1 alarm,
- 1 event,
with 1 device per minute along with instance APIs (IM, TDR).
# File Upload Tests
The file uploading was tested in the following configuration by sending:
- 1 MB / 10 MB / 100 MB files
with 1 device, and the results are as follow:
| File Sizes | 1MB | 10MB | 100MB |
|---|---|---|---|
| Threads | 100 | 100 | 100 |
| 90%ile Response time | 1503,9 | 13403,3 | 205613,9 |
| Achieved Bandwidth | 106,8944444 | 119,2277778 | 96,0555556 |
| DA CPU-Utilization | 0,6 | 0,6 | 0,6 |
| APIM Capacity-Utilization | 0,7 | 0,6 | 0,8 |
| AuthZ CPU-Utilization | 0,25 | 0,05 | 0,05 |
# Commentary
File upload processing is a CPU-intensive task. The file upload processor resides in an app service plan that is shared by the data access API. The file upload process includes the data access API which shares resources on the same app service plan. At times, the CPU in this app service plan spiked to 100% utilization which caused the file upload process to time out.
The development team has implemented retry patterns in the file upload processor that will attempt to upload the file again upon failure. The development team did observe rare occurrences of missed files, i.e. files that remain in the IoT hub temporary folder because the retry limit had been reached.