Cloud Region and Instance Release Notes Archive

Cloud Region and Instance 22.2.3

New Features

[109696] - Enable business background application managing object models for respective tenancy

Any business background application can manage object models (create, modify, delete, get) for the respective tenantId provided in the header. Note: application delegation ="". In addition, users can create conditions for the info model with property tenantId, which means that resource "object_model" is now grant accessible.

Resolved Issues

[111998] - Fixed multilevel covariance for related models

The issue, when a user was not able to create covariant-related models, was fixed. Now multiple levels of inheritance or multiple base types with covariant related models are possible.

Cloud Region and Instance 22.2.2

New Features

[110317] - Ability Platform now supports 1.22 version of cluster

New Functionality

We have updated our Kubernetes security layer to synchronize our policies with new Azure Kubernetes Service cluster and now we recommend using 1.22 version of AKS to work with 22.2.3 version of Ability Platform.

Resolved Issues

[101461] - 504 error when too big traffic on Object Storage

Object Storage returns 429 (Too Many Requests) in case of too big traffic.

[104536] - Incompatible empty result of LKV query when using TSIv1 and Redis

Empty results from Last Know Query always have the same common format:

{
  "data": []
}

[110836] - Missing Object Storage logs in Azure App Insights

Operations on Object Storage API are being logged to Azure App Insights.

[111721] - 500 error when calling Method Invocation API from the back office application

The user is able to get the expected response from the API of commanding-api when calling Method Invocation API from back office application.

[112463] - Better background c2d processing and scaling

Changing a minimum number of pods from 0 to 1 for DCS background processors. It secures for faster background c2d processing and scaling.

[112852] - Fix for duplicate values in response when retrieving latest versions of types

In some cases when the user was trying to retrieve the latest versions of multiple type definitions, also other versions were retrieved. Now only the latest versions are returned when calling this API.

[112872] - Allowing creating type with 0.0.0 version via Device API

Removing validation of type version regarding 0 in Device Configuration Service (before treated as invalid and considered as an exception). Since now it is possible to create types with 0.0.0 as the version via Device API.

Cloud Region and Instance 22.2.1

New Features

[76035] - Rotating communication keys every 120 days

New Functionality

Following the security requirement that asymmetric keys should not be used for longer than a year, the communication key (encryption key for 'persistent' secrets) will be rotated once every 120 days. Only the current and previous version of the encryption key will be accepted.

[102812] - Implementing dynamic DCS scaling in AKS based on metrics

New Functionality

DCS processors are now automatically scaled based on the demand. This allows dynamic, automatic adjusting of a number of DCS pods running on Kubernetes/AKS based on a number of incoming messages to be processed. As a result DCS fully supports cost-optimized development deployments and also high-load production deployments.

[108268] - IM API: Remove References Improvement

New Functionality

New reference delete API in Information Model to Remove all outgoing references based on their name where the source is the given objectId. New endpoint significantly improves the performance of reference delete operation raised as an issue in BUG#107358.

Resolved Issues

[107125] - Subscriptions API: Deleting subscription with unrecognized type should return Bad Request but returns Internal Server Error

During hot data subscription deleting, when the type parameter does not match any of the available types, API returns 400 (Bad Request).

[109798] - Eliminate the risk of PM database crashing due to oversizing Group document

Corrected the mechanism of removing a device from a Group, which could create a risk of oversize Cosmos DB document, and finally disable the database.

[111300] - System.FormatException reported by TelemetryHotProcessor on EL-03

When sending telemetry event which is not valid and DPP validation step is off, there are no exceptions.

[111723] - Getting FluentValidation.ValidationException

No validation exceptions are thrown when the Validation step on DPP is turned off.

[112280] - Missing data on EL Hot Path

The message with a specific payload (Variable telemetry without "variable" field) is processed without any error.

Cloud Region and Instance 22.1.1

New Features

[76157][82974] - Data Access migration to Azure Kubernetes Service

New Functionality

Data Access components:

• Remote Command Queue Processor,
• Telemetry Hot Path Processor,
• Telemetry Cold Path Processor,
• Telemetry Last Known Value Processor,
• Data Access API,

were moved from Azure Function runtime to Azure Kubernetes Service hosting environment.

It is a part of the general Ability Platform goal of containerizing components to achieve a uniform deployment model, significant cost reduction, and better scaling capabilities.

[82309] - Extension of the common data definitions in TDR

New Functionality

This feature introduces the extension of the common definitions in TDR for data. It will allow users to create common primitive data definitions and reuse them within type/model definition instead of defining the same parameters each time. In the future, primitive definitions will be extended by complex ones.

[83517] - Structural validation of Alarms and Events

New Functionality

Ability Platform supports structural validation of incoming telemetry data. Until now, only variables were a subject of verification against corresponding types. This feature introduces verification also for telemetry of type alarm and event.

If there are no validation schemas defined for alarms/events, all alarms and events are passed through.

[84165] - Increased Tenant capacity per instance

New Functionality

The introduction of dynamic policies in the Azure B2C service allowed us to increase tenant capacity per instance:

• The total number of policies (static and dynamic) remains the same - 1000.
• Users can create an unlimited number of tenants using dynamic policies.
• Tenants using the dynamic policy can be associated with only one identity provider, Azure Active Directory.
• Tenants using the static policy retain the same functionality.

Improvements in Principal Manager APIs:

• We have introduced a new API GET /b2c/tenant?tenantName={name} to improve the developer's experience. It returns an Ability Tenant Id and OpenId configuration URL based on the provided name.
• The new staticTenantAssignment flag enables managing policy for tenant via APIs; POST /tenants, PUT /tenants/{tenantId}, DELETE /tenants/{tenantId}, PUT /tenants/{tenantId}/identityProviders/{ipId}
• The staticTenantAssignment property is optional, and its default value is true to maintain backward compatibility.
• Adding pagination and sort options to API GET /contracts.

Limitations

The total number of Azure B2C policies is 1000.

The dynamic policy is valid only for the Azure Active Directory identity provider.

[87280] - New Azure Container Registry Workflow

New Functionality

In earlier Ability Cloud versions, one common Azure Container Registry was used for all purposes. This feature introduced a new, safer, and more reliable process of storing containers. Development ACR and Production ACR are separated. The first one is now more cost-efficient due to the new retention policy of stored data. The second one keeps containers chosen to be released, and they are stored with a digital signature. Strict identity access management is applied to both resources.

[88339] - Adding support of semantic versioning for 'map' properties within Type Definition

New Functionality

In earlier versions, every change in map property required major version update. Now, map properties have similar versioning rules as other properties, supporting non-breaking updates of types (e.g. when adding a new optional part of map, there is no need to upgrade with a major version).

[88866] - AuthZ migration from Azure App Service Plan to Azure Kubernetes Service

New Functionality

AuthZ (instance) component has been moved from Azure App Service Plan to Azure Kubernetes Service:

• performance achievements: AuthZ can process twice more requests in a given period,
• cost reduction: due to automatically scaling the number of pods and keeping the minimal number at the same time.

[104199] - Migration of regional endpoint

New Functionality

Introduced new endpoints for Principal Manager API and Region public B2C/OAuth to streamline naming convention of Ability Platform endpoints for the end-user:

PM API Endpoint

changed from https://abiapiapmxxxxx.azure-api.net

to https://api-xxxxx.dev.abilityplatform.abb

Region Public B2C/OAuth Endpoint

changed from https://api-xxxx.dev.abilityplatform.abb/public/api/oauth2/token

to https://api-public-xxxx.dev.abilityplatform.abb/public/api/oauth2/token

Portal Endpoint

changed from https://api-xxxx.dev.abilityplatform.abb/Portal

to https://api-public-xxxx.dev.abilityplatform.abb/Portal

• Instance endpoints remain unchanged and follow naming convention https://api-xxxxx.dev.abilityplatform.abb.
• Old PM API Endpoints will work as well, and change can be made at a later date (whenever convenient).

Diagram

Required Migrations

Required change: Region Public B2C/OAuth Endpoint

from https://api-xxxx.dev.abilityplatform.abb/public/api/oauth2/token

to https://api-public-xxxx.dev.abilityplatform.abb/public/api/oauth2/token

Required change: Portal Endpoint

from https://api-xxxx.dev.abilityplatform.abb/Portal

to https://api-public-xxxx.dev.abilityplatform.abb/Portal

Change that can be made at a later date: PM API Endpoint

from https://abiapiapmrxxxx.euwdev.azure-api.net

to https://api-xxxx.dev.abilityplatform.abb

[104250] - Audit Logging on Region was moved to Azure Kubernetes Service

New Functionality

Implementation of this feature is a part of the general Ability Platform goal of containerizing components to achieve a uniform deployment model, significant cost reduction, and better scaling capabilities.
In this case, we moved Audit Logging on Region from Azure Function runtime to Azure Kubernetes Service hosting environment.

Limitations

[77948][106301] DPP stores excessive duplicate values

Misconfiguration of DPP may lead to an increased number of duplicate values, characterized by an identical timestamp.

The default configuration is designed to handle most cases, but it may be helpful to instruct the Operations Team to redefine the number of minimum replicas in some rare cases. See the tab Scaling and Performance in the Data Ingestion section for more info.

21.4.2

New Features

[90200] - Configurable Telemetry Data Routing

New Functionality

This feature enables Ability Platform clients to define custom telemetry data routes based on incoming telemetry message characteristics. Additionally, telemetry messages now support filtering by headers and payload. The platform will still support four built-in routes used for telemetry: hot, warm, cold, and last known value. However, clients may opt out of using any of these or use some of these partially, for a subset of messages. Clients may also create their own custom sinks (currently the Event Hub is the only supported type), which are deployed outside of the Ability Platform and may be used freely by the client. Updates to the routing configuration is executed by a request to the Ability Platform Operations Team.

[95282] - Enhancements to non-validated telemetry data in Data Processing Pipeline

New Functionality

Functionality has been introduced to prevent telemetry data from being pushed into predefined data storage locations (built-in sinks) when data validation is disabled. As a result, no data which would violate the ABB Ability Platform compliance rules will be distributed into the cold, hot, warm, or last known value storage.

[96149] - Device API v2 - new action name for accepting batch requests

New Functionality

The topic and action names for accepting batch requests has been updated so that the specifications stay consistent with other V2 APIs. From "action": "batch" to "action": "batch.execute"

[99799] - Optimized AKS idle resources usage to fit into 2xF8 or 4xF4 nodes

New Functionality

Optimized autoscaling configuration and deployment options used by Kubernetes (AKS) to reduce the number of nodes used by the cluster. As a result, a reduced number of cores (from 24 to 16) leads to a reduction in costs.

Resolved Issues

[99266] - Cost optimization: Data Processing Pipeline storage type update

The default Azure Storage Account type was updated to V1.

With a manual downgrade procedure executed by the Operations team, the daily costs of storage used by the processing pipeline can be reduced at least by a factor of 10.

[101471] - Unexpected AuthZ exceptions in the instance

We have fixed an issue with Gremlin Proxy initialization that was generating multiple exceptions unexpectedly.

[102529] - DataAccess response rendering took longer than expected

A problem was found in the parsing algorithm where a large number of records (10,000) were requested from the Time Series Insights data store using the DataAccess API. The parsing algorithm did not provide efficient results. We have made a modification to this module to improve the time required to parse the data record and improve the response. Because the improvement is proportional to the number of records to be processed, larger data sets will see larger improvements.

[103401] - Data Access queries for warm storage did not return results when missing parameters

The issue is now fixed, and TSI returns proper results, even when there is a missing ability-messageType or msgType parameter.

[103856] - Clustering algorithm does not always return the right match under heavy load

The problem with the "Clustering" matching mode is now fixed, and it is safe to use both modes. Still, the default value of the matching algorithm is "Simplified".
"Clustering" should be used when the number of defined filters exceeds 1000 entries.

[105019] -Ingestion error is not captured in DPP while sending wrong data type Telemetry

The issue is now fixed. In order to make it easier to find, the following DPP logs format was changed:

Old format: tid_{tenant}_did_{device}_oid_{object} New format: oid_{object}_did_{device}_tid_{tenant}

[105881] - Hot Path Subscription API filter changes

With this fix, the following changes were introduced:

• The body of the variable telemetry event now is automatically added to the Service Bus message property value (for variables only) if it is of JSON type integer, number, boolean or string. In the case of string the value size must not exceed the configured size. Arrays, and strings exceeding the configured size, are never added to the message properties. This allows creating subscriptions with filters for, for example, numeric value thresholds, etc.
• Property size (for string variables) can be configured with the VariableValueMaxPropertySizeBytes environment variable. The default size is 1 KB, max size is 32 KB. Adding value to the message properties can be completely switched off by setting this environment variable to any negative value, for example, -1.
• Fixed an issue when all values were added as strings, now they're added as the corresponding types, which enables creating Hot Path subscriptions with filters likevalue >= 2 AND value < 12.01, etc.See Message properties format.

Known Issues

[62908] - Principle Manager API fails to remove tenants - BadGateway

The problem can occur based on concurrent requests to the principal manager API. The Principal Manager APIs use Azure B2C services to create Applications for business entities, e.g. Application, Solution, etc. The workflow in the PM is sequential and dependent on the result of the B2C operation. After a successful answer from the B2C operation, the request is processed to provide the respective response to the caller.
For any B2C-related request, some buffer time needs to be provided to complete the action. It is advised to maintain a gap of 60 secs between two requests.

[73963] - A new application created by the principal manager can lack secrets due to latency issues

Known Issue: Occasionally, a newly created application using principal manager service is created without secrets causing the app to become unusable because a bearer token cannot be obtained.

Workaround: If this is observed, create the application again after waiting approx. 60 secs.

[74595] - User cannot access applications when the "read" permission is limited to "user" delegation

Known Issue:

• Query apps endpoint - passing 'use****r' instead of 'User' for delegation parameter will return empty results.

• Get apps endpoint - passing 'User' instead of 'user' for delegation parameter returns empty results.

Workaround: When querying for applications using the "Query apps" or "Get apps" endpoint, limited to user delegation, pass (delegation='user' OR delegation='User') for delegation parameter to get the expected results

[75339] - Sorting functionality in Principal Manager APIs is case sensitive

Sorting functionality implemented as part of the Pagination & Searching feature in Principal Manager APIs is case sensitive. For example: when trying to sort a set of tenants {ABB01, Robotics01, abb02, Volvo01, robotics02, volvo02}, the ascending sorting should look as follows {ABB01, Robotics01, Volvo01, abb02, robotics02, volvo02}.

[77345] - When creating a solution or resource, Principle Manager service sporadically returns a 400 Bad Gateway response code

Known Issue: When creating a solution or a resource, Microsoft Graph API occasionally will return a "400 Bad Gateway" response code with the message, "One or more of your reply URLs is not valid". As a result, the Solution is not created. Workaround: The end user will need to resubmit the request.

[79098] - TSI storage doubled (costs) and max throughput decreased when DPP status code processing is enabled

Data quality decoration increases the total size of the telemetry message significantly. An original message may look like this:

{  
 
"objectId": "2B129E4C-0944-4534-8E8B-DEB49D8AF0AC",  
 
  "model": "abb.somedomain.somemodel",  
 
  "variable": "SomeVariableName",  
 
  "timestamp": "2018-05-217T23:00:00Z",  
 
  "value": 42,  
 
  "quality" : 1073741954  
 
}  

After quality decoding it may look like this:

{  
 
  "objectId": "2B129E4C-0944-4534-8E8B-DEB49D8AF0AC",  
 
  "model": "abb.somedomain.somemodel",  
 
  "variable": "SomeVariableName",  
 
  "timestamp": "2018-05-217T23:00:00Z",  
 
  "value": 42,  
 
  "quality" : 1073741954,  
 
  "qualityFlags" : {  
 
       "validity" : "uncertain",  
 
       "limit" : "low",  
 
       "historian" : "interpolated"  
 
  }  
 
} 

This in turn has a direct impact on:

• The total capacity of data that Ability Platform ingress pipeline may accept (Azure Event Hubs limits that to 20MB/sec.)
• The total amount of data being stored to TSI, which has a direct impact on the cost of the system

Data quality decoration is turned off by default and not recommended for use at this time.  A future update is planned to provide better control over the extra space used when this feature is enabled.

[81758] - Device API v2 model.create - missing rollback of update ownership in case of request fails

Despite the failure of model.create request,  the model ownership could be updated.

[81955] - New APIM DataAccess file routes return 404 error

APIM DataAccess file routes return 404 error - "Resource Not Found". Microsoft developers determined this to be a bug in the APIM service.

[82194] - Device delegation background app is available on API definition and Portal to be used BL

Currently creating an object model or data reinjection is not allowed for a business's own background application due to missing permission for such device delegation operations.

[85925] - DCS default message senders configuration increases response times

Devices that request more than 50 DCS responses at a time may see response delays with the default configuration of C2D senders. Times can be improved by reducing the number of senders to 2. Overall usage and performance requirements should be evaluated before and after making modifications to DCS settings.

[87885] - Tenant delete operation through APIM works slower causing timeouts and 504 errors

This issue is predominant when we try to delete a tenant having 1000 users or more, Delete Tenant operation is taking more than 4min to complete, and 500 request time out error is given to the client-side, whereas on the server-side the job continues and deletes the tenant and associated users.

[88162] - Cannot log in to Principal Manager

Known Issue: In some cases, users with multiple group memberships, have such a claim size making the request URI too long.

Workaround:

1. Create a new active directory.

2. Create new test credentials for the user and on board him to the tenant.

3. Make the user as tenant admin.

4. Assign the grants required for the user.

[90542] - Missing data while retrieving Last Known Value (v1/Data/variables)

Known Issue: Under some circumstances, data for Last Known Value might be lost.

Workaround: To prevent data loss on production environments, increase the Azure Cache for Redis SKU to P1 Premium. However, the increase in cost is significant, so the operation should be done only if necessary.

[91525] - Missing validation of "access" property

Any integer value is accepted (enum values ignored). Permission is created with the default value "create" even if access is not provided at all.

[103006] - file uploader is not transferring orphaned files after upgrade

The file uploader can be affected by intermittent network failures or when the Ability environment is being updated. If one of these is the case and no files are being transferred, the program will retry the file upload procedure up to three times by default at 15-second intervals. If these are also unsuccessful, the program will retry once every hour. The time to recover is a combination of how many files must be processed, the size of the files, and the scheduling algorithm for retries. This can result in a significant delay in re-processing files before they are visible in object storage.

[103623] - Low-severity security vulnerabilities in B2C - Newtonsoft.Json library

Security scan found low-severity security vulnerabilities in B2C - Newtonsoft.Json library.

[105096] - The issue with the application delegation parameter set to "none"

There is an issue with getting the correct access token for application if the delegation parameter is set to none.
It is not recommended to use the value none at all. Instead, use an empty string.

[105641] - Router upgrade can duplicate or misorder values in the cold storage during the migration process

The introduction of a telemetry router requires establishing a new infrastructure. Internal data routes had to be changed, and data from legacy Event Hubs had to be migrated to the new infrastructure. To maintain a balance between a smooth upgrade process, data integrity, and completeness, we provided a set of so-called migration services. However, this causes that a small fraction of data in our target storages may appear duplicated or out of order.
This only concerns the upgrade time and short time after the upgrade, when the data from legacy infrastructure is transferred to a new one (usually up to a few minutes).

21.4.1

Resolved Issues

[78585] - Principal Manager API - Enum values can be provided and filtered, ignoring case sensitivity

Any enum properties (Application Delegation or Type) should be automatically treated as case insensitive in QEL filtering. This applies to all Principal Manager APIs.

Despite enum values being case insensitive, providing any other values which do not match the enum type returns the error message “400 Bad request” with details. The issue is now corrected.

[97902] - TSI query failed to retrieve the Last State Value when the value property of JSON responses contained an array

We have fixed the issue, so the Last Known Value query now works correctly.

[96146] - B2C doesn't forward domain_hint parameter to Azure AD

Instead of the **domain_hint **parameter forwarding users to the ABB-branded login page, it redirects them to the Microsoft-branded Azure AD common login page.

This prevents users from being able to use their private email addresses in the login form.
The issue is now corrected.

21.3.1

New Features

[96207] - Privacy Notice removed

New Functionality

Every new user accessing their application was being redirected to accept the ABB Ability Platform Admin Portal Privacy Notice. We have removed the notice from the user login flow for custom applications.

[84515] - ASE Deprecation - Migration of environments

New Functionality

This change aligns with the previously released changes to the region API and provides a cost-optimization-driven removal of App Service Environment (ASE) used internally for hosting services inside Ability Platform. It will disable VNet peering of client's applications and testing machines with Ability Platform instance virtual networks and API Management service. Compared to VNet peering, IP allowlisting will give secure access to APIs needed by client applications and testing machines. In order to configure allowlisting, Clients need to provide Operations with their application and test machine IP addresses. It is important that this is done before the update.

An alternative method is under evaluation to allow the continued use of VNet peering (at an incremental cost). Please contact the operations team for additional details on the status of this option before upgrading if this method is required.

[82135] - App Service Environment Deprecation

New Functionality

App Service Environment was replaced with App Service network isolation maintaining the same level of security. There is no reduction of any functionality, the replacement has a significant impact on operational costs reduction.

[82491] - Support for filtering by type version in Type Definitions

New Functionality

The new filtering parameter was added to TDR endpoints with the capability to filter for the version. This allows users to query types with specific major versions or types with specific full semantic versions in TDR.

[85409] - Increased Cosmos DB limit of max properties in object model payload

New Functionality

Cosmos DB limit is increased:

• from 368 max number of properties in object model payload up to 9000 for simple properties and up to 3000 for complex properties. The increased number of properties is allowed for object model payloads with max size up to 64 KB allowed in IoT Hub C2D message size. Whichever limit is crossed first will cause an error.

Limitations

Cosmos DB limit is extended:

• from 368 max number of properties (386) in object model payload up to 9000 for simple properties and up to 3000 for complex properties. Increased number of properties is allowed for object model payloads with max size up to 64KB.

Resolved Issues

[68309] - Unable to search for a file using a user token after the upload

Fixing the issue when searching for files uploaded via Edge via requests using a user token when the number of objects exceeds 500.

[78357] - Exception "Container is disposed and should not be used" in App Insights under heavy load

The problem generating exceptions: "Container is disposed and should not be used" is now fixed.

[86541] - Index for global/object files is not effective

The new index allows delete and save operations to execute with minimal delay.

[91737] - Redis key scan takes a long time

The Redis key scan mechanism was optimized to decrease the scan time significantly.

[92685] - Aggregated data query stopped by the circuit breaker

Querying data older than retention time returns an empty result, not a 500 error code: "Time Series Insights Warm Data API Failure".

[93624] - Inconsistency in Data Processing Pipeline JSON schema validation

Official JSON schema for alarm and event type of telemetry didn't accept arrays. However, arrays were accepted for telemetry types. The behavior is now unified, and the platform officially accepts arrays of alarms and events.

[93688] - File upload is not transferring all files from IoT devices to long-term storage

During issues of related services (and other internal failures), FileUploadProc Azure Function will enter the indefinite regression retry loop until service becomes available again.

The auth token caching issue was also fixed.

[94673] - Application crash with "NullReferenceException"

DPP validation uses data from Type Definition Registry. The reported problem occurred in situations when at the same time:

• Type Definition for particular objectId didn't have variables defined (TDR type definition response didn't contain any variables, which was interpreted as null),
• Type Definition for objectId was marked as extensible.

Such combination caused an application to crash with NullReferenceException, which is now fixed.

Known Issues

[68309] - Unable to search for a file using a user token after the upload

Known Issue: When searching for files uploaded via Edge, requests using a user token fail when the number of objects exceeds 500.

Workaround: When querying, the objectid, along with the path, can be passed in QEL format to overcome this limitation.

[78357] - Exception "Container is disposed and should not be used" in App Insights under heavy load

Under heavy load, the exception: "Container is disposed and should not be used" could appear in Application Insights.

[78585] - Principal Manager API - Enum values can be provided and filtered, ignoring case sensitivity

Enum properties (Application Delegation or Type) are case sensitive in QEL filter.

[81955] - New APIM DataAccess file routes return 404 error

APIM DataAccess file routes return 404 error - "Resource Not Found". Microsoft developers determined this to be a bug in the APIM service.

[86541] - Index for global/object files is not effective

The more files is stored in the storage, the longer it takes to delete and save files.

[90542] - Missing data while retrieving Last Known Value (v1/Data/variables)

Known Issue: Under some circumstances, data for Last Known Value might be lost.

Workaround: To prevent data loss on production environments, increase the Azure Cache for Redis SKU to P1 Premium. However, the increase in cost is significant, so the operation should be done only if necessary.

[91525] - Missing validation of "access" property

Any integer value is accepted (enum values ignored).
Permission is created with the default value "create" even if access is not provided at all.

[91737] - Redis key scan takes a long time

Scanning for Redis keys could take a very long time as there were many keys to check. The issue was especially noticeable as the number of keys exceeded 800,000.

[92685] - Aggregated data query stopped by the circuit breaker

Querying data older than retention time returns a 500 error code: "Time Series Insights Warm Data API Failure".

[93688] - File upload is not transferring all files from IoT devices to long-term storage

During issues of related services (like the unavailability of AuthZ or DataAccess), FileUploadProc Azure Function retried the failed operation five times and then marked the uploaded blob as poisoned and forgot about it. This caused that some client files weren't uploaded because of infrastructure issues.

[94170] - Data Access fails with an error "System.Exception : CloudBlockBlob not found"

Known Issue:

The issue is related to the technical arrangements of Azure Blob storage:

"Polling works as a hybrid between inspecting logs and running periodic container scans. Blobs are scanned in groups of 10,000 at a time with a continuation token used between intervals."

The blob trigger events to be processed are created based on the logs inspection, which doesn't guarantee that all events are captured or a fixed processing time.

Workaround: For more reliable processing of the blob triggers, please consider using a queue trigger or an Event Grid.

[94257] - Uploaded files might get stuck in temporary storage instead of being moved to object storage

In some rare cases where issues occur during file upload, resulting in files getting stuck in temporary storage.

The file will never be moved to object storage. The reason could be, for example, a problem with authorization.

21.2.3

Resolved Issues

[95780] - Unintentional update on Object Model that failed with reference validation on referenced object

Attempt to update the object model that was referenced with the covariant approach was not validated properly (the correct operation was rejected). It is now fixed to support covariance when validating incoming references during object model update operation.

21.2.2

Resolved Issues

[94143] - Principal Manager Time Conversion error from Graph API endpoint

A modification was made within the Graph API endpoints used by the principal manager, which caused a time conversion error. This issue is resolved.

21.2.1

New Features

[14481] - Switching object model to different type definition is now supported

New Functionality

Users are now able to change compatible type structures upon ObjectModel update (PUT "/objects/{objectId}/models/{modelId}" in API V1) in one operation. Before, an Information Modeling service user needed to drop the object model first and recreate it with the same object, with the new type.

[54374] - Type version upgrade/downgrade upon ObjectModel update (Device API v2)

New Functionality

It is now possible to upgrade/downgrade the type version of ObjectModel by requests model.create and model.update.

[56549] - Message compression for C2D and D2C communication (Device API v2)

New Functionality

C2D and D2C message body compression is now available for Device API V2. The feature supports 'gzip' and 'deflate' compression options.

[62357] - Akamai Web Application Firewall implementation on Region Stamp

New Functionality

The use of Azure Web Application Firewall has been deprecated in favor of the Akamai Web Application Firewall (WAF) provided as a service for ABB by Akamai. Akamai WAF is used for Admin Portal and for incoming communication from B2C.

[74509] - Restricting access to Regional APIs as a result of PoC Portal deprecation

New Functionality

Due to the deprecation of the PoC Principal Manager Portal, access to Region API endpoints is now restricted. OAuth Proxy API was migrated to a new URL.

Required Migrations

Please update your application configuration to start using the new URL of the OAuth proxy provided by the Operations Team.

Deprecated Functionality

Region APIs are no longer available over the public internet.

Several Region APIs (for example used by B2C in OAuth flow) are still published for public access. The rest of Region APIs (for example Principal Manager APIs, OAuth proxy) are available only for whitelisted caller IP addresses.

Please contact Operations Team to have your application or test machine whitelisted.

OAuth proxy URL is updated to following the naming convention of the other Region APIs.

[75059] - Support for filtering/listing/ordering by LastModified root property in InfoModel DSL query

New Functionality

It is now possible to filter/list/order objects by the last modified date (LastModified property).

[76533] - Enabling Microsoft autoscaling on CosmosDB databases

New Functionality

Microsoft now provides automatic scaling of the throughput (RU/s) of databases up or down based on the data workloads. At the same time, Microsoft provides full availability of the resource and its SLAs.

[78703] - Sending model.create queries within unique groups

New Functionality

The feature extends model.create and model.update requests to allow change
to a different typeId within a unique group, when type version is
specified in the payload.

[81457] - C2D message propagation enhancement by Device Configuration Service

New Functionality

C2D message propagation enhancement by:

• Implementing a caching mechanism to reduce the number of interactions with IoT Hub
• Implementing a dynamic reschedule time frame, which means the dynamic number of seconds/minutes which the rescheduled message should be pushed forward

[82030] - Alarm and Event definitions added to Type Definition Registry

New Functionality

Alarm and Event definition (get/post/delete) is introduced in Type Definition Registry. Users are now able to define Alarms & Events in TDR and reuse them when creating/updating model definitions/type definitions/object mode (holistic view)/extension.

[82447] - Telemetry quality decoration now available for alarms and events

New Functionality

Telemetry quality decoration is now enabled for alarms and events. Previously data quality decoration was available only for variables.

[87008] - New header property 'context' in Device API v2 and limitation of header size

New Functionality

The new header property 'context' is added to all Device API v2 actions and returned in all acknowledgments and request-response notifications.

• It is optional. If provided, it is guaranteed to be passed to all C2D messages generated as an outcome of this action.

A new validation rule is added that checks the size of all header properties - size must not exceed 32 kB.

• If header size is too large, unsuccessful acknowledgment is sent back with error code 'platform_event_validation_error' and details 'Request message header properties size limit of 32 kB exceed'.

Limitations

Limit of D2C/C2D message header size - 32 kB

Resolved Issues

[56109] - Privileges of a user not updated until the reauthentication

When permissions are updated for users, their reauthentication is required for changes to take effect. A user could access and modify privileged data even after the privileges were revoked. The ability to interact with the privileged functions was available until the user logged off. The issue has been resolved, and the lifetime of the access token is revised to 1 hour.

[67991] - Azure functions save sensitive data in application settings without using the Key Vault

All sensitive data like credentials and connection strings are now stored in the Azure Key Vault.

[78288] - Principal Manager test cases failing due to error in event logging

Principal Manager Tests were occasionally failing because events within Audit Log were not logged. This issue is resolved in this version.

[78455] - Platform APIs may occasionally return 5xx errors due to networking or Azure services issues

Platform code was enhanced to add internal retries policies that should cover short connectivity or availability issues.

This should limit the number of externally reported errors, but such errors still can happen for longer issues. The client's code should still implement retry policies and retry failed calls based on the component's SLA parameters.

[79644] - Data Access queries returned 500/504 errors

Fixed problem with querying TSIv1 with more than allowed concurrent requests, which can lead to 500/504 errors from Data Access.

[80721] - Information Model service fails with 502 bad gateway error

The issue was resolved by optimizing the conditions in the query filter in AuthZ.

[81548] - Downloading files requires a filter change (adding 'dt' to the filter string)

Filter parsing in all Data Access components was optimized. This improved system stability and reliability and introduced the following change in the API format due to a library update:

Before this change, Data Access allowed filters format supported the following "timestamp > '2019-01-01T00:00:00Z'"

After - It is required that all filter format include the following format change: "timestamp > dt'2019-01-01T00:00:00Z'"

Affected endpoints:

• POST request for variables data: /api/v1/data/variables
• POST request for events data: /api/v1/data/events
• POST request for alarms data: /api/v1/data/alarms
• object storage search for files: /api/v1/storage/object/files/search
• global storage search for files: /api/v1/storage/global/files/search
• create subscription for variables: /api/v1/subscriptions/variables
• create subscription for events: /api/v1/subscriptions/events
• create subscription for alarms: /api/v1/subscriptions/alarms

[81890] - Device API header property 'target' was changed

Device API v2 header property 'target' was changed

from:

Represents the path to a connected device that wishes to receive the acknowledgment (if requested). If not provided, defaults to empty string

to:

Represents target recipient for the acknowledgment/response messages. It consists of names of connected devices (principals) separated by a forward slash that have to be registered identities. If present and principals are valid, it is guaranteed to be passed to all c2d messages generated as an outcome of this action. If not provided, defaults to empty string.

[82304] - DPP DataQuality should not be calculated, when quality property is not present

Data Processing Pipeline is capable of decoding quality field value into a human-readable format.

Although the quality field is optional, there were always some activities happening with decoding the value, even though it may have had been empty.

This is now fixed and we first verify if the quality field is at all present in the telemetry payload before executing any analysis.

[84444] - Error while getting bearer token for apps with several rules in grant

When a grant with multiple rules was being assigned to an App, and a Bearer Token was being requested, the call to Principal Manager was resulting in a 500 internal server error.

[84719] - Bootstrap failure during upgrade from 19.0.9 to Berlin

The Bootstrap failure occurred due to not retrieving proper permissions for the Tenant. The issue is fixed by fetching distinct permissions from the list.

[85111] - BadHttpRequestException in TLS when AuthZ calls PM

When AuthZ calls PM, a sporadic 500 BadHttpRequestException can occur. Having that in mind, the MinRequestBodyDataRate in AuthZ was changed to null.

[86315] - Data Access APIs calls for Last Known Value are failing due to long response time

Last Known Value queries without the model and variable name against the data endpoints (/variables, /alarms, /events) are improved.

An inefficient Last Known Value query is a query that may contain some of the following constructs:

• does not contain model
• variable filter contains STARTS_WITH parameter
• query multiple objectIds

Specifying the model and avoiding STARTS_WITH operators in Last Known Value queries, will make the query much faster.

[86320] - DSL query not working in Data Access Alarm API

When telemetry data contained "[" as a first character (e.g. "[TEMP M] -  Measured temperature is above alarm limit.") it was treated by Data Access as an array, and parsing such data to json array failed. As a bug fix json array handling in Data Access has been fixed, so:

• "[TEMP M] -  Measured temperature is above alarm limit." is treated as a string message
• "[12, 45, 71]" or "["value 1", "value 2", "value 2"]" are treated as json arrays

[86542] - File upload processor is not keeping up with files pushed to IoT Hub

The file upload processor couldn't keep up with the files being uploaded. All new files are retried indefinitely, but older orphaned files were not recoverable (they won't be retried). When you upgrade your environment to 21.2.1, all the receipts will be deleted after 30 days. Then, all the files uploads without the receipts will be pushed again.

It is recommended to change the cleaning policy to 2 days in order to speed up the processing of the stuck files.

[86916] - Too many logs and traces from Authproxy component

Authproxy component was logging too many information level statements and B2C traces, making it difficult to debug issues using App Insights. This issue has been resolved by moving the information logs to debug and disabling B2C traces.

[87171] - File transfer receipts are kept indefinitely, but source files are deleted wasting resources

Data Access is not keeping forever logs of files transferred from the GTW storage account to long-term storage.

[88058] - Unable to update public key to null

As per the existing design in PM, the public key of a device could not be updated to null. This restriction has now been removed to support a business use case of resetting the public key.

[88312] - Fixed issue with unsuccessful delete of files

Fixed issue with unsuccessful delete of files, where there is a lot of files in the storage.

When there were too many results in metadata storage, the continuation token was ignored and the file wasn't delated.

[89265] - GlobalStorage supported uploading files up to 134MB

In this release of Ability Platform Cloud, we extended the file size that is supported in GlobalStorage uploads from 134 to 250MB, which is now in sync with ObjectStorage file upload capabilities.

[90228] - File DELETE fails to delete

Fixed issue with unsuccessful delete of files in case when a client has a large number of them.

[91834] - Changes in 'compression' property support

Data Processing Pipeline supports decompression. This functionality is enabled through headers, which are decorating telemetry messages.

Before, the default configuration was expecting "compression" as a header triggering decompression functionality in the cloud.

Code changes switched that default setup, which required "ability-compression".

After this fix, both "compression" and "ability-compression" headers are supported by default. Using any of these would have the same impact on the processing - triggering decompression within the processing pipeline.

Known Issues

[62908] - Principle Manager API fails to remove tenants - BadGateway

The problem can occur based on concurrent requests to the principal manager API. The Principal Manager APIs use Azure B2C services to create Applications for business entities, e.g. Application, Solution, etc. The workflow in the PM is sequential and dependent on the result of the B2C operation. After a successful result from the B2C operation, the request is further processed to provide the respective response to the caller.

For any B2C-related request, some buffer time needs to be provided so that the action can be completed.

It is advised to maintain a gap of 60 secs between two requests.

[68309] - Unable to search for a file using a user token after upload

Know Issue: When searching for files uploaded via Edge, requests using a user token are failing when the number of objects exceeds 500.

Workaround: When querying, the objectid, along with the path, can be passed in QEL format to overcome this limitation.

[73963] - A Latency issue can cause new application created by principal manager to be constructed without secrets

Known Issue: Occasionally a newly created application using principal manager service will be created without secrets causing the app to become unusable because a bearer token cannot be obtained.

Workaround: If this is observed, create the application again after about waiting 60 secs.

[74595] - User cannot access applications when the "read" permission is limited to "user" delegation

Known Issue:

• Query apps endpoint - passing 'use****r' instead of 'User' for delegation parameter will return empty results.
• Get apps endpoint - passing 'User' instead of 'user' for delegation parameter returns empty results.

Workaround: When querying for applications using the "Query apps" or "Get apps" endpoint, limited to user delegation, pass (delegation='user' OR delegation='User') for delegation parameter to get the expected results

[75339] - Sorting functionality in Principal Manager APIs is case sensitive

Sorting functionality implemented as part of the Pagination & Searching feature in Principal Manager APIs is case sensitive.

For example: when trying to sort a set of tenants {ABB01, Robotics01, abb02, Volvo01, robotics02, volvo02}, the ascending sorting should look as follows {ABB01, Robotics01, Volvo01, abb02, robotics02, volvo02}.

[76007] - DSL query escape sequence handling for backward slash() in property value filter is not consistent

Known Issue: When using the backslashes ("") in the object model properties and then trying to query them using DSL, the user cannot obtain it by a single escape character ("\"), which is expected behavior.

Workaround: The workaround is to use double escaping in the DSL query ("\\").

For example, having property:

{
"browseName": {
"value": "some\\path"
}
}

one needs to use the DSL:

models(...).hasProperty("browseName", "some\\\\\\\\path")

[77345] - When creating a solution or resource, Principle Manager service sporadically returns a 400 Bad Gateway response code

Known Issue: When creating a solution or a resource, Microsoft Graph API occasionally will return a "400 Bad Gateway" response code with the message, "One or more of your reply URLs is not valid". As a result, the Solution is not created.

Workaround: The end user will need to resubmit the request.

[79098] - TSI storage doubled (costs) and max throughput decreased when DPP status code processing is enabled

Data quality decoration increases the total size of the telemetry message significantly.

An original message may look like this:

{
"objectId": "2B129E4C-0944-4534-8E8B-DEB49D8AF0AC",
"model": "abb.somedomain.somemodel",
"variable": "SomeVariableName",
"timestamp": "2018-05-217T23:00:00Z",
"value": 42,
"quality" : 1073741954
}

After quality decoding it may look like this:

{
"objectId": "2B129E4C-0944-4534-8E8B-DEB49D8AF0AC",
"model": "abb.somedomain.somemodel",
"variable": "SomeVariableName",
"timestamp": "2018-05-217T23:00:00Z",
"value": 42,
"quality" : 1073741954,
"qualityFlags" : {
    "validity" : "uncertain",
    "limit" : "low",
    "historian" : "interpolated"
    }
}

This in turn has a direct impact on:

• The total capacity of data that Ability Platform ingress pipeline may accept (Azure Event Hubs limits that to 20MB/sec.)
• The total amount of data being stored to TSI, which has a direct impact on the cost of the system

Data quality decoration is turned off by default and not recommended for use at this time.

A future update is planned to provide better control over the extra space used when this feature is enabled.

[81955] - New APIM DataAccess file routes return 404 error

APIM DataAccess file routes return 404 error - "Resource Not Found". Microsoft developers determined this to be a bug in the APIM service.

[82194] - Device delegation background app is available on API definition and Portal to be used BL

Currently creating an object model or data reinjection is not allowed for a business's own background application due to missing permission for such device delegation operations.

[85925] - DCS default message senders configuration increases response times

Devices that request more than 50 DCS responses at a time may see response delays with the default configuration of C2D senders. Times can be improved by reducing the number of senders to 2. Overall usage and performance requirements should be evaluated before and after making modifications to DCS settings.

[86541] - Deleting files takes a significant amount of time

The more files you store in the storage, the longer it takes to delete a file.

[87885] - Tenant delete operation through APIM works slower causing timeouts and 504 errors

This issue is predominant when we try to delete a tenant having 1000 users or more, Delete Tenant operation is taking more than 4min to complete, and 500 request time out error is given to the client-side, whereas on the server-side the job continues and deletes the tenant and associated users.