File Upload v2

PUB Upload a file

from/module/{module_id}/action/file/upload

Use this endpoint to upload a file to the cloud.

Parameters:

Name	Type	Description	Accepted values
module_id	string	Module identifier	Name of module

Message format

Properties:

Key	Value	Notes	Mandatory
correlationId	<string>	Unique String (ex: uuid)	No
ack	<string>	And value in ("positive", "negative", "all", "none")	No
target	<string>		No
context	<string>		No
timestamp	<string>	Date time in ISO-8601 format	No
objectId	<string>	Format: uuid	No
model	<string>		No

These properties can be included in the request by adding them to the topic to which you are sending the messages. See the Message Properties section to find out more.

Here is an example of the properties used:

from/module/{module_id}/action/file/upload&ack=all

Payload information:

Key	Value	Mandatory
fromFile	<string>	Yes
toFile	<string>	Yes
removeOnComplete	<boolean>	No
objectId	<string>	Yes
model	<string>	No

Here is an example payload:

{
	"fromFile": "{from_file_name}",
	"toFile": "{to_file_name}",
	"removeOnComplete": {false/true},
	"objectId": "{object_id}"
}

SUB Ack for file upload

to/module/{module_id}/ack/file/upload

Use this endpoint to check whether the file was uploaded successfully.

Parameters:

Name	Type	Description	Accepted values
module_id	string	Module identifier	Name of module

Format of successful acknowledgment

{
  "isSuccess": true,
  "correlationId": "dd92fc0a-2a63-409e-b1e9-ffe1bafc0b32",
  "objectId": "0d7a78c3-6aa1-4325-b30b-e99534868e38",
  "message": "[
	{ 
	    "isSuccess ": true,
	    "consumer ": "$cloud ", 
	    "messageToProducer ":  "File upload successful " 
	}
  ]"
}

Format of failing acknowledgment

{
  "isSuccess": false,
  "correlationId": "dd92fc0a-2a63-409e-b1e9-ffe1bafc0b32",
  "objectId": "0d7a78c3-6aa1-4325-b30b-e99534868e38",
  "message": [
	{ 
	    "isSuccess ": false, 
	    "consumer ": "All", 
	    "messageToProducer ":  "file files/test1.txt does not exist" 
	}
  ]
}

File sharing between Edge Modules

The file transfer feature supports the transfer of files among edge modules. There will be at least two modules that are involved for the file transfer. These modules are in general referred to as the Producer and Consumer modules as defined above.. The current document explains file transfer feature only, and briefly mentions file upload wherever applicable, as the broker topics used for both the features are the same. Currently, the file transfer is supported over Module API V2.

File transfer requests

There is no separate File Transfer module. The Edge Proxy handles all the file transfer requests.

Definitions

File Upload: A file to be uploaded from a BU module to the Cloud via the Edge Proxy.
File Transfer: A file to be locally transferred (on an Edge) from the Producer to Consumers via the Edge Proxy.
Producer: The creator of a file which will be considered for file upload and/or file transfer, and one of the BU modules.
Consumer: A BU module that receives the producer transferred file via the Edge Proxy.

The diagram below briefly highlights the tasks performed at various modules:

A file transfer request is sent. The proxy handles the request, and sends a notification to the consumer(s) for a new file. At the same time, the proxy sends an acknowledgment which is retrieved by the producer to confirm that the file has been transferred successfully.

Acquiring Source File

The producer module is responsible for obtaining the source file from the var/ability/modules/<producer-module-id>/files/ directory. The file transfer feature has no role in acquiring the source file. The BU modules shouldn't alter the file permissions in such a way that would render the Edge Proxy unable to perform the file transfer successfully.

File Transfer Sequence Diagram:

File Transfer

The Producer module sends a file transfer request over the topic and payload as seen below. Here is an example based on the diagram in the introduction section.

Topic: from/module/{module_id}/action/file/upload

Payload: Here is a sample payload. Refer to the specification above for further details.

{
  "consumers": ["Consumer1", "Consumer2"],
  "fromFile": "from.txt",
  "toFile": "to.txt",
  "removeOnComplete": false,
  "correlationId": "12345678-8eab-438d-ab6a-768de1fe1234",
  "objectId":"6acd4f8a-8eab-438d-ab6a-768de1fe9f3a",
  "model" : "abb.ability.device"
}

Payload Fields:

• consumers: This field provides a list of consumers that are supposed to receive the source file from the producer. The field can also be used to perform file upload. Various combinations are explained below.

"consumers": ["Consumer1", "Consumer2"] : Proxy transfer the source file to both the consumers.
"consumers": ["$cloud, "Consumer1", "Consumer2"] : Proxy performs the source file upload, as well as transfer the file to the consumers.
"consumers": ["$cloud] : Proxy performs the source file upload.
"consumers": not available / null : Proxy performs the source file upload, backward compatibility.
"consumers": [] : Proxy returns negative ack, as the consumers field exists but consumer list is empty.

• fromFile: Source file name from producer under the producer's /files directory.

• toFile: Destination file name from producer under the consumer's /files directory.

• removeOnComplete: The field that informs the agent whether to retain or remove the file from the producer post a successful file transfer.

• correlationId: Any new, random guid for tracking purposes of this request.

• objectId: The object id the producer module. This is required only for file upload scenario. For file transfer the object id field is not required.

• model: The 'model' provided by the producer module. This is required only for file upload scenario. For file transfer the model field is not required.

Once the Edge Proxy receives the file transfer request, it performs input validations such as: whether the source file (fromFile field in payload) exists, whether there is enough free space for the file transfer to complete successfully, whether the consumer modules exist, and so on.

The Edge Proxy then copies the files to the target consumer's /files directories with the name taken from the toFile field in the payload. Once the file copy is complete, the Edge Proxy notifies the consumer modules that a new file is available for each of the consumer modules with the topic and payload listed below:

Topic: to/module/{module_id}/local/file

Payload: Here is a sample payload:

{
  "publisherID": "Producer",
  "file": "to.txt"
}

The Edge Proxy does not expect acknowledgment for the notification message from the consumer modules need. This is to notify the consumers of an incoming file. This can be seen in the first basic architecture diagram.

Once the file copy is complete for all the consumers, the Edge Agent will check for the removeOnComplete field. If it is set to true, the Edge Agent will remove the fromFile from the producer.

It then sends an acknowledgment to the producer using this topic:

Topic: to/module/{module_id}/ack/file/upload

Note: The acknowledgment is a combined acknowledgment for file transfer for all the consumers. This can be seen in the sample payloads below.

Scenario: Producer sends a file for file transfer to Consumer1 and file upload to $cloud

Payload:

{
  "consumers": ["Consumer1", "$cloud"],
  "fromFile": "from.txt",
  "toFile": "toboth.txt",
  "removeOnComplete": false,
  "objectId":"6acd4f8a-8eab-438d-ab6a-768de1fe9f3a",
  "correlationId": "12345678-8eab-438d-ab6a-768de1fe1234",
}

Ack:

{ "isSuccess": true,
  "id": "12345678-8eab-438d-ab6a-768de1fe1234",
  "objectId": "6acd4f8a-8eab-438d-ab6a-768de1fe9f3a",
  "message": "[\n {\n \"isSuccess\": true,\n \"consumer\": \"Consumer1\",\n \"messageToProducer\": \"File transfer successful for consumer Consumer1\"\n },\n {\n \"isSuccess\": true,\n \"consumer\": \"$cloud\",\n \"messageToProducer\": \"File upload successful\"\n }\n]"
}

Scenario: Producer sends a file for file transfer to Consumer1 and Consumer2

Payload:

{
  "consumers": ["Consumer1", "Consumer2"],
  "fromFile": "from.txt",
  "toFile": "tolocal.txt",
  "removeOnComplete": false,
  "correlationId": "12345678-8eab-438d-ab6a-768de1fe1234",
}

Ack:

{ "isSuccess": true,
  "correlationId": "12345678-8eab-438d-ab6a-768de1fe1234",
  "message": "[\n {\n \"isSuccess\": true,\n \"consumer\": \"Consumer1\",\n \"messageToProducer\": \"File transfer successful for consumer Consumer1\"\n },\n {\n \"isSuccess\": true,\n \"consumer\": \"Consumer2\",\n \"messageToProducer\": \"File transfer successful for consumer Consumer2\"\n }\n]"
}

Scenario: Producer sends a file, which does not exist, for file transfer

Payload:

{
  "consumers": ["Consumer1", "Consumer2"],
  "fromFile": "nofile.txt",
  "toFile": "to.txt",
  "removeOnComplete": false,
  "id": "12345678-8eab-438d-ab6a-768de1fe1234",
}

Ack:

{ "isSuccess": false,
  "correlationId": "12345678-8eab-438d-ab6a-768de1fe1234",
  "message": "[\n {\n \"isSuccess\": false,\n \"consumer\": \"All\",\n \"messageToProducer\": \" file files/nofile.txt does not exist\"\n }\n]"
}

Additonal Information:

• The current file transfer implementation means that the amount of space which can be taken up on the drive is capped at 80% of the drive's total storage capacity. This is explained in the example scenario below:

  Let's say we have 1 producer, 8 consumers, and a 1 GB file.

  The 1 GB file needs to be sent to each consumer, meaning that at least 8 GB of free space is required to initiate the file transfer. However, the current implementation has a cutoff point which is reached when 80% of the total space on the drive is used up. This means that for the file transfer to take place, at least 10 GB of free space should be available. (8 GB for 8 consumers with a buffer of 20% free space).

• Overwriting files with the same toFile name is not supported.

• A number of validations is in place on the fromFile and toFile fields, meaning that the following characters and strings are not supported in the filename fields: { "*", ".*", "*.", ".sh", "~", "..", "./" }.

• Files can be transferred from the producer to itself, that is, by adding itself as a consumer in the payload. However, the fromFile name and the toFile name cannot be the same, as overwriting files with same name is not supported.

• Because $cloud is used for uploading files as a consumer name, no module on the edge can take the same name. Additionally, All cannot be used as a module name, as the failure acknowledgment uses All for some general File Upload/File Transfer parameter-level errors.