Files SFTP Connector Guide

The Files SFTP connector provides a quick and simple way to achieve seamless integration between applications that consume data in CSV, SWIFT, or XML files using an SFTP protocol and other applications in your ecosystem.

For example, a financial institution is adding a service to enable customers to send payment requests from their mobile app. For this purpose, a REST API "payment" request is sent from the mobile device to the organization URL.

The OpenLegacy service, based on the Files-sFTP connector, serializes the "payment" data from the request into either CSV, SWIFT, or XML file and sends the file using an sFTP protocol so that other applications in the organization ecosystem can fulfill the request.

In this guide, you will learn how to implement scenarios, similar to the scenario above, using the OpenLegacy File-sFTP solution.

The guide outline

To customize the OpenLegacy solution to your application, perform the following steps:

Use the CLI to generate an asset containing a model of the generated CSV, SWIFT, or XML files. See Steps 1 - 4 below.
Upload the asset to an OpenLegacy Hub project; see Step 5 below.
Generate a microservice on top of the uploaded asset in the OL Hub and deploy it in your destination platform; see Step 6 below.

Prerequisites

Download and install OpenLegacy CLI for Windows or macOS.
Sign up to OpenLegacy Hub and obtain your login credentials.
After you are logged in to OL Hub, generate an API key to be able to log in to the hub from the CLI.
Prepare an XSD file describing the field metadata of the XML asset to be generated using the Files SFTP connector.

Step 1. Create a Files-SFTP module

First, open the CLI and create the Files-sFTP module. For example:

> ol create module files-sftp-module --connector files-sftp

$ ol create module files-sftp-module --connector files-sftp

Step 2. Test the connection to the SFTP server

Run the ol test connection command to verify that your SFTP server is accessible and the connection properties at your disposal are correct. The command sends a connection request to the server and returns a success message when the server is accessible and an error message when it's not.

Navigate to the module folder you just created. For example:
```
> cd files-sftp-module
```
```
$ cd files-sftp-module
```
From under the module directory, display the help menu of the ol test connection command to review the Files SFTP connector connection properties. Refer to Options for the description of each option.
```
> ol test connection --help
```
```
$ ol test connection --help
```
For a full definition of each option, see ol test connection options.

Run ol test connection with the selected connection properties to the SFTP server. For example:

> ol test connection --host localhost --port 22 --user demo --password demo --connect-timeout 3000 --session-timeout 3000 --file-name-pattern CMI{dateTime}.CCT --base-path upload/buf

$ ol test connection --host localhost --port 22 --user demo --password demo --connect-timeout 3000 --session-timeout 3000 --file-name-pattern CMI{dateTime}.CCT --base-path upload/buf

❗️
The Files SFTP connector supports the following authentication modes to the SFTP server:

Provide the username and password in the --user and --password options.

Provide the username and the path to a local private key file in the --user and --private-key-path options.

Provide the username only in the --user option and set the private key in $HOME/.ssh/private_key.

Step 3. Add an asset to the Files SFTP module

After verifying the connection parameters, you can create a Files-sFTP asset by executing the ol add command. To create the asset, provide to the ol add command a path to a file containing a description of the asset metadata and structure. Depending on the generated file type provide one of the following:
To generate an XML file provide an XSD file describing the XML scheme definition.
To generate a CSV or a SWIFT file provide a CSV file describing the file metadata and structure.

The ol add help menu displays the connector property list:

  --verbose                                             Enables verbose mode
  --verbose-debug                                       Enables debug mode
  --verbose-trace                                       Enables trace mode
  --additional VALUE                                    Additional properties
  --file-path TEXT                                      Define path to the input file
  --file-content TEXT                                   Define input data
  --out-file-path TEXT                                  Define path to the output file
  --operation-name TEXT                                 Define name for this operation
  --file-type TEXT                                      Define the file type
  --raw                                                 flag to add raw data (BYTE_ARRAY) default Asset to the module
  --csv-append-mode                                     flag to enable/disable append mode for csv producer
  --base-packages TEXT
  --live TEXT                                           (default: true)
  --prettify-xml TEXT                                   (default: true)
  --base-path TEXT
  --file-name-pattern TEXT
  --temp-extension TEXT
  --file-name-place-holder TEXT
  --encoding TEXT                                       (default: UTF-8)
  --save-file TEXT                                      (default: true)
  --line-separator TEXT                                 [OS, LF, CRLF] (default: OS)
  --generate-null-elements TEXT                         (default: false)
  --host TEXT
  --port TEXT                                           (default: 22)
  --user TEXT
  --password TEXT
  --connect-timeout TEXT                                (default: 1000)
  --session-timeout TEXT                                (default: 3000)
  --private-key-path TEXT
  --passphrase TEXT
  --private-key-data TEXT
  --host-finger-print TEXT
  --known-hosts-path TEXT
  --timeout TEXT                                        (default: 30000)
  --swift.block1 TEXT                                   (default: F01RZBB9155XXXX0000000000)
  --csv.has-header TEXT                                 (default: true)
  --csv.delimiter TEXT                                  (default: ,)
  --csv.skip-header-lines TEXT                          (default: 0)
  --csv.skip-footer-lines TEXT                          (default: 0)
  --csv.predefined-header TEXT
  --csv.predefined-footer TEXT
  --csv.generate-header-from-field-names TEXT           (default: false)
  --csv.generate-header-from-field-original-names TEXT  (default: true)
  --csv.field-names-line-placeholder TEXT               (default: {csv_field_names_line_placeholder})
  --csv.custom-header-footer-new-line-delimiter TEXT    (default: )
  --csv.quotations-mode TEXT                            (default: false)
  --csv.indent-collection-string TEXT
  --csv.collection-item-identifier TEXT
  --csv.null-placeholder TEXT
  --pool.enabled TEXT                                   (default: true)
  --pool.max-size TEXT                                  (default: 10)
  --profile TEXT                                        Configuration profile
  -h, --help                                            Show this message and exit

Provide to the ol add command a file path to the XSD or CSV file.

For example in a scenario where payment data is serialized into an XML file, add an asset by the name of remit_payment and provide as its data source an XSD file containing the metadata for this operation by the name of iso-20022.xsd.
```
> ol add --file-path ../iso-20022.xsd --operation-name remit_payments --file-type XML
```
```
$ ol add --file-path ../iso-20022.xsd --operation-name remit_payments --file-type XML
```

Step 4. Test the Files SFTP asset

After adding the Files SFTP asset, you can test the asset by executing the ol test asset command to verify that the target file is created.

For example, in a scenario where the remit_payment asset is tested, you want to verify that the request to post a payment was transferred to the SFTP server.

From your module folder root go to assets/remit_payment/test_data/case1/:

Open in.json:

case-1> notepad in.json

case-1$ notepad in.json

in.json contains the asset input fields required for performing the test.
For example, in a scenario where the remit_payment asset is tested, in.json contains the asset input fields required for posting the payment.

{
    "conceptBO":
    {
        "key": "",
        "amount": 0,
        "type": "",
        "numDays": 0,
        "subtype": "",
        "applyCapacityPayment": "false",
        "description": "",
        "requiredCollection": "false",
        "isVirtual": "false"
    }
}

Fill in the values of the input fields in in.json.
In the remit_payment asset example, fill in the details on the payment: for the amount key, type 5 and for the numDays key 3.
Now you are ready to run ol test asset. For example:
```
> test asset remit_payment
```
```
$ test asset remit_payment
```
The test result is an XML file sent via SFTP to the remote location you specified using the --base-path and the --file-name-pattern properties.

Connect to the above remote location with an SFTP client and open the generated file. In our example the XML file should look like this:
```
<?xml version="1.0" encoding="UTF-8"?>
<ConceptBO>
    <amount>5</amount>
    <numDays>3</numDays>
    <applyCapacityPayment>false</applyCapacityPayment>
    <requiredCollection>false</requiredCollection>
    <isVirtual>false</isVirtual>
</ConceptBO>
```

Step 5: Push the module to the OL Hub

Now that you are sure that the connecting properties are accurate and that a file can be created and sent via SFTP, you can push the finalized module to the hub.

❗️
Make sure you are logged in to the OL Hub before executing the ol push module command.

Run the ol push module command from the module folder root.
```
[module-name]> ol push module
```
```
[module-name]$ ol push module
```

The CLI displays the command success status. For example:

Module files-sftp-module pushed successfully to the HUB

Step 6: Continue in OL Hub

Your Files SFTP module is now on the OL Hub and can be included in projects.

Next, generate a microservice on top of your uploaded asset in the OL Hub.

Connector Options

ol add options

--file-path

Set the path to the parser source file:

--file-path path/to/data/source

--file-content

The type of the data source file [XSD, CSV]

--file-content XSD

--out-file-path

Set the path to the produced file

--out-file-path path/to/produced/file

--operation-name

The name of the created asset. If not provided, the name of the source file will be used as the asset name.

--operation-name

--file-type

The produced file type: [CSV, XML SWIFT], default: XML

--file-type CSV

--raw

Include the --raw flag to add the raw-data-transformer asset to the Files SFTP module. The asset contains a single field called content used for storing the entire message contents in a byte array format. The single-field asset is used when the input message does not require pre-processing and can be passed in its original content and format to the Files sFTP queue.

--raw

ol test connection options

❗️
The ol test connection options can also be used with the ol add command.

--prettify-xml

Trim leading spaces to improve the XML file readability (default:true).

--prettify-xml false

--base-path

The root directory of the produced file. The value overrides the module-level base path specified in module.json.

--base-path file/root/directory

--file-name-pattern

The directory and name of the produced file. The --file-name-pattern value is concatenated to the --base-path value to create the full path.

--file-name-pattern path/to/file

--encoding

The produced file character encoding (default: UTF-8).

--encoding EBCDIC

--line-separator

The character used as a line separator in the produced file [OS, LF, CRLF]. Select OS when the operating system type determines whether LF or CRLF is selected (default: OS).

--line-separator LF

--generate-null-elements

Include XML elements with a Null value in the produced XML file (default: false).

--generate-null-elements true

--host

The hostname or IP of the remote FTP server

--host https://192.86.32.238

--port

The SFTP server port number (default: 22)

--port 67258

--user

The host user login credential

--user demo-user

--password

The host password login credential

--password P@ssw0rd

--connect-timeout

The maximum time in milliseconds to wait for connection to the SFTP server before aborting (default: 1000).

--connect-timeout 2000

--session-timeout

The maximum time in milliseconds to wait for a successful FTP session creation before aborting (default: 3000).

--session-timeout 2000

--private-key-path

A Local path to the SSH private key file.

--private-key-path path/to/private-key-file

--passphrase

The passphrase to decrypt the SSH private key

--passphrase 3ed04c7f887dc04fe11ad1f58f0473c88edf966502d66aff43a3583569c945de

--private-key-data

The SSH private key

--private-key-data 6JCG34x2a040op1BfSwPicBNUNCuk9Ht1qWMgWoMJWJpownAAi

--host-finger-print

The SSH host key fingerprint

--host-finger-print 97:3c:ae:76:73:f3:ef:a7:18:02:6a:c6:57:43:82:f6

--known-hosts-path

The path to the known hosts file

--known-hosts-path path/to/knownhosts

--swift.block1

The text stored in the basic header block (block 1) of the SWIFT message (default:F01RZBB9155XXXX0000000000)

--swift.block1 G02SACC0266XXXX0000000000

--csv.delimiter

The character that separates fields in the CSV file (default: ",")

--csv.delimiter ;

--csv.skip-header-lines

The number of header lines to skip in the CSV file (default: 0)

--csv.skip-footer-lines

The number of footer lines to skip in the CSV file (default: 0)

--csv.predefined-header

The produced CSV header section template describes the header section structure and contents. The template can contain:

A placeholder for the CSV header row (the row that contains the column names) to signify where the header row is located in the context of the header section. The placeholder string is set by the csv.field-names-line-placeholder property, e.g. "[fnlpe]"
A placeholder to indicate where a new line in the header section begins. The placeholder string is set by the csv.custom-header-footer-new-line-delimiter property, e.g. "||".
Literal text. e.g. "This is a custom multiline header"

--csv.predefined-header This is a custom||multiline||header.||[FNLPE]

--csv.predefined-footer

The produced CSV footer section template describes the footer section structure and contents.
The template can contain:

A placeholder to indicate where a new line in the footer section begins. The placeholder string is set by the csv.custom-header-footer-new-line-delimiter property, e.g. "||".
Literal text. e.g. "This file was generated by OpenLegacy"

--csv.predefined-footer This is a custom||multiline||footer||This file has been auto-generated by OpenLegacy

--csv.generate-header-from-field-names

Generate a header row that consists of the field names (default:false).

--csv.generate-header-from-field-names true

--csv.generate-header-from-field-original-names

Generate a header row that consists of the field original-name attribute. (default:true)

--csv.generate-header-from-field-original-names true

--csv.field-names-line-placeholder

A placeholder for the CSV header row (the row that contains the column names) to signify where the header row is located in the context of the header section, e.g., see csv.predefined-header. Default: {csv_field_names_line_placeholder}

--csv.field-names-line-placeholder [FNLPE]

--csv.custom-header-footer-new-line-delimiter

A placeholder to indicate where a new line in the header or footer section begins. See csv.predefined-header.

--csv.custom-header-footer-new-line-delimiter ||

--csv.quotations-mode

Producer: The quotation mark character can wrap string values. Consequently those string values can include the csv.delimiter character. (default:false)

--csv.quotations-mode true

--csv.indent-collection-string

A string designating a csv row as storing the members of a collection variable. The collection nesting level is indicated by the number of times the string is repeated. For example, if the collection is [n1, n2, n3] and the string is "foo", the csv row is generated as foo, n1, n2, n3. In contrast, if the collection is a collection nested in another collection, the csv row is generated as foofoo, n1, n2, n3.

--csv.indent-collection-string foo

--csv.collection-item-identifier

A string designating a csv row as storing the members of a collection variable. The collection nesting level is indicated by the number that appears at the end of the string. For example, if the collection is [o1, o2, o3,...] and the string is "bar" the csv row is generated as bar1, o1, o2, o3,... In contrast if the collection is a collection nested in another collection the csv row is generated as bar2, n1, n2, n3,....

--csv.collection-item-identifier bar

--csv.null-placeholder

The character designating an empty value in the produced CSV file

--csv.null-placeholder

--pool.enabled

Enable thread pool (default:true)

--pool.enabled false

--pool.max-size

The maximal number of threads allowed in the thread pool (default: 10)

--pool.max-size 20

--profile

The name of the group of connection properties to be used with the ol test connection or ol add commands See: Connection Profiles.

--profile dev

Files SFTP Connector Guide

The guide outline

Prerequisites

Step 1. Create a Files-SFTP module

Step 2. Test the connection to the SFTP server

❗️
The Files SFTP connector supports the following authentication modes to the SFTP server:

Step 3. Add an asset to the Files SFTP module

Step 4. Test the Files SFTP asset

Step 5: Push the module to the OL Hub

❗️
Make sure you are logged in to the OL Hub before executing the ol push module command.

Step 6: Continue in OL Hub

Connector Options

ol add options

ol test connection options

❗️
The ol test connection options can also be used with the ol add command.

The guide outline

Prerequisites

Step 1. Create a Files-SFTP module

Step 2. Test the connection to the SFTP server

❗️The Files SFTP connector supports the following authentication modes to the SFTP server:

Step 3. Add an asset to the Files SFTP module

Step 4. Test the Files SFTP asset

Step 5: Push the module to the OL Hub

❗️Make sure you are logged in to the OL Hub before executing the ol push module command.

Step 6: Continue in OL Hub

Connector Options

ol add options

ol test connection options

❗️The ol test connection options can also be used with the ol add command.

❗️
The Files SFTP connector supports the following authentication modes to the SFTP server:

❗️
Make sure you are logged in to the OL Hub before executing the ol push module command.

❗️
The ol test connection options can also be used with the ol add command.