Mainframe Dataset Connector Guide

Create MF-Dataset assets

The MF-Dataset connector includes a prepackaged asset containing the metadata necessary for fetching data from the MF Dataset Records file. In contrast to the prepackaged asset, the assets that you add to the module using the ol add command are entity-type assets containing only the metadata describing the business fields required by the data consumer.

To create an MF-Dataset asset, add to the module the entity-type asset describing the business data you want to expose. Then, refer to that asset from the sort2-operation asset that is prepackaged with the MF-Dataset connector. Finally, to test the asset, insert input parameters to the in.json file of sort2-operation.

Step 1. Create MFDS module

First, create the MF-Dataset module.

> ol create module mfds-module --connector mainframe-dataset

$ ol create module mfds-module --connector mainframe-dataset

The prepackaged sort2-operation asset is generated under the assets directory in your module folder. You will need to use it later.

Step 2. View the properties of the MFDS connector

After creating your connector module, you can learn what properties you need to populate to continue creating the asset.

Navigate to the module folder:
```
> cd mfds-module
```
```
$ cd mfds-module
```
From under the module directory, display the help menu of the ol add command:
```
[module-name]> ol add --help
```
```
[module-name]$ ol add --help
```

The ol add help menu displays the connector property list:

Usage: ol add [OPTIONS]

  Add Asset to Module metadata

Options:
 --verbose                               Enables verbose mode
 --verbose-debug                         Enables debug mode
 --verbose-trace                         Enables trace mode
 --additional VALUE                      Additional properties
 --source-path TEXT                      Provide local copybook source (required)
 --indicator-value TEXT                  Provide records indicator key
 --update-assets                         Add a collection or structure of the copybook type to the default assets
 --host TEXT                             Define the mainframe dataset host address (required)
 --port TEXT                             Define the mainframe dataset port number (required) (default: 21)
 --user TEXT                             Specify the mainframe dataset username (required)
 --password TEXT                         Specify the mainframe dataset password (required)
 --output-file-prefix TEXT               Specify the prefix to be used when fetching the output file
 --output-file-suffix TEXT               Specify the suffix to be used when fetching the output file
 --file-name TEXT                        Specify file name
 --record-offset TEXT                    Specify record position
 --record-read-length TEXT               Specify length to read for record length
 --record-length TEXT                    Specify record length - only when length is fixed
 --default-job-card TEXT                 Specify the default job card
 --records TEXT
 --code-page TEXT                        Specify the mainframe dataset code page (default: cp037)
 --spaces-are-null TEXT                  (default: true)
 --exclude-spaces-are-null TEXT          Exclude type from treat spaces as null[STRING, NUMBER, DATA, ENUM, BOOLEAN, CHARACTER] (default: [])
 --treat-empty-string-as-low-value TEXT  (default: false)
 --space-char TEXT                       (default: 40)
 --timeout TEXT                          Specify the timeout for FTP commands (default: 0)
 --local-passive-mode TEXT               Specify if FTP client should work in local passive mode (default: true)
 --output-poll-interval TEXT             (default: 500)
 --output-poll-timeout TEXT              (default: 10000)
 --connector-mode TEXT                   [SEND_RECEIVE, SEND_AWAIT_COMPLETE, SEND_ONLY] (default: SEND_RECEIVE)
 --records-have-fixed-length TEXT        Specify if records have fixed length (default: false)
 --response-format TEXT                  [ASCII, HEX] (default: ASCII)
 --var0 TEXT
 --hlq TEXT
 --version TEXT
 --cust-id TEXT
 --file-id TEXT
 --inc TEXT
 --file-param TEXT
 --record-id-position TEXT               The position, in the record, of the beginning of the data to be collected as ID
 --record-id-length TEXT                 The number of bites to collect as ID
 --outfil-name TEXT                      (default: OUT1)
 --pool.pool.enabled TEXT                Specify if enabling the ftp client pool (default: true)
 --pool.pool.max-size TEXT               Specify the maximum size of the connection pool (default: 10)
 --pool.pool.idle-time TEXT              Specify the maximum idle time for the connection in ms (default: 180000)
 --ssl.enabled TEXT                      (default: false)
 --ssl.protocol TEXT                     Default secure socket protocol name (default: TLS)
 --ssl.implicit TEXT                     The security mode. (True - Implicit Mode / False - Explicit Mode) (default: false)
 --ssl.client-certificate-path TEXT      The path of the client certificate
 --ssl.store-pass TEXT                   The store pass of the client certificate
 --bidi.response-enabled TEXT            Enable bidi logic on response field (default: false)
 --bidi.request-enabled TEXT             Enable bidi logic on request field (default: false)
 --profile TEXT                          Configuration profile
 --delete-response-file                  Specify if FTP file will be deleted by the end of the process 
  -h, --help                             Show this message and exit

For more information on the MFDS connection properties, see Properties below.

Step 3. Test the connection to the MFDS remote data source

Run the ol test connection command with the connection properties below to verify that the data source is accessible and that the connection properties are correct:

[module-name]> ol test connection --host 192.86.32.238 --port 2025 --user testuser --password testpass --output-file-prefix prefix --output-file-suffix suffix --records-have-fixed-length true --record-offset 8 --record-length 1 --file-name AAVSAM4

[module-name]$ ol test connection --host 192.86.32.238 --port 2025 --user testuser --password testpass --output-file-prefix prefix --output-file-suffix suffix --records-have-fixed-length true --record-offset 8 --record-length 1 --file-name AAVSAM4

Step 4. Add an entity-type asset to the MFDS module

After you verified the connection parameters, you can create the asset. The MFDS module you created in step 1 contains a generic prepackaged asset specifying the metadata necessary for accessing the information stored on an MF Dataset records file. Now you can add the asset specifying the metadata required for the customized service you want to create.

The connector requires a local copybook file as a data source for the connector.

For example the user-info-new copybook file (CPY):

01 id                        PIC X(6).
01 fillers                   PIC X(1).
01 fullName.
   03 LAST-NAME              PIC X(15).
   03 FIRST-NAME             PIC X(10).
01 userAddress               PIC X(24).
01 city                    PIC X(15).
01 cityCode                  PIC X(3).
01 zipCode                   PIC 9(6).

Add the asset by performing the ol add command as in the example below:

[module-name]> ol add --source-path ./COMPANY.cpy --indicator-value C1

[module-name]$ ol add --source-path ./COMPANY.cpy --indicator-value C1

The ol add command adds the asset folder inside the module directory. The folder contains the asset metadata file ([asset-name].json) and the asset testing resources.

Step 5. Test the MFDS asset

Test the asset you added to verify that you can request and receive actual business data from the data source.

The asset you just added is an entity-type asset and cannot be tested directly. For that purpose, refer to your entity-type asset from the in.json file of the prepackaged sort2-operation asset.
sort2-operation describes the information required for querying information from an MF-dataset records file. In addition, specify in in.json a condition that filters the query results.

Open assets/sort2-operation/test_data/case-1/in.json:

{
"var0" : "000100",
"fileName" : "OUT1",
"conditions" : [ {
"entity" : "com.openlegacy.entity.userinfonew.UserInfoNew",
"field" : "id",
"symbol" : "LESS",
"value" : "C'&V00'"
} ],
"operator" : null,
"hlq" : "replace with hlq",
"version" : "V0000001",
"custId" : "replace with cust id",
"fileId" : "AAVSAM2",
"inc" : "&HLQ..&VERSION..INCLUDES",
"fileParam" : "&HLQ..&VERSION..&CUSTID..FILEPARM"
}

Set the entity key to the asset's qualified name specified in the generated entity JSON file. "qualifiedName" : "com.openlegacy.entity.userinfonew.UserInfoNew"
Create the filter condition "id < C'&V00'" by setting the field, symbol, and value keys.
Set the qualifiers that resolve dataset names on the mainframe side. In the example above, the qualifiers are hlq, version, custId, and fileId. To obtain the qualifiers and their values for your mainframe system, consult with your MF system programmer.

Run ol test asset for sort2-operation:

[module-name]> ol test asset sort2-operation

[module-name]$ ol test asset sort2-operation

The following output is displayed:

{
    "records": [
    {
        "id": "000010",
        "fullname":
        {
            "lastName": "Alebert",
            "firstName": "John",
            "@clazz_name@": "com.openlegacy.entity.fullname.Fullname"
        },
        "useraddress": "134 St Park Street",
        "city": "California",
        "citycode": "CA",
        "zipcode": 961540,
        "@clazz_name@": "com.openlegacy.entity.userinfonew.UserInfoNew"
    },
    {
        "id": "000020",
        "fullname":
        {
            "lastName": "Bradford",
            "firstName": "Billie",
            "@clazz_name@": "com.openlegacy.entity.fullname.Fullname"
        },
        "useraddress": "222 Baker Boulevard",
        "city": "Baltimore",
        "citycode": "MD",
        "zipcode": 357020,
        "@clazz_name@": "com.openlegacy.entity.userinfonew.UserInfoNew"
    },
    {
        "id": "000030",
        "fullname":
        {
            "lastName": "Carson",
            "firstName": "Cameron",
            "@clazz_name@": "com.openlegacy.entity.fullname.Fullname"
        },
        "useraddress": "333 Crenshaw Blvd.",
        "city": "Cupertino",
        "citycode": "CA",
        "zipcode": 961540,
        "@clazz_name@": "com.openlegacy.entity.userinfonew.UserInfoNew"
    }]
}

Properties

--source-path

The path to a local data-source copybook (CPY) file. The path is used by the CLI ol add command to access the file and parse it to extract its metadata.

--source-path ./user-info-new.cpy

--indicator-value

The indicator value identifies the record type of each record in the file. A VSAM file can contain multiple record types (e.g., account, customer, credit card).

--indicator-value E4A28599

--record-offset

The starting position of the record indicator value in the record.

--record-offset 8

--record-length

The number of characters reserved in the record for the record indicator.

--record-length 4

--pool.pool.enabled

Enable the ftp client connection pool

--pool.pool.enabled true

--pool.pool.max-size

The maximum connection number in the pool

--pool.pool.max-size 8

--timeout

Specify the timeout for FTP commands

--timeout 42000

--output-poll-interval

The time interval between output log polls in milliseconds

--output-poll-interval 300000

--output-poll-timeout

The maximum duration of an output log polling

--output-poll-timeout 1000

--local-passive-mode

FTP client transfer mode is set to passive (default: true)

--local-passive-mode true

--default-job-card

The default job card is used if the service source code does not specify the job card (job card is mandatory for the JCL to work).

--default-job-card "OL5SORT1 JOB A123,'CICS COBOL',CLASS=A,MSGCLASS=H,NOTIFY=&SYSUID"

--records-have-fixed-length

Whether the records have fixed length (default: false). If the record indicator is not specified, set the option to true.

--records-have-fixed-length true

--output-file-suffix

The suffix of the job output file. The prefix and suffix are used for retrieving the file via ftp.

--output-file-suffix suffix

--output-file-prefix

The prefix of the job output file. The prefix and suffix are used for retrieving the file via ftp.

--output-file-prefix prefix

--host

The mainframe dataset host address

--host 192.86.32.238

--user

The mainframe dataset username

--user demo

--password

The mainframe dataset password (when using a SSL client certificate do not add a password)

--password 12345

--code-page

The output file code page

--code-page 037

--port

The mainframe dataset port number

--port 2025

--delete-response-file

Whether the FTP file will be deleted after the asset response (default: true).

--delete-response-file false

--ssl.client-certificate-path

The path of the client certificate (required when using mTLS)

--ssl.store-pass

The store pass of the client certificate (required when using mTLS)

Runtime behavior

The MFDS connector receives a data request from the data consumer, converts it into JCL statements, and transfers them to the mainframe data source via FTP. The JCL statements trigger the MF job and return the Job ID via FTP.

Then, the connector, using FTP commands, polls the job log in predefined intervals and checks if the job ended successfully. If it does, the connector retrieves the output file via FTP, processes the data, and sends it back to the data consumer.