Kafka JSON Connector Guide

The Kafka JSON connector enables clients to connect to a Kafka stream processor and publish their content to a specified topic.

The Kafka JSON connector data flow consists of the following steps:

1. A Client sends a JSON object specifying the data to be pushed to the Kafka topic.
2. The object is then serialized according to the Kafka asset definition as a Kafka message.
3. The message is then pushed to the Kafka topic in JSON format.

The guide outline

To configure the OpenLegacy connector to work with your Kafka implementation, perform the following steps:

1. Use the CLI to generate an asset containing a model of the Kafka message input parameters; see steps 1 - 4 below.
2. Upload the asset to an OpenLegacy Hub project; see step 5 below. 
3. Generate a microservice on top of the uploaded asset in the OL Hub and deploy it in your destination platform; see step 6 below. 

Prerequisites

• Download and install OpenLegacy CLI for Windows or macOS.
• Sign in to OpenLegacy Hub and obtain your login credentials.
• After you are logged in to OL Hub, generate an API key to be able to log in to the hub from the CLI.
• Prepare a Kafka message example identical in format to the messages that the microservice is supposed to create. 

Create Kafka JSON module and assets

To create a Kafka JSON asset, you will perform the following steps in the OpenLegacy CLI:

1. Create a Kafka JSON module using the Kafka JSON connector.
2. Test the connection to the remote Kafka server. 
3. Create a Kafka JSON asset. 
4. Test the created asset by sending a test message to the Kafka topic.

Step 1. Create a Kafka JSON module

First, open the CLI and create the Kafka JSON module. 

[module-name]> ol create module kafka-json-module --connector kafka-json

[module-name]$ ol create module kafka-json-module --connector kafka-json

Step 2. Test the connection to the remote Kafka server

Run the ol test connection command to verify that the connection properties are correct and that the remote Kafka server is accessible. 

1. Navigate to the module folder:

   WindowsmacOS

   [module-name]> cd kafka-json-module
   

   [module-name]$ cd kafka-json-module
   

2. From under the module directory, display the help menu of the ol test connection command to review the Kafka JSON connector connection properties.

   WindowsmacOS

   > ol test connection --help
   

   $ ol test connection --help
   

The ol test connection help menu displays the Kafka connector connection properties.

Options:
  --verbose                             Enables verbose mode
  --verbose-debug                       Enables debug mode
  --verbose-trace                       Enables trace mode
  --timeout TEXT                        Specify delivery.timeout.ms (default: 30000)
  --acks TEXT                           Specify ack strategy ONE/NONE/ALL[ONE, NONE, ALL] (default: ONE)
  --topic TEXT                          Specify kafka topic
  --depending-on-mode TEXT              Specify depending on mode (default: true)
  --client.bootstrap-servers TEXT       A comma-separated list of host and port pairs
  --client.group-id TEXT                Specifies the name of the consumer group a Kafka consumer belongs to
  --client.additional-properties VALUE
  --client.poll-timeout TEXT            Specify no commit poll timeout in milliseconds (default: 10)
  --profile TEXT                        Configuration profile
  -h, --help                            Show this message and exit

To learn more about the ol test connection properties see below.

3. Run ol test connection:
   WindowsmacOS

   [module-name]> ol test connection --client.bootstrap-servers kafka-sandbox.openlegacy.com:9092 --client.group-id kafka-demo --topic test
   

   [module-name]$ ol test connection --client.bootstrap-servers kafka-sandbox.openlegacy.com:9092 --client.group-id kafka-demo --topic test
   

Step 3. Add an asset to the Kafka JSON module

After verifying the connection parameters, you can create a Kafka JSON asset by executing the ol add command. To create the asset, provide to the ol add command a JSON file containing an example of the Kafka message.  The connector parses this example to compose the input and output metadata model of the Kafka asset. 

1. Review the properties of ol add command:
   WindowsmacOS

   [module-name]> ol add --help
   

   [module-name]$ ol add --help
   

The ol add help menu displays the connector property list.

Options:
  --verbose                             Enables verbose mode
  --verbose-debug                       Enables debug mode
  --verbose-trace                       Enables trace mode
  --additional VALUE                    Additional properties
  --program-path TEXT                   Provide the cobol program path. Required when parsing multiple copybooks without cobol file
  --asset-name TEXT                     Provide asset name when parsing multiple copybooks without cobol file
  -i, --input TEXT                      Provide input copybook file path Or multiple input copybooks file paths Or input copybooks directory
  -o, --output TEXT                     Provide output copybook file path Or multiple output copybooks file paths Or output copybooks directory
  --source-path TEXT                    Provide local source or directory to parse cobol files
  --entity                              Indicate parse as entity
  --strict                              strictly parse a resource
  --input-super-class TEXT              Provide input entity super class
  --output-super-class TEXT             Provide output entity super class
  --timeout TEXT                        Specify delivery.timeout.ms (default: 30000)
  --acks TEXT                           Specify ack strategy ONE/NONE/ALL[ONE, NONE, ALL] (default: ONE)
  --topic TEXT                          Specify kafka topic
  --depending-on-mode TEXT              Specify depending on mode (default: true)
  --client.bootstrap-servers TEXT       A comma-separated list of host and port pairs
  --client.group-id TEXT                Specifies the name of the consumer group a Kafka consumer belongs to
  --client.additional-properties VALUE
  --client.poll-timeout TEXT            Specify no commit poll timeout in milliseconds (default: 10)
  --profile TEXT                        Configuration profile
  -h, --help                            Show this message and exit

2. Provide to the ol add command a JSON source file with a Kafka message example:
   WindowsmacOS

   [module-name]> ol add --source-path ./acc_balance.json
   

   [module-name]$ ol add --source-path ./acc_balance.json
   

To learn more about the ol add properties see below.

Step 4. Test the Kafka JSON asset

After adding the Kafka JSON asset, you can test the asset by executing the ol test asset command to verify that you can push messages to the Kafka topic.

1.  From your module folder root go to assets/acc_balance/test_data/case1/:

   WindowsmacOS

   [module-name]> cd assets/acc_balance/test_data/case1/
   

   [module-name]$ cd assets/acc_balance/test_data/case1/
   

2. To view and edit the test input data, open in.json:

   WindowsmacOS

   case-1> notepad in.json
   

   case-1$ notepad in.json
   

in.json contains the information sent to the Kafka topic. 

An example for path parameters:

{
  "select" : {
    "id" : 2,
    "accountNumber" : 20,
    "balance" : 120,
    "owner" : "John"
  }
}

3. In in.json, type the values of the input properties. In the example above, all field values are populated.

4. Now you are ready to run ol test asset:

   WindowsmacOS

   > ol test asset acc_balance
   

   $ ol test asset acc_balance
   

The response from the test is displayed:

Test asset for test case-1 succeeded.
response:
{ }

5. Search the relevant topic in Kafka and confirm that the message you sent as part of the test appears in the message list.

Step 5: Push the module to the OL Hub

Now that you are sure that the Kafka JSON assets are operating as they should, that the connecting properties are accurate, and that the Kafka message is listed in the topic, you can push the finalized module to the hub.

❗️

Make sure you are logged in to the OL Hub before executing ol push module.

1.  Run the ol push module command from the module folder root:

   WindowsmacOS

   [module-name]> ol push module
   

   [module-name]$ ol push module
   

2.  The CLI displays the command success status:

   Module kafka-json-module pushed successfully to the HUB
   

Step 6: Continue in OL Hub

Your Kafka module is now on the OL Hub and can be included in projects. 

Next, generate a microservice on top of your uploaded asset in the OL Hub.

Connector Options

--additional

key-value pairs of dynamic options

--additional [missing_value]

--source-path

The path to a local data-source COBOL file. The path is used by the CLI ol add command to access the file and parse it to extract its metadata. The property is not required if the input, the output and the program-path properties are populated.

--source-path ./acc_balance.json

--program-path

The Kafka topic name. If not specified the account level name specified in the topic option is used.

--program-path test-demo

--asset-name

The name of the new asset. If none is provided the name of the data source specified in the source-path option is used.

--asset-name

--timeout

The client waiting period for Kafka to return a confirmation response after a message was pushed (default: 30000).

--timeout 5000

--acks

The number of acknowledgments the client requires the leader to have received before considering a request complete [ONE, NONE, ALL] (default: ONE).

--acks none

--topic

The Kafka topic to be used when no topic name is provided in the source-path and program-path properties.

--topic default-input

--client.bootstrap-servers

A comma-separated list of host and port pairs

--client.bootstrap-servers kafka-sandbox.openlegacy.com:9092

--client.group-id

The name of the consumer group a Kafka consumer belongs to

--client.group-id consumer-cluster-a

--client.additional-properties

key-value pairs describing configuration properties of the client library

--client.additional-properties max.block.ms=60000 max.request.size=1048576

--profile

The connection profile of ol add or ol test connection (see: Connection Profiles).

--profile dev