Installation steps

Gluesync SQL to NoSQL for Apache Kafka

Prerequisites

In order to have Gluesync working on your Kafka instance you will need to have:

  • as many topics created one per each of the entities you’re looking to source changes from;

  • valid user credentials with permissions to post to the above mentioned topics;

  • valid certificate in case of SASL authentication.

Basic configuration example

This module can be customized by using a configuration file, in JSON format. The file name to use must be specified as parameter when launching the app, with the -f or --file tokens. The file should be composed by union of common configuration file (see here Installation steps) and source specific configuration:

{
  ...
  // (source section omitted)
  ...
  "targetHost": "KAFKAIPADDRESS",
  "targetPort": "9092",
  "targetUsername": "KafkaUsername",
  "targetPassword": "KafkaPassword",
  ...

  "kafka": {
    "topicPrefix": "MYTOPICPREFIX_",
    "useTransactions": true,
    "saslMechanism": "SCRAM-SHA-256",
    "securityProtocol": "SASL_SSL",
    "certificatesType": "PEM",
    "certificatesPath": "/path/to/certificate.pem",
    "certificatesPassword": "certificatePassword"
  },
}

These following are the parameteres used to configure your Kafka target:

  • targetHost: the host address of your Kafka cluster;

  • targetPort: the port exposed by your Kafka cluster;

  • targetUsername: username to use to connect against Kafka;

  • targetPassword: password to use to connect against Kafka.

Apache Kafka specific configurations are listed under the kafka property:

  • topicPrefix (optional): string value that defines a prefix used in conjunction with the entity name in order to build the Kafka topic name;

  • useTransactions (optional): defaults to false. If set to true it enables the usage of Kafka transactions, look at this documentation in order to know more about it;

  • saslMechanism (required if auth in place): defaults to SCRAM-SHA-256, accepts the following parameters:

    • GSSAPI,

    • OAUTHBEARER,

    • PLAIN,

    • SCRAM-SHA-256,

    • SCRAM-SHA-512.

  • securityProtocol (required if auth in place): defaults to SASL_SSL, accepts the following parameters:

    • PLAINTEXT,

    • SASL_PLAINTEXT,

    • SASL_SSL,

    • SSL.

  • certificatesType (required if auth in place): defaults to PEM, accepts the following parameters:

    • PEM,

    • JKS.

  • certificatesPath (required if auth in place): defaults to null, the path to your certificates;

  • certificatesPassword (required if auth in place): defaults to null, the password used to lock your certificate;