Installation Steps
Gluesync SQL to NoSQL for Apache Kafka
Prerequisites
In order to have Gluesync working on your Kafka instance you will need to have:
-
as many topics created one per each of the entities you’re looking to source changes from;
-
valid user credentials with permissions to post to the above mentioned topics;
-
valid certificate in case of SASL authentication.
Basic configuration example
This module can be customized by using a configuration file, in JSON format.
The file name to use must be specified as parameter when launching the app, with the -f
or --file
tokens.
The file should be composed by union of common configuration file (see here Installation Steps) and source specific configuration:
{
...
// (source section omitted)
...
"targetHost": "KAFKAIPADDRESS",
"targetPort": "9092",
"targetUsername": "KafkaUsername",
"targetPassword": "KafkaPassword",
...
"kafka": {
"topicPrefix": "MYTOPICPREFIX_",
"useTransactions": true,
"saslMechanism": "SCRAM-SHA-256",
"securityProtocol": "SASL_SSL",
"certificatesType": "PEM",
"certificatesPath": "/path/to/certificate.pem",
"certificatesPassword": "certificatePassword"
},
}
These following are the parameteres used to configure your Kafka target:
-
targetHost: the name of the region where your bucket belongs to;
-
targetPort: the port exposed by your Kafka cluster;
-
targetUsername: username to use to connect against Kafka;
-
targetPassword: password to use to connect against Kafka.
Apache Kafka specific configurations are listed under the kafka
property:
-
topicPrefix (optional): string value that defines a prefix used in conjunction with the entity name in order to build the Kafka topic name;
-
useTransactions (optional): defaults to
false
. If set totrue
it enables the usage of Kafka transactions, look at this documentation in order to know more about it; -
saslMechanism (required if auth in place): defaults to
SCRAM-SHA-256
, accepts the following parameters:-
GSSAPI
, -
OAUTHBEARER
, -
PLAIN
, -
SCRAM-SHA-256
, -
SCRAM-SHA-512
.
-
-
securityProtocol (required if auth in place): defaults to
SASL_SSL
, accepts the following parameters:-
PLAINTEXT
, -
SASL_PLAINTEXT
, -
SASL_SSL
, -
SSL
.
-
-
certificatesType (required if auth in place): defaults to
PEM
, accepts the following parameters:-
PEM
, -
JKS
.
-
-
certificatesPath (required if auth in place): defaults to
null
, the path to your certificates; -
certificatesPassword (required if auth in place): defaults to
null
, the password used to lock your certificate;