JDBC Sync connector errors with JSON data.

Problem

Users may encounter difficulties when attempting to use the JDBC sink connector to transfer JSON data from a Redpanda topic to a JDBC database, particularly when the JSON data doesn't include explicit schema information.

Error Message

The following error may occur when trying to use the JDBC sink connector with JSON data lacking schema information:

Caused by: org.apache.kafka.connect.errors.DataException: JsonConverter with schemas.enable requires "schema" and "payload" fields and may not contain additional fields. If you are trying to deserialize plain JSON data, set schemas.enable=false in your converter configuration.

Resolution

To resolve this issue, follow these steps:

1. Use Schema Mapping: The JDBC sink connector requires a schema for the data it processes. Use the schema mapping approach as documented in the RedPanda documentation.

2. Structure Your JSON Data: Ensure your JSON data includes both schema and payload fields. Here's an example of properly structured JSON data:

   {
     "schema": {
       "type": "struct",
       "fields": [
         {
           "type": "int64",
           "optional": false,
           "field": "sensor_id"
         },
         {
           "type": "string",
           "optional": false,
           "field": "metric"
         },
         {
           "type": "int32",
           "optional": false,
           "field": "value"
         }
       ]
     },
     "payload": {
       "sensor_id": 1,
       "metric": "Temperature",
       "value": 40
     }
   }

3. Configure the Connector: In your connector configuration, ensure you have the following settings:

   {
     "value.converter": "org.apache.kafka.connect.json.JsonConverter",
     "value.converter.schemas.enable": "true"
   }

Important Notes

• It is currently not possible to send plain JSON without a schema/payload structure using the JDBC sink connector in Redpanda.
• This limitation is being worked on in KIP-301, but it is not yet implemented.

Alternative Approach

If you cannot modify your JSON data to include schema information, consider using a different connector or implementing a pre-processing step to add the required schema information to your JSON data before it reaches the JDBC sink connector.

 

If you need additional assistance please reach out to Redpanda Support.