1、 Background

Using drainer to synchronize to Kafka, an error is reported:

 ["fail to produce message to kafka, please check the state of kafka server"] [error="kafka: Failed to produce message to topic test-tidb: kafka server: Message was too large, server rejected it to avoid allocation error."]

2、 Settle

Error reason: if a large transaction is executed in tidb, the generated binlog data will be large, which may exceed the message size limit of Kafka.

Solution: you need to adjust the configuration parameters of Kafka cluster, as shown below.

message.max.bytes=1073741824
replica.fetch.max.bytes=1073741824
fetch.message.max.bytes=1073741824