Kafka Connect 的堆空间不足

java jvm apache-kafka apache-kafka-connect

2022-09-01 00:18:54

启动 Kafka Connect （）后，我的任务在以下情况下立即失败：connect-standalone

java.lang.OutOfMemoryError: Java heap space
    at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
    at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
    at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:93)
    at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71)
    at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:154)
    at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:135)
    at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:343)
    at org.apache.kafka.common.network.Selector.poll(Selector.java:291)
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:260)
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:232)
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:180)
    at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:193)
    at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:248)
    at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1013)
    at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:979)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.pollConsumer(WorkerSinkTask.java:316)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:222)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:170)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:142)
    at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:140)
    at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:175)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

在一些Kafka文档中提到了堆空间，告诉您使用“默认值”尝试它，并且仅在出现问题时才修改它，但没有修改堆空间的说明。

答案 1

当你有卡夫卡问题

java.lang.OutOfMemoryError: Java heap space

这并不一定意味着这是一个记忆问题。一些像Kafka管理工具一样，在尝试连接到SSL端口时会掩盖真正的错误。真正的（被掩盖的）错误是！kafka-topics.shSSL handshake failed

请参阅此问题：https://issues.apache.org/jira/browse/KAFKA-4090

解决方案是在命令中包含一个属性文件（因为这将是），并且绝对包含以下行：kafka-topics.sh--command-config

security.protocol=SSL

答案 2

您可以通过设置环境变量来控制最大堆大小和初始堆大小。KAFKA_HEAP_OPTS

以下示例将起始大小设置为 512 MB，最大大小设置为 1 GB：

KAFKA_HEAP_OPTS="-Xms512m -Xmx1g" connect-standalone connect-worker.properties connect-s3-sink.properties

运行 Kafka 命令（如）时，将调用该脚本，如果尚未设置，则该脚本会在环境变量中设置默认堆大小 256 MB。connect-standalonekafka-run-classKAFKA_HEAP_OPTS