Kafka failed to map 1073741824 bytes for committing reserved memory - apache-kafka

Question

I am installing kafka on an aws t2 instance(one that has 1gb of memory).
(1) I download kafka_2.11-0.9.0.0
(2) I run zookeeper bin/zookeeper-server-start.sh config/zookeeper.properties
(3) I try running bin/kafka-server-start.sh config/server.properties
and I get
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000c0000000, 1073741824, 0) failed; error='Cannot allocate memory' (errno=12)
.#
.# There is insufficient memory for the Java Runtime Environment to continue.
.# Native memory allocation (mmap) failed to map 1073741824 bytes for committing reserved memory.
.# An error report file with more information is saved as:
.# /home/machine_name/kafka_2.11-0.9.0.0/hs_err_pid9161.log
I checked all propertes in the server.properties config file and in the documentation for properties that could try to do something like this but coudn't find anything.
Does anyone know why is kafka trying to allocated 1 gb when starting?

Answer 1

Kafka defaults to the following jvm memory parameters which mean that kafka will allocate 1GB at startup and use a maximum of 1GB of memory:
-Xmx1G -Xms1G
Just set KAFKA_HEAP_OPTS env variable to whatever you want to use instead. You may also just edit ./bin/kafka-server-start.sh and replace the values.

Answer 2

also if you have lower memory heap then try to reduce the size
-Xmx400M -Xms400M for both zookeeper and kafka

Answer 3

This issue might also relate to the maximum number of memory map areas allocated. It throws exactly the same error.
To remedy this you can run the following:
sysctl -w vm.max_map_count=200000
You want to set this in relation to your File Descriptor Limits. In summary, for every log segment on a broker, you require two map areas - one for index and one for time index.
For reference see the Kafka OS section: https://kafka.apache.org/documentation/#os

Answer 4

I was getting Java IO Exception: Map failed while starting Kafka-server. By analyzing previous logs it looks like it was failed because of insufficient memory in the java heap while loading logs. I changed the maximum memory size but it was not able to fix it. Finally, doing more research on google, I got to know that I had downloaded 32-bit version of java so downloading 64-bit version of java solved my problem.

Answer 5

Pass the KAFKA_HEAP_OPTS argument with your required memory value to run with.
Make sure to pass it in quotes - KAFKA_HEAP_OPTS="-Xmx512M -Xms512M"
docker run -it --rm --network app-tier -e KAFKA_HEAP_OPTS="-Xmx512M -Xms512M" -e KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper-server:2181 bitnami/kafka:latest kafka-topics.sh --list --bootstrap-server kafka-server:9092