Kafka processes footprint
We have a node configured as a Universal node. On startup this spins up 2 additional Kafka processes. As per the documentation I would expect with a default heap size of 1Gb. We see this alongside an additional process that utilises around 1.4Gb. We need to understand expected memory footprint for the application and any supporting processes to accurately size production and test hardware.
***Edited by Moderator Marissa to update SR Details***
Pegasystems Inc.
PL
Hello,
That is right: default JVM Heap for Kafka-broker is 1GB unless stated differently in prconfig.xml file.
To answer your questions better I'd like to ask for output of those commands:
jps -l
ps -ef | grep java
top (as you already did, but matched in time with executing commands above)
Also please check if values in your prconfig.xml relect default ones.
Regards,
Michal
Pegasystems Inc.
The Kafka element itself is pretty clear, the supporting Cassandra process is not. How much memory is this expected to consume? How do we size accordingly for this in terms of both memory and disk space?
Pegasystems Inc.
PL
Regarding Cassandra:
1. How it can be tuned to suit your needs:
Cassandra heap can be configured via the following prconfig.xml settings:
<env name="dnode/cassandra_heap_new_size" value="400M" />
<env name="dnode/cassandra_max_heap_size" value="2G" />
Regarding Cassandra:
1. How it can be tuned to suit your needs:
Cassandra heap can be configured via the following prconfig.xml settings:
<env name="dnode/cassandra_heap_new_size" value="400M" />
<env name="dnode/cassandra_max_heap_size" value="2G" />
Note that you need to set both properties if you want to override the Cassandra defaults.
2. How is it counted by default.
This article from datastax exlpain all in details:
https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsTuneJVM.html
For impatient:
Cassandra uses the following rules to calculate what the heap should be
# set max heap size based on the following
# max(min(1/2 ram, 1024MB), min(1/4 ram, 8GB))
# calculate 1/2 ram and cap to 1024MB
# calculate 1/4 ram and cap to 8192MB
# pick the max
I hope this helps. Should you need more info - please ask.
Regards,
Michal
Pegasystems Inc.
In terms of the Cassandra process - in a deployment where I am not using the decision component why is this running? Is this because of the "Universal" node classification that I have applied?