Service Configuration

The most flexible way to start Kafka Monitor is to run kafka-monitor-start.sh with a config file, which allows you to instantiate multiple Service or App that are already implemented in Kafka Monitor and tune their configs to monitor your clusters. See here for an example config file.

In this section we list the configurations of the Service classes that have been implemented in Kafka Monitor. You can also visit the source code for up-to-date configures of each service (e.g. ProduceServiceConfig).

Produce Service Configuration Parameters

Name	Description	Type	Default	Valid Values	Importance
bootstrap.servers	A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. The client will make use of all servers irrespective of which servers are specified here for bootstrapping—this list only impacts the initial hosts used to discover the full set of servers. This list should be in the form host1:port1,host2:port2,.... Since these servers are just used for the initial connection to discover the full cluster membership (which may change dynamically), this list need not contain the full set of servers (you may want more than one, though, in case a server is down).	string			high
zookeeper.connect	Zookeeper connect string.	string			high
produce.partitioner.class	KMPartitioner class that corresponds to the partitioner used in the target cluster.	class	class com.linkedin.kmf.partitioner.NewKMPartitioner		high
topic	Topic to be used by the service.	string	kafka-monitor-topic		medium
produce.producer.class	Producer class that will be instantiated as producer in the produce service. It can be NewProducer, or full class name of any class that implements the KMBaseProducer interface.	string	com.linkedin.kmf.producer.NewProducer		low
produce.producer.id	Client id that will be used in the record sent by produce service.	string	kmf-producer		low
produce.record.delay.ms	The gap in ms between records sent to the same partition by produce service.	int	100		low
produce.record.size.byte	Size of record in bytes sent by produce service.	int	100		low
produce.sync	If true, and if this is supported by the producer class, messages are sent synchronously	boolean	true		low
produce.thread.num	Number of threads that produce service uses to send records.	int	5	[1,...]	low

Consume Service Configuration Parameters

Name	Description	Type	Default	Valid Values	Importance
bootstrap.servers	A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. The client will make use of all servers irrespective of which servers are specified here for bootstrapping—this list only impacts the initial hosts used to discover the full set of servers. This list should be in the form host1:port1,host2:port2,.... Since these servers are just used for the initial connection to discover the full cluster membership (which may change dynamically), this list need not contain the full set of servers (you may want more than one, though, in case a server is down).	string			high
zookeeper.connect	Zookeeper connect string.	string			high
consume.latency.sla.ms	The maximum latency of message delivery under SLA. Consume availability is measured as the fraction of messages that are either lost or whose delivery latency exceeds this value	int	20000		medium
consume.consumer.class	Consumer class that will be instantiated as consumer in the consume service. It can be NewConsumer, OldConsumer, or full class name of any class that implements the KMBaseConsumer interface.	string	com.linkedin.kmf.consumer.NewConsumer		low
consume.latency.percentile.granularity.ms	This is used to derive the bucket number used to configure latency percentile metric. The latency at the specified percentile should be multiple of this value.	int	1		low
consume.latency.percentile.max.ms	This is used to derive the bucket number used to configure latency percentile metric. Any latency larger than this max value will be rounded down to the max value.	int	5000		low
topic	Topic to be used by the service.	string	kafka-monitor-topic		low

Topic Management Service Parameters

Name	Description	Type	Default	Valid Values	Importance
zookeeper.connect	Zookeeper connect string.	string			high
topic	Topic to be used by the service.	string	kafka-monitor-topic		medium
topic-management.partitionsToBrokersRatio	Determines the number of partitions per broker when a topic is created or rebalanced. ceil(nBrokers * partitionsToBrokerRatio) is used to determine the actual number of partitions when partitions are added or removed.	double	2.0	[1,...]	low
topic-management.partitionsToBrokersRatioThreshold	The expected ratio of partitions / brokers. When the actual ratio falls below this threshold new partitions are created.	double	1.5	[1.0,...]	low
topic-management.rebalance.interval.ms	The gap in ms between the times the cluster balance on the monitored topic is checked. Set this to a large value to disable automatic topic rebalance.	int	600000	[10,...]	low
topic-management.replicationFactor	When a topic is created automatically this is the replication factor used.	int	1	[1,...]	low
topic-management.topicCreationEnabled	When true this automatically creates the topic mentioned by "topic" with replication factor "topic-management.replicationFactorand min ISR of max(topic-management.replicationFactor-1, 1) with number of brokers * "topic-management.partitionsToBrokersRatio" partitions.	boolean	true		low
topic-management.topicFactory.class.name	The name of the class used to create topics. This class must implement com.linkedin.kmf.topicfactory.TopicFactory.	string	com.linkedin.kmf.topicfactory.DefaultTopicFactory		low

Jetty Service Config

Name	Description	Type	Default	Valid Values	Importance
jetty.port	Port to be used by the Jetty service.	int	8000		low

Default Metrics Reporter Service Parameters

Name	Description	Type	Default	Valid Values	Importance
report.metrics.list	A list of objectName/attributeName pairs used to filter the metrics that will be exported. Only metrics that match any pair in the list will be exported. Each pair is in the form objectName:attributeName, where objectName and attributeName can contain wild card. If no objectName/attributeName is specified, all metrics with JMX prefix kmf.services will be reported	list	[kmf.services:*:*]		medium
report.interval.sec	The interval in second by which DefaultMetricsReporterService will report the metrics values.	int	1		low