经验首页 前端设计 程序设计 Java相关 移动开发 数据库/运维 软件/图像 大数据/云计算 其他经验
当前位置：技术经验 » 大数据/云/AI » Apache Kafka » 查看文章
Kafka使用jmxtrans+influxdb+grafana监控JMX指标
来源：cnblogs　　作者：湮灭时空　　时间：2019/4/11 8:59:02　　对本文有异议

　　最近在搞Kafka集群监控，之前也是看了网上的很多资料。之所以使用jmxtrans+influxdb+grafana是因为界面酷炫，可以定制化，缺点是不能操作Kafka集群，可能需要配合Kafka Manager一起使用。

　　环境信息

　　CentOS Linux release 7.6.1810 (Core)

　　jdk1.8.0_201

　　zookeeper-3.4.14

　　kafka_2.11-2.2.0

　　开启Kafka JMX端口

　　JMX（Java Management Extensions，即Java管理扩展）是一个为应用程序、设备、系统等植入管理功能的框架。JMX可以跨越一系列异构操作系统平台、系统体系结构和网络传输协议，灵活的开发无缝集成的系统、网络和服务管理应用。Kafka做为一款Java应用，已经定义了丰富的性能指标，(可以参考Kafka监控指标)，通过JMX可以轻松对其进行监控。

　　在${KAFKA_HOME}/bin/路径下修改kafka-run-class.sh脚本，第一行增加JMX_PORT=9999即可。

 

  1. JMX_PORT=9999

 

　　重启Kafka

  1. ./bin/kafka-server-stop.sh
  2. ./bin/kafka-server-start.sh -daemon ./config/server.properties

　　重启后查看Kafka以及JMX端口状态

  1. ps -ef | grep kafka
  2. root       8273      1 99 02:32 pts/0    00:00:09 /opt/jdk1.8.0_201/bin/java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 ......  kafka.Kafka ./config/server.properties
  4. netstat -anop | grep 9999
  5. tcp6       0      0 :::9999                 :::*                    LISTEN      8273/java            off (0.00/0/0)

　　安装InfluxDB

　　InfluxDB是一个时间序列数据库，用于处理海量写入与负载查询。InfluxDB旨在用作涉及大量时间戳数据的任何用例（包括DevOps监控，应用程序指标，物联网传感器数据和实时分析）的后端存储。

　　下载InfluxDB rpm安装包

  1. wget https://dl.influxdata.com/influxdb/releases/influxdb-1.7.5.x86_64.rpm
  2. --2019-04-10 02:52:30--  https://dl.influxdata.com/influxdb/releases/influxdb-1.7.5.x86_64.rpm
  3. Resolving dl.influxdata.com (dl.influxdata.com)... 54.192.151.21, 54.192.151.81, 54.192.151.87, ...
  4. Connecting to dl.influxdata.com (dl.influxdata.com)|54.192.151.21|:443... connected.
  5. HTTP request sent, awaiting response... 200 OK
  6. Length: 46536692 (44M) [application/octet-stream]
  7. Saving to: ‘influxdb-1.7.5.x86_64.rpm’
  9. 100%[================================================================================================================================================================================>] 46,536,692   440KB/s   in 60s    
  11. 2019-04-10 02:53:37 (756 KB/s) - ‘influxdb-1.7.5.x86_64.rpm’ saved [46536692/46536692]

　　安装rpm包

  1. rpm -ivh influxdb-1.7.5.x86_64.rpm 
  2. Preparing...                          ################################# [100%]
  3. Updating / installing...
  4.    1:influxdb-1.7.5-1                 ################################# [100%]
  5. Created symlink from /etc/systemd/system/influxd.service to /usr/lib/systemd/system/influxdb.service.
  6. Created symlink from /etc/systemd/system/multi-user.target.wants/influxdb.service to /usr/lib/systemd/system/influxdb.service.

　　启动InfluxDB

  1. service influxdb start
  2. Redirecting to /bin/systemctl start influxdb.service

　　查看InfluxDB状态

  1. ps -ef | grep influxdb
  2. influxdb   8475      1  2 03:01 ?        00:00:00 /usr/bin/influxd -config /etc/influxdb/influxdb.conf
  3. root       8486   7007  0 03:02 pts/0    00:00:00 grep --color=auto influxdb
  5. service influxdb status
  6. Redirecting to /bin/systemctl status influxdb.service
  7. ● influxdb.service - InfluxDB is an open-source, distributed, time series database
  8.    Loaded: loaded (/usr/lib/systemd/system/influxdb.service; enabled; vendor preset: disabled)
  9.    Active: active (running) since Wed 2019-04-10 03:01:48 EDT; 22s ago
  10.      Docs: https://docs.influxdata.com/influxdb/
  11.  Main PID: 8475 (influxd)
  12.    CGroup: /system.slice/influxdb.service
  13.            └─8475 /usr/bin/influxd -config /etc/influxdb/influxdb.conf
  15. Apr 10 03:01:48 node1 influxd[8475]: ts=2019-04-10T07:01:48.375804Z lvl=info msg="Starting precreation service" log_id=0EiWgWRl000 service=shard-precreation check_interval=10m advance_period=30m
  16. Apr 10 03:01:48 node1 influxd[8475]: ts=2019-04-10T07:01:48.375810Z lvl=info msg="Starting snapshot service" log_id=0EiWgWRl000 service=snapshot
  17. Apr 10 03:01:48 node1 influxd[8475]: ts=2019-04-10T07:01:48.375816Z lvl=info msg="Starting continuous query service" log_id=0EiWgWRl000 service=continuous_querier
  18. Apr 10 03:01:48 node1 influxd[8475]: ts=2019-04-10T07:01:48.375826Z lvl=info msg="Starting HTTP service" log_id=0EiWgWRl000 service=httpd authentication=false
  19. Apr 10 03:01:48 node1 influxd[8475]: ts=2019-04-10T07:01:48.375830Z lvl=info msg="opened HTTP access log" log_id=0EiWgWRl000 service=httpd path=stderr
  20. Apr 10 03:01:48 node1 influxd[8475]: ts=2019-04-10T07:01:48.375936Z lvl=info msg="Listening on HTTP" log_id=0EiWgWRl000 service=httpd addr=[::]:8086 https=false
  21. Apr 10 03:01:48 node1 influxd[8475]: ts=2019-04-10T07:01:48.375949Z lvl=info msg="Starting retention policy enforcement service" log_id=0EiWgWRl000 service=retention check_interval=30m
  22. Apr 10 03:01:48 node1 influxd[8475]: ts=2019-04-10T07:01:48.376138Z lvl=info msg="Listening for signals" log_id=0EiWgWRl000
  23. Apr 10 03:01:48 node1 influxd[8475]: ts=2019-04-10T07:01:48.376389Z lvl=info msg="Storing statistics" log_id=0EiWgWRl000 service=monitor db_instance=_internal db_rp=monitor interval=10s
  24. Apr 10 03:01:48 node1 influxd[8475]: ts=2019-04-10T07:01:48.376534Z lvl=info msg="Sending usage statistics to usage.influxdata.com" log_id=0EiWgWRl000

　　使用InfluxDB客户端

  1. influx
  2. Connected to http://localhost:8086 version 1.7.5
  3. InfluxDB shell version: 1.7.5
  4. Enter an InfluxQL query
  5. >

　　创建用户和数据库

  1. > CREATE USER "admin" WITH PASSWORD 'admin' WITH ALL PRIVILEGES
  1. > create database "jmxDB"

　　创建完成InfluxDB的用户和数据库暂时就够用了，其它简单操作如下，后面会用到

  1. #创建数据库
  2. create database "db_name"
  3. #显示所有的数据库
  4. show databases
  5. #删除数据库
  6. drop database "db_name"
  7. #使用数据库
  8. use db_name
  9. #显示该数据库中所有的表
  10. show measurements
  11. #创建表，直接在插入数据的时候指定表名
  12. insert test,host=127.0.0.1,monitor_name=test count=1
  13. #删除表
  14. drop measurement "measurement_name"
  15. #退出
  16. quit

　　安装jmxtrans

　　jmxtrans的作用是自动去jvm中获取所有jmx格式数据，并按照某种格式（json文件配置格式）输出到其他应用程序（本例中的influxDB）。

　　下载jmxtrans rpm安装包

  1. wget http://central.maven.org/maven2/org/jmxtrans/jmxtrans/270/jmxtrans-270.rpm
  2. --2019-04-10 03:18:14--  http://central.maven.org/maven2/org/jmxtrans/jmxtrans/270/jmxtrans-270.rpm
  3. Resolving central.maven.org (central.maven.org)... 151.101.40.209
  4. Connecting to central.maven.org (central.maven.org)|151.101.40.209|:80... connected.
  5. HTTP request sent, awaiting response... 200 OK
  6. Length: 18750744 (18M) [application/x-rpm]
  7. Saving to: ‘jmxtrans-270.rpm’
  9. 100%[================================================================================================================================================================================>] 18,750,744 342KB/s in 43s 

    2019-04-10 03:18:59 (422 KB/s) - ‘jmxtrans-270.rpm’ saved [18750744/18750744]

　　安装rpm包

rpm -ivh jmxtrans-270.rpm
Preparing... ################################# [100%]
Updating / installing...
1:jmxtrans-270-1 ################################# [100%]

 

　　jmxtrans相关路径

 

  1. jmxtrans安装目录：/usr/share/jmxtrans
  2. json文件默认目录：/var/lib/jmxtrans/
  3. 日志路径：/var/log/jmxtrans/jmxtrans.log

 

　　配置json，jmxtrans的github上有一段示例配置

  1. {
  2.   "servers" : [ {
  3.     "port" : "1099",
  4.     "host" : "w2",
  5.     "queries" : [ {
  6.       "obj" : "java.lang:type=Memory",
  7.       "attr" : [ "HeapMemoryUsage", "NonHeapMemoryUsage" ],
  8.       "resultAlias":"jvmMemory",
  9.       "outputWriters" : [ {
  10.         "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
  11.         "url" : "http://127.0.0.1:8086/",
  12.         "username" : "admin",
  13.         "password" : "admin",
  14.         "database" : "jmxDB",
  15.         "tags"     : {"application" : "kafka"}
  16.       } ]
  17.     } ]
  18.   } ]
  19. }

　　host：监控服务器

　　port：jmx端口

　　obj：对应jmx的ObjectName，就是我们要监控的指标

　　attr：对应ObjectName的属性，可以理解为我们要监控的指标的值

　　resultAlias：对应metric 的名称，在InfluxDB里面就是MEASUREMENTS名

　　tags：对应InfluxDB的tag功能，对与存储在同一个MEASUREMENTS里面的不同监控指标可以做区分，我们在用Grafana绘图的时候会用到，建议对每个监控指标都打上tags

 

　　启动jmxtrans

  1. service jmxtrans start
  2. Starting JmxTrans...

　　查看日志没有报错即为成功

  1. tail /var/log/jmxtrans/jmxtrans.log
  2. INFO   | jvm 1    | 2019/04/10 04:44:31 |   Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 10 threads.
  3. INFO   | jvm 1    | 2019/04/10 04:44:31 |   Using job-store 'org.quartz.simpl.RAMJobStore' - which does not support persistence. and is not clustered.
  4. INFO   | jvm 1    | 2019/04/10 04:44:31 | 
  5. INFO   | jvm 1    | 2019/04/10 04:44:31 | 2019-04-10 04:44:31 [WrapperSimpleAppMain] INFO  org.quartz.impl.StdSchedulerFactory - Quartz scheduler 'ServerScheduler' initialized from an externally opened InputStream.
  6. INFO   | jvm 1    | 2019/04/10 04:44:31 | 2019-04-10 04:44:31 [WrapperSimpleAppMain] INFO  org.quartz.impl.StdSchedulerFactory - Quartz scheduler version: 1.8.6
  7. INFO   | jvm 1    | 2019/04/10 04:44:31 | 2019-04-10 04:44:31 [WrapperSimpleAppMain] INFO  org.quartz.core.QuartzScheduler - JobFactory set to: com.googlecode.jmxtrans.guice.GuiceJobFactory@23822296
  8. 2019-04-10 04:44:31 [WrapperSimpleAppMain] level com.googlecode.jmxtrans.JmxTransformer [JmxTransformer.java:177] - Starting Jmxtrans on : /var/lib/jmxtrans
  9. 2019-04-10 04:44:31 [WrapperSimpleAppMain] level org.quartz.core.QuartzScheduler [QuartzScheduler.java:519] - Scheduler ServerScheduler_$_node11554885871753 started.
  10. INFO   | jvm 1    | 2019/04/10 04:44:31 | 2019-04-10 04:44:31 [WrapperSimpleAppMain] INFO  c.googlecode.jmxtrans.JmxTransformer - Starting Jmxtrans on : /var/lib/jmxtrans
  11. INFO   | jvm 1    | 2019/04/10 04:44:31 | 2019-04-10 04:44:31 [WrapperSimpleAppMain] INFO  org.quartz.core.QuartzScheduler - Scheduler ServerScheduler_$_node11554885871753 started.

　　附上两段通用的json文件

　　base_127.0.0.1.json

  1. {
  2.     "servers": [{
  3.         "port": "9999",
  4.         "host": "127.0.0.1",
  5.         "queries": [{
  6.             "obj": "kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec",
  7.             "attr": ["Count", "OneMinuteRate"],
  8.             "resultAlias": "BytesInPerSec",
  9.             "outputWriters": [{
  10.                 "@class": "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
  11.                 "url": "http://127.0.0.1:8086/",
  12.                 "username": "admin",
  13.                 "password": "admin",
  14.                 "database": "jmxDB",
  15.                 "tags": {
  16.                     "application": "BytesInPerSec"
  17.                 }
  18.             }]
  19.         }, {
  20.             "obj": "kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec",
  21.             "attr": ["Count", "OneMinuteRate"],
  22.             "resultAlias": "BytesOutPerSec",
  23.             "outputWriters": [{
  24.                 "@class": "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
  25.                 "url": "http://127.0.0.1:8086/",
  26.                 "username": "admin",
  27.                 "password": "admin",
  28.                 "database": "jmxDB",
  29.                 "tags": {
  30.                     "application": "BytesOutPerSec"
  31.                 }
  32.             }]
  33.         }, {
  34.             "obj": "kafka.server:type=BrokerTopicMetrics,name=BytesRejectedPerSec",
  35.             "attr": ["Count", "OneMinuteRate"],
  36.             "resultAlias": "BytesRejectedPerSec",
  37.             "outputWriters": [{
  38.                 "@class": "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
  39.                 "url": "http://127.0.0.1:8086/",
  40.                 "username": "admin",
  41.                 "password": "admin",
  42.                 "database": "jmxDB",
  43.                 "tags": {
  44.                     "application": "BytesRejectedPerSec"
  45.                 }
  46.             }]
  47.         }, {
  48.             "obj": "kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec",
  49.             "attr": ["Count", "OneMinuteRate"],
  50.             "resultAlias": "MessagesInPerSec",
  51.             "outputWriters": [{
  52.                 "@class": "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
  53.                 "url": "http://127.0.0.1:8086/",
  54.                 "username": "admin",
  55.                 "password": "admin",
  56.                 "database": "jmxDB",
  57.                 "tags": {
  58.                     "application": "MessagesInPerSec"
  59.                 }
  60.             }]
  61.         }, {
  62.             "obj": "kafka.network:type=RequestMetrics,name=RequestsPerSec,request=FetchConsumer",
  63.             "attr": ["Count"],
  64.             "resultAlias": "RequestsPerSec",
  65.             "outputWriters": [{
  66.                 "@class": "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
  67.                 "url": "http://127.0.0.1:8086/",
  68.                 "username": "admin",
  69.                 "password": "admin",
  70.                 "database": "jmxDB",
  71.                 "tags": {
  72.                     "request": "FetchConsumer"
  73.                 }
  74.             }]
  75.         }, {
  76.             "obj": "kafka.network:type=RequestMetrics,name=RequestsPerSec,request=FetchFollower",
  77.             "attr": ["Count"],
  78.             "resultAlias": "RequestsPerSec",
  79.             "outputWriters": [{
  80.                 "@class": "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
  81.                 "url": "http://127.0.0.1:8086/",
  82.                 "username": "admin",
  83.                 "password": "admin",
  84.                 "database": "jmxDB",
  85.                 "tags": {
  86.                     "request": "FetchFollower"
  87.                 }
  88.             }]
  89.         }, {
  90.             "obj": "kafka.network:type=RequestMetrics,name=RequestsPerSec,request=Produce",
  91.             "attr": ["Count"],
  92.             "resultAlias": "RequestsPerSec",
  93.             "outputWriters": [{
  94.                 "@class": "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
  95.                 "url": "http://127.0.0.1:8086/",
  96.                 "username": "admin",
  97.                 "password": "admin",
  98.                 "database": "jmxDB",
  99.                 "tags": {
  100.                     "request": "Produce"
  101.                 }
  102.             }]
  103.         }, {
  104.             "obj": "java.lang:type=Memory",
  105.             "attr": ["HeapMemoryUsage", "NonHeapMemoryUsage"],
  106.             "resultAlias": "MemoryUsage",
  107.             "outputWriters": [{
  108.                 "@class": "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
  109.                 "url": "http://127.0.0.1:8086/",
  110.                 "username": "admin",
  111.                 "password": "admin",
  112.                 "database": "jmxDB",
  113.                 "tags": {
  114.                     "application": "MemoryUsage"
  115.                 }
  116.             }]
  117.         }, {
  118.             "obj": "java.lang:type=GarbageCollector,name=*",
  119.             "attr": ["CollectionCount", "CollectionTime"],
  120.             "resultAlias": "GC",
  121.             "outputWriters": [{
  122.                 "@class": "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
  123.                 "url": "http://127.0.0.1:8086/",
  124.                 "username": "admin",
  125.                 "password": "admin",
  126.                 "database": "jmxDB",
  127.                 "tags": {
  128.                     "application": "GC"
  129.                 }
  130.             }]
  131.         }, {
  132.             "obj": "java.lang:type=Threading",
  133.             "attr": ["PeakThreadCount", "ThreadCount"],
  134.             "resultAlias": "Thread",
  135.             "outputWriters": [{
  136.                 "@class": "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
  137.                 "url": "http://127.0.0.1:8086/",
  138.                 "username": "admin",
  139.                 "password": "admin",
  140.                 "database": "jmxDB",
  141.                 "tags": {
  142.                     "application": "Thread"
  143.                 }
  144.             }]
  145.         }, {
  146.             "obj": "kafka.server:type=ReplicaFetcherManager,name=MaxLag,clientId=Replica",
  147.             "attr": ["Value"],
  148.             "resultAlias": "ReplicaFetcherManager",
  149.             "outputWriters": [{
  150.                 "@class": "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
  151.                 "url": "http://127.0.0.1:8086/",
  152.                 "username": "admin",
  153.                 "password": "admin",
  154.                 "database": "jmxDB",
  155.                 "tags": {
  156.                     "application": "MaxLag"
  157.                 }
  158.             }]
  159.         }, {
  160.             "obj": "kafka.server:type=ReplicaManager,name=PartitionCount",
  161.             "attr": ["Value"],
  162.             "resultAlias": "ReplicaManager",
  163.             "outputWriters": [{
  164.                 "@class": "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
  165.                 "url": "http://127.0.0.1:8086/",
  166.                 "username": "admin",
  167.                 "password": "admin",
  168.                 "database": "jmxDB",
  169.                 "tags": {
  170.                     "application": "PartitionCount"
  171.                 }
  172.             }]
  173.         }, {
  174.             "obj": "kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions",
  175.             "attr": ["Value"],
  176.             "resultAlias": "ReplicaManager",
  177.             "outputWriters": [{
  178.                 "@class": "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
  179.                 "url": "http://127.0.0.1:8086/",
  180.                 "username": "admin",
  181.                 "password": "admin",
  182.                 "database": "jmxDB",
  183.                 "tags": {
  184.                     "application": "UnderReplicatedPartitions"
  185.                 }
  186.             }]
  187.         }, {
  188.             "obj": "kafka.server:type=ReplicaManager,name=LeaderCount",
  189.             "attr": ["Value"],
  190.             "resultAlias": "ReplicaManager",
  191.             "outputWriters": [{
  192.                 "@class": "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
  193.                 "url": "http://127.0.0.1:8086/",
  194.                 "username": "admin",
  195.                 "password": "admin",
  196.                 "database": "jmxDB",
  197.                 "tags": {
  198.                     "application": "LeaderCount"
  199.                 }
  200.             }]
  201.         }, {
  202.             "obj": "kafka.network:type=RequestMetrics,name=TotalTimeMs,request=FetchConsumer",
  203.             "attr": ["Count", "Max"],
  204.             "resultAlias": "TotalTimeMs",
  205.             "outputWriters": [{
  206.                 "@class": "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
  207.                 "url": "http://127.0.0.1:8086/",
  208.                 "username": "admin",
  209.                 "password": "admin",
  210.                 "database": "jmxDB",
  211.                 "tags": {
  212.                     "application": "FetchConsumer"
  213.                 }
  214.             }]
  215.         }, {
  216.             "obj": "kafka.network:type=RequestMetrics,name=TotalTimeMs,request=FetchFollower",
  217.             "attr": ["Count", "Max"],
  218.             "resultAlias": "TotalTimeMs",
  219.             "outputWriters": [{
  220.                 "@class": "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
  221.                 "url": "http://127.0.0.1:8086/",
  222.                 "username": "admin",
  223.                 "password": "admin",
  224.                 "database": "jmxDB",
  225.                 "tags": {
  226.                     "application": "FetchFollower"
  227.                 }
  228.             }]
  229.         }, {
  230.             "obj": "kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Produce",
  231.             "attr": ["Count", "Max"],
  232.             "resultAlias": "TotalTimeMs",
  233.             "outputWriters": [{
  234.                 "@class": "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
  235.                 "url": "http://127.0.0.1:8086/",
  236.                 "username": "admin",
  237.                 "password": "admin",
  238.                 "database": "jmxDB",
  239.                 "tags": {
  240.                     "application": "Produce"
  241.                 }
  242.             }]
  243.         }, {
  244.             "obj": "kafka.server:type=ReplicaManager,name=IsrShrinksPerSec",
  245.             "attr": ["Count"],
  246.             "resultAlias": "ReplicaManager",
  247.             "outputWriters": [{
  248.                 "@class": "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
  249.                 "url": "http://127.0.0.1:8086/",
  250.                 "username": "admin",
  251.                 "password": "admin",
  252.                 "database": "jmxDB",
  253.                 "tags": {
  254.                     "application": "IsrShrinksPerSec"
  255.                 }
  256.             }]
  257.         }]
  258.     }]
  259. }
View Code

　　topicA_1.json

  1. {
  2.     "servers": [{
  3.         "port": "9999",
  4.         "host": "127.0.0.1",
  5.         "queries": [{
  6.             "obj": "kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=topicA",
  7.             "attr": ["Count"],
  8.             "resultAlias": "topicA",
  9.             "outputWriters": [{
  10.                 "@class": "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
  11.                 "url": "http://127.0.0.1:8086/",
  12.                 "username": "admin",
  13.                 "password": "admin",
  14.                 "database": "jmxDB",
  15.                 "tags": {
  16.                     "application": "BytesInPerSec"
  17.                 }
  18.             }]
  19.         }, {
  20.             "obj": "kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec,topic=topicA",
  21.             "attr": ["Count"],
  22.             "resultAlias": "topicA",
  23.             "outputWriters": [{
  24.                 "@class": "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
  25.                 "url": "http://127.0.0.1:8086/",
  26.                 "username": "admin",
  27.                 "password": "admin",
  28.                 "database": "jmxDB",
  29.                 "tags": {
  30.                     "application": "BytesOutPerSec"
  31.                 }
  32.             }]
  33.         }, {
  34.             "obj": "kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec,topic=topicA",
  35.             "attr": ["Count"],
  36.             "resultAlias": "topicA",
  37.             "outputWriters": [{
  38.                 "@class": "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
  39.                 "url": "http://127.0.0.1:8086/",
  40.                 "username": "admin",
  41.                 "password": "admin",
  42.                 "database": "jmxDB",
  43.                 "tags": {
  44.                     "application": "MessagesInPerSec"
  45.                 }
  46.             }]
  47.         }, {
  48.             "obj": "kafka.log:type=Log,name=LogEndOffset,topic=topicA,partition=*",
  49.             "attr": ["Value"],
  50.             "resultAlias": "topicA",
  51.             "outputWriters": [{
  52.                 "@class": "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
  53.                 "url": "http://127.0.0.1:8086/",
  54.                 "username": "admin",
  55.                 "password": "admin",
  56.                 "database": "jmxDB",
  57.                 "tags": {
  58.                     "application": "LogEndOffset"
  59.                 }
  60.             }]
  61.         }]
  62.     }]
  63. }
View Code

　　安装Grafana

　　Grafana是一个跨平台的开源的度量分析和可视化工具，可以通过将采集的数据查询然后可视化的展示，并及时通知。

　　下载jmxtrans rpm安装包

  1. wget https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana-6.0.2-1.x86_64.rpm
  2. --2019-04-10 04:53:15--  https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana-6.0.2-1.x86_64.rpm
  3. Resolving s3-us-west-2.amazonaws.com (s3-us-west-2.amazonaws.com)... 52.218.144.92
  4. Connecting to s3-us-west-2.amazonaws.com (s3-us-west-2.amazonaws.com)|52.218.144.92|:443... connected.
  5. HTTP request sent, awaiting response... 200 OK
  6. Length: 56002012 (53M) [application/x-redhat-package-manager]
  7. Saving to: ‘grafana-6.0.2-1.x86_64.rpm’
  9. 100%[================================================================================================================================================================================>] 56,002,012 177KB/s in 2m 52s 

    2019-04-10 04:56:08 (318 KB/s) - ‘grafana-6.0.2-1.x86_64.rpm’ saved [56002012/56002012]

　　安装rpm包

  1. rpm -ivh grafana-6.0.2-1.x86_64.rpm 
  2. warning: grafana-6.0.2-1.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID 24098cb6: NOKEY
  3. error: Failed dependencies:
  4.     fontconfig is needed by grafana-6.0.2-1.x86_64
  5.     urw-fonts is needed by grafana-6.0.2-1.x86_64

　　缺少依赖，下载依赖

  1. yum install --downloadonly --downloaddir=./ fontconfig
  3. yum localinstall fontconfig-2.13.0-4.3.el7.x86_64.rpm
  5. yum install --downloadonly --downloaddir=./ urw-fonts
  7. yum localinstall urw-fonts-2.4-16.el7.noarch.rpm
  9. rpm -ivh grafana-6.0.2-1.x86_64.rpm
  10. warning: grafana-6.0.2-1.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID 24098cb6: NOKEY
  11. Preparing...                          ################################# [100%]
  12. Updating / installing...
  13.    1:grafana-6.0.2-1                  ################################# [100%]
  14. ### NOT starting on installation, please execute the following statements to configure grafana to start automatically using systemd
  15.  sudo /bin/systemctl daemon-reload
  16.  sudo /bin/systemctl enable grafana-server.service
  17. ### You can start grafana-server by executing
  18.  sudo /bin/systemctl start grafana-server.service
  19. POSTTRANS: Running script

　　启动Grafana

  1. service grafana-server start
  2. Starting grafana-server (via systemctl):                   [  OK  ]

　　打开浏览器

  1. http://127.0.0.1:3000

　　先输入默认用户名密码admin/admin

　　设置新密码

　　点击Add data source

 

　　选择InfluxDB

 

　　输入连接信息后点击Save & Test

　　通过后点击Back返回

　　左侧 + 可以创建或引入仪表盘

　　类似于数据库SQL语句，查询相应的指标

　　计算平均每秒数值可以使用如上语法，用当前值减1分钟之前的值再除以60

 

　　具体展示效果就看各位的审美能力，这里就不贴出来了。至此，Kafka的JMX指标监控就完成了。

原文链接：http://www.cnblogs.com/ymsk/p/10685015.html

 友情链接：直通硅谷 　点职佳 　北美留学生论坛

Apache Kafka热门文章

  • Kafka监控工具汇总
  • Kraft模式下Kafka脚本的使用
  • Kafka web console安装
  • kafka_2.11-2.0.0_安装部署
  • ELK日志方案--使用Filebeat收集日志并输出到Kafka
  • Kafka相关内容总结（存储和性能）
  • 一次flume exec source采集日志到kafka因为单条日志数据非常大同步失败的踩坑带来的思考
  • Kafka使用jmxtrans+influxdb+grafana监控JMX指标
  • Micronaut 微服务中使用 Kafka
  • 什么是Kafka？

    • Apache Kafka推荐文章

  • 异源数据同步 → DataX 为什么要支持 kafka？
  • 解锁强强组合: 使用 Kafka + ClickHouse 快速搭建流数据实时处理平台(DoubleCloud 博客)
  • Kafka 线上性能调优
  • Kafka原理剖析之「位点提交」
  • Kafka 社区KIP-382中文译文（MirrorMaker2/集群复制/高可用/灾难恢复）
  • kafka源码阅读之MacBook Pro M1搭建Kafka2.7版本源码运行环境
  • Kafka干货之「零拷贝」
  • 扫盲Kafka？看这一篇就够了！
  • 从一次Kafka宕机说起（JVM hang）
  • 从Kafka中学习高性能系统如何设计

    • 本站QQ群：前端 618073944 | Java 606181507 | Python 626812652 | C/C++ 612253063 | 微信 634508462 | 苹果 692586424 | C#/.net 182808419 | PHP 305140648 | 运维 608723728

    W3xue 的所有内容仅供测试，对任何法律问题及风险不承担任何责任。通过使用本站内容随之而来的风险与本站无关。
    关于我们  |  意见建议  |  捐助我们  |  报错有奖  |  广告合作、友情链接（目前9元/月）请联系QQ：27243702 沸活量
    皖ICP备17017327号-2　皖公网安备34020702000426号