经验首页 前端设计 程序设计 Java相关 移动开发 数据库/运维 软件/图像 大数据/云计算 其他经验
当前位置:技术经验 » 程序设计 » Elasticsearch » 查看文章
ELKF(elasticsearch、logstash、kibana、filebeat)搭建及收集nginx日志特昂糖-
来源:cnblogs  作者:特昂糖-  时间:2024/6/19 15:14:09  对本文有异议

1、elasticsearch

1.1、根目录下新建data文件夹

1.2、修改elasticsearch.yml文件,添加以下内容

  1. path.data: /home/wwq/elk/elasticsearch-8.13.4/data
  2. path.logs: /home/wwq/elk/elasticsearch-8.13.4/logs

 

1.3、修改jvm.options文件,新增以下内容

  1. -Xms2g
  2. -Xmx2g

 

1.4、启动

  1. bin/elasticsearch 前台启动
  2. bin/elasticsearch -d 后台启动

 

1.5、在日志中查看初始密码

  1. ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  2. ? Elasticsearch security features have been automatically configured!
  3. ? Authentication is enabled and cluster connections are encrypted.
  4. ?? Password for the elastic user (reset with `bin/elasticsearch-reset-password -u elastic`):
  5. qP3wO0GZ+pdomIp_ShHL
  6. ?? HTTP CA certificate SHA-256 fingerprint:
  7. 98aef4ebc491b232a4c1cbf1cbfe7b73e1e4ebb8567caa174097f5c69f2b41fd
  8. ?? Configure Kibana to use this cluster:
  9. ? Run Kibana and click the configuration link in the terminal when Kibana starts.
  10. ? Copy the following enrollment token and paste it into Kibana in your browser (valid for the next 30 minutes):
  11. eyJ2ZXIiOiI4LjEzLjQiLCJhZHIiOlsiMTk5Ljk5LjAuMTo5MjAwIl0sImZnciI6Ijk4YWVmNGViYzQ5MWIyMzJhNGMxY2JmMWNiZmU3YjczZTFlNGViYjg1NjdjYWExNzQwOTdmNWM2OWYyYjQxZmQiLCJrZXkiOiJmcF9ES3BBQlR6c3lRM0RMSU4teDoxclFRLXZraFREYUdYZmNiN2pQbXBBIn0=
  12. ?? Configure other nodes to join this cluster:
  13. ? On this node:
  14. ? Create an enrollment token with `bin/elasticsearch-create-enrollment-token -s node`.
  15. ? Uncomment the transport.host setting at the end of config/elasticsearch.yml.
  16. ? Restart Elasticsearch.
  17. ? On other nodes:
  18. ? Start Elasticsearch with `bin/elasticsearch --enrollment-token <token>`, using the enrollment token that you generated.
  19. ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

 

1.6、修改密码

  1. # 重置密码
  2. bin/elasticsearch-reset-password --username elastic -i

 

1.7、启动脚本

  1. #!/bin/bash
  2. #chkconfig: 345 63 37
  3. #description: elasticsearch
  4. #processname: elasticsearch-7.10.2
  5. # 这个目录是你Es所在文件夹的目录
  6. export ES_HOME=/home/wwq/elk/elasticsearch-8.13.4
  7. case $1 in
  8. start)
  9. cd $ES_HOME
  10. ./bin/elasticsearch -d -p pid
  11. exit
  12. !
  13. echo "elasticsearch is started"
  14. ;;
  15. stop)
  16. pid=`cat $ES_HOME/pid`
  17. kill -9 $pid
  18. echo "elasticsearch is stopped"
  19. ;;
  20. restart)
  21. pid=`cat $ES_HOME/pid`
  22. kill -9 $pid
  23. echo "elasticsearch is stopped"
  24. sleep 1
  25. cd $ES_HOME
  26. ./bin/elasticsearch -d -p pid
  27. exit
  28. !
  29. echo "elasticsearch is started"
  30. ;;
  31. *)
  32. echo "start|stop|restart"
  33. ;;
  34. esac
  35. exit 0

 

2.kibana

2.1、编辑kibana.yml

  1. server.port: 5601 # kibana的监听端口,可通过浏览器访问
  2. server.host: "0.0.0.0" # kibana监听本地IP,全零为本地所有网卡
  3. i18n.locale: "zh-CN"

 

2.2、启动

  1. 启动进程
  2. ./kibana
  3. 后台启动
  4. nohup ./kibana &

 

2.3、浏览器访问,配置即可

http://127.0.0.1:5601

2.4、设置开机自启

  1. vim /lib/systemd/system/kibana.service
  1. [Unit]
  2. Description=kibana
  3. After=network.target
  4. ?
  5. [Service]
  6. Type=simple
  7. User=tomcat
  8. ExecStart=/home/tomcat/elk/kibana-8.13.4/bin/kibana
  9. PrivateTmp=true
  10. ?
  11. [Install]
  12. WantedBy=multi-user.target
  1. systemctl enable kibana #开机自启
  2. systemctl start kibana #启动
  3. systemctl stop kibana #停止
  4. systemctl restart kibana #重启

 

 

3.logstash

3.1、获取es秘钥

  1. [tomcat@wwq bin]$ ./elasticsearch-keystore show xpack.security.http.ssl.keystore.secure_password
  2. warning: ignoring JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.402.b06-1.el7_9.x86_64/jre; using bundled JDK
  3. 04nkv5WkSLuB854KnE-Kxg

3.2、配置logstash

  1. vim /home/wwq/elk/logstash-8.13.4/config/logstash-sample.conf

 

  1. # Sample Logstash configuration for creating a simple
  2. # Beats -> Logstash -> Elasticsearch pipeline.
  3. ?
  4. input{
  5. beats{
  6. port => 5044
  7. }
  8. }
  9. ?
  10. output{
  11. elasticsearch{
  12. hosts => ["https://192.168.1.223:9200"]
  13. index => "%{[fields][logcategory]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
  14. user => "elastic"
  15. password => "123456"
  16. ssl_certificate_verification => true
  17. truststore => "/home/tomcat/elk/elasticsearch-8.13.4/config/certs/http.p12"
  18. truststore_password => "04nkv5WkSLuB854KnE-Kxg"
  19. }
  20. }

3.3、开机自启

  1. vim /lib/systemd/system/logstash.service
  1. [Unit]
  2. Description=logstash
  3. ?
  4. [Service]
  5. User=tomcat
  6. ExecStart=/home/tomcat/elk/logstash-8.13.4/bin/logstash -f /home/tomcat/elk/logstash-8.13.4/config/logstash-sample.conf
  7. Restart=always
  8. ?
  9. [Install]
  10. WantedBy=multi-user.target
  1. systemctl enable logstash #开机自启
  2. systemctl start logstash #启动
  3. systemctl stop logstash #停止
  4. systemctl restart logstash #重启

4、filebeat

4.1、nginx日志json格式

  1.  
  1. log_format log_json '{"@timestamp":"$time_iso8601",'
  2. '"server_addr":"$server_addr",'
  3. '"server_name":"$server_name",'
  4. '"server_port":"$server_port",'
  5. '"server_protocol":"$server_protocol",'
  6. '"client_ip":"$remote_addr",'
  7. '"client_user":"$remote_user",'
  8. '"status":"$status",'
  9. '"request_method": "$request_method",'
  10. '"request_length":"$request_length",'
  11. '"request_time":"$request_time",'
  12. '"request_url":"$request_uri",'
  13. '"request_line":"$request",'
  14. '"send_client_size":"$bytes_sent",'
  15. '"send_client_body_size":"$body_bytes_sent",'
  16. '"proxy_protocol_addr":"$proxy_protocol_addr",'
  17. '"proxy_add_x_forward":"$proxy_add_x_forwarded_for",'
  18. '"proxy_port":"$proxy_port",'
  19. '"proxy_host":"$proxy_host",'
  20. '"upstream_host":"$upstream_addr",'
  21. '"upstream_status":"$upstream_status",'
  22. '"upstream_cache_status":"$upstream_cache_status",'
  23. '"upstream_connect_time":"$upstream_connect_time",'
  24. '"upstream_response_time":"$upstream_response_time",'
  25. '"upstream_header_time":"$upstream_header_time",'
  26. '"upstream_cookie_name":"$upstream_cookie_name",'
  27. '"upstream_response_length":"$upstream_response_length",'
  28. '"upstream_bytes_received":"$upstream_bytes_received",'
  29. '"upstream_bytes_sent":"$upstream_bytes_sent",'
  30. '"http_host":"$host",'
  31. '"http_cookie":"$http_cooke",'
  32. '"http_user_agent":"$http_user_agent",'
  33. '"http_origin":"$http_origin",'
  34. '"http_upgrade":"$http_upgrade",'
  35. '"http_referer":"$http_referer",'
  36. '"http_x_forward":"$http_x_forwarded_for",'
  37. '"http_x_forwarded_proto":"$http_x_forwarded_proto",'
  38. '"https":"$https",'
  39. '"http_scheme":"$scheme",'
  40. '"invalid_referer":"$invalid_referer",'
  41. '"gzip_ratio":"$gzip_ratio",'
  42. '"realpath_root":"$realpath_root",'
  43. '"document_root":"$document_root",'
  44. '"is_args":"$is_args",'
  45. '"args":"$args",'
  46. '"connection_requests":"$connection_requests",'
  47. '"connection_number":"$connection",'
  48. '"ssl_protocol":"$ssl_protocol",'
  49. '"ssl_cipher":"$ssl_cipher"}';
  50. access_log logs/access_json.log log_json;

4.2、配置filebeat.yml

  1. vim /home/wwq/elk/filebeat-8.13.4/filebeat.yml

 

  1. ###################### Filebeat Configuration Example #########################
  2. ?
  3. # This file is an example configuration file highlighting only the most common
  4. # options. The filebeat.reference.yml file from the same directory contains all the
  5. # supported options with more comments. You can use it as a reference.
  6. #
  7. # You can find the full configuration reference here:
  8. # https://www.elastic.co/guide/en/beats/filebeat/index.html
  9. ?
  10. # For more available modules and options, please see the filebeat.reference.yml sample
  11. # configuration file.
  12. ?
  13. # ============================== Filebeat inputs ===============================
  14. ?
  15. filebeat.inputs:
  16. ?
  17. # Each - is an input. Most options can be set at the input level, so
  18. # you can use different inputs for various configurations.
  19. # Below are the input specific configurations.
  20. ?
  21. # filestream is an input for collecting log messages from files.
  22. - type: filestream
  23. ?
  24. # Unique ID among all inputs, an ID is required.
  25. id: my-filestream-id
  26. ?
  27. # Change to true to enable this input configuration.
  28. enabled: false
  29. ?
  30. # Paths that should be crawled and fetched. Glob based paths.
  31. paths:
  32. - /var/log/*.log
  33. #- c:\programdata\elasticsearch\logs\*
  34. ?
  35. # Exclude lines. A list of regular expressions to match. It drops the lines that are
  36. # matching any regular expression from the list.
  37. # Line filtering happens after the parsers pipeline. If you would like to filter lines
  38. # before parsers, use include_message parser.
  39. #exclude_lines: ['^DBG']
  40. ?
  41. # Include lines. A list of regular expressions to match. It exports the lines that are
  42. # matching any regular expression from the list.
  43. # Line filtering happens after the parsers pipeline. If you would like to filter lines
  44. # before parsers, use include_message parser.
  45. #include_lines: ['^ERR', '^WARN']
  46. ?
  47. # Exclude files. A list of regular expressions to match. Filebeat drops the files that
  48. # are matching any regular expression from the list. By default, no files are dropped.
  49. #prospector.scanner.exclude_files: ['.gz$']
  50. ?
  51. # Optional additional fields. These fields can be freely picked
  52. # to add additional information to the crawled log files for filtering
  53. #fields:
  54. # level: debug
  55. # review: 1
  56. ?
  57. # ============================== Filebeat modules ==============================
  58. ?
  59. filebeat.config.modules:
  60. # Glob pattern for configuration loading
  61. path: ${path.config}/modules.d/*.yml
  62. ?
  63. # Set to true to enable config reloading
  64. reload.enabled: false
  65. ?
  66. # Period on which files under path should be checked for changes
  67. #reload.period: 10s
  68. ?
  69. # ======================= Elasticsearch template setting =======================
  70. ?
  71. setup.template.settings:
  72. index.number_of_shards: 1
  73. #index.codec: best_compression
  74. #_source.enabled: false
  75. ?
  76. ?
  77. # ================================== General ===================================
  78. ?
  79. # The name of the shipper that publishes the network data. It can be used to group
  80. # all the transactions sent by a single shipper in the web interface.
  81. #name:
  82. ?
  83. # The tags of the shipper are included in their own field with each
  84. # transaction published.
  85. #tags: ["service-X", "web-tier"]
  86. ?
  87. # Optional fields that you can specify to add additional information to the
  88. # output.
  89. #fields:
  90. # env: staging
  91. ?
  92. # ================================= Dashboards =================================
  93. # These settings control loading the sample dashboards to the Kibana index. Loading
  94. # the dashboards is disabled by default and can be enabled either by setting the
  95. # options here or by using the `setup` command.
  96. #setup.dashboards.enabled: false
  97. ?
  98. # The URL from where to download the dashboards archive. By default this URL
  99. # has a value which is computed based on the Beat name and version. For released
  100. # versions, this URL points to the dashboard archive on the artifacts.elastic.co
  101. # website.
  102. #setup.dashboards.url:
  103. ?
  104. # =================================== Kibana ===================================
  105. ?
  106. # Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
  107. # This requires a Kibana endpoint configuration.
  108. setup.kibana:
  109. ?
  110. ?
  111. # ------------------------------ Logstash Output -------------------------------
  112. #output.logstash:
  113. # The Logstash hosts
  114. #hosts: ["localhost:5044"]
  115. ?
  116. # Optional SSL. By default is off.
  117. # List of root certificates for HTTPS server verifications
  118. #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
  119. ?
  120. # Certificate for SSL client authentication
  121. #ssl.certificate: "/etc/pki/client/cert.pem"
  122. ?
  123. # Client Certificate Key
  124. #ssl.key: "/etc/pki/client/cert.key"
  125. filebeat.inputs:
  126. - type: log
  127. enabled: true
  128. paths:
  129. #nginx日志目录
  130. - /home/wwq/nginx-1.25.5/logs/access_json.log
  131. fields:
  132. logcategory: nginx
  133. json:
  134. keys_under_root: true
  135. overwrite_keys: true
  136. message_key: "message"
  137. add_error_key: true
  138. ?
  139. output.logstash:
  140. hosts: ["192.168.1.200:5044"]
  141. #
  142. # ================================= Processors =================================
  143. processors:
  144. - add_host_metadata:
  145. when.not.contains.tags: forwarded
  146. - add_cloud_metadata: ~
  147. - add_docker_metadata: ~
  148. - add_kubernetes_metadata: ~
  149. ?
  150. # ================================== Logging ===================================
  151. ?
  152. # Sets log level. The default log level is info.
  153. # Available log levels are: error, warning, info, debug
  154. #logging.level: debug
  155. ?
  156. # At debug level, you can selectively enable logging only for some components.
  157. # To enable all selectors use ["*"]. Examples of other selectors are "beat",
  158. # "publisher", "service".
  159. #logging.selectors: ["*"]
  160. ?
  161. # ============================= X-Pack Monitoring ==============================
  162. # Filebeat can export internal metrics to a central Elasticsearch monitoring
  163. # cluster. This requires xpack monitoring to be enabled in Elasticsearch. The
  164. # reporting is disabled by default.
  165. ?
  166. # Set to true to enable the monitoring reporter.
  167. #monitoring.enabled: false
  168. ?
  169. # Sets the UUID of the Elasticsearch cluster under which monitoring data for this
  170. # Filebeat instance will appear in the Stack Monitoring UI. If output.elasticsearch
  171. # is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.
  172. #monitoring.cluster_uuid:
  173. ?
  174. # Uncomment to send the metrics to Elasticsearch. Most settings from the
  175. # Elasticsearch output are accepted here as well.
  176. # Note that the settings should point to your Elasticsearch *monitoring* cluster.
  177. # Any setting that is not set is automatically inherited from the Elasticsearch
  178. # output configuration, so if you have the Elasticsearch output configured such
  179. # that it is pointing to your Elasticsearch monitoring cluster, you can simply
  180. # uncomment the following line.
  181. #monitoring.elasticsearch:
  182. ?
  183. # ============================== Instrumentation ===============================
  184. ?
  185. # Instrumentation support for the filebeat.
  186. #instrumentation:
  187. # Set to true to enable instrumentation of filebeat.
  188. #enabled: false
  189. ?
  190. # Environment in which filebeat is running on (eg: staging, production, etc.)
  191. #environment: ""
  192. ?
  193. # APM Server hosts to report instrumentation results to.
  194. #hosts:
  195. # - http://localhost:8200
  196. ?
  197. # API Key for the APM Server(s).
  198. # If api_key is set then secret_token will be ignored.
  199. #api_key:
  200. ?
  201. # Secret token for the APM Server(s).
  202. #secret_token:
  203. ?
  204. ?
  205. # ================================= Migration ==================================
  206. ?
  207. # This allows to enable 6.7 migration aliases
  208. #migration.6_to_7.enabled: true

4.3、启动

  1. filebeat -e -c filebeat.yml

4.4、开机自启

  1. vim /lib/systemd/system/filebeat.service
  1. [Unit]
  2. Description=filebeat
  3. Wants=network-online.target
  4. After=network-online.target
  5. ?
  6. [Service]
  7. User=tomcat
  8. ExecStart=/home/tomcat/elk/filebeat-8.13.4/filebeat -e -c /home/tomcat/elk/filebeat-8.13.4/filebeat.yml
  9. Restart=always
  10. ?
  11. [Install]
  12. WantedBy=multi-user.target
  1. systemctl enable filebeat #开机自启
  2. systemctl start filebeat #启动
  3. systemctl stop filebeat #停止
  4. systemctl restart filebeat #重启

 

 5、页面访问kibana及配置

 

 

原文链接:https://www.cnblogs.com/teangtang/p/18256316

 友情链接:直通硅谷  点职佳  北美留学生论坛

本站QQ群:前端 618073944 | Java 606181507 | Python 626812652 | C/C++ 612253063 | 微信 634508462 | 苹果 692586424 | C#/.net 182808419 | PHP 305140648 | 运维 608723728

W3xue 的所有内容仅供测试,对任何法律问题及风险不承担任何责任。通过使用本站内容随之而来的风险与本站无关。
关于我们  |  意见建议  |  捐助我们  |  报错有奖  |  广告合作、友情链接(目前9元/月)请联系QQ:27243702 沸活量
皖ICP备17017327号-2 皖公网安备34020702000426号