扫一扫,微信登陆

 青浦修电脑 青浦笔记本维修 青浦手机维修 青浦电器维修

搜索
查看: 550|回复: 0

ELK日志收集

[复制链接]

1万

主题

1万

帖子

5万

积分

论坛元老

Rank: 8Rank: 8

积分
56206
发表于 2022-9-5 08:10:05 | 显示全部楼层 |阅读模式
搭建ELK ; t% O0 s# `" c, n& v
ELK是由elasticsearch、logstash、kibana三个开源软件组成的一个组合体,ELK是elastic公司公司研发的一套完整的日志收集、分析和展示的企业级解决方案,在这三个软件当中,每个软件用于完成不同的功能,官方域名为elastic.io,ELK stack的主要优点:
/ x9 S/ `) V. P # I" i1 r- t$ P
处理方式灵活:elasticsearch是实时全文索引,具有强大的搜索功能配置相当简单:elasticsearch的API全部使用JSON接口,logstash使用模块配置,kibana的配置文件部分更简单检索性能高效:基于优秀的设计,虽然每次查询都是实时,但是也可以达到百亿数据的查询秒级响应。集群线性扩展:elasticsearch和logstash都可以灵活线性扩展前端操作绚丽:kibana的前端设计比较绚丽,而且操作简单, e- S* V/ ]! \' ~4 H
Elasticsearch , X6 `! s/ N+ N( W
elasticsearch是一个高度可扩展的开源全文搜索和分析引擎,它可实现数据的实时全文搜索、支持分布式可实现高可用、提供API接口,可以处理大规模日志数据,比如nginx、tomcat、系统日志等功能。
1 U1 u9 H* @" Q, C4 ^: i * k7 Z/ q7 T' \8 q2 d
elasticsearch的特点:
6 [, k1 l& U' ^  @6 f
: x) ?9 r6 ?! I# R! w" |+ H( n实时收索、实时分析分布式架构、实时文件存储文档导向,所有对象都是文档高可用,易扩展,支持集群,分片与复制接口友好,支持json
' m0 j$ P- t" ]$ k9 n部署elasticsearch
8 _, R! X. r  pGitHub - elastic/elasticsearch: Free and Open, Distributed, RESTful Search Engine,基于java开发
( l' a8 n" t4 E0 a4 @ ) `5 y+ z$ L; j0 j
centos系统关闭服务器的防火墙和selinux,ubuntu关闭防火墙,保持各服务器时间同步- x# ]: m0 S* w3 |+ Z1 E
: i9 R: d0 }, u9 \: x
服务器1:172.20.22.24/ r8 {# f3 V7 N! A& t
" Y; u  I& c2 b/ O
服务器2:172.20.22.27, d1 a$ ~/ B/ ?, i- F
* f' B/ i' Y! r+ p  I0 ?
服务器3:172.20.22.28
& J5 K3 l+ |$ Q- X' f* E7 s
& M. n: Q2 x( s3 K' i3 d###ubuntu  [1 l; h& E6 x9 a; P- K* v- m
# apt install -y ntpdate* _$ J1 l  M5 F3 |% u
# rm -f /etc/localtime
! E0 @: Q- u9 r6 X) ?% N# ln -s /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
( I  g6 i! `/ ]* M4 c# hwclock --systohc  i1 G" x6 t0 @
# ntpdate -u ntp1.aliyun.com/ X* f0 W2 z' W) W3 w! k1 _$ E% L
###设置内核参数# M2 L. i4 W  S& ?! J
# vim /etc/security/limits.conf( V$ [/ k* P% d+ g, H& h
*                soft        nofile                500000
" T; b: d/ x# [. b* O*                hard        nofile                5000006 N: [9 a* Q: Q- n, {" D3 \
# vim /etc/security/limits.d/20-nproc.conf
) q: _  p% n8 c! H1 e% m*          soft    nproc     4096: I  ?* L( d3 [% W% F
elasticsearch soft    nproc     unlimited
4 k7 k9 h' j4 ~  broot       soft    nproc     unlimited
/ D: [, y: i# }! O; Z- P8 p- U###安装jdk
3 `$ a+ p( h8 w6 o6 \3 H# apt install -y openjdk-8-jdk
0 `1 M) r% L. e9 f2 I* g8 @3 K2 n5 `! ^# k% S: Z* o
###每个节点都安装
9 z7 I) m4 W9 h" j9 E# ls -lrt elasticsearch-7.12.1-amd64.deb- J1 G2 D" a( W
# dpkg -i elasticsearch-7.12.1-amd64.deb6 K4 D0 k2 t5 \+ D% g
###节点1配置文件) O; s  K% Z( d. f& U
# grep '^[^#]' /etc/elasticsearch/elasticsearch.yml, I! U7 }, ~# N1 w5 C$ \
cluster.name: m63-elastic        #集群名称
: z7 U% D5 O3 L4 knode.name: node1                 #当前节点在集群内的节点名称
# v5 z; C! e3 L* ^path.data: /data/elasticsearch   #数据保存目录1 j* n* `0 j8 u4 O  P7 D
path.logs: /data/elasticsearch   #日志保存目录
& ^1 s: g" C  V7 tbootstrap.memory_lock: true      #服务启动的时候锁定足够的内存,防止数据写入swap/ I) f$ y1 M9 @+ g- U, s8 {
network.host: 172.20.22.24       #监听IP: O% H. w% Y& P5 ]+ q/ }
http.port: 9200                  #监听端口: v. N3 }; c( l: z& ?  O7 q9 a) ]+ W
###集群中node节点发现列表
3 L8 s5 d$ ]& e. w5 odiscovery.seed_hosts: ["172.20.22.24", "172.20.22.27","172.20.22.28"]
) g3 k6 v: q& v: l0 D###集群初始化哪些节点可以被选举为master! y7 U+ n3 _8 Z3 y  K1 i
cluster.initial_master_nodes: ["172.20.22.24", "172.20.22.27","172.20.22.28"]
4 ~! ~9 C8 b3 |" W9 xaction.destructive_requires_name: true4 K! b+ d4 _, V: V, C
# mkdir /data/elasticsearch -p
; C$ s9 w" d$ X% P# chown -R elasticsearch. /data/elasticsearch9 X. U. S  e6 |. x- w) v1 a
# systemctl start elasticsearch.service
1 \2 M; {9 s% B+ I9 ^- k5 D###节点2
6 s  k) U9 E, F% x% F$ z. x$ q# grep '^[^#]' /etc/elasticsearch/elasticsearch.yml
( T7 S" M$ k4 m7 b7 mcluster.name: m63-elastic
1 }% u3 D4 |: y- anode.name: node23 i* J! O+ w7 `, ]! L) j
path.data: /data/elasticsearch/ G3 F$ @5 p/ F
path.logs: /data/elasticsearch
/ i: U# v: a+ ^network.host: 172.20.22.27& y0 X: k" K; T: {: y) q
http.port: 92000 ]  ]8 n" [; j9 x+ [' d
discovery.seed_hosts: ["172.20.22.24", "172.20.22.27","172.20.22.28"]
) g, v7 I/ N) v2 Fcluster.initial_master_nodes: ["172.20.22.24", "172.20.22.27","172.20.22.28"]
( K4 Q" ^- `; d; z3 j4 Baction.destructive_requires_name: true
0 e* Q( S4 R) t# A# mkdir /data/elasticsearch -p2 E  g7 s9 D& e! _9 Z: ]
# chown -R elasticsearch. /data/elasticsearch8 E/ B- A) |$ S
# systemctl start elasticsearch.service
& u- Z$ o2 a7 |( q###节点3
0 V! V" z. [0 O) a0 g' Z5 f$ t) r# grep '^[^#]' /etc/elasticsearch/elasticsearch.yml
- Y0 A  N0 u, F! j5 ycluster.name: m63-elastic" B( `9 k) Z: E% ]  M# F
node.name: node3
( Q: J2 ?, e/ ?2 g8 T2 tpath.data: /data/elasticsearch
/ h) D4 u: H/ u5 e: G- b6 [) p/ ]3 ]. Ypath.logs: /data/elasticsearch
; ^: y9 |6 o& y( y6 z8 `3 M7 A2 ^network.host: 172.20.22.28, F" K6 z7 i/ I0 r! j$ j
http.port: 9200
& J) N+ }/ ~3 ?  G0 ddiscovery.seed_hosts: ["172.20.22.24", "172.20.22.27","172.20.22.28"]9 w+ {+ T& B5 ?  O7 C! |; T
cluster.initial_master_nodes: ["172.20.22.24", "172.20.22.27","172.20.22.28"]
. B! {5 J: |& ]2 ]action.destructive_requires_name: true
+ e% G' D/ W' K0 Q8 V  v) J/ f1 }# mkdir /data/elasticsearch -p
& O3 h5 }7 Y! D& x4 z% b: r# chown -R elasticsearch. /data/elasticsearch
5 h/ z# D) S% s# systemctl start elasticsearch.service
+ J6 c! u! _! S浏览器访问验证
$ D4 @- h: P  Z; y+ A; [2 ohttp://$IP:9200
8 s/ g% V9 ?9 A$ Q
6 K. q: {! T6 u# B2 h+ \! ]
9 j+ N$ f! F( g% E; [/ s/ C3 e! m: T
+ {& g9 l0 N& u2 u$ \& xLogstash
" W% v+ e; R5 Z' {( LLogstash是一个具有实时传输能力的数据收集引擎,其可以通过插件实现日志收集和转发,支持日志过滤,支持普通log、自定义json格式的日志解析,最终把经过处理的日志发送给elasticsearch。
1 v* p4 ]! Y2 U3 r; c. ^ : ]) }3 V' N: a6 j
部署Logstash # d. J/ ?1 r% X2 h
Logstash是一个开源的数据收集引擎,可以水平伸缩,而且logstash是整个ELK当中用于最多插件的一个组件,其可以接收来自不同来源的数据并统一输出到指定的且可以是多个不同目的地+ y( M  F1 {! r* c
% j8 |2 C2 f  [" A& W
https://github.com/elastic/logstash #GitHub  ]! _: ]6 k! n# d/ [

* [5 j- c$ U& u# aElastic Stack and Product Documentation | Elastic; z) D' ]: r, e( ?. o/ C% D3 d

7 Z  D$ s( `; f4 R  C: S环境准备:关闭防火墙和selinux,并且安装java环境
7 K: W: Z( I7 l ( N( y, f6 \) m4 k! D( s  X& J) T
# apt install -y openjdk-8-jdk
8 j1 ?4 E  `: T7 z7 G0 }' ?1 J# ls -lrt logstash-7.12.1-amd64.deb
8 W# u, T$ g! U7 u* n$ d  c# dpkg -i logstash-7.12.1-amd64.deb. _; d$ L" q( {( w$ v
###启动测试
' `  E" C5 w1 L: F5 O# /usr/share/logstash/bin/logstash -e  'input { stdin {} } output { stdout {}}'   ##标准输入和标准输出
! t8 r/ A  D: h: t6 c+ i* Vhello world!~" b! R+ t+ e" ^& W, L- g+ r* C: t' v* q# d
{
0 A3 k; G/ ]) Q: M      "@version" => "1",
7 e4 H( `" y4 S- h) H    "@timestamp" => 2022-04-13T06:16:32.212Z,
: a" V  u, ?3 d2 n1 {9 @, w          "host" => "jenkins-slave",
9 c( I5 C+ \) y9 u2 u       "message" => "hello world!~"
  G' S' F# K& w& @9 b1 f% t+ ~! p; E}
2 Z5 M3 l. v: y7 A: G###通过配置文件启动
4 S2 y6 v# f2 P/ S# cd /etc/logstash/conf.d/0 C1 u: {7 x  u, {
# cat test.conf
4 X& j& c. \% L1 n$ ?+ N4 jinput {
  ~8 W/ `% @# ]6 B7 m, y  stdin {}- W# W) S  ^2 F8 }! f  D+ w9 ~, b
}
0 K7 K/ O7 o# ooutput {
6 u4 V4 s0 i2 z, U3 g* S  stdout {}* _' i6 o+ D0 b& x4 K
}7 B4 [" a' q" _7 g9 j. Y
6 ]$ R  l: C0 I2 C8 a2 c/ R
###通过指定配置文件启动
3 H- {5 `  w- M% A) Y5 |/ G( U; z3 f# /usr/share/logstash/bin/logstash -f test.conf -t   ##检查配置文件语法
# S$ f2 d( S  i) f: u) u# /usr/share/logstash/bin/logstash -f test.conf
* W/ G4 t; [" m( ]
; E" c. m3 V$ K# ~& E- w####输出到elasticsearch/ F( S; B. o# ^9 u
# cat test.conf
" E; P9 d; S4 Y$ uinput { 8 d7 J* \9 Z* E& ]$ N
  stdin {}
0 i1 a& F1 g: K; ?. t; l" S: m7 R}% c' Z7 y% g3 \! E
output {
7 T* @$ r; ]. D  #stdout {}' z: |4 R# R7 l* u, f% z
  elasticsearch {
$ B( A. z0 L8 Q0 Y- q; Y    hosts => ["172.20.22.24:9200"]( T9 Z2 F, i, ]2 Y( T0 A+ C
    index => "magedu-m63-test-%{+YYYY.MM.dd}"4 d! G6 |5 a& j! _2 Z' U! _
  }" H+ m, e3 ~# Z1 S% R
}
0 w' K& ?- _6 U$ s# /usr/share/logstash/bin/logstash -f test.conf
4 M3 @7 X2 Z$ z0 @version1# W; u8 P. l/ {, x. l. g
version2" T# j$ O. }4 [+ O
version34 x  {$ A1 q9 c8 J  L' B0 b
test10 p# _# U8 `6 ~" V  k
test2
7 b1 P% o9 e( P& ]) H. E* @test3
4 s/ k) Y) J: t7 e9 c/ \: u* c, T/ Z6 s, _9 ~( B
####elasticsearch服务器查看收集到的数据1 u9 L% l& ^; `* d
# ls -lrt /data/elasticsearch/nodes/0/indices/
8 X+ O/ D6 }; F/ X+ K! w% b& ytotal 4
! i- n& q7 j- y1 [% A  R0 idrwxr-xr-x 4 elasticsearch elasticsearch 4096 Apr 13 14:36 DyCv8w7mTleuAvlItAJlWA
" S3 d/ s% w$ \; }kibana
# D. ?, x) R  [. e; N3 U+ ]* Gkibana为elasticsearch提供一个查看数据的web界面,其主要是通过elasticsearch的API接口进行数据查找,并进行前端数据可视化的展现,另外还可以针对特定格式的数据生成相应的表格、柱状图、饼图等
2 ^9 P# C! c. X% h' {5 K ! ?) B" @5 Q& y) S+ `7 q$ x( Z
部署kibana
0 A! @# a; K8 G3 r/ N# H# ls -lrt kibana-7.12.1-amd64.deb5 Z- V, N3 }- z3 J! H
# dpkg -i kibana-7.12.1-amd64.deb- W# S1 l0 L* Z' h/ o
# grep "^[^$|#]" /etc/kibana/kibana.yml
: W7 r+ w$ E. h: B+ t  c- X( bserver.port: 5601( _) X2 ^' f& d* o! k+ U
server.host: "172.20.22.24"/ X4 t5 ?5 C$ P/ M# f5 F8 m
elasticsearch.hosts: ["http://172.20.22.27:9200"]( [; s  T3 u2 H. {# d
i18n.locale: "zh-CN"
" ~- r2 C; M/ M9 a8 t# systemctl restart kibana
$ `3 W1 G) R+ s5 J; s" Z& O$ n( `% L  c. y浏览器访问http://172.20.22.24:5601
; K2 G2 i. n8 g& h  K $ w( u' z7 U& ^6 b2 N- t
Stack Management-->索引模式-->创建索引模式# E, b& ^0 T  [# ^  x# y7 \8 b5 L

. }) ~6 C: w" v5 v4 V. l% d2 ~* |6 r
4 b; w4 M/ d9 x选择时间字段' n0 @5 K9 ?# X  W

7 B* C6 o; i% A1 p- f查看对应创建的索引日志信息
7 A$ x9 g* B5 Y5 w# R* I$ a3 G' C( g  M. h
! N6 Z. x9 a6 B6 s9 f
; ?2 x6 z7 o, h) [3 M" A5 p
收集tomcat日志
; L1 e$ X" J9 U$ ]& |6 A收集tomcat服务器的访问日志以及tomcat错误日志进行实时统计,在kibana页面进行搜索展现,每台tomcat服务器要安装logstash负责收集日志,然后将日志转发给elasticsearch进行分析,再通过kibana在前端展现# D- z7 R! H5 w+ m& A, P& V

2 ~( W4 t1 D0 z9 W% P$ ~部署tomcat 2 y1 w8 |  Z8 n8 V
####tomcat1,172.20.22.30; p5 \5 [9 C# J5 U- a
# apt install -y openjdk-8-jdk- }. y" V4 C: J8 o# b, `, i& ]5 D3 {7 L
# ls -lrt apache-tomcat-8.5.77.tar.gz
; g* r% A( ?6 {# h-rw-r--r-- 1 root root 10559655 Apr 13 21:44 apache-tomcat-8.5.77.tar.gz5 S4 |5 b& n7 k5 c+ b5 ?: t* ~/ p
# tar xf apache-tomcat-8.5.77.tar.gz -C /usr/local/src/; |) D0 F* [/ s" K+ z4 Y
# ln -s /usr/local/src/apache-tomcat-8.5.77 /usr/local/tomcat
' |+ I* Q# t" C# cd /usr/local/tomcat% y7 {$ a* E. Y( m0 _& [6 A
###修改tomcat日志格式为json
; m1 }) l7 C: x# vim conf/server.xml, p$ m! g$ J# E) P% }% ]/ [
....& m' x. T$ i1 t# x

& W  w2 y; I8 H# J1 R) q0 c....
0 O$ v/ {& J4 M8 T( N( L# mkdir /usr/local/tomcat/webapps/myapp
  M% f: m) a. D7 Y& x& P& f# echo "web1 172.20.22.30" > /usr/local/tomcat/webapps/myapp/index.html$ i' f0 Q# C# O) h9 v! ~
# ./bin/catalina.sh start2 K* l5 ~) E3 J' U3 ?# M% Z8 J1 ]; B1 {5 L

6 P5 [( `4 h+ G1 a4 h2 F###访问测试! @: \6 ]# N) X* A3 |: L. h
# curl http://172.20.22.30:8080/myapp/
4 |% C1 N' X) q! W###查看访问日志
3 D, |' X1 Q8 O( D7 H# tail -f /usr/local/tomcat/logs/tomcat_access_log.2022-04-13.log6 R4 h$ S8 _; E( o% Y8 ~8 U
; \& i( R) H5 z2 r9 W4 T, l7 `( j
####tomcat2,172.20.22.26
) a* [, S9 @6 M# apt install -y openjdk-8-jdk
. i8 I% D, @' s( F; V# ls -lrt apache-tomcat-8.5.77.tar.gz
* n: Y2 W  A1 y! W5 X& |-rw-r--r-- 1 root root 10559655 Apr 13 21:44 apache-tomcat-8.5.77.tar.gz, d0 u0 I9 k' S3 u
# tar xf apache-tomcat-8.5.77.tar.gz -C /usr/local/src/
! Z# d/ ]1 \0 d$ B. S' `& C# ln -s /usr/local/src/apache-tomcat-8.5.77 /usr/local/tomcat. {; z4 K1 z0 I: V0 P
# cd /usr/local/tomcat
/ G& K4 l; E) F3 N+ X0 y! ]###修改tomcat日志格式为json
! t6 `) J6 p$ w' T! r# vim conf/server.xml
# o) H& w$ k. B3 G. y" M, F. L....
9 y1 n  n9 L8 x
* K" Q9 @' _) p. }+ A....! W, `2 }+ B% S( l2 B2 Q
# mkdir /usr/local/tomcat/webapps/myapp
; V6 t: ~0 Q1 I8 N: s8 N% E3 \" [# echo "web2 172.20.22.26" > /usr/local/tomcat/webapps/myapp/index.html
% `& K, Q  b! Y$ G# ./bin/catalina.sh start6 B2 P  F+ v$ Y' D2 R' e; V1 M

9 q2 j9 K$ J. `  k- e###访问测试2 F  |5 x3 k" I9 U5 B0 f1 L9 F
# curl http://172.20.22.26:8080/myapp/, j. i6 b, H2 f3 k
###查看访问日志! B9 w( E' b4 U3 `9 o
# tail -f /usr/local/tomcat/logs/tomcat_access_log.2022-04-14.log
$ g  I6 L5 p$ \$ A# C! L5 d+ T部署logstash
+ ~  a3 S9 I7 ^; v. k在tomcat服务器安装logstash收集tomcat和系统日志
! `9 t, L6 B( S4 N. W 2 L* E& v* R) g/ i
####tomcat1,172.20.22.30( e; h0 l; L" I  R- U3 P
# ls -lrt logstash-7.12.1-amd64.deb5 d5 p7 u: e3 R9 [, X
# dpkg -i logstash-7.12.1-amd64.deb
& b' S* T3 h1 _# vim /etc/systemd/system/logstash.service* E+ w0 Y/ V' M- [! b8 t
...: g+ B9 x1 ]* t) `
User=root
% g( |' z- F" P& C& ~. Q: a2 D" Y; }Group=root1 t% k3 ^( n" g( x5 e& c
...
6 G" ^6 G, L8 E0 _2 A" W1 v3 j2 f# cd /etc/logstash/conf.d6 v4 y" p! g' J* L
# cat tomcat.conf
9 s( ^% e) G: [% B+ i: Ginput { - J1 X  N! t/ w' M0 @( P5 B; s( @
  file {
+ Y5 }) A+ w) Y; e    path => "/usr/local/tomcat/logs/tomcat_access_log*.log"2 y* B# f: k; U( {
    type => "tomcat-log"
7 b4 K1 B2 m. f' I    start_position => "beginning"7 R& @/ I3 L+ O0 M% @
    stat_interval => "3"6 W7 a1 |7 {  `6 R
  }
9 b+ `6 t/ X( _% b9 D# p  file {
1 j8 d1 J" S) T1 W+ ~; D, ]    path => "/var/log/syslog"
. m7 p/ y3 x' \$ k& S: ~    type => "systemlog"
3 H/ e  }" O; f3 o    start_position => "beginning"1 U. H1 n& a% u" S, o6 ]! n
    stat_interval => "3"
' w, F9 {7 V% Z2 F5 u' {7 a0 }( ]  }
5 Q/ b0 L: G  O6 o& U  R}
; V; V8 J- C- j  Woutput {1 t8 o2 z7 n& M0 K3 D' b! |- I
  if [type] == "tomcat-log" {; z# l7 ]& v4 s- b; v
  elasticsearch {
  @6 Q5 _2 ]1 Z2 w9 T    hosts => ["172.20.22.24:9200","172.20.22.27:9200"]" C4 B) R7 U( q( s: O% v4 v
    index => "elk-tomcat-%{+YYYY.MM.dd}"
) e3 d5 }+ }! t2 Y  }}7 G: @) v5 A, i7 a# T. n
  if [type] == "systemlog" {
4 [, O( t# |) i* ?* i5 Q  elasticsearch {
5 x" `7 Y, \6 t! z& Q& G) u    hosts => ["172.20.22.27:9200","172.20.22.27:9200"]: ]7 y% b8 c, ]3 l
    index => "elk-syslog-%{+YYYY.MM.dd}"8 A+ A" V5 S# J  I) i
  }}# |; i0 O' R( O
}8 W+ A; a1 ?0 g9 F% V# o
, R1 Y; B+ D# L
# /usr/share/logstash/bin/logstash -f tomcat.conf -t4 B( C) d' z, p* l; G. z1 l  N% p
# systemctl daemon-reload3 z) {" _, X  O
# systemctl start logstash.service
8 }+ _" x- k0 r! d# scp tomcat.conf root@3172.20.22.265 a( Y& M% z- F1 z/ d
# p; H. _4 j3 c1 h4 z
####tomcat2,172.20.22.26( _7 ?" |$ @0 x( |* L0 A3 B8 P
# ls -lrt logstash-7.12.1-amd64.deb# d4 k# I6 e9 }; H6 a  ~5 ]
# dpkg -i logstash-7.12.1-amd64.deb8 R- m9 S2 ~- L
# vim /etc/systemd/system/logstash.service
/ W, A: ?: Z) o2 u: s! n...
( z0 G( ~" q- W. AUser=root  C7 j. f: |$ T$ A7 D) f/ Z) x
Group=root
% T( O- e  @4 l8 Y9 c1 }...
! @- ~# E5 K+ f4 @# a0 T9 s# systemctl daemon-reload
, q7 K1 r$ i5 a5 n0 M' u# systemctl daemon-reload1 I- v4 f3 M7 m) a( I1 K
# systemctl start logstash.service
7 K; Z/ H  k" E. l& i" s3 H+ Y8 G8 L通过kibana展现3 a9 P/ G$ J$ A4 A
# J; I1 m5 I- r# X' I2 W3 X$ e
% v" i- F, t) y6 e4 H" O, x% B
收集Java日志 # Q+ |6 h% @& i1 q; O& @
使用codec的multiline插件实现多行匹配,这是一个可以将多行进行合并的插件,而且可以使用what指定将匹配到的行与前面的行合并还是和后面的行合并
- k6 J/ K2 B" J8 B. O! j& _ $ o5 |4 |) E' H: w9 q
Multiline codec plugin | Logstash Reference [8.1] | Elastic8 O, u5 r" e4 G' ]- F! w
% k  C( H+ V/ E5 w4 T
添加logstash配置文件 , P. |5 A* R% M! e" _- J( v4 O& Q$ z
###收集logstash自身的日志,172.20.22.26
  P; W6 W/ W  s* q" ^# cd /etc/logstash/conf.d
8 ]; y6 g1 u0 w: g9 u# cat java.conf 8 q5 s$ c$ D/ g$ l0 @$ E  P) i0 G& r' y
input {4 v8 W( c* K; k
  file {! Z' j, Z, S" ^% t* D& H" Y+ O
    path => "/var/log/logstash/logstash-plain.log"* ~; P. E8 A1 d4 [- L2 b1 G/ _$ ^2 z6 ~' w
    type => "logstash-log"* }$ \& |+ N. f9 m% i
    start_position => "beginning"
; r* a5 K' p% m# S- p+ g3 K    stat_interval => "3"; F+ o3 A! i, d" I+ R' C/ D6 x7 ]4 |
    codec => multiline {
: T" S5 q# [! A; G* E      pattern => "^\["
( k5 F  u4 t/ S! A7 A* V      negate => true9 z1 n4 a- H; r9 H3 ~
      what => "previous"
( _! [# X& e2 c, h7 m0 x) e# y   }}% T6 e$ w' X  ~( e- f7 t5 |1 Z% T/ m1 G
}: N) r5 c) l8 f
output {5 F; ?: `7 ^. f% X
  if [type] == "logstash-log" {
* q/ y3 o- {, V5 l" w; H. L9 ~5 M+ K2 i  elasticsearch {9 Y) `; j% O& _' }
    hosts => ["172.20.22.24"]5 C" x! E9 G) z) A
    index => "logstash-log-%{+YYYY.MM.dd}"6 `$ ^% [2 H0 Y& z; ?+ S, t
  }}3 I  g6 k+ o- f7 Q( C& V6 ~$ H
}  w2 l8 I7 r! l) m
% Y- t' c0 j$ F2 M' P
# /usr/share/logstash/bin/logstash -f java.conf -t) F  l) Z" P1 H/ f
# systemctl restart logstash.service
$ ~7 w- g0 a7 h1 d. Y. z
" V; y$ M, F/ c$ t, O###收集logstash自身的日志,172.20.22.30* K0 [! K, g+ _; ]- g  W
# cd /etc/logstash/conf.d
- Y7 S( J  h. L0 d1 ?4 b# cat java.conf 2 k$ Y* {% n* M0 T& j7 R# a/ o
input {$ ?& \) s# r4 G5 T  S
  file {/ U0 V* w9 S' i2 N4 z( ^
    path => "/var/log/logstash/logstash-plain.log"$ a: p. M0 }" I; a; S
    type => "logstash-log") Q/ \0 s; m1 E/ C$ o/ Z3 @
    start_position => "beginning": ~  T* K$ G4 X" {
    stat_interval => "3"
  g& ~) @: [, f9 G    codec => multiline {1 {/ b) n8 K$ O
      pattern => "^\["5 j: p; a" i7 L
      negate => true  T. {; U) f/ p/ r
      what => "previous"
& g/ a! G8 J6 |/ B) v/ i5 l   }}
. H  |% w7 ], ?3 R6 ?}5 R3 m; J* \  n8 t, a  q
output {
: z* f0 B/ ]1 G  if [type] == "logstash-log" {1 X  i4 s9 d5 T- v6 z
  elasticsearch {
2 B2 Z$ G+ B0 M# V2 G* f    hosts => ["172.20.22.24"]! V( k# X6 ?0 I; s
    index => "logstash-log-%{+YYYY.MM.dd}"* @8 h! Q4 E( I  ]$ V# Q
  }}
) G# @9 Y5 M* }9 j- K! s$ ^" T: m}4 |% n6 [- r" \3 c; _0 h

1 o* C( n7 I9 e* l4 v7 s# /usr/share/logstash/bin/logstash -f java.conf -t
+ @; p6 r$ P* y7 Z: c5 V$ O# systemctl restart logstash.service
6 ?  Z, o  z" _/ r查看kibana收集到的日志& n4 I  n2 l/ i' |3 c& Q# L3 V

9 y9 j: Z% n/ B' H5 ~" B! q5 }
: t. C: X1 X" ]% l' A+ i
, S- Q  H! S7 s4 g' s9 nfilebeat结合redis、logstash收集nginx日志
7 A/ g! y7 p% \! {! Y' |# n% b使用filebeat收集日志发送到logstash1,再由logstash1发送到redis,最后再由logstash2发送到elasticsearch* F( j: s$ ~: ^' d2 i
& S0 D' m+ K* P7 c, F
web1:172.20.22.30,部署好nginx、filebeat、llogstash. ]) @' }7 t  O
/ N: e+ M3 ~! u
web2:172.20.22.26,部署好nginx、filebeat、llogstash
" _- f* I# Z& W7 z
9 g0 l$ p) ]5 Q0 P8 `logstash服务器2:172.20.22.23,redis服务器:172.20.23.157
8 K6 V  I1 q5 S! N
1 F. p5 {& _- v; `! q6 g* ?4 Bnginx服务器相关配置
9 @7 ~6 D, Q6 R2 P部署nginx
' P, v1 c  r6 u! r! z. j# wget http://nginx.org/download/nginx-1.18.0.tar.gz# I6 T: x0 }9 x& o
# tar xf nginx-1.18.0.tar.gz! l  |5 h3 h% f3 X* C* ]0 J# u
# cd nginx-1.18.0
% N% Z) y% l* N  Z# ./configure --prefix=/usr/local/nginx --with-http_ssl_module
% Z) j% @+ v/ U* s. o# make -j4 && make install
* J& z  B% b" z# /usr/local/nginx/sbin/nginx
" B$ X, |6 R# ~0 U部署配置logstash
5 [0 E" P* w6 L1 j把filebeat收集到的日志信息发送到redis
( F! M- @$ F$ V+ X  {* c
1 y! m" j4 |% O" m/ H2 k# apt install -y openjdk-8-jdk1 T" r) F- n( K. g1 ~: `" Z
# dpkg -i logstash-7.12.1-amd64.deb" E' k0 ?; l$ e. n
# cat /etc/logstash/conf.d/beats-to-redis.conf
- M. M$ _+ A0 Q6 linput {3 V; D  n( @1 M4 |% p
  beats {
# U! G+ Q3 Y& D( F; x) v    port => 5044
0 A! }; j; i+ r& C& k* i: _    codec => "json"
6 t  R1 r. f3 |7 E: q, y$ P  }- R1 A) W( z1 _' D) S
  beats {/ i$ O  {5 [2 W( l
    port => 5045- n% j: x2 R' v$ h/ q' |0 b
    codec => "json"
( Z0 S( ~- x! y/ J# i2 Y  }
  L5 K" a9 B: ]8 d}& J. n6 f9 n- J3 S
output {
) b1 i& |/ F* P; O3 L6 v0 E1 M  if [fields][project] == "filebeat-systemlog" {
. C* B2 f  S* C& j7 ~9 v, I. U1 f1 r2 T    redis {$ R) c# e; C+ s& S) Z/ M
      data_type => "list"$ i" I$ D8 e$ j' X; \2 d! a
      key => "filebeat-redis-systemlog"
" i9 n* b$ E; T: V& H# M      host => "172.20.23.157"
, b- m" r0 Y. w9 y0 F& [2 M, x      port => "6379"
4 l. y# n- Z0 ^      db => "0". t& z" [5 M; \' H; T  I
      password => "12345678"" h- y: N9 N8 G6 e8 d
  }}, Y: L4 _- b6 i
  if [fields][project] == "filebeat-nginx-accesslog" {
; C# }2 D4 Q7 ?3 t2 F) E+ X# O    redis {- b  f- e0 m1 s5 h8 e( w; ^; D
      data_type => "list"
4 F* \0 s+ q9 d  p* D      key => "filebeat-redis-nginx-accesslog"
- U+ F- v- ^% j- G      host => "172.20.23.157"
8 E7 u) c6 Q4 n      port => "6379") }$ c) Y7 p' R3 ~* c) R
      db => "1"1 V; n3 {; Y! g. c% p
      password => "12345678"( ]; c& N: O, s) ^0 z7 h5 K3 J6 L
  }}
8 Q9 ^9 t% i0 Y' H' P) b( c  if [fields][project] == "filebeat-nginx-errorlog" {9 U, n8 @0 w: G. j" Y' F
    redis {1 K6 B+ I! E' u2 X
      data_type => "list"
2 f3 n. @5 A7 q5 {( `      key => "filebeat-redis-nginx-errorlog", W' `0 x9 u$ c1 a- e& L+ S
      host => "172.20.23.157"5 {8 |- A/ l: ~# N
      port => "6379"' H- g/ R" y4 o: d) S
      db => "1"
; `; J' L! a! s0 G$ k0 n9 K      password => "12345678"
, c7 z, j* |6 p  }}9 P6 l  a( B3 T
}
0 ~  `/ a/ S6 t3 k3 Q# systemctl start logstash7 ~5 j# C6 O2 [* Q; j
# scp /etc/logstash/conf.d/beats-to-redis.conf root@172.20.22.26:/etc/logstash/conf.d/ + \# f" h) s/ x
部署配置filebeat
5 D+ g' n6 {! n/ g5 d. E通过filebeat收集日志信息发送到logstash
: c; b' u0 S& w* ~& C8 l3 o  C3 |
2 \( g4 P9 h- u( V; J; O* C# dpkg -i filebeat-7.12.1-amd64.deb2 I+ C% y9 o$ |/ [( x' n4 Q/ D
# grep -v "#" /etc/filebeat/filebeat.yml | grep "^[^$]"/ S+ j/ k6 W6 B. B& l/ p
filebeat.inputs:& s  ~+ Z, f8 f- S1 \. m
- type: log
1 t/ O2 g6 W: n# c" w/ {; {  enabled: true' C5 r1 R7 I, O1 V$ f
  paths:4 [& ^, X0 B2 g: u  n7 U# l
    - /var/log/syslog
9 a% R$ e1 H0 p2 k8 ~/ L  fields:
- C5 t2 U# d0 B" _    project: filebeat-systemlog
! \; S; J5 a! y* V0 `* E$ a8 r- type: log
# J4 v. d; e; E3 {/ {  enabled: true/ t2 o0 p" Q! l$ r% |4 _/ ?- O: H
  paths:
1 f+ u  d( k6 K: Q. p9 v    - /usr/local/nginx/logs/access.log
: ~* c: I  u* @9 A" F  fields:
6 W* H- w+ _9 m+ m4 R3 e7 @! x    project: filebeat-nginx-accesslog1 M4 s  L# m% G& g& A# i
- type: log* V+ i' e, k8 \: d
  enabled: true
+ v  |! m0 a. {. r7 D+ x2 J+ {  paths:
4 s( c7 c0 R4 q: e% r* E% ~8 l    - /usr/local/nginx/logs/error.log6 s( h2 y) n9 u+ N( O
  fields:4 i- a0 `3 c, X6 |5 c
    project: filebeat-nginx-errorlog! }1 b; v$ y$ b8 r
filebeat.config.modules:
9 _  [/ ^, R; D. V( ]  path: ${path.config}/modules.d/*.yml; W+ n) W1 ?/ B5 o  W" P8 H
  reload.enabled: false
8 E) w* x& r; x' a: A, e$ `- hsetup.template.settings:
4 @2 d4 y1 J7 {: E2 l  index.number_of_shards: 1) |: Z$ i2 \6 ?  \
setup.kibana:& `; Y% ?$ J& `' V. G8 D, w
processors:7 C% p" U2 @+ b" q9 M2 m
  - add_host_metadata:
, B& Q+ C4 ^( j# o$ t      when.not.contains.tags: forwarded
4 j4 e# g( e2 [6 t0 K  - add_cloud_metadata: ~
1 R! U. c5 t8 U* b) X, Z* ?  - add_docker_metadata: ~
# _: c8 A7 i% M! A, z, Y  - add_kubernetes_metadata: ~* I! d( [! A9 ]0 m! K+ L; ^
output.logstash:  z3 W: {! }5 J1 }2 T1 Z8 l. L- G
  hosts: ["172.20.22.30:5044","172.20.22.30:5045"]# ^  o" J8 O# }1 d  t
  enabled: true
8 K3 u& z4 j# C1 \3 a3 R  T  worker: 2
5 D+ k+ ~$ Z. r, E( o6 L( o$ z  compression_level: 3
# v7 j; C" n8 ~$ e% V2 Y* d( H0 h3 @: [) Y  loadbalance: true
/ [4 k, K- [- [+ a/ w* D8 p9 ^$ H* G- ]
# systemctl start filebeat( H5 c' w7 h$ T# r$ s1 T- n+ m
# scp /etc/filebeat/filebeat.yml root@172.20.22.26:/etc/filebeat/
$ L' M: C" G7 ^* |( Slogstash服务器配置 % t' Z" B) N8 C4 P7 Y
logstash服务器2:172.20.22.23,把redis缓存的日志发送到elasticsearch1 w% A1 s* y0 G; y
/ d0 H, \% J7 i! A  I
# apt install -y openjdk-8-jdk% V# ^" N/ p6 g( H2 f2 k: a
# dpkg -i logstash-7.12.1-amd64.deb
/ a' U% @5 u8 ?) u5 d' e$ E# cat /etc/logstash/conf.d/redis-to-es.conf - ^& o, J8 l6 A; N. t( L
input {
& K6 ~8 d0 H! i; d- B! ~4 v4 s  redis {' j  S2 t* `+ s! \
    data_type => "list"
& C& b; p5 Y9 `    key => "filebeat-redis-nginx-accesslog"
  ~/ s- R  Y2 C+ x3 ^0 j    host => "172.20.23.157"
: u9 k' s  K# `  T3 p, _    port => "6379"  _+ v0 x% r" V% U& f2 K; y2 k
    db => "1"5 D7 E8 c% B- z
    password => "12345678"
9 t9 Z5 k# x4 E1 W  }& s* p0 _$ W! y! f
  redis {$ s: F- M5 V: S! R& z6 C0 k- a
    data_type => "list"
8 B( }3 X! R/ K/ I6 V    key => "filebeat-redis-nginx-errorlog"
8 [+ o4 g: n% o) @    host => "172.20.23.157"' i# A% ]% b! `; @5 l7 ^$ O! c7 R( |
    port => "6379"
' r% R- Y' P# m, Q: d4 p    db => "1"
) A3 ^! s- F7 R- H" q* |    password => "12345678"$ A- O: Y+ n/ |2 \
  }
* y2 }) x4 R0 e- M  redis {
+ h# y' \  F! m- V' `    data_type => "list"# y& a; V3 ?4 R" o' Q
    key => "filebeat-redis-systemlog"
) h) I  Y: P" o4 L6 ?    host => "172.20.23.157"
  A$ q) l! C$ p& A    port => "6379". d" W5 t2 ]& ~0 }6 Y2 i
    db => "0"9 F, X0 B; |! P: N  ]  Y% c* ]
    password => "12345678"
  }8 v0 D- u1 l+ \. X+ ?3 f) A  }
! S; L5 B' w7 ]* H8 H}
1 R$ Q8 x2 v$ G2 ooutput {
- G6 X1 W( z# \# N& L# v  if [fields][project] == "filebeat-systemlog" {! C6 I' Y+ U. y! c$ r( e
    elasticsearch {7 O+ u' x8 Y; m) `% f# [) a. W
      hosts => ["172.20.22.28:9200"]4 G2 M8 G5 l. f  B
      index => "filebeat-systemlog-%{+YYYY.MM.dd}"
  ?: r9 j- L0 A9 k! R" q  }}( r( ?% V( l3 s
  if [fields][project] == "filebeat-nginx-accesslog" {
, x6 ?- _( d( h2 |. u    elasticsearch {
$ u  q7 w0 S6 W8 `* G9 x7 J* P2 v      hosts => ["172.20.22.28:9200"]7 E" U* g8 j# G% ^: F) _
      index => "filebeat-nginx-accesslog-%{+YYYY.MM.dd}"/ |# ]3 T* y: _
  }}# v! x- c3 Y: D8 z! F6 S
  if [fields][project] == "filebeat-nginx-errorlog" {
# \5 a& F( ~$ L: ~6 e- i4 f    elasticsearch {4 V; @7 K' r2 Q' [0 P1 S/ O
      hosts => ["172.20.22.28:9200"]5 o: W2 s% J) A) A; [0 h2 s
      index => "filebeat-nginx-errorlog-%{+YYYY.MM.dd}"& h  t( R% Z" v( M$ }
  }}
/ p& X* r) C' i6 U" C; ?6 [3 b}
5 i& l0 R0 k+ I$ I# systemctl restart logstash.service
( R" R7 R) U) @- yredis安装配置
1 B" u& A  k2 o6 Oredis服务器:172.20.23.157,
  z7 o: t/ P3 |$ Y
$ t1 Z, ^. g+ N4 }. I& M) Z# yum install -y redis
7 ^5 G' w# u4 f# R, @) _# vim /etc/redis.conf
& d9 c. ]5 u3 _- V####修改以下配置项1 J5 }" i6 U9 g: p% N/ x
bind 0.0.0.0& M7 v7 }& V  S) \: p. A0 }# H1 t
....: Y. q, \, f* z* z' d
save ""
8 {3 Y0 g" ~' k, Q9 t7 y....
" \9 ]: y! P# f" H3 S1 X7 q# Trequirepass 12345678
; d- h5 a# q& ~4 v1 t$ Q$ z7 J. c....2 I% ~! K4 A  ?: l% x
# systemctl start redis4 \4 n- M$ _" h! y
###测试连接redis: Q# w( r1 z3 b  g0 G
# redis-cli
( N6 {5 b' j/ N8 h; W127.0.0.1:6379> auth 12345678
) Y$ t# S5 O3 ?OK3 }) s8 I4 S6 q/ a8 a( B1 {: k
127.0.0.1:6379> ping
3 t4 x+ s" `1 o; q9 N# _% E, o" \PONG
: M; h# ^9 ?3 y7 _! N& y+ T2 ~& K3 d5 _) f: z- J
###验证收集到的日志信息
2 |$ @/ m5 A; J2 V3 l0 k127.0.0.1:6379[1]> keys */ r7 ~( u, ?9 }9 T
1) "filebeat-redis-nginx-accesslog"* p9 n: K5 L9 M1 W5 ~9 `0 a
2) "filebeat-redis-nginx-errorlog". V$ h9 ~6 [4 I! E7 u
127.0.0.1:6379[1]> select 01 c2 q( A, x5 R/ ~1 u- f8 |! V
OK' |1 f5 R' m% l$ l+ x- Q! s' H
127.0.0.1:6379> keys *+ F: Q' M5 G6 B5 l% U. d/ c
1) "filebeat-redis-systemlog"   Z$ E! D& f6 |4 v9 B1 ]
通过head插件验证生成的索引
8 F% ]# o% U$ E5 X- i9 B# R0 A' j* S+ a- C

2 Y% F; k# }, O; R, `kibana验证收集到的日志信息 # ^2 L8 j2 J6 \- l  ]% M

本帖子中包含更多资源

您需要 登录 才可以下载或查看,没有账号?立即注册

x
回复

使用道具 举报

您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

Copyright © 2001-2013 Comsenz Inc.Powered by Discuz!X3.4( 沪ICP备18024137号 )
快速回复 返回顶部 返回列表