命令记不住

登陆

Logstash 日志收集

  |   129 浏览

Logstash日志收集

在学习Logstash之前，我们需要先了解以下几个基本概念:

logstash收集日志基本流程: input-->codec-->filter-->codec-->output

  • 1.input:从哪里收集日志。
  • 2.filter:发出去前进行过滤
  • 3.output:输出至Elasticsearch或Redis消息队列
  • 4.codec:输出至前台，方便边实践边测试
  • 5.数据量不大日志按照月来进行收集
#通常使用rubydebug方式前台输出展示以及测试
[root@linux-node3 ～]# /opt/logstash/bin/logstash -e 'input { stdin {} } output { stdout{codec => rubydebug} }'

hello  #输入
{
       "message" => "hello",
      "@version" => "1",
    "@timestamp" => "2016-09-01T08:16:36.354Z",
          "host" => "linux-node3.com"
}

1.Logstach实践案例

以下所有收集的日志写入node4的Redis，最后node4通过logstash写入ES,具体架构图如下:
如果数据量不大需要直接写入elasticsearch，可将案例Redis改为elasticsearch即可。在后面我也会放出实际的案例。
4

es收集架构

2.Logstash安装

Logstash需要Java环境，所以直接使用yum安装。
1.安装java

[root@linux-node1 ~]# yum install java
[root@linux-node1 ~]# java -version
openjdk version "1.8.0_101"
OpenJDK Runtime Environment (build 1.8.0_101-b13)
OpenJDK 64-Bit Server VM (build 25.101-b13, mixed mode)

2.下载并安装GPG key

[root@linux-node1 ~]# rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch

3.添加logstash的yum仓库

#添加logstash的yum仓库
[root@linux-node1 ~]# cat /etc/yum.repos.d/logstash.repo
[logstash-2.3]
name=Logstash repository for 2.3.x packages
baseurl=https://packages.elastic.co/logstash/2.3/centos
gpgcheck=1
gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1

4.安装Logstash

[root@linux-node1 ~]# yum install -y logstash
声明:如果需要前台查看测试结果，在output加入如下:
    stdout {
        codec => "rubydebug"
        }
    
#执行命令：
/opt/logstash/bin/logstash -f /etc/logstash/conf.d/syslog.conf 

#执行完毕，将文件放置/etc/logstash/conf.d目录,logstash会自动读取相关配置文件
如果无法读取，可将/etc/init.d/logstash里USER和GROUP修改为root

3.Logstash收集Rsyslog日志

1.修改rsyslog.conf配置文件

 [root@linux-node3 elasticsearch]#vim /etc/rsyslog.conf  
 *.* @@192.168.90.203:514
 [root@linux-node3 elasticsearch]# systemctl restart rsyslog

2.编写收集rsyslog日志，写入至node4的Redis(Redis配置请自行谷歌,这里不在介绍)

[root@linux-node3 conf.d]# cat rsyslog.conf
input {
    syslog {
        type => "system_rsyslog"
        host => "192.168.90.203"
        port => "514"
        }
}


output {
    redis {
        host => "192.168.90.204"
        port=> "6379"
        db => "6"
        data_type => "list"
        key => "system_rsyslog"
       }
}

4.Logstash收集tcp日志

1.编写收集tcp网络日志

[root@linux-node3 conf.d]# cat tcp.conf
input {
    tcp {
        type => "tcp_port_6666"
        host => "192.168.90.203"
        port => "6666"
        mode => "server"
        }
}

output {
    redis {
        host => "192.168.90.204"
        port => "6379"
        db => "6"
        data_type => "list"
        key => "tcp_port_6666"
        }

}

2.往666端口发送数据几种方式：

echo "heh" |nc 192.168.90.203 6666
nc 192.168.90.203 6666 </etc/resolv.conf
echo hehe >/dev/tcp/192.168.90.203/6666

5.Logstash收集JAVA日志

es是java服务，收集es需要注意换行问题
1.编写收集Elasticsearch访问日志

[root@linux-node3 conf.d]# cat java.conf
input {
    file {
        type => "access_es"
        path => "/var/log/elasticsearch/xuliangwei.log"
        codec => multiline {
            pattern => "^\["
            negate => true
            what => "previous"
            }
        }
}

output {
    redis {
        host => "192.168.90.204"
        port => "6379"
        db => "6"
        data_type => "list"
        key => "access_es"
        }
}

6.Logstash收集Nginx日志

1.安装Nginx

yum install nginx

1.nginx改成json格式输出日志

#http段加如下信息(日志位置根据业务自行调整)
    log_format json '{ "@timestamp": "$time_local", '
         '"@fields": { '
         '"remote_addr": "$remote_addr", '
         '"remote_user": "$remote_user", '
         '"body_bytes_sent": "$body_bytes_sent", '
         '"request_time": "$request_time", '
         '"status": "$status", '
         '"request": "$request", '
         '"request_method": "$request_method", '
         '"http_referrer": "$http_referer", '
         '"body_bytes_sent":"$body_bytes_sent", '
         '"http_x_forwarded_for": "$http_x_forwarded_for", '
         '"http_user_agent": "$http_user_agent" } }';
    access_log  /var/log/nginx/access_json.log  json;

2.编写收集Nginx访问日志

[root@linux-node3 conf.d]# cat nginx.conf
input {

    file {
        type => "access_nginx"
        path => "/var/log/nginx/access_json.log"
        codec => "json"
        }
}

output {
    redis {
        host => "192.168.90.204"
        port => "6379"
        db => "6"
        data_type => "list"
        key => "access_nginx"
        }
}
推荐阅读