大型企业架构部署之构建企业级ELK日志分析系统

ELK架构流程图

前言

ELK（elasticsearch、logstash、kibana）可以作为日志收集及分析的一整套系统。想了解更多，请打开Elastic官网。

环境准备

系统：Centos7
内存：4G
处理器：2核
Java：jdk-8u241-linux-x64.tar.gz
Elasticsearch：elasticsearch-6.2.4.rpm
Kibana：kibana-6.2.4-x86_64.rpm
Logstash：logstash-6.2.4.rpm
Nginx：nginx-1.16.1-1.el7.ngx.x86_64.rpm

一、部署Java环境

1.下载JDK包

到官网找到jdk-8u241-linux-x64.tar.gz安装包并下载到本地，然后上传到服务器
注意：一定要是tar包，别下错了。（其实找显哥哥要是最方便的）

#附官网地址：
https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

2.上传并解压JDK包

[root@elkstack ~]# ls -l
total 189988
-rw-r--r-- 1 root root 194545143 Mar 31 22:26 jdk-8u241-linux-x64.tar.gz
[root@elkstack ~]# tar xf jdk-8u241-linux-x64.tar.gz 
[root@elkstack ~]# ll
total 189988
drwxr-xr-x 7 10143 10143       245 Dec 11 18:39 jdk1.8.0_241
-rw-r--r-- 1 root  root  194545143 Mar 31 22:26 jdk-8u241-linux-x64.tar.gz

3.移动并设置软连接

[root@elkstack ~]# mv jdk1.8.0_241 /usr/local/
[root@elkstack ~]# ln -s /usr/local/jdk1.8.0_241 /usr/local/jdk

4.设置Java环境变量并检查是否成功

[root@elkstack ~]# export JAVA_HOME=/usr/local/jdk1.8.0_241
[root@elkstack ~]# export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
[root@elkstack ~]# export PATH=$PATH:$JAVA_HOME/bin
[root@elkstack ~]# ln -s /usr/local/jdk1.8.0_241/bin/java /usr/bin/java
[root@elkstack ~]# source /etc/profile
[root@elkstack ~]# java -version
java version "1.8.0_241"
Java(TM) SE Runtime Environment (build 1.8.0_241-b07)
Java HotSpot(TM) 64-Bit Server VM (build 25.241-b07, mixed mode)

二、部署Elasticsearch

1.下载并安装elasticsearch

[root@elkstack ~]# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.4.rpm

也可以到Elastic官网下载：直达官网

[root@elkstack ~]# rpm -ivh elasticsearch-6.2.4.rpm 
warning: elasticsearch-6.2.4.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
Preparing...                          ################################# [100%]
Creating elasticsearch group... OK
Creating elasticsearch user... OK
Updating / installing...
   1:elasticsearch-0:6.2.4-1          ################################# [100%]
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable elasticsearch.service
### You can start elasticsearch service by executing
 sudo systemctl start elasticsearch.service

2.创建elasticsearch数据目录和日志目录

[root@elkstack ~]# mkdir -p /data/es-data
[root@elkstack ~]# chown -R elasticsearch:elasticsearch /data/es-data
[root@elkstack ~]# mkdir -p /log/es-log
[root@elkstack ~]# chown -R elasticsearch:elasticsearch /log/es-log

3.修改配置文件elasticsearch.yml

找到这些参数并修改以下内容

[root@elkstack ~]# vim /etc/elasticsearch/elasticsearch.yml
path.data: /data/es-data
path.logs: /log/es-log
bootstrap.memory_lock: false
network.host: 0.0.0.0
http.port: 9200
#下面这两行是新写入的，原本没有
http.cors.enabled: true
http.cors.allow-origin: "*"

4.启动elasticsearch并查看状态（必须有9200端口）

[root@elkstack ~]# systemctl start elasticsearch.service
[root@elkstack ~]# systemctl status elasticsearch.service
[root@elkstack ~]# systemctl enable elasticsearch.service

[root@elkstack ~]# netstat -antp |grep 9200
tcp6       0      0 :::9200                 :::*                    LISTEN      10082/java

三、部署Kibana

1.下载并安装kibana

[root@elkstack ~]# wget https://artifacts.elastic.co/downloads/kibana/kibana-6.2.4-x86_64.rpm
[root@elkstack ~]# rpm -ivh kibana-6.2.4-x86_64.rpm 
warning: kibana-6.2.4-x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:kibana-6.2.4-1                   ################################# [100%]

2.修改配置文件kibana.yml

找到这些参数并修改以下内容

[root@elkstack ~]# vim /etc/kibana/kibana.yml
server.port: 5601
server.host: "localhost"
elasticsearch.url: "http://localhost:9200"
kibana.index: ".kibana"

3.启动kibana并检查状态（必须有5601端口）

[root@elkstack ~]# systemctl start kibana.service 
[root@elkstack ~]# systemctl status kibana.service
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; disabled; vendor preset: disabled)
   Active: active (running) since Sun 2020-04-05 00:04:43 CST; 11ms ago
 Main PID: 10246 (node)
   CGroup: /system.slice/kibana.service
           └─10246 /usr/share/kibana/bin/../node/bin/node --no-warnings /usr/share/kibana/bin/../s...

Apr 05 00:04:43 elkstack systemd[1]: Started Kibana.
[root@elkstack ~]# netstat -lntp |grep 5601
tcp        0      0 127.0.0.1:5601          0.0.0.0:*               LISTEN      10246/node 

四、部署Logstash

1.下载并安装logstash

[root@elkstack ~]# wget https://artifacts.elastic.co/downloads/logstash/logstash-6.2.4.rpm
[root@elkstack ~]# rpm -ivh logstash-6.2.4.rpm 
warning: logstash-6.2.4.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:logstash-1:6.2.4-1               ################################# [100%]
Using provided startup.options file: /etc/logstash/startup.options
Successfully created system startup script for Logstash

2.创建logstash的数据和日志目录

[root@elkstack ~]# mkdir -p /data/ls-data
[root@elkstack ~]# chown -R logstash:logstash /data/ls-data
[root@elkstack ~]# mkdir -p /log/ls-log
[root@elkstack ~]# chown -R logstash:logstash /log/ls-log

3.修改配置文件logstash.yml

找到这些参数并修改以下内容

path.data: /data/ls-data
path.config: /etc/logstash/conf.d
path.logs: /log/ls-log

4.启动logstash并查看状态（一定要变绿）

[root@elkstack ~]# systemctl start logstash
[root@elkstack ~]# systemctl status logstash
● logstash.service - logstash
   Loaded: loaded (/etc/systemd/system/logstash.service; disabled; vendor preset: disabled)
   Active: active (running) since Sun 2020-04-05 00:13:39 CST; 11ms ago
 Main PID: 10421 (logstash)
   CGroup: /system.slice/logstash.service
           ├─10421 /bin/bash /usr/share/logstash/bin/logstash --path.settings /etc/logstash
           ├─10429 /bin/bash /usr/share/logstash/bin/logstash --path.settings /etc/logstash
           ├─10430 /bin/bash /usr/share/logstash/bin/logstash --path.settings /etc/logstash
           └─10431 /bin/bash /usr/share/logstash/bin/logstash --path.settings /etc/logstash

Apr 05 00:13:39 elkstack systemd[1]: Started logstash.
[root@elkstack ~]# systemctl enable logstash

5.测试logstash是否安装成功

1.给logstash创建软连接（执行的时候不需要绝对路径）

[root@elkstack ~]# ln -s /usr/share/logstash/bin/logstash /bin/

2.执行命令测试
当执行命令之后，等待片刻会有The stdin plugin is now waiting for input:提示，然后回车或者输入内容会有输错内容，如下所示则成功。

[root@elkstack ~]# logstash -e 'input { stdin { } } output { stdout {} }'
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
.....
...此次省略多行数据...
......
The stdin plugin is now waiting for input:
abc
{
    "@timestamp" => 2020-04-04T16:15:51.586Z,
       "message" => "abc",
      "@version" => "1",
          "host" => "elkstack"
}
lixian
{
    "@timestamp" => 2020-04-04T16:15:55.438Z,
       "message" => "lixian",
      "@version" => "1",
          "host" => "elkstack"

五、Nginx代理（使用域名访问）

1.下载并安装nginx

[root@elkstack ~]# wget http://nginx.org/packages/rhel/7/x86_64/RPMS/nginx-1.16.1-1.el7.ngx.x86_64.rpm
[root@elkstack ~]# yum localinstall -y nginx-1.16.1-1.el7.ngx.x86_64.rpm

2.给elasticsearch添加代理

[root@elkstack ~]# vim /etc/nginx/conf.d/elasticsearch.conf
server {
listen 81;
server_name www.elk.com;
location / {
    proxy_pass http://localhost:9200;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection 'upgrade';
    proxy_set_header Host $host;
    proxy_cache_bypass $http_upgrade;
} 
}

3.给kibana添加代理

server {
listen 80;
server_name www.elk.com;
location / {
    proxy_pass http://localhost:5601;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection 'upgrade';
    proxy_set_header Host $host;
    proxy_cache_bypass $http_upgrade;
}
}

3.检查并重启nginx服务

[root@elkstack ~]# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
[root@elkstack ~]# systemctl restart nginx

六、浏览器输入域名访问