• 欢迎访问显哥博客,本网站纯属学习技术,绝无商业用途,欢迎小伙伴们共同学习!研究技术!QQ:52249909 加我QQ
  • 世界75亿人,这么小的概率,能认识你,是我一生的幸运,不妨加个QQ接触一下:52249909 加我QQ

大型企业架构部署之构建企业级ELK日志分析系统

Linux架构 lixian 2年前 (2020-04-05) 302次浏览 0个评论 扫描二维码
文章目录[隐藏]

ELK架构流程图

大型企业架构部署之构建企业级ELK日志分析系统

前言

ELK(elasticsearch、logstash、kibana)可以作为日志收集及分析的一整套系统。想了解更多,请打开Elastic官网

环境准备

系统:Centos7
内存:4G
处理器:2核
Java:jdk-8u241-linux-x64.tar.gz
Elasticsearch:elasticsearch-6.2.4.rpm
Kibana:kibana-6.2.4-x86_64.rpm
Logstash:logstash-6.2.4.rpm
Nginx:nginx-1.16.1-1.el7.ngx.x86_64.rpm

一、部署Java环境

1.下载JDK包

到官网找到jdk-8u241-linux-x64.tar.gz安装包并下载到本地,然后上传到服务器
注意:一定要是tar包,别下错了。(其实找显哥哥要是最方便的)

#附官网地址:
https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

2.上传并解压JDK包

[root@elkstack ~]# ls -l
total 189988
-rw-r--r-- 1 root root 194545143 Mar 31 22:26 jdk-8u241-linux-x64.tar.gz
[root@elkstack ~]# tar xf jdk-8u241-linux-x64.tar.gz 
[root@elkstack ~]# ll
total 189988
drwxr-xr-x 7 10143 10143       245 Dec 11 18:39 jdk1.8.0_241
-rw-r--r-- 1 root  root  194545143 Mar 31 22:26 jdk-8u241-linux-x64.tar.gz

3.移动并设置软连接

[root@elkstack ~]# mv jdk1.8.0_241 /usr/local/
[root@elkstack ~]# ln -s /usr/local/jdk1.8.0_241 /usr/local/jdk

4.设置Java环境变量并检查是否成功

[root@elkstack ~]# export JAVA_HOME=/usr/local/jdk1.8.0_241
[root@elkstack ~]# export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
[root@elkstack ~]# export PATH=$PATH:$JAVA_HOME/bin
[root@elkstack ~]# ln -s /usr/local/jdk1.8.0_241/bin/java /usr/bin/java
[root@elkstack ~]# source /etc/profile
[root@elkstack ~]# java -version
java version "1.8.0_241"
Java(TM) SE Runtime Environment (build 1.8.0_241-b07)
Java HotSpot(TM) 64-Bit Server VM (build 25.241-b07, mixed mode)

二、部署Elasticsearch

1.下载并安装elasticsearch

[root@elkstack ~]# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.4.rpm

也可以到Elastic官网下载:直达官网

[root@elkstack ~]# rpm -ivh elasticsearch-6.2.4.rpm 
warning: elasticsearch-6.2.4.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
Preparing...                          ################################# [100%]
Creating elasticsearch group... OK
Creating elasticsearch user... OK
Updating / installing...
   1:elasticsearch-0:6.2.4-1          ################################# [100%]
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable elasticsearch.service
### You can start elasticsearch service by executing
 sudo systemctl start elasticsearch.service

2.创建elasticsearch数据目录和日志目录

[root@elkstack ~]# mkdir -p /data/es-data
[root@elkstack ~]# chown -R elasticsearch:elasticsearch /data/es-data
[root@elkstack ~]# mkdir -p /log/es-log
[root@elkstack ~]# chown -R elasticsearch:elasticsearch /log/es-log

3.修改配置文件elasticsearch.yml

找到这些参数并修改以下内容

[root@elkstack ~]# vim /etc/elasticsearch/elasticsearch.yml
path.data: /data/es-data
path.logs: /log/es-log
bootstrap.memory_lock: false
network.host: 0.0.0.0
http.port: 9200
#下面这两行是新写入的,原本没有
http.cors.enabled: true
http.cors.allow-origin: "*"

4.启动elasticsearch并查看状态(必须有9200端口)

[root@elkstack ~]# systemctl start elasticsearch.service
[root@elkstack ~]# systemctl status elasticsearch.service
[root@elkstack ~]# systemctl enable elasticsearch.service
注意:如果查看状态,显示是红色,没有变绿,操作步骤:
(1)执行which java命令查看Java的目录在哪,然后将这个目录创建软连接到/usr/bin目录下
(2)ln -s /usr/local/jdk1.8.0_241/bin/java /usr/bin/java
(2)重启systemctl restart elasticsearch.service,再次 systemctl status elasticsearch.service查看状态
如果前面执行过了,一般情况下,会变绿的……
[root@elkstack ~]# netstat -antp |grep 9200
tcp6       0      0 :::9200                 :::*                    LISTEN      10082/java

大型企业架构部署之构建企业级ELK日志分析系统

三、部署Kibana

1.下载并安装kibana

[root@elkstack ~]# wget https://artifacts.elastic.co/downloads/kibana/kibana-6.2.4-x86_64.rpm
[root@elkstack ~]# rpm -ivh kibana-6.2.4-x86_64.rpm 
warning: kibana-6.2.4-x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:kibana-6.2.4-1                   ################################# [100%]

2.修改配置文件kibana.yml

找到这些参数并修改以下内容

[root@elkstack ~]# vim /etc/kibana/kibana.yml
server.port: 5601
server.host: "localhost"
elasticsearch.url: "http://localhost:9200"
kibana.index: ".kibana"

3.启动kibana并检查状态(必须有5601端口)

[root@elkstack ~]# systemctl start kibana.service 
[root@elkstack ~]# systemctl status kibana.service
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; disabled; vendor preset: disabled)
   Active: active (running) since Sun 2020-04-05 00:04:43 CST; 11ms ago
 Main PID: 10246 (node)
   CGroup: /system.slice/kibana.service
           └─10246 /usr/share/kibana/bin/../node/bin/node --no-warnings /usr/share/kibana/bin/../s...

Apr 05 00:04:43 elkstack systemd[1]: Started Kibana.
[root@elkstack ~]# netstat -lntp |grep 5601
tcp        0      0 127.0.0.1:5601          0.0.0.0:*               LISTEN      10246/node 

四、部署Logstash

1.下载并安装logstash

[root@elkstack ~]# wget https://artifacts.elastic.co/downloads/logstash/logstash-6.2.4.rpm
[root@elkstack ~]# rpm -ivh logstash-6.2.4.rpm 
warning: logstash-6.2.4.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:logstash-1:6.2.4-1               ################################# [100%]
Using provided startup.options file: /etc/logstash/startup.options
Successfully created system startup script for Logstash

2.创建logstash的数据和日志目录

[root@elkstack ~]# mkdir -p /data/ls-data
[root@elkstack ~]# chown -R logstash:logstash /data/ls-data
[root@elkstack ~]# mkdir -p /log/ls-log
[root@elkstack ~]# chown -R logstash:logstash /log/ls-log

3.修改配置文件logstash.yml

找到这些参数并修改以下内容

path.data: /data/ls-data
path.config: /etc/logstash/conf.d
path.logs: /log/ls-log

4.启动logstash并查看状态(一定要变绿)

[root@elkstack ~]# systemctl start logstash
[root@elkstack ~]# systemctl status logstash
● logstash.service - logstash
   Loaded: loaded (/etc/systemd/system/logstash.service; disabled; vendor preset: disabled)
   Active: active (running) since Sun 2020-04-05 00:13:39 CST; 11ms ago
 Main PID: 10421 (logstash)
   CGroup: /system.slice/logstash.service
           ├─10421 /bin/bash /usr/share/logstash/bin/logstash --path.settings /etc/logstash
           ├─10429 /bin/bash /usr/share/logstash/bin/logstash --path.settings /etc/logstash
           ├─10430 /bin/bash /usr/share/logstash/bin/logstash --path.settings /etc/logstash
           └─10431 /bin/bash /usr/share/logstash/bin/logstash --path.settings /etc/logstash

Apr 05 00:13:39 elkstack systemd[1]: Started logstash.
[root@elkstack ~]# systemctl enable logstash

5.测试logstash是否安装成功

1.给logstash创建软连接(执行的时候不需要绝对路径)

[root@elkstack ~]# ln -s /usr/share/logstash/bin/logstash /bin/

2.执行命令测试
当执行命令之后,等待片刻会有The stdin plugin is now waiting for input:提示,然后回车或者输入内容会有输错内容,如下所示则成功。

[root@elkstack ~]# logstash -e 'input { stdin { } } output { stdout {} }'
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
.....
...此次省略多行数据...
......
The stdin plugin is now waiting for input:
abc
{
    "@timestamp" => 2020-04-04T16:15:51.586Z,
       "message" => "abc",
      "@version" => "1",
          "host" => "elkstack"
}
lixian
{
    "@timestamp" => 2020-04-04T16:15:55.438Z,
       "message" => "lixian",
      "@version" => "1",
          "host" => "elkstack"

五、Nginx代理(使用域名访问)

1.下载并安装nginx

[root@elkstack ~]# wget http://nginx.org/packages/rhel/7/x86_64/RPMS/nginx-1.16.1-1.el7.ngx.x86_64.rpm
[root@elkstack ~]# yum localinstall -y nginx-1.16.1-1.el7.ngx.x86_64.rpm

2.给elasticsearch添加代理

[root@elkstack ~]# vim /etc/nginx/conf.d/elasticsearch.conf
server {
listen 81;
server_name www.elk.com;
location / {
    proxy_pass http://localhost:9200;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection 'upgrade';
    proxy_set_header Host $host;
    proxy_cache_bypass $http_upgrade;
} 
}

3.给kibana添加代理

server {
listen 80;
server_name www.elk.com;
location / {
    proxy_pass http://localhost:5601;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection 'upgrade';
    proxy_set_header Host $host;
    proxy_cache_bypass $http_upgrade;
}
}

3.检查并重启nginx服务

[root@elkstack ~]# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
[root@elkstack ~]# systemctl restart nginx

六、浏览器输入域名访问

注意:本地hosts文件要域名解析

大型企业架构部署之构建企业级ELK日志分析系统
大型企业架构部署之构建企业级ELK日志分析系统


本站博主 , 版权所有丨如未注明 , 均为原创
转载请注明原文链接:大型企业架构部署之构建企业级ELK日志分析系统
喜欢 (0)
发表我的评论
取消评论
表情 贴图 加粗 删除线 居中 斜体 签到