ELK(Elasticsearch+Logstash+Kibana) Apacheのグラフ化まで│システムガーディアン株式会社

JSON形式のデータからなんでもグラフにして遊んじゃおう！

今回はElasticsearch, Logstash, Kibanaを使ってローカルにデータ分析基盤を簡単に作りたいと思います。

1 役割
- 1.1 環境
2 Y軸とX軸の設定をしよう

役割

Logstash(データ抽出・転送) ⇒ Elasticsearch(分析基盤) ⇒ Kibana(表示)

こんな役割と流れになっています。ElasticsearchはMySQLなどのRDBMSみたいなものと考えて貰ってOKです！クエリのように、インデックスされたデータの抽出も出来ますよ。

環境

ESXi
CentOS7
CPU2, メモリ10GB

Apache WEBサーバのインストールと設定

Apacheのインストール

# yum install httpd
# systemctl enable httpd
# systemctl start httpd

# yum install httpd

# systemctl enable httpd

# systemctl start httpd

ログフォーマットの登録

# vi /etc/httpd/conf/httpd.conf


<IfModule log_config_module>
    #
    # The following directives define some format nicknames for use with
    # a CustomLog directive (see below).
    #
    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
    LogFormat "%h %l %u %t \"%r\" %>s %b" common


※以下を追記

    LogFormat "%h %l %u %t \"%r\" %s %b %D \"%{Referer}i\" \"%{User-agent}i\"" virtual

# vi /etc/httpd/conf/httpd.conf

# The following directives define some format nicknames for use with

# a CustomLog directive (see below).

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

LogFormat "%h %l %u %t \"%r\" %>s %b" common

※以下を追記

LogFormat "%h %l %u %t \"%r\" %s %b %D \"%{Referer}i\" \"%{User-agent}i\"" virtual

バーチャルホストの設定

# vi /etc/httpd/conf.d/yuutest.local.conf


<VirtualHost *:80>
    ServerName yuutest.local
    DocumentRoot /var/www/vhosts/yuutest.local/httpdocs
    ErrorLog logs/yuutest.local-error_log
    CustomLog logs/yuutest.local-access_log virtual env=!no_log
</VirtualHost>

# vi /etc/httpd/conf.d/yuutest.local.conf

ServerName yuutest.local

DocumentRoot /var/www/vhosts/yuutest.local/httpdocs

ErrorLog logs/yuutest.local-error_log

CustomLog logs/yuutest.local-access_log virtual env=!no_log

</VirtualHost>

環境に合わせて下さいね。

公開ディレクトリの作成

# mkdir -p /var/www/vhosts/yuutest.local/httpdocs
# echo "yuutest.local" > /var/www/vhosts/yuutest.local/httpdocs/index.html

1 2	# mkdir -p /var/www/vhosts/yuutest.local/httpdocs # echo "yuutest.local" > /var/www/vhosts/yuutest.local/httpdocs/index.html

反映させましょう！

# httpd -t
# systemctl restart httpd

1 2	# httpd -t # systemctl restart httpd

http://yuutest.local/

何回かアクセスしてログを作っておいて下さいね。

Elasticsearchのインストール

リポジトリの設定

# vi /etc/yum.repos.d/elastic.repo

[elasticsearch-5.x]
name=Elasticsearch repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

# vi /etc/yum.repos.d/elastic.repo

[elasticsearch-5.x]

name=Elasticsearch repository for 5.x packages

baseurl=https://artifacts.elastic.co/packages/5.x/yum

gpgcheck=1

gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch

enabled=1

autorefresh=1

type=rpm-md

インストール

# yum install java-1.8.0-openjdk-devel
# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
# yum install elasticsearch kibana logstash

# yum install java-1.8.0-openjdk-devel

# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

# yum install elasticsearch kibana logstash

ElasticSearchの設定

# vi /etc/elasticsearch/elasticsearch.yml

#network.host: 192.168.0.1
network.host: 0.0.0.0

# vi /etc/elasticsearch/elasticsearch.yml

#network.host: 192.168.0.1

network.host: 0.0.0.0

Kibana設定

# vi /etc/kibana/kibana.yml


#server.port: 5601

# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
#server.host: "localhost"
server.host: "0.0.0.0"

# vi /etc/kibana/kibana.yml

#server.port: 5601

# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.

# The default is 'localhost', which usually means remote machines will not be able to connect.

# To allow connections from remote users, set this parameter to a non-loopback address.

#server.host: "localhost"

server.host: "0.0.0.0"

Elasticsearch Systemdカスタマイズ

# vi /usr/lib/systemd/system/elasticsearch.service


[Service]

(略)

LimitMEMLOCK=infinity

# vi /usr/lib/systemd/system/elasticsearch.service

[Service]

(略)

LimitMEMLOCK=infinity

Elasticsearchの対応

# vi /etc/elasticsearch/jvm.options


-Xms2g
-Xmx2g

↓変更

#-Xms2g
#-Xmx2g
-Xms6g
-Xmx6g

# vi /etc/elasticsearch/jvm.options

-Xms2g

-Xmx2g

↓変更

#-Xms2g

#-Xmx2g

-Xms6g

-Xmx6g

Elasticsearchの起動

# systemctl daemon-reload
# systemctl restart elasticsearch

1 2	# systemctl daemon-reload # systemctl restart elasticsearch

ステータスの対応

# systemctl status elasticsearch


● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: active (running) since 火 2018-05-22 19:09:52 JST; 6s ago
     Docs: http://www.elastic.co
  Process: 10791 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
 Main PID: 10793 (java)
   CGroup: /system.slice/elasticsearch.service
           mq10793 /bin/java -Xms4g -Xmx4g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+Alw...

 5月 22 19:09:52 localhost.localdomain systemd[1]: Starting Elasticsearch...
 5月 22 19:09:52 localhost.localdomain systemd[1]: Started Elasticsearch.
 5月 22 19:09:52 localhost.localdomain elasticsearch[10793]: OpenJDK 64-Bit Server VM warning: If the number of processors is expected to inc...eads=N
Hint: Some lines were ellipsized, use -l to show in full.

# systemctl status elasticsearch

● elasticsearch.service - Elasticsearch

Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)

Active: active (running) since 火 2018-05-22 19:09:52 JST; 6s ago

Docs: http://www.elastic.co

Process: 10791 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)

Main PID: 10793 (java)

CGroup: /system.slice/elasticsearch.service

mq10793 /bin/java -Xms4g -Xmx4g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+Alw...

5月 22 19:09:52 localhost.localdomain systemd[1]: Starting Elasticsearch...

5月 22 19:09:52 localhost.localdomain systemd[1]: Started Elasticsearch.

5月 22 19:09:52 localhost.localdomain elasticsearch[10793]: OpenJDK 64-Bit Server VM warning: If the number of processors is expected to inc...eads=N

Hint: Some lines were ellipsized, use -l to show in full.

Elasticsearchはメモリが足りないなどで、ステータスを確認して下さい。

Elasticsearch+Logstash+Kibana 起動・自動起動設定

systemctl start kibana
systemctl start logstash
systemctl enable elasticsearch
systemctl enable kibana
systemctl enable logstash

systemctl start kibana

systemctl start logstash

systemctl enable elasticsearch

systemctl enable kibana

systemctl enable logstash

起動・ポート状況確認

# netstat -plntu

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1242/sshd
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1831/master
tcp        0      0 0.0.0.0:5601            0.0.0.0:*               LISTEN      2496/node ←kibana確認
tcp6       0      0 :::9200                 :::*                    LISTEN      10793/java ←elasticsearch確認
tcp6       0      0 :::80                   :::*                    LISTEN      2968/httpd
tcp6       0      0 :::9300                 :::*                    LISTEN      10793/java ←確認
tcp6       0      0 :::22                   :::*                    LISTEN      1242/sshd
tcp6       0      0 ::1:25                  :::*                    LISTEN      1831/master
tcp6       0      0 127.0.0.1:9600          :::*                    LISTEN      2669/java
udp        0      0 127.0.0.1:323           0.0.0.0:*                           613/chronyd
udp6       0      0 ::1:323                 :::*                                613/chronyd

# netstat -plntu

Active Internet connections (only servers)

Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name

tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1242/sshd

tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1831/master

tcp 0 0 0.0.0.0:5601 0.0.0.0:* LISTEN 2496/node ←kibana確認

tcp6 0 0 :::9200 :::* LISTEN 10793/java ←elasticsearch確認

tcp6 0 0 :::80 :::* LISTEN 2968/httpd

tcp6 0 0 :::9300 :::* LISTEN 10793/java ←確認

tcp6 0 0 :::22 :::* LISTEN 1242/sshd

tcp6 0 0 ::1:25 :::* LISTEN 1831/master

tcp6 0 0 127.0.0.1:9600 :::* LISTEN 2669/java

udp 0 0 127.0.0.1:323 0.0.0.0:* 613/chronyd

udp6 0 0 ::1:323 :::* 613/chronyd

Logstashのコンフィグ設定

# vi /etc/logstash/conf.d/10_apache_access_log.conf


input {
    file {
        path => "/var/log/httpd/yuutest.local-access_log"
        start_position => "beginning"
    }
}

filter {
    grok {
        match => {
            "message" => "%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{URIPATH:requestpath:string}%{URIPARAM:reqparam}(?: HTTP/%{NUMBER:httpversion})?|%{CISCO_REASON:verb}%{URIPATHPARAM:requestpath:string} HTTP/%{NUMBER:httpversion}?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes:int}|-) (?:%{NUMBER:response_time:int}|-) %{QS:referrer} %{QS:agent}"
        }
    }
    date {
        match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]
    }
    geoip {
        source => ["clientip"]
    }
}

output {
    elasticsearch {
        hosts => ["localhost:9200"]
        index => "yuutest.local-%{+YYYYMMdd}"
    }
}

# vi /etc/logstash/conf.d/10_apache_access_log.conf

input {

file {

path => "/var/log/httpd/yuutest.local-access_log"

start_position => "beginning"

}

filter {

grok {

match => {

"message" => "%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{URIPATH:requestpath:string}%{URIPARAM:reqparam}(?: HTTP/%{NUMBER:httpversion})?|%{CISCO_REASON:verb}%{URIPATHPARAM:requestpath:string} HTTP/%{NUMBER:httpversion}?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes:int}|-) (?:%{NUMBER:response_time:int}|-) %{QS:referrer} %{QS:agent}"

}

date {

match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]

}

geoip {

source => ["clientip"]

}

output {

elasticsearch {

hosts => ["localhost:9200"]

index => "yuutest.local-%{+YYYYMMdd}"

}

パーミッション設定

# chgrp logstash -R /var/log/httpd
# chmod 640 -R /var/log/httpd

1 2	# chgrp logstash -R /var/log/httpd # chmod 640 -R /var/log/httpd

しなくてもデフォルトで読めるとは思います。

設定を読み込ませる

# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/10_apache_access_log.conf

1	# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/10_apache_access_log.conf

インデックス状況の確認

# curl 'localhost:9200/_cat/indices?v'

health status index                  uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   yuutest.local-20180525 iF0UbAImTLGt9l8E1SHwxw   5   1        124            0    301.1kb        301.1kb
yellow open   .kibana                DcNCDEm3QQSOkc5MNAp-lQ   1   1         69           31      125kb          125kb

# curl 'localhost:9200/_cat/indices?v'

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size

yellow open yuutest.local-20180525 iF0UbAImTLGt9l8E1SHwxw 5 1 124 0 301.1kb 301.1kb

yellow open .kibana DcNCDEm3QQSOkc5MNAp-lQ 1 1 69 31 125kb 125kb

yuutest.local-（＾－＾）がインデックスされていますね。

実際にはこんな風にJSON形式でインデックスされています。

# curl 'localhost:9200/yuutest.local-20180525/_search?q=*&pretty'


{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 124,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "yuutest.local-20180525",
        "_type" : "logs",
        "_id" : "AWOWZ6S8xrgPWWhLbaj-",
        "_score" : 1.0,
        "_source" : {
          "agent" : "\"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36\"",
          "geoip" : { },
          "auth" : "-",
          "ident" : "-",
          "verb" : "GET ",
          "message" : "192.168.100.3 - - [25/May/2018:17:24:45 +0900] \"GET / HTTP/1.1\" 304 - 142 \"-\" \"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36\"",
          "requestpath" : "/",
          "tags" : [
            "_geoip_lookup_failure"
          ],
          "path" : "/var/log/httpd/yuutest.local-access_log",
          "referrer" : "\"-\"",
          "@timestamp" : "2018-05-25T08:24:45.000Z",
          "response" : "304",
          "clientip" : "192.168.100.3",
          "@version" : "1",
          "host" : "localhost.localdomain",
          "httpversion" : "1.1",
          "response_time" : 142,
          "timestamp" : "25/May/2018:17:24:45 +0900"
        }
      },
      {
        "_index" : "yuutest.local-20180525",
        "_type" : "logs",
        "_id" : "AWOWZ6S8xrgPWWhLbakC",
        "_score" : 1.0,
        "_source" : {

(略)

# curl 'localhost:9200/yuutest.local-20180525/_search?q=*&pretty'

{

"took" : 1,

"timed_out" : false,

"_shards" : {

"total" : 5,

"successful" : 5,

"skipped" : 0,

"failed" : 0

"hits" : {

"total" : 124,

"max_score" : 1.0,

"hits" : [

{

"_index" : "yuutest.local-20180525",

"_type" : "logs",

"_id" : "AWOWZ6S8xrgPWWhLbaj-",

"_score" : 1.0,

"_source" : {

"agent" : "\"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36\"",

"geoip" : { },

"auth" : "-",

"ident" : "-",

"verb" : "GET ",

"message" : "192.168.100.3 - - [25/May/2018:17:24:45 +0900] \"GET / HTTP/1.1\" 304 - 142 \"-\" \"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36\"",

"requestpath" : "/",

"tags" : [

"_geoip_lookup_failure"

"path" : "/var/log/httpd/yuutest.local-access_log",

"referrer" : "\"-\"",

"@timestamp" : "2018-05-25T08:24:45.000Z",

"response" : "304",

"clientip" : "192.168.100.3",

"@version" : "1",

"host" : "localhost.localdomain",

"httpversion" : "1.1",

"response_time" : 142,