0%

ES+Logstash+Kibana

下载ELK包

ELK 7.12

ELK是一整套解决方案,ELK由ElasticSearch、Logstash和Kiabana三个开源工具组成

Elasticsearch + Kibana

Logstash

准备环境及启用工具

Ubuntu 20.04.2 LTS MATE

sudo apt install openjdk-11-jre-headless curl pluma -y

ELK搭建教程

搭建ELK日志分析系统

Elasticserach

修改elasticserach配置文件

vi conf/elasticserach.yml

1
2
3
4
5
6
node.name: node-1
network.host 0.0.0.0
http.port: 9200
discovery.seed_hosts: ["127.0.0.1"]
cluster.initial_master_nodes: ["node-1"]
action.destructive_requires_name: true

Firefox

安装es-head插件

插件 –> elasticvue

ES基本命令

1
2
3
4
5
6
7
8
1、使用_cat API检测集群是否健康,确保9200端口号可用:
curl http://localhost:9200/_cat/health?v

2、获取集群的节点列表
curl http://localhost:9200/_cat/nodes?v

3、查看所有索引
curl http://localhost:9200/_cat/indices?v

ELK-ElasticSearch索引详解

Kibana

修改kibana配置文件

vi conf/kibana.yml

1
2
3
4
5
6
7
server.port: 5601
server.host: "localhost"
server.name: "kibana"
elasticsearch.hosts: ["http://localhost:9200"]
kibana.index: ".kibana"
elasticsearch.pingTimeout: 1500
elasticsearch.requestTimeout: 30000

Kibana官方中文手册

Logstash

修改logstash配置文件

vi conf/logstash.yml

1
2
3
path.data: /home/User/Desktop/logstash-7.12.0/data

path.logs: /home/User/Desktop/logstash-7.12.0/logs

测试编写grok规则

问题解决

解决max virtual memory areas vm.max_map_count [65530] is too low

  • sudo vi /etc/sysctl.conf
  • vm.max_map_count=262144
  • sudo /sbin/sysctl -p

解决memory is not locked

sudo vi /etc/security/limits.conf

1
2
3
4
5
6
* soft nofile 65536
* hard nofile 65536
* soft nproc 32000
* hard nproc 32000
* hard memlock unlimited
* soft memlock unlimited

sudo vi /etc/systemd/system.conf

1
2
3
DefaultLimitNOFILE=65536
DefaultLimitNPROC=32000
DefaultLimitMEMLOCK=infinity

重启系统,启动es

创建桌面执行图标

vi es.desktop

1
2
3
4
5
6
7
8
9
#!/usr/bin/env xdg-open
[Desktop Entry]
Type=Application
Name=ES
Comment=Tool
Exec=bash /home/User/Desktop/elasticsearch-7.12.0/bin/elasticsearch
Icon=/home/User/Desktop/elasticsearch-7.12.0/icon-elasticsearch.svg
StartupNotify=true
Terminal=true

相关脚本

因某些txt文本太大,直接打开不现实,so,要搞个脚本去读取基本格式,用作给Logstash匹配用

读取txt文本前10行

python3 read_txt_library.py CSDN_600w

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
#!/usr/bin/env python3
# -*- coding:utf-8 -*-

import sys

DIR = "./Social_Library/"

LIBRARY_NAME = sys.argv[1]

if LIBRARY_NAME[-4:] == ".txt":
LOCATION = DIR+LIBRARY_NAME
else:
LOCATION = DIR+LIBRARY_NAME+'.txt'

with open(LOCATION, 'rb') as f:
for i in range(10):
print(f.readline().decode('utf8').rstrip("\n"))

欢迎关注我的其它发布渠道