Logstash 与 filebeat 配置

 

 

mutate插件可以对事件中的数据进行修改,包括rename、update、replace、convert、split、gsub、uppercase、lowercase、strip、remove_field、join、merge等功能。

1、rename

对于已经存在的字段,重命名其字段名称。

 

filter {
    mutate {
        rename => ["syslog_host", "host"]
    }
}

2、update

更新字段内容,如果字段不存在,不会新建

 

filter {
    mutate {
        update => { "sample" => "My new message" }
    }
}

3、replace

与 update 功能相同,区别在于如果字段不存在则会新建字段

 

filter {
    mutate {
        replace => { "message" => "%{source_host}: My new message" }
    }
}

4、convert

数据类型转换。

 

filter {
    mutate {
        convert => ["request_time", "float"]
    }
}

5、gsub

gsub 提供了通过正则表达式实现文本替换的功能。

 

filter {
    mutate {
        gsub => [
            # replace all forward slashes with underscore
            "fieldname", "/", "_",
            # replace backslashes, question marks, hashes, and minuses
            # with a dot "."
            "fieldname2", "[\\?#-]", "."
        ]
    }
}

6、uppercase/lowercase

大小写转换

 

filter {
    mutate {
        uppercase => [ "fieldname" ]
    }
}

7、split

将提取到的某个字段按照某个字符分割

 

filter {
    mutate {
        split => ["message", "|"]
    }
}

针对字符串 "123|321|adfd|dfjld*=123",可以看到输出结果:

 

{
    "message" => [
        [0] "123",
        [1] "321",
        [2] "adfd",
        [3] "dfjld*=123"
    ],
    "@version" => "1",
    "@timestamp" => "2014-08-20T15:58:23.120Z",
    "host" => "raochenlindeMacBook-Air.local"
}

8、strip

类似 trim,只去除首尾的空白字符

 

filter {
    mutate {
        strip => ["field1", "field2"]
    }
}

9、remove_field

删除字段:

 

filter {
    mutate {
        remove_field => [ "foo_%{somefield}" ]
    }
}

10、join

将类型为 array 的字段中的 array 元素使用指定字符为分隔符聚合成一个字符串。
如我们可以将 split 分割的结果再重新聚合起来:

 

filter {
    mutate {
        split => ["message", "|"]
    }
    mutate {
        join => ["message", ","]
    }
}

输出结果:

 

{
    "message" => "123,321,adfd,dfjld*=123",
    "@version" => "1",
    "@timestamp" => "2014-08-20T16:01:33.972Z",
    "host" => "raochenlindeMacBook-Air.local"
}

11、merge

对于几个类型为 array 或 hash 或 string 的字段,我们可以使用 merge 合并

 

filter {
    mutate {
        merge => [ "dest_field", "added_field" ]
    }
}

需要注意的是,array 和 hash 两个字段是不能 merge 的

注意:建议正则放在单引号内,例如'^\[?[0-9][0-9]:?[0-9][0-9]|^[[:graph:]]+'

样例 描述

单个字符

 

x

单个字符

.

任何字符

[xyz]

字符类

[^xyz]

非字符类

[[:alpha:]]

ASCII字符类

[[:^alpha:]]

非ASCII字符类

\d

Perl字符类

\D

非Perl字符类

\pN

Unicode字符类(一个字母的名称)

\p{Greek}

Unicode字符类

\PN

非Unicode字符类(一个字母的名称)

\P{Greek}

非Unicode字符类

复合类型

 

xy

x|y

重复类型

 

x*

以x开头

x+

一个或者多个x

x?

零或一个x

x{n,m}

n or n+1 or … or m x, prefer more

x{n,}

n or more x, prefer more

x{n}

exactly n x

x*?

zero or more x, prefer fewer

x+?

one or more x, prefer fewer

x??

zero or one x, prefer zero

x{n,m}?

n or n+1 or … or m x, prefer fewer

x{n,}?

n or more x, prefer fewer

x{n}?

exactly n x

分组

 

(re)

numbered capturing group (submatch)

(?P<name>re)

named & numbered capturing group (submatch)

(?:re)

non-capturing group

(?i)abc

set flags within current group, non-capturing

(?i:re)

set flags during re, non-capturing

(?i)PaTTeRN

case-insensitive (default false)

(?m)multiline

multi-line mode: ^ and $ match begin/end line in addition to begin/end text (default false)

(?s)pattern.

let . match \n (default false)

(?U)x*abc

ungreedy: swap meaning of x* and x*?x+ and x+?, etc (default false)

空字符串

 

^

at beginning of text or line (m=true)

$

at end of text (like \z not \Z) or line (m=true)

\A

at beginning of text

\b

at ASCII word boundary (\w on one side and \W\A, or \z on the other)

\B

not at ASCII word boundary

\z

at end of text

转义序列

 

\a

bell (same as \007)

\f

form feed (same as \014)

\t

horizontal tab (same as \011)

\n

newline (same as \012)

\r

carriage return (same as \015)

\v

vertical tab character (same as \013)

\*

literal *, for any punctuation character *

\123

octal character code (up to three digits)

\x7F

two-digit hex character code

\x{10FFFF}

hex character code

\Q...\E

literal text ... even if ... has punctuation

ASCII字符类

 

[[:alnum:]]

alphanumeric (same as [0-9A-Za-z])

[[:alpha:]]

alphabetic (same as [A-Za-z])

[[:ascii:]]

ASCII (same as \x00-\x7F])

[[:blank:]]

blank (same as [\t ])

[[:cntrl:]]

control (same as [\x00-\x1F\x7F])

[[:digit:]]

digits (same as [0-9])

[[:graph:]]

graphical (same as [!-~] == [A-Za-z0-9!"#$%&'()*+,\-./:;<=>?@[\\\]^_` {|}~])

[[:lower:]]

lower case (same as [a-z])

[[:print:]]

printable (same as [ -~] == [ [:graph:]])

[[:punct:]]

punctuation (same as [!-/:-@[-`{-~])

[[:space:]]

whitespace (same as [\t\n\v\f\r ])

[[:upper:]]

upper case (same as [A-Z])

[[:word:]]

word characters (same as [0-9A-Za-z_])

[[:xdigit:]]

hex digit (same as [0-9A-Fa-f])

支持Perl字符类

 

\d

digits (same as [0-9])

\D

not digits (same as [^0-9])

\s

whitespace (same as [\t\n\f\r ])

\S

not whitespace (same as [^\t\n\f\r ])

\w

word characters (same as [0-9A-Za-z_])

\W

not word characters (same as [^0-9A-Za-z_])

 

 

架构一:
filebeat -> logstash1 -> redis -> logstash2 -> elasticsearch(集群) -> kibana
这里就不写安装程序的步骤了相信大家都没有难度:
(软件安装可自行设计)
230,安装filebeat, logstash1 ,elasticsearch
232,安装logstash2, redis, elasticsearch  ,kibana

注意:filebeat文件很注重文件格式
1,配置filebeat文件:
[root@localhost filebeat]# cat /etc/filebeat/filebeat.yml
filebeat:
  prospectors:
   # - #每个日志文件的开始
   #   paths: #定义路径
   #     - /var/www/logs/access.log #绝对路径
   #   input_type: log #日志类型为log
   #   document_type: api4-nginx-accesslog # 此名称要与logstash定义的名称相对应,logstash要使用此名称做type判断使用
    -
      paths:
        - /opt/apps/huhu/logs/ase.log
      input_type: log
      document_type: "ase-ase-log"
      encoding: utf-8
      tail_files: true  #每次最后一行
      multiline.pattern: '^\[' #分割符
      multiline.negate: true
      multiline.match: after    #最后合并
      #tags: ["ase-ase"]

    -
      paths:   #收集json格式日志
        - /var/log/nginx/access.log
      input_type: log
      document_type: "nginx-access-log"
      tail_files: true
      json.keys_under_root: true      
      json.overwrite_keys: true  

  registry_file: /var/lib/filebeat/registry
output:      #输出到230
  logstash:
    hosts: ["192.168.0.230:5044"]

shipper:
  logging:
    to_files: true
    files:
      path: /tmp/mybeat

 2.配置230:logstash-->input-redis
[root@web1 conf.d]# pwd
/etc/logstash/conf.d
[root@web1 conf.d]# cat nginx-ase-input.conf 
input {
        beats {
        port => 5044
        codec => "json"
        }}

output {                         
        if [type] == "nginx-access-log" {
        redis {                            #nginx日志写到redis信息
                data_type => "list"
                key => "nginx-accesslog"
                host => "192.168.0.232"
                port => "6379"
                db => "4"
                password => "123456"
        }}
        if [type] == "ase-ase-log" {
        redis {                            #写到ase日志写到redis信息
                data_type => "list"
                key => "ase-log"
                host => "192.168.0.232"
                port => "6379"
                db => "4"
                password => "123456"
        }}      

}

  3.redis写到elstach里,232服务器配置:logstash-->output-->resid->elasticsearch
[root@localhost conf.d]# pwd
/etc/logstash/conf.d
[root@localhost conf.d]# cat nginx-ase-output.conf 
input {
        redis {
               type => "nginx-access-log"
                data_type => "list"
                key => "nginx-accesslog"
                host => "192.168.0.232"
                port => "6379"
                db => "4"
                password => "123456"
                codec  => "json"
        }

        redis {
                type => "ase-ase-log"
                data_type => "list"
                key => "ase-log"
                host => "192.168.0.232"
                port => "6379"
                db => "4"
                password => "123456"
        }
}

output {
    if [type] == "nginx-access-log" { 
        elasticsearch {  
            hosts => ["192.168.0.232:9200"] 
            index => "nginx-accesslog-%{+YYYY.MM.dd}" 
    }}
    if [type] == "ase-ase-log" {
            elasticsearch {
                hosts => ["192.168.0.232:9200"]
                index => "ase-log-%{+YYYY.MM.dd}"
        }}
}

4,在232上配置elsaticsearch--->kibana
在kibana上找到ELS的索引即可。

架构二:
filebeat -> redis -> logstash --> elsasctic --> kibana  #缺点filebeat写进redis有限制,占时还没找到多个写入。

1.feilebeat配置:
[root@localhost yes_yml]# cat filebeat.yml 
filebeat:
  prospectors:
   # - #每个日志文件的开始
   #   paths: #定义路径
   #     - /var/www/logs/access.log #绝对路径
   #   input_type: log #日志类型为log
   #   document_type: api4-nginx-accesslog # 此名称要与logstash定义的名称相对应,logstash要使用此名称做type判断使用
    -
      paths:
        - /opt/apps/qpq/logs/qpq.log
      input_type: log
      document_type: "qpq-qpq-log"
      encoding: utf-8
      tail_files: true
      multiline.pattern: '^\['
      multiline.negate: true
      multiline.match: after
   #tags: ["qpq-qpq-log"]
  registry_file: /var/lib/filebeat/registry

output:
  redis:
      host: "192.168.0.232"
      port: 6379
      db: 3
      password: "123456"
      timeout: 5
      reconnect_interval: 1
      index: "pqp-pqp-log"

shipper:
  logging:
    to_files: true
    files:
      path: /tmp/mybeat

2.由232redis-->els--kibana
[root@localhost yes_yml]# cat systemlog.conf 
input {
   redis {
        type => "qpq-qpq-log"
        data_type => "list"
        key => "qpq-pqp-log"
        host => "192.168.0.232"
        port => "6379"
        db => "3" 
        password => "123456"
        }}
output {
   if [type] == "qpq-qpq-log"{
      elasticsearch {  
            hosts => ["192.168.0.232:9200"] 
            index => "qpq-qpq-log-%{+YYYY.MM.dd}" 

 }

}
}

3.在232上配置elsaticsearch--->kibana
在kibana上找到ELS的索引即可

 

filebeat 执行命令

filebeat -e -c filebeat.yml

 

 

 

 

 

wangxiaoming CSDN认证博客专家 架构 Spring Boot Redis
博客是很好的总结和记录工具,如果有问题,来不及回复,关注微信公众号:程序员开发者社区,获取我的联系方式,向我提问,也可以给我发送邮件,联系 1275801617@qq.com
©️2020 CSDN 皮肤主题: Age of Ai 设计师: meimeiellie 返回首页
实付0元
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、C币套餐、付费专栏及课程。

余额充值