Skip to content
This repository was archived by the owner on Feb 27, 2019. It is now read-only.

Commit 41e4d29

Browse files
jehuty0shiftjehuty0shift
committed
adding filebeat guide PaaS Logs documentation
Signed-off-by: Babacar Diassé <babacar.diasse@ovh.net>
1 parent e39135e commit 41e4d29

File tree

3 files changed

+267
-0
lines changed

3 files changed

+267
-0
lines changed

‎en/_posts/2016-02-25-filebeat-logs.md

+267
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,267 @@
1+
---
2+
layout: post
3+
title: "Shipping logs to PaaS Logs with Filebeat"
4+
categories: Logs
5+
author: Pierre de Paepe
6+
lang: en
7+
---
8+
9+
Filebeat is an [open source](https://github.com/elastic/beats/tree/master/filebeat) file harvester, mostly used to fetch logs files and feed them into logstash. Together with Logstash, Filebeat is a really powerful tool that allows you to parse and send your logs to PaaS logs in a elegant and non intrusive way (except installing filebeat of course ;-).
10+
11+
This guide will describe how to ask OVH to host your own dedicated Logstash on PaaS Logs and how to setup Filebeat on your system to forward your logs to it. It will also present you with some configuration setup you can use on Logstash to further structure your logs. Note that in order to complete this tutorial, you should have at least :
12+
13+
- [Activated the Paas Logs lab and created an account.](/kb/en/logs/quick-start.html#account)
14+
- [created at least one Stream and get its token.](/kb/en/logs/quick-start.html#streams)
15+
16+
Once you have done theses two steps, you can dig into this one. Be prepared.
17+
18+
----------
19+
20+
#1 Simple Logstash 2.x Configuration on PaaS Logs
21+
22+
This simple configuration is here only to make it easier for you to see your logs, a couple of chapters later, you will find more advanced configurations that will breakdown your code. If you are already familiar with Logstash configuration on PaaS Logs, you can skip this one. Otherwise, it is a good start point to get it up and running.
23+
On PaaS Logs manager, in Inputs section:
24+
25+
1. Click on blue + icon
26+
2. Give a name, a short description, select "Logstash 2.x" as engine, then click on the blue floppy disk icon to save the entry.
27+
3. Attach your graylog stream to logstash by clicking on `Subscription` then on `Attach this stream`. Please refer to [this guide first](https://community.runabove.com/kb/en/logs/quick-start.html#streams) if you need to create a new one.
28+
3. Once attached, click on "Networking", then set "5044" as exposed port. If you change it, you will have to also change it in the input section of your Logstash configuration. Click on "Add" to add it. You might want to also add the IPs where your logs come from, so the hosted input will only trust these IPs.
29+
4. Now please click on "Configuration", then fill the following snippet.
30+
31+
On input section, add:
32+
33+
input {
34+
beats {
35+
port => 5044
36+
ssl => true
37+
ssl_certificate => "/etc/ssl/private/server.crt"
38+
ssl_key => "/etc/ssl/private/server.key"
39+
}
40+
}
41+
42+
On filter section, add:
43+
44+
filter {
45+
}
46+
47+
Once configured, You can launch your logstash by clicking on "Start" button. At the end the procedure, a hostname will appear in green meaning your input is started. You will need this hostname for Filebeat configuration.
48+
49+
#2 Setup Filebeat in your system
50+
51+
Filebeat supports <b>many platforms</b> as listed here [https://www.elastic.co/downloads/beats/filebeat](https://www.elastic.co/downloads/beats/filebeat)
52+
Following section will give the Debian one as an example.
53+
54+
you can decide to setup Filebeats from package or to compile it from source (you will need the latest [go compiler](https://golang.org/) to compile it) or just download the generic Linux binary to start immediately.
55+
56+
For this part head to [Filebeat download website](https://www.elastic.co/downloads/beats/filebeat) to download the best version for your distribution. Just choose the Linux 64 bit if you don't know which one to choose.
57+
58+
The following configuration files have been tested on the latest version of Filebeat available at the time of writing (1.1.2).
59+
The Debian installation package will install the config file in the following directory : /etc/filebeat/filebeat.yml
60+
61+
62+
63+
#3 Configure Filebeat on your system
64+
65+
Filebeat expect a configuration file named **filebeat.yml** .
66+
67+
Following example will be for Apache logs and syslog files but you can easily prospect anything else. The trick is to attach a type to any file you parse so that in Logstash, you will be able to select the correct Grok for your file. You will see in the next chapter how to parse your logs depending on the type you send. For the configuration to work, the important part is to replace *hosts: ["c002-my-paas-logs-hostname.in.laas.runabove.com:5044"]* with the hostname given by PaaS Logs. You should also put the SSL Certificate authority of the dedicated inputs in a file, (ex : /usr/local/etc/filebeat/laas-ca.crt). The input SSL CA is exposed below.
68+
69+
####Filebeat configuration:
70+
71+
```
72+
############################# Filebeat ######################################
73+
filebeat:
74+
# List of prospectors to fetch data.
75+
prospectors:
76+
# Each - is a prospector. Below are the prospector specific configurations
77+
# Paths that should be crawled and fetched. Glob based paths.
78+
# To fetch all ".log" files from a specific level of subdirectories
79+
# /var/log/*/*.log can be used.
80+
# For each file found under this path, a harvester is started.
81+
# Make sure not file is defined twice as this can lead to unexpected behaviour.
82+
-
83+
paths:
84+
- /var/log/apache2/access.log
85+
input_type: log
86+
document_type: apache
87+
fields_under_root: true
88+
-
89+
paths:
90+
- /var/log/apache2/error.log
91+
input_type: log
92+
document_type: apache-error
93+
fields_under_root: true
94+
-
95+
paths:
96+
- /var/log/syslog
97+
input_type: log
98+
document_type: syslog
99+
fields_under_root: true
100+
101+
# Name of the registry file. Per default it is put in the current working
102+
# directory. In case the working directory is changed after when running
103+
# filebeat again, indexing starts from the beginning again.
104+
registry_file: /var/lib/filebeat/registry
105+
############################# Output ##########################################
106+
# Configure what outputs to use when sending the data collected by the beat.
107+
# Multiple outputs may be used.
108+
output:
109+
### Logstash as output
110+
logstash:
111+
# The Logstash hosts
112+
hosts: ["c002-my-paas-logs-hostname.in.laas.runabove.com:5044"]
113+
worker: 1
114+
tls:
115+
# List of root certificates for HTTPS server verifications
116+
certificate_authorities:
117+
- /usr/local/etc/filebeat/laas-ca.crt
118+
############################# Logging #########################################
119+
# There are three options for the log ouput: syslog, file, stderr.
120+
# Under Windos systems, the log files are per default sent to the file output,
121+
# under all other system per default to syslog.
122+
logging:
123+
# Send all logging output to syslog. On Windows default is false, otherwise
124+
# default is true.
125+
to_syslog: false
126+
# Write all logging output to files. Beats automatically rotate files if rotateeverybytes
127+
# limit is reached.
128+
to_files: true
129+
# To enable logging to files, to_files option has to be set to true
130+
files:
131+
# The directory where the log files will written to.
132+
path: /var/log/
133+
# The name of the files where the logs are written to.
134+
name: filebeat.log
135+
# Configure log file size limit. If limit is reached, log file will be
136+
# automatically rotated
137+
rotateeverybytes: 10485760 # = 10MB
138+
# Number of rotated log files to keep. Oldest files will be deleted first.
139+
keepfiles: 7
140+
# Sets log level. The default log level is error.
141+
# Available log levels are: critical, error, warning, info, debug
142+
level: info
143+
```
144+
145+
####SSL CA Certificate
146+
```bash
147+
-----BEGIN CERTIFICATE-----
148+
MIIDozCCAougAwIBAgIJALxR4fTZlzQMMA0GCSqGSIb3DQEBCwUAMGgxCzAJBgNV
149+
BAYTAkZSMQ8wDQYDVQQIDAZGcmFuY2UxDjAMBgNVBAcMBVBhcmlzMQwwCgYDVQQK
150+
DANPVkgxCzAJBgNVBAYTAkZSMR0wGwYDVQQDDBRpbi5sYWFzLnJ1bmFib3ZlLmNv
151+
bTAeFw0xNjAzMTAxNTEzMDNaFw0xNzAzMTAxNTEzMDNaMGgxCzAJBgNVBAYTAkZS
152+
MQ8wDQYDVQQIDAZGcmFuY2UxDjAMBgNVBAcMBVBhcmlzMQwwCgYDVQQKDANPVkgx
153+
CzAJBgNVBAYTAkZSMR0wGwYDVQQDDBRpbi5sYWFzLnJ1bmFib3ZlLmNvbTCCASIw
154+
DQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAL03NApk8fl82L4cH7XW+c+8k8dX
155+
xDWLaHl5sfxXqUghmbz5+O5GHPRecxZifcyxsgiw8kUh/wxkqu4+ac4HK0Anod9i
156+
h6VpT7zSTgdFfmJcOxkrcJ9cfVScvWN/4fYZGkGXJHiu+GHmZU1906P2q/OOibpg
157+
/FVvRo/+xoo4RI/uGBrezeSzDjq6vjPY0+eSTtBqb0h459Bguvv2gxV+u8PfpZEk
158+
ELU9KxGlgbikkMTV/Q7zfMEG+4e6A7xxoM33Bh0DhsIALLtBSd6jed5YiYQL2ke2
159+
OMIqwWrOnoccSp46TmDOd62NAESd2hif3Cwd/jbM/D/dfGetW99DrpH/7jUCAwEA
160+
AaNQME4wHQYDVR0OBBYEFFaAcbmGh/ObAeMhYQb3Norh0I1yMB8GA1UdIwQYMBaA
161+
FFaAcbmGh/ObAeMhYQb3Norh0I1yMAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcNAQEL
162+
BQADggEBADts4SsO+01wjv5BF22kUCPoiNzZW00PYXwjKQf/4oaqJgHVAb2NcnlO
163+
/p86eKzkPjTAH4B1PeGaSAGTt63C8h52ec4UgkjtZ5kf7pfmeH2ZDNVSSHYgoeDF
164+
7nXPyUtwDmHHrwoWJPalL/lo6eWXu/1oaioBvctFGwQf1yTIOCsHktu5rQlOAXn8
165+
r4IOeC764Hsupu2IjaLkyp+WBb6mRIS4B3ubDM8Vuc8tc7GC0B+5jXhOQRu9ZNfO
166+
3Xulb5Vk3AYF6s8TQ3ALK4doCupTUPX4XMXbtBH3XA8Rp7/dLo4oMWQrDpGzP5ys
167+
2kv1X/+sZvjaR0Eezj2owsqR3slqSZ0=
168+
-----END CERTIFICATE-----
169+
```
170+
171+
###Launch Filebeat
172+
173+
Launch the Filebeat binary or service to test your config file and head to your apache website for exemple to send some logs.
174+
you will see this kind of logs in Graylog :
175+
176+
![beat_graylog](/kb/images/2016-02-25-filebeat-logs/beats_graylog.png)
177+
178+
179+
Note the type value (apache or syslog or apache-error) that indicate the source file of the log message. You can easily display only your apache access logs for example by typing in the search bar `type:apache`.
180+
181+
182+
#4 OK i get it, but i want some magic powers now.
183+
184+
It's cool we have our logs but we can make them even more useful. By specifying the right configuration in Logstash, we can parse it and enrich the log messages with custom fields.
185+
For this you have to tweak two items:
186+
- the filter configuration in Logstash
187+
- the Grokpatterns configuration in Logstash
188+
189+
The main idea is to define custom fields in Grok patterns and to use these Groks in the Filter Section of Logstash. Head to the Logstash Configuration interface in the Paas Logs Manager by clicking on `Configuration` in your input panel. Here are some valid custom Grok you can use for Apache and Syslog :
190+
191+
####Grok Pattern configuration
192+
193+
```
194+
OVHCOMMONAPACHELOG %{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion_num:float})?|%{DATA:rawrequest})" %{NUMBER:response_int:int} (?:%{NUMBER:bytes_int:int}|-)
195+
OVHCOMBINEDAPACHELOG %{OVHCOMMONAPACHELOG} %{QS:referrer} %{QS:agent}
196+
```
197+
198+
a Grok pattern is a pattern in the following form %{SYNTAX:SEMANTIC}. This pattern will allow you to specify the fields in a log of line in the order in which they appear. Note that we customize the fields by specifying the actual type of number types and by suffixing them with '\_num' or '\_int' as explained in the [PaaS Logs fields convention tutorial](/kb/logs/2016-02-28-field-naming-conventions.md).
199+
Now that the Grok are defined, you can use them freely in your Logstash filter configuration.
200+
201+
####Logstash Filter Configuration
202+
203+
204+
```
205+
filter {
206+
if [type] == "apache" {
207+
Grok {
208+
match => { "message" => "%{OVHCOMMONAPACHELOG}" }
209+
patterns_dir => "/opt/logstash/patterns"
210+
}
211+
if ("_grokparsefailure" in [tags]) {
212+
mutate {
213+
remove_tag => [ "_grokparsefailure" ]
214+
}
215+
Grok {
216+
match => [ "message", "%{OVHCOMBINEDAPACHELOG}" ]
217+
patterns_dir => "/opt/logstash/patterns"
218+
named_captures_only => true
219+
}
220+
}
221+
}
222+
if [type] == "syslog" {
223+
Grok {
224+
match => { "message" => "%{SYSLOGBASE}" }
225+
}
226+
}
227+
}
228+
```
229+
230+
In this configuration you can see how Logstash will parse your logs. It will use the type field you defined before in the Filebeat configuration. If it matches "apache" for example, it will try to match the log line with the Grok COMMONAPACHELOG, if the Grok fail, it will add a tag `_grokparsefailure`. We use this tag to relaunch the Grok parsing by using another Grok We use this tag to relaunch the grok parsing by using another Grok. This is how you can specify several Grok for diverse messages that could be in one log file.
231+
232+
Note also how the syslog part of the filter use the default Grok Pattern SYSLOGBASE provided by Logstash to parse the syslog lines sent by Filebeat. There is a lot of Grok Patterns already available in Logstash, check the links at the end to know how you can effortlessly parse any kind of log source.
233+
234+
Once the configuration is done, click on 'Update Configuration' at the bottom of the page. You can really easily test your Configuration afterwards by using the button `Test` on the Input Panel. This will launch a task that will check if the Input and Filter part of the configuration are valid. You will see the following output if it is :
235+
```
236+
Configuration OK
237+
```
238+
239+
Once done, restart the input and wait for it to be active. Don't worry you won't lose any logs in the meantime, Filebeat tracks automatically the offset of the last log sent in the log file. Get to your stream to watch your brand new and shiny parsed logs lines.
240+
This is what you can have in Graylog when you use these filters :
241+
242+
![filter_graylog](/kb/images/2016-02-25-filebeat-logs/filter_graylog.png)
243+
244+
245+
As you can see, response code got its own field, as the bytes transmitted that you can already use in a graph to monitor the global traffic going through one particular page or website. you can also see all the traffic requested by a particular IP, and easily find the kind of content or webpage requested.
246+
247+
#5 Conclusion and useful resources
248+
249+
Filebeat is a really useful tool to send the content of your current log files to PaaS Logs. Combined with the filter in Logstash, it offers a clean and easy way to send your logs without changing the configuration of your software. There is a lot you can do with Logstash and Filebeat. Don't hesitate to check the links below to master these tools.
250+
251+
252+
- Configuration's details : [https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-configuration-details.html](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-configuration-details.html)
253+
- Getting started : [https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-getting-started.html](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-getting-started.html)
254+
- Grok Patterns Documentation : [https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html](https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html)
255+
- Current Grok Pattern reference : [https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns](https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns)
256+
- Even a logstash_forwarder to filebeat tutorial : [https://www.elastic.co/guide/en/beats/filebeat/current/migrating-from-logstash-forwarder.html](https://www.elastic.co/guide/en/beats/filebeat/current/migrating-from-logstash-forwarder.html)
257+
258+
259+
#Getting Help
260+
261+
- Getting Started : [Quick Start](/kb/en/logs/quick-start.html)
262+
- Documentation : [Guides](/kb/en/logs)
263+
- Mailing List : [paas.logs-subscribe@ml.ovh.net](mailto:paas.logs-subscribe@ml.ovh.net)
264+
- Visit our community: [community.runabove.com](https://community.runabove.com)
265+
- Create an account: [PaaS Logs Beta](https://cloud.runabove.com/signup/?launch=paas-logs)
266+
267+

‎images/2016-02-25-filebeat-logs/beats_graylog.png

110 KB
Loading

‎images/2016-02-25-filebeat-logs/filter_graylog.png

78.5 KB
Loading

0 commit comments

Comments
 (0)