Header RSS Feed
 
If you only want to see the articles of a certain category, please click on the desired category below:
ALL Android Backup BSD Database Hacks Hardware Internet Linux Mail MySQL Monitoring Network Personal PHP Proxy Shell Solaris Unix Virtualization VMware Windows Wyse

Install and configure Elastic Filebeat through Ansible
Friday - Aug 11th 2017 - by - (0 comments)

Whilst I'm currently building an ELK stack for centralized logging and visualizing these logs, I came also across Filebeat. Filebeat is basically a log parser and shipper and runs as a daemon on the client. 

So far the first tests using Nginx access logs were quite successful.

Now I wanted to go one step further and automatically deploy Filebeat through an Ansible playbook. This playbook should also be used to automatically configure the "logs to be followed", called "prospectors" in Filebeat terminology.

Well, the following playbook does it. With the current code, it checks if there is Nginx and/or HAProxy installed on the target machine and automatically configures the prospectors and of course also the output (a Logstash receiver in my setup):

$ cat /srv/ansible/playbooks/filebeat/filebeat.yml
- name: ANSIBLE - Filebeat installation and configuration by www.claudiokuenzler.com
  hosts: '{{ target }}'
  roles:
    - yaegashi.blockinfile

  sudo: yes

  tasks:

  - name: APT - Add elastic.co key
    apt_key: url="https://artifacts.elastic.co/GPG-KEY-elasticsearch"
    when: ansible_distribution == "Ubuntu"

  - name: APT - Add elastic.co repository
    apt_repository: repo="deb https://artifacts.elastic.co/packages/5.x/apt stable main" filename="elastic-5.x" update_cache=yes
    when: ansible_distribution == "Ubuntu"

  - name: FILEBEAT - Install Filebeat
    apt: pkg=filebeat
    when: ansible_distribution == "Ubuntu"
 
  - name: FILEBEAT - Copy base filebeat config file
    copy: src=/srv/ansible/setup-files/filebeat/filebeat.yml dest=/etc/filebeat/filebeat.yml

  - name: FILEBEAT - Set shipper name
    lineinfile: "dest=/etc/filebeat/filebeat.yml state=present regexp='^name:' line='name: {{ ansible_hostname }}' insertafter='# Shipper Name'"
 
  - name: FILEBEAT - Configure Logstash output
    blockinfile:
      dest: /etc/filebeat/filebeat.yml
      insertafter: '# Logstash output'
      marker: "# {mark} -- Logstash output configured by Ansible"
      block: |
        output.logstash:
          hosts: ["logstashreceiver.example.com:5044"]


  - name: FILEBEAT - Check if Nginx is installed
    command: dpkg -l nginx
    register: nginxinstalled

  - name: FILEBEAT - Configure Nginx Logging
    blockinfile:
      dest: /etc/filebeat/filebeat.yml
      insertafter: 'filebeat.prospectors:'
      marker: "# {mark} -- Nginx logging configured by Ansible"
      block: |
        - input_type: log
          paths:
            - /var/log/nginx/*.log
          document_type: nginx-access
    when: nginxinstalled.rc == 0

  - name: FILEBEAT - Check if HAProxy is installed
    command: dpkg -l haproxy
    register: haproxyinstalled

  - name: FILEBEAT - Configure HAProxy Logging
    blockinfile:
      dest: /etc/filebeat/filebeat.yml
      insertafter: 'filebeat.prospectors:'
      marker: "# {mark} -- HAProxy logging configured by Ansible"
      block: |
        - input_type: log
          paths:
            - /var/log/haproxy.log
          document_type: haproxy
    when: haproxyinstalled.rc == 0

  - name: FILEBEAT - Restart filebeat
    service: name=filebeat state=restarted

Of course this only works when the correct "template" is used (see "FILEBEAT - Copy base filebeat config file"). This is a minimal config file prepared with certain lines:

$ cat /srv/ansible/setup-files/filebeat/filebeat.yml
#=========================== Filebeat prospectors =============================

filebeat.prospectors:

#================================ General =====================================

# Shipper Name

#================================ Outputs =====================================

# Logstash output

#================================ Logging =====================================

 I let the playbook run on a test machine (which has Nginx and HAProxy installed):

ansible-playbook playbooks/filebeat.yaml --extra-vars "target=testmachine"
[DEPRECATION WARNING]: Instead of sudo/sudo_user, use become/become_user and make sure become_method is 'sudo'
(default).
This feature will be removed in a future release. Deprecation warnings can be disabled by setting
deprecation_warnings=False in ansible.cfg.

PLAY [ANSIBLE - Filebeat installation and configuration] ***********************

TASK [setup] *******************************************************************
ok: [testmachine]

TASK [APT - Add elastic.co key] ************************************************
ok: [testmachine]

TASK [APT - Add elastic.co repository] *****************************************
ok: [testmachine]

TASK [FILEBEAT - Install Filebeat] *********************************************
ok: [testmachine]

TASK [FILEBEAT - Copy base filebeat config file] *******************************
changed: [testmachine]

TASK [FILEBEAT - Set shipper name] *********************************************
changed: [testmachine]

TASK [FILEBEAT - Configure Logstash output] ****************
skipping: [testmachine]

TASK [FILEBEAT - Check if Nginx is installed] **********************************
changed: [testmachine]

TASK [FILEBEAT - Configure Nginx Logging] **************************************
changed: [testmachine]

TASK [FILEBEAT - Check if HAProxy is installed] ********************************
changed: [testmachine]

TASK [FILEBEAT - Configure HAProxy Logging] ************************************
changed: [testmachine]

TASK [FILEBEAT - Restart filebeat] *********************************************
changed: [testmachine]

PLAY RECAP *********************************************************************
testmachine     : ok=13   changed=8    unreachable=0    failed=0  

On the testmachine itself, the Filebeat config was correctly set:

root@testmachine:~# cat /etc/filebeat/filebeat.yml
#=========================== Filebeat prospectors =============================

filebeat.prospectors:
# BEGIN -- HAProxy logging configured by Ansible
- input_type: log
  paths:
    - /var/log/haproxy.log
  document_type: haproxy
# END -- HAProxy logging configured by Ansible
# BEGIN -- Nginx logging configured by Ansible
- input_type: log
  paths:
    - /var/log/nginx/*.log
  document_type: nginx-access
# END -- Nginx logging configured by Ansible

#================================ General =====================================

# Shipper Name
name: testmachine

#================================ Outputs =====================================

# Logstash output
# BEGIN -- Logstash output configured by Ansible
output.logstash:
  hosts: ["logstashreceiver.example.com:5044"]
# END -- Logstash output configured by Ansible

#================================ Logging =====================================

 Now that Logstash received these logs and added them into ElasticSearch, I am able to see them in Kibana:

Filebeat logs in Kibana


 

Add a comment

Show form to leave a comment

Comments (newest first):

No comments yet.

Go to Homepage home
Linux Howtos how to's
Monitoring Plugins monitoring plugins
Links links

Valid HTML 4.01 Transitional
Valid CSS!
[Valid RSS]

7393 Days
until Death of Computers
Why?