Logstash to Logstash log forwarding using the Lumberjack output plugin

Written by Claudio Kuenzler - 5 comments

Published on May 28th 2020 - Listed in ELK Elasticsearch Filebeat Logstash

---

In a usual ELK setup, one would run application servers logging to a central Logstash server and this Logstash writes log events into an Elasticsearch index.

But sometimes it might be required to forward logs from a central Logstash to another Logstash server. A practical example: Your applications run on different premises or cloud providers but you still want to have all the logs in a central place. For such a scenario a "local" Logstash server, which is used as a forwarder to the central Logstash server, can be used. The question is: Which output plugin should be used to forward the logs?

Once upon a time: logstash-forwarder

A couple of years ago, the logstash-forwarder project was created to solve exactly this: Pick up logs and forward them to one or more Logstash servers "listening for our messages".

However logstash-forwarder was replaced by Filebeat:

The filebeat project replaces logstash-forwarder. Please use that instead.

Interestingly this project was renamed from "lumberjack" to "logstash-forwarder" as can be read in the README:

This project was recently renamed from 'lumberjack' to 'logstash-forwarder' to make its intended use clear. The 'lumberjack' name now remains as the network protocol, and 'logstash-forwarder' is the name of the program.

We'll come back to the lumberjack name later again...

Filebeat output?

So if Filebeat should be used instead, is there an actual logstash-output-filebeat plugin available? No, there isn't (as of this writing). The idea behind using Filebeat as a forwarder to another Logstash server would be for the local Logstash to write local logs into files. These files are then picked up by Filebeat and sent to the central Logstash server.

Sounds messy? It is. Not only will you lose time by writing disks to log (by Logstash) and reading them again (by Filebeat), it also means to install a second daemon (Filebeat) which uses resources on that local Logstash server. 

It would surely work, but do we really want to involve another daemon? Nope.

Finding the right Logstash output plugin

The challenge now is to find the right Logstash output plugin. Sure, there are the basic tcp and udp output plugins, which simply connect to a Logstash listener on the given port - but is that the right choice?

Going through the list of Logstash output plugins, one name catches the eye: lumberjack.

Wait, wasn't lumberjack the original name of the logstash-forwarder? A quick read in the lumberjack output plugin documentation doesn't really show a use case for this output, but it's certainly worth a try.

Using lumberjack as log forwarder

The lumberjack output basically uses the lumberjack protocol, which was picked up and replaced by Filebeat (as mentioned above). This means: Our local Logstash can use lumberjack as output and our central Logstash can use the beats input to receive these log events. 

In order to use the lumberjack output, SSL certificate exchange between the local and the central Logstash servers must happen. The ssl_certificate parameter is required, it won't work without it. The SSL key and certificate need to be configured on the central (receiving) Logstash server and configured in the beats intput definition. On the local (sending) Logstash, it's enough to use the same SSL certificate.

To create a very basic SSL key and certificate on the central Logstash server (make sure to set CN to your central Logstash FQDN):

root@central:~# openssl req -x509 -batch -nodes -newkey rsa:2048 -keyout lumberjack.key -out lumberjack.cert -subj /CN=logstash.example.com

This command creates two files:

• lumberjack.key: The SSL key
• lumberjack.cert: The SSL certificate (needs to be copied to the local Logstash, too)
  

Add a comment

Show form to leave a comment

Comments (newest first)

ck from Switzerland wrote on Oct 27th, 2020:

Cliff, you could export your local hostname as an environment variable and then use this in Logstash mutate filter to add a field "logstash_forwarder". First create an environment variable:

ckadm@mintp ~ $ export myhost=$(hostname)
ckadm@mintp ~ $ env|grep myhost
myhost=mintp

Then add the field in Logstash using the mutate filter:

filter {
mutate {
add_field => { "logstash_forwarder" => "${myhost}" }
}
}

See Using Environment Variables in the Configuration.

---

Cliff from Australia wrote on Oct 27th, 2020:

Any idea how to get the details (ip/hostname) of the forwarding logstash instance added to the lumberjack output?

I know I can use a mutate filter to add fields, but for the life of me I cannot find any fields/variables that contain the details of the forwarding logstash instance.

---

ck from Switzerland wrote on Aug 5th, 2020:

Hi Yan. Yes, I would even recommend using certificates from your own CA. I just created self-signed certificates here as an example. Something else which should be mentioned: The certificate created in the article was only valid for 30 days (by default) - we ran into communication issues using the Lumberjack output once the certificate expired. We then replaced it with a 10y certificate.

---

Yan from wrote on Aug 4th, 2020:

Hey thanks for the useful post.

Just wondering. Do you know if I can use certificates from my CA instead of the selfsigned? I actually tried to replace them but having certificate related errors.

---

Jamsux from Salvador / Bahia / Brazil wrote on Jul 17th, 2020:

Hello!
I very Liked this Post. Very good Written.
Congratulations man!

---