Which companies use Logstash

Find out details about individual Logstash filters

Hopefully everyone who has tried it will know that Logstash is very high-performance. However, there are some tricks that can be used to write the configuration in such a way that events are processed even faster. So far, the only problem was to find out whether the tuning worked or just made things worse. Since version 5 Logstash has got its own API, which makes it possible to get a lot of information about the Logstash service. This can be reached by default on port 9600 of the Logstash host. (Depending on the version, it may have to be activated in.)

# curl localhost: 9600 / _node / stats? pretty ... "process": {"open_file_descriptors": 79, "peak_open_file_descriptors": 80, "max_file_descriptors": 16384, "mem": {"total_virtual_in_bytes": 3390824448}, " cpu ": {" total_in_millis ": 176030000000," percent ": 3," load_average ": {" 1m ": 0.0," 5m ": 0.02," 15m ": 0.11}} ...

Newer versions of Logstash 5 even provide details about individual filters. The only problem with displaying the performance details is that Logstash gives each filter a random ID and an assignment is not possible. However, if you follow the Logstash Issues on Github, you will see that there is definitely a way to set this ID - it just hasn't been documented yet.
The following configuration gives the specified filter an ID that is not stored in Elasticsearch and therefore cannot be seen in Kibana. However, it is very visible via the Logstash API.

filter {if [program] == "kibana" {json {id => "kibana-json" source => "message" target => "kibana"}}}

The API then delivers the corresponding output.

{"id": "kibana-json", "events": {"duration_in_millis": 908, "in": 394, "out": 394}, "name": "json"}

If you don't want to use input so that Logstash regularly requests its own API, you can also use check_logstash, which has already made it into the Icinga 2 ITL as a check command. I would be happy to receive feedback on both the plugin and the integration. And at this point, thank you again to those who have already contributed. First and foremost Jordan Sissel.
Vagrant boxes that already contain the configuration shown are also available on Github. They may not have progressed as far as I would like, but they can certainly serve as a basis for your own experiments. Here, too, I look forward to your feedback.
Anyone who would like to learn more about Logstash and the Elastic Stack should register for one of our training courses on the subject. However, if you have not yet heard of Vagrant, you will find it in another one of our training courses.

Thomas was a system administrator at an Austrian university and was particularly responsible for Linux and Unix. But since 2013 he has preferred to see the big wide world and has therefore joined the NETWAYS Consulting team. He also wants to spread as far as possible how and how easily personal communication can be securely encrypted, so that there is no constant complaint about the lack of data protection, but something that is finally done about it. In the meantime he has become a logstash guy at NETWAYS and keeps ...