Using Kibana detections to trigger slack alerts
In Kibana verion 7.6 a new feature of the SIEM app was released (in beta). This feature, called “detections”, allows the user to generate an alert for each result of a given search.
How these alerts work
These “signal detection rule” searches run on a schedule, and any results of the search will create a new alert in a special Kibana detections index. Helpfully, there are a number of built-in alerts based on the Mitre ATT&CK framework that are available for you to enable right off the bat. You can also build your own alerts simply by building a search and setting the interval at which it should run (defaults to 5 minutes). Because the timestamp of logs may be slightly off, it also has the option to extend the timeframe for the search that it runs (defaults to 1 minute). Because of this, a default search would run every 5 minutes, and look at the logs for the past 6 minutes when it runs.
Throwing alerts to slack
Since I don’t want to be eternally glued to the alerts dashboard in Kibana, I build a quick and dirty python program that is able to query the alert index on Elasticsearch and send a message to a designated slack channel.
Since the timing of when the alerts run vs. when the python program runs could end up missing some alerts if not set up correctly, I’ve added the ability for the program to keep track of the last alerts it sent, and it will not send those alerts a 2nd time. This way I can have some overlap in the timing of when an alert is created and when the program is querying for alerts.
Limitations of the Kibana search functionality
Because of the limited functionality of Kibana search, and the fact that every single result of a search is returned as a separate alert, you are severly limited in the complexity of the alerts you can build. For instance, if you wanted to build an alert that fires when there are more than 10 failed logins from the same user in the last 5 minutes, this type of alert is not possible with the current functionality of the Kibana Query Language (KQL). To be able to make these complex alerts, you would want some functions that are seen in Splunk such as sum, count, evals, and subsearches. An example Splunk search for this alert is shown below:
{search for failed logins} | count by user | search count > 10
Workarounds for these limitations
The current workaround that I’m building into my python program would allow some basic rate limiting of alerts, to ensure that I don’t get 10 slack messages at once that tell me a website is down. Because the amount of logs I work with is pretty small, I can have an alert that is thrown every time theres a failed login attempt and it won’t cause any issues. For a larger organization this would not be feasable since it would likely would overwhelm you with alerts of people fat-fingering their password.
In order to limit the amount of custom checks that I have to build into my code, I am also finely tuning the alerts that I enable. I want to ensure that we are only alerted when we actually need to check on something, and not every time a server has an error logged to syslog.
Why dont I just use elastalert?
The main reason I don’t use elastalert is due to how user-unfriendly it is. To add alerts you must SSH into the server and edit some config files to add a new alert, this isn’t really accessible for the average user. I have looked at the elastalert Kibana plugin that someone created, but the development pace of Kibana is far faster than that of the plugin. At the time of writing this, the last commit to the master branch of the plugin was 9 months ago. This means that to use this plugin I would need to be running a far older version of Kibana.
I wanted the ability to add new alerts to be easy and intuitive for new people to learn, and the new detections feature allows just that. All I need to have to add a new alert is just a normal search, meaning that anyone who knows how to search in Kibana can just as easily make an alert.
Code release
I currently don’t have my Github repo for this program set to public, but I am thinking about releasing it for other people to use and modify for their own needs. When I decide to release it, I will post a link to it here.
If you have any questions about this post you can contact me by email (my email address is in the sidebar).
Note for anyone wanting to set up the detections feature in Kibana:
You will first need to enable https on your Elasticsearch cluster and use https to connect Kibana to Elasticsearch. You will then need to enable API keys, you should then be able to access the detections feature.