Here at xforty technologies we’ve been using a log aggregation toolkit called ELK. This toolkit is a collection of open source software packages that collect logs and parse them into structured snippets (logstash), store them in a searchable index (elasticsearch), and allow for the searching and dashboard reporting (kibana). Making the setup work well takes some effort, but the results are incredible. We stood up an ELK system and have been running it for about a year.
What can we do with it?
Search
Through my twenty or so years doing software development and systems support I’ve made use of the linux command line tools to parse logs and produce digestible information. Ad-hoc use of grep, sed, awk, wc were the go-to. Maintaining a sharp command of these tools requires regular use. Go 6 months without solving a log parsing problem you’ll be spending some time rooting through cheat-sheets and manual pages.
ELK, using the kibana front end can easily search through log data. Using an easy to digest query format you can quickly find what you’re looking for. We used this very basic Kibana feature for probably six months without diving into any of the more advanced features. Our ELK setup provides very quick searching of our data. Searches usually return sub-second. We’ve been able to support users, troubleshoot problems, identify misbehaving hosted applications as well as identify nefarious scans all with the simple searching interface. Kibana can do far more than simple searching, however.
Simple Visualization
After squashing a performance issue, I decided to tinker with the visualization feature to try to answer some questions I have had like:
What are the IP addresses most dropped from our firewalls?
Over the past seven days, the top five blocked IP addresses all clocked in at over 11,000 blocked connection attempts, the leader 62.138.2.123 with over 27,000 connection attempts blocked.
What are the most commonly scanned tcp/udp ports?
We figured the best indicator of this was to sum up the number of drops per tcp/udp port. Okay, this one is pretty interesting. The most commonly scanned port seems to be telnet, port 23. Port 5060, Session Initiation Protocol is up there on the list as well.
We were all pretty surprised to see that port 23, telnet is the clear leader. The only times we see telnet behind another port number is when our time window is short (say 15 minutes) and there’s scan of another port happening.
NOTE: Click on the image to blow it up.
A Note on Scans
There are scans and there are big scans. Run of the mill scans are happening all the time. A few times an hour it seems. These scans are not particularly aggressive or intrusive. We see clients run a sweep of our networks looking for this service or that. The most popular surprisingly seems to be telnet. Lagging far behind telnet are session initiation protocol (SIP), secure shell (ssh),
Every now and then we’ll see a monster scan, a leave no stone unturned level scan. Every IP/ port combination is exhaustively tapped asking the question “is this thing on?”. One such scan asked this very question 2.87 million times over the source of about fifteen hours. Another knocked on the door 821K times in one 24 hour period.
It’s a jungle out there.
Conclusion
If you’ve got log retention or analysis needs either for IT best practices regulation and compliance requirements the good folks over at elastic have got a good solution for you. At xforty we were able to improve our ability to support customers, understand the makeup of our network traffic and identify problems with our firewall policy.