An introduction to the basics of log analysis, including what exactly it is, what its applications are and how you can do it yourself with tools and techniques.
Log analysis is a branch of data analysis that involves drawing insights from log files. It's a staple in the IT industry, where almost every product and service generates massive logs for a variety of processes. Although it may sound complicated, it's surprisingly straightforward if you know the basics.
In this article, we'll introduce you to the basics of log analysis, including what exactly it is, what its applications are, and how you can do it yourself using a variety of tools and techniques.
What Is Log Analysis?
Log analysis is the process of interpreting computer-generated records called logs. Logs can contain a variety of information about how a digital product or service is used, so the applications of log analysis are endless.
Examples of logs might include:
- Sign-in and sign-out requests on a website
- Transactions made on a currency exchange
- Calls made to an informational API
- Various other industry-specific actions
Applications
Logs are often used for monitoring, auditing, or debugging purposes. As a result, the applications of log analysis will usually fall under one of these categories. Let's review all three of them in more depth.
Monitoring
Logs can be used to monitor the usage of a product or service, often for security reasons. For example, consider a private web service that allows partners of a company to make requests to their internal databases. By analyzing the logs which are created when making these requests, the company itself can identify malicious usage patterns.
Auditing
Logs may also be used for auditing purposes, as is especially common in the financial industry. Here, consider a regulated currency exchange that allows users to trade between currencies. If a regulator suspects that the exchange has been incorrectly processing trades, it may request access to the exchange's logs, so as to see the transaction history.
Debugging
Logs are especially popular for debugging computer processes, which is one of the reasons they are so frequently used. If a programmer observes that a product or service is malfunctioning, he or she can refer to the appropriate logs to find reasons why that may be the case.
When to Do Log Analysis
When you choose to do log analysis will depend on why you are doing it. If monitoring is the main reason you are creating and analyzing logs, you may wish to perform log analysis on an ongoing basis, so as to identify malicious usage patterns as they happen. On the other hand, log analysis for auditing or debugging purposes may only be necessary sporadically — such as if a complaint is made or unexpected behavior occurs.
How to Do Log Analysis
It's remarkably easy to get started with log analysis, thanks to the plethora of tools and techniques available to the general public. Below, we'll look at some of these tools and techniques to give you an idea of what this type of analysis involves.
Techniques
Like with any type of data analysis, your success with log analysis ultimately comes down to the techniques you use to interpret data. In log analysis, five common techniques (also known as processes) are normalization, pattern recognition, classification and tagging, correlation analysis, and artificial ignorance.
Normalization
Normalization is the process of cleaning logs so that they adhere to the same standards or formats. For example, if logs from various sources contain varying datetime formats, they should be normalized before proceeding.
Pattern recognition
Pattern recognition is the process of identifying patterns in logs, so that individual log entries can be handled appropriately. For example, consider the logs collected by an ecommerce platform. Log entries that refer to users signing in should be separated from log entries that refer to users signing out.
Classification and tagging
Classification and tagging is another process that involves categorizing individual log entries. In this case, log entries should be further classified; for example, based on keywords that may be present in the entries themselves.
Correlation analysis
Correlation analysis is the process of finding log entries that are correlated. This may refer to identifying which entries pertain to a specific event, or identifying which entries (pertaining to separate events) are correlated. As in other types of data analysis, identifying correlations is an essential step in drawing meaningful conclusions from logs.
Artificial ignorance
Artificial ignorance is the process of "ignoring" entries which are not useful for analysis. In web-based applications, artificial ignorance may be used to identify which logs relate to intended usage patterns. With the help of artificial ignorance, it is possible to significantly reduce the number of logs which must be analyzed, which can speed up automatic analysis processes or even make manual analysis a possibility.
Tools
There are plenty of log analysis tools on the market, which allow you to quickly and easily import, normalize, and process data. In terms of paid solutions, some of the most popular are:
- Splunk: This free and paid platform aides in all areas of data analysis, including log analysis.
- Retrace: This popular SaaS solution takes your logs and finds ways to improve app performance.
- Sumologic: This dedicated log management tool is purpose-built for cloud applications.
As for open-source solutions, some notable choices are:
- Graylog: This open-core solution is — once again — a dedicated log management tool.
- GoAccess: This free offering helps to both visualize and analyze logs.
- Logz.io: This free offering targets those with cloud-based products.
All in all, there are plenty of tools out there. The tool you ultimately choose for log analysis will likely depend on factors like your budget and technology stack, as many of these solutions offer similar functionality.
Final Thoughts
Log analysis is an extremely valuable skill for tech companies that collect plenty of logs. Its applications are almost endless: allowing analysts to monitor, audit, and debug their offerings. While there are a variety of techniques that may be used for log analysis — including normalization, pattern recognition, and correlation analysis — there are dozens of tools on the market (both free and paid!) ready to help.
