The logging ecosystem or a logging infrastructure is the set of all components and parts that work together to generate, filter, normalize, and store log messages. The purpose of this logging system is to use logs for solving particular problems. For example, the logs can help to find out the source of the attack. This article defines each component of logging ecosystem and illustrates how they work.
What Do I Need to Know About Log Message Generation?
A log message is generated in response to some sort of stimuli that depends on the source of that log message. For instance, the disk storage system generates log messages when failures occur, firewalls will have ACL accept-and-deny messages, and Unix systems will have user’s login and logout messages. Logs can be generated by various sources including:
- Windows and Unix operating systems
- Antivirus programs
- Virtual Private Network (VPN) server
- Wireless Access Points (WAPs)
To enable logging on your devices, you need to follow some basic steps that are listed below.
- Turn on logging on your device. Each device can have a different
- Configure your device so that it can send log messages.
- Configure that device to send “loghost” for further analysis and collection.
Log Messages Filtering
Log messages filtering and normalization occur once a computer system or device is configured properly to generate log messages. Log message filtering is the act of either including or excluding log messages based on the contents in the log message. Some devices/sources offer this feature natively or you can employ an additional agent that will intercept log messages and filter them in accordance with the user-defined rules. However, filtering depends on the needs of your enterprise. For instance, if you are a system administrator and has a responsibility to ensure the smooth running of your enterprise’s critical systems, you need to check logs to make sure that everything is working as expected.
Log Message Normalization
Log message normalization is the process of converting the disparately formatted log messages into a common format. The normalized log message is called an event, which is a currency of the logging system. The event is stored in the relational database for reporting and analysis purposes. Besides, normalized data can easily be manipulated and understandable.
Log Message Collection
Organizations deploy a log server to collect all log messages generated by a variety of systems and network devices. A single log server can serve the small organization (See Figure 1 below). If your organization has numerous branches spread in different countries, then you must have a distributed set of log servers.
Logging in the Cloud
Logging in the cloud or cloud logging is the vital application of cloud computing. Rather, it has been emerged as the latest cloud deployment model—namely, Logging as a Service (LaaS). Various providers today use LaaS to deliver logging services in a cloud ecosystem. For example, the cloud service provider assigns you a hostname or IP address to pinpoint your Syslog configuration.
Log analysis is performed to analyze log data to make it meaningful. You need to put all your data at a single place or correlate log messages. In a distributed system, you can have a wide range of log collectors and you should correlate log messages received at one collector with those arrived at another collector.
As a result, it has been realized that the logging ecosystem is indispensable for pinpointing problems. Logs are used to detect threats as well as utilized for forensic purposes. The logging ecosystem involves several components including generation, filtering, normalization, and analysis of log data. Understanding of each component is necessary to use the logging system effectively. You also need to understand Log Management to further understand this subject.