The Analytics Agent can be configured to receive syslog messages using TCP transport and standard syslog format adhering to RFC 5424. The Analytics Agent can listen on a TCP port for syslog messages coming from a remote source or from the same host on which the Analytics Agent is present.
There are two steps to set this up:
- Configure your web server, such as Apache, to send logs to the Analytics Agent. There are two ways to direct the syslog data to the Analytics Agent:
- Write the log messages to the usual log files and then read and forward to the agent from the log file. See Read syslog Messages From the Log File and Send to Analytics Agent
- Send the log data directly to the Analytics Agent without writing to a file first. See Send syslog Data Directly to the Analytics Agent.
- Configure the Analytics Agent to receive and parse the logs. There are two ways to configure syslog message collection for Log Analytics:
Supported Environment Details
- Linux only.
- Network protocol is TCP only
- A template for Apache Web Server syslog format is shipped with the Analytics Agent. You can bring any log data in through syslog over TCP as long as you provide the correct configuration to parse the log message.
- One job file per Analytics Agent can be enabled to receive syslog messages over TCP.
Both the syslog utility and analytics-agent should have root access if the port where the analytics-agent is listening for syslog messages is lower than 1024.
Send syslog Data Directly to the Analytics Agent
You can use the Linux Logger utility to direct your Apache access and error log messages to the syslog daemon directly. Using this method, the logs are not written to the usual access log and error log apache files or to the
/var/log/message file. Instead, the log lines are sent directly to a syslog daemon which then forwards the data to the analytics-agent.
Configure the following:
- Configure Apache to delegate logs to
- Locate and open
httpd.conf, the Apache configuration file. This is typically located at
Add a new CustomLog directive to send access logs and error logs to the syslog and comment out the line that sends to the access_logs.
This directive uses the logger utility to send messages with facility local6, tag 'httpd' and log format combined.
The facility code specifies the type of program that is logging the message. Messages with different facilities may be handled differently.
The tag 'httpd' in the httpd.conf directly relates to the programname in the rsyslog.conf (see next step) for filtering out which messages need to be sent. For example, there can be other programs writing to that particular port, but we only want to send the logs from the httpd program to analytics-agent.
- Locate and open
- Configure the rsyslog client.
rsyslog.conf, typically located at
Add the following lines above "RULES" or "var/log/messages" filter so Apache doesn't log to the
Make sure you replace 514 with the port where the analytics-agent is listening for syslog messages. This must match the port specified in the job file.
Restart apache and rsyslog, and look at /var/log/messages for any rsyslog errors.
Read syslog Messages From the Log File and Send to Analytics Agent
In this case, the Apache server writes logs to the usual access log and error log files, and the rsyslog daemon is configured to read from these log files and forward the log data to analytics agent. This method preserves the original log files. In this case, the access and the error log messages are logged in the
In this example, the rsyslog client is configure to read from a specific file and forward the message with facility local6 and severity level info over the specified port (default port is 514).
- Locate and edit rsyslog.conf, typically located at
In the begin forwarding section of the rsyslog.conf file, add the following lines:
$InputFileName: path to the log file you want to tail.
local6.info: Use the Analytics Agent IP address if your analytics agent is not local to the controller. If needed, replace 514 with the port where the analytics-agent is listening for syslog messages. This must match the port specified in the job file or the source rule.
Restart rsyslog and review /var/log/messages for any rsyslog errors.
Configure Log Analytics TCP Source Rule
Using the Centralized Log Management UI, you can configure a source rule to extract log analytics fields from syslog messages over TCP.
- Access the Centralized Log Management UI from your Controller by clicking Analytics > Configuration > Log Analytics.
- On the Source Rules tab, click +Add.
- In the Add Source Rule panel, select the Create from source template and select From Network Connection as the collection type. For example, select the default Apache syslog template
Several log format templates are shipped with the Analytics Agent. You can create a new source rule for any log format over syslog TCP as long as you configure it correctly.
- Click Next to see the Add Source Configuration wizard.
- Specify the collection details, such as the name of the source rule, source type and enter the TCP Port where the Analytics Agent is listening.
When you specify From Network Connection as the collection type, the grok pattern for the syslog header (which is appended to the log messages) is automatically added at the beginning of the grok Message Pattern:
Confirm that the value for Multiline Format is None.
Configure field extraction and field management as for any other source rule. See Configure Log Analytics Using Source Rules for more details.
Configure Log Analytics TCP Job File
When selecting and configuring the port where the analytics agent will listen for the syslog data, make sure it does not conflict with anything else active in network. If no port number is provided, port 514 is used. Both the syslog utility and analytics-agent should have root access to send logs to port 514 (binding to ports less than 1024 requires root access).
To allow the analytics-agent to listen at a port, specify the log file source property and associated parameters for type=syslog. For example, add the following to the appropriate job file:
A job file for apache commons is included in the analytics distribution at
Your job file should look similar to the following: