Amazon EC2 Application Logs

This page describes how to install and configure the Log Collector to collect logs from applications running on non-Kubernetes® Linux hosts, such as Amazon Elastic Compute Cloud (EC2) and bare-metal Linux machines. Hosts must be running on Linux. The Log Collector sends logs from the host to the Common Ingestion Service (CIS) directly or through the Cisco AppDynamics Distribution of OpenTelemetry Collector.

For more information about onboarding logs using OpenTelemetry™, see Auto-Instrument Your Applications using OpenTelemetry Operator for Kubernetes.

Prerequisites

If you plan to connect the Log Collector to the Cisco AppDynamics Distribution of OpenTelemetry Collector, install Cisco AppDynamics Distribution of OpenTelemetry Collector on the host first. See Deploy the Cisco AppDynamics Distribution of OpenTelemetry Collector in Amazon EC2 (Linux).

Configuration Options

This service reads configurations from the following variables set in the /opt/appdynamics/appdynamics.conf file:

Name	Description	Required
`APPD_LOGCOL_BASE_URL`	The endpoint the Log Collector sends data to. Must have `https://` prefix.	Yes, if you deploy the Log Collector to send logs directly to the Common Ingestion Service; otherwise, No.
`APPD_LOGCOL_CLIENT_ID`	The client ID that the Log Collector uses to authenticate with the Common Ingestion Service.	Yes, if you deploy the Log Collector to send logs directly to the Common Ingestion Service; otherwise, No.
`APPD_LOGCOL_CLIENT_SECRET`	The client secret (in plaintext) that the Log Collector uses to authenticate with the Common Ingestion Service.	Yes, if you deploy the Log Collector to send logs directly to the Common Ingestion Service; otherwise, No.
`APPD_LOGCOL_TOKEN_URL`	The URL that the Log Collector retrieves Oauth2 tokens from. Must have `https://` prefix.	Yes, if you deploy the Log Collector to send logs directly to the Common Ingestion Service; otherwise, No.
`APPD_LOGCOL_SELF_LOGGING_LEVEL`	The logging level for the Log Collector's self-logging.	No
`APPD_LOGCOL_COLLECTORS_LOGGING_ENABLED`	Enables or disables collection self-logging from the Cisco Cloud Observability collectors running on the host (Cisco AppDynamics Distribution of OpenTelemetry Collector, Infrastructure Collector, and Log Collector).	No
`APPD_LOGCOL_SSL_ENABLED`	Enables or disables SSL communication on the export of both application logs and the Log Collector's self-logs and self-telemetry to the Cisco AppDynamics Distribution of OpenTelemetry Collector or to the Common Ingestion Service. Valid values: `true`, `false`. Default: `false`.	No
`APPD_LOGCOL_SSL_CERTIFICATE_AUTHORITIES`	List of your root CA certificates.	No
`APPD_LOGCOL_SSL_CERTIFICATE`	Full pathname of your certificate for SSL client authentication.	No
`APPD_LOGCOL_SSL_KEY`	Full pathname of your private client certificate SSL key.	No
`AGENT_MANAGEMENT_ENABLE`	Enables or disables Agent Management and observability of the Log Collector. Default: `false`.	No
`SERVICE_URL`	Agent Management service URL. Format: `https://<tenant-url>/rest/agent/service`	Yes, if Agent Management is enabled; otherwise, No
`OAUTH_ID`	The client ID that the Log Collector uses to authenticate with Agent Managment.	Yes, if Agent Management is enabled; otherwise, No
`OAUTH_URL`	The URL that the Log Collector retrieves Oauth2 tokens from for Agent Management. Must have `https://` prefix.	Yes, if Agent Management is enabled; otherwise, No
`OAUTH_SECRET`	The client secret (in plaintext) that the Log Collector uses to authenticate with Agent Management.	Yes, if Agent Management is enabled; otherwise, No
`OAUTH_TENANT_ID`	Your tenant ID, from the Account Management portal.	Yes, if Agent Management is enabled; otherwise, No
`AGENT_DEPLOYMENT_NAME`	Your own deployment name for this instance of the Log Collector. You use this value to target this instance in UQL queries.	Yes, if Agent Management is enabled; otherwise, No
`AGENT_PLATFORM_NAME` NEW	Full name of the EC2 instance that this Log Collector instance is running on. The Log Collector tries to set this variable automatically using a metadata query. If it can't, it sets this variable to the EC2 instance ID (the default value). In the rare cases where it can't retrieve the EC2 instance name, you can override the default value by explicitly setting this variable.	No

Example /opt/appdynamics/appdynamics.conf file:

APPD_LOGCOL_BASE_URL=<data-endpoint>
APPD_LOGCOL_CLIENT_ID=<client-id>
APPD_LOGCOL_CLIENT_SECRET=<secret>
APPD_LOGCOL_TOKEN_URL=<auth-endpoint>

CODE

The Log Collector uses a static configuration file that you don't need to modify: /opt/appdynamics/appdlogcol/filebeat.yaml.

Create the Configuration File

If you plan to configure the Log Collector to send logs directly to the Common Ingestion Service, specify the Log Collector's configuration options as follows:

On your host, do the following steps:

Create the file /opt/appdynamics/appdynamics.conf if it doesn't already exist:
```
mkdir -p /opt/appdynamics
touch /opt/appdynamics/appdynamics.conf
```
BASH

In /opt/appdynamics/appdynamics.conf, add your authentication credentials. The Log Collector uses these credentials to authenticate with the Common Ingestion Service:

cat > /opt/appdynamics/appdynamics.conf << EOF
APPD_LOGCOL_BASE_URL=<data-endpoint>
APPD_LOGCOL_CLIENT_ID=<client-id>
APPD_LOGCOL_CLIENT_SECRET=<secret>
APPD_LOGCOL_BASE_TOKEN_URL=<auth-endpoint>
EOF

BASH

Install the Log Collector on the Host

Before starting this step, you must create the /opt/appdynamics/appdynamics.conf file described in Create the Configuration File.

Centos/RHEL

Add the Cisco Cloud Observability repository:

sudo cat > /etc/yum.repos.d/artifactory.repo << EOF
[Artifactory]
name=Artifactory
baseurl=https://appdynamics.jfrog.io/artifactory/rpm-hosted/
enabled=1
gpgcheck=0

CODE

Refresh the repository:
```
sudo yum update
```
CODE
Install the Cisco AppDynamics Distribution of OpenTelemetry Collector:
```
sudo yum install appdlogcol
```
CODE
The installation should start the Cisco AppDynamics Distribution of OpenTelemetry Collector as a service named appdlogcol.service. To check the status, run the following command:
```
systemctl status appdlogcol.service
```
BASH

Debian

Add the Cisco Cloud Observability repository:

sudo sh -c "echo 'deb [trusted=yes] https://appdynamics.jfrog.io/artifactory/deb-hosted/ appdynamics main' >> /etc/apt/sources.list"

CODE

Refresh the repository:
```
sudo apt-get update
```
CODE
Install the Cisco AppDynamics Distribution of OpenTelemetry Collector:
```
sudo apt-get install appdlogcol
```
CODE
The installation should start the Cisco AppDynamics Distribution of OpenTelemetry Collector as a service named appdlogcol.service. To check the status, run the following command:
```
systemctl status appdlogcol.service
```
BASH

Configure Default Log Collection

By default, the Log Collector collects logs from subdirectories within /var/log. You only need to create a subdirectory within /var/log and configure your application to write logs to that subdirectory. No additional configuration is required.

To disable default log collection, rename the default logs configuration file, default_logs.yml, (present in /opt/appdynamics/appdlogcol/inputs.d) to default_logs.yml-disabled

Configure Custom Log Collection

If your application is not writing logs to a file in /var/log, you can configure your own input source so that the Log Collector can collect logs from there.

Known Limitation: If the logs are inside a user's home directory, the Log Collector cannot collect logs from there. Workaround: Pipe those logs either to a directory within /var/log (since this is the default collection point) or to any other location outside of the home directory and follow the steps below to configure a custom collection point.

In /opt/appdynamics/appdlogcol/inputs.d, add an input file named <filename>.yml with configuration to collect logs from your custom log source.
The new configuration is applied automatically.
Sample <filename>.yml:

- type: filestream
  # Unique ID among all inputs, an ID is required.
  id: filestream-id-log-gen-1

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /usr/local/etc/log-gen-1/logs/application.log
  parsers:
    - multiline:
        type: pattern
        pattern: '^\d{4}\-\d{2}\-\d{2}'
        negate: true
        match: after
  fields:
    service.name: log-gen-1
    messageParser:
      log4J:
        enabled: true
      	pattern: "%d{yyyy-MM-dd HH:mm:ss} %-5p %m%n"

CODE

Configure the service.name field.
Add suitable message parser and multiline pattern configurations for your logs. See Log Collector Settings.

To disable your custom log collection, rename the default logs configuration file, <filename>.yml, (present in /opt/appdynamics/appdlogcol/inputs.d) to <filename>.yml-disabled

Enable or Disable Self-Logging

You can enable or disable the collection of self-logging logs from Splunk AppDynamics collectors (namely, the Infra collector, Cisco AppDynamics Distribution of OpenTelemetry Collector, or the Log Collector). By default, collection of self-logging logs is disabled.

To enable:

Set APPD_LOGCOL_COLLECTORS_LOGGING_ENABLED to true in /opt/appdynamics/appdynamics.conf:
```
APPD_LOGCOL_COLLECTORS_LOGGING_ENABLED=true
```
CODE

Restart the appdlogcol service with the following commands:

sudo systemctl restart appdlogcol
sudo systemctl daemon-reload

BASH

Update the Configuration

You can update the Log Collector's configuration options as follows:

To update the input sources, you can add a new file (named filename.yml) to the /opt/appdynamics/appdlogcol/inputs.d folder, or edit an existing file. Restart is not required.
To update filebeat.yaml, restart is required after changes are made.
To update /opt/appdynamics/appdynamics.conf, restart is required after changes are made.

To restart, run these commands:

systemctl restart appdlogcol.service
sudo systemctl daemon-reload

BASH

View Diagnostics

View Logs

The Log Collector runs as a system service named appdlogcol.service. Logs from this service are written by syslog to /var/logs. To view the logs, run the following command:

journalctl -u appdlogcol.service

BASH

View Status

To check the status of the Log Collector service on the host, run this command:

systemctl status appdlogcol.service

CODE

Set Self-Logging Levels

Set the environment variable APPD_LOGCOL_SELF_LOGGING_LEVEL in /opt/appdynamics/appdynamics.conf file. Default: info.
For example:
```
APPD_LOGCOL_SELF_LOGGING_LEVEL=debug
```
CODE

Restart the Log Collector:

sudo systemctl restart appdlogcol
sudo systemctl daemon-reload

BASH

Find Logs on the Observe Page

Entity-centric pages related to this log collection don't show log messages right now. Workaround: navigate to Explore > Logs and search for log messages whose attributes match these values:

Attribute	Key	Value
Service Name	`service.name`	Value might be any of these: For default log collection, the value is the parent directory name of log file. For custom log collection, the value is as set in the custom log collection config `service.name` field. For logs from Cisco Cloud Observability collectors, the value is the collector service name. Example: `appdlogcol.service`
Service Instance ID	`service.instance.id`	EC2 instance ID

Log Collector Metrics from EC2

You can send Log Collector metrics from this EC2 instance to Cisco Cloud Observability. The Log Collector instance will not be visible on the Agent Management console, but you can find the metrics associated with it using UQL. To enable the Log Collector metrics, set environment variables in /opt/appdynamics/appdynamics.conf:

AGENT_MANAGEMENT_ENABLE=true
SERVICE_URL=https://<tenant-url>/rest/agent/service
OAUTH_ID=<client-id>
OAUTH_URL=https://<tenant-url>/auth/<tenant-id>/default/oauth2/token
OAUTH_SECRET=<client-secret>
OAUTH_TENANT_ID=<tenant-id>
AGENT_DEPLOYMENT_NAME=<custom-deployment-name>

For a description of Log Collector metrics, see Log Collector Observability.

This document contains references to third-party documentation. Splunk AppDynamics does not own any rights and assumes no responsibility for the accuracy or completeness of such third-party documentation.

OpenTelemetry™ and Kubernetes® (as applicable) are trademarks of The Linux Foundation®.