Amazon ECS-on-EC2 Application Logs

This page explains how to collect application logs from Amazon Elastic Container Service (Amazon ECS) backed by EC2.

Prerequisites

An Amazon ECS cluster with Amazon EC2 infrastructure, not AWS Fargate (serverless) infrastructure
The Amazon EC2 instances joined to the Amazon ECS cluster have:
- Linux OS (currently only Linux installation is supported)
- X86_64 or amd64 architecture

Configuration Options

The Log Collector gets its configuration from the following environment variables in the task definition file:

Name	Description	Required
`APPD_LOGCOL_BASE_URL`	Defines the endpoint the collector sends data to.	Yes, if you deploy the Log Collector to send logs directly to the Common Ingestion Service; otherwise, No.
`APPD_LOGCOL_CLIENT_ID`	Defines the client ID for authenticating to the Cisco Cloud Observability Common Ingestion Service.	Yes, if you deploy the Log Collector to send logs directly to the Common Ingestion Service; otherwise, No.
`APPD_LOGCOL_CLIENT_SECRET`	Defines the secret string in plaintext for authenticating to the Cisco Cloud Observability Common Ingestion Service.	Yes, if you deploy the Log Collector to send logs directly to the Common Ingestion Service; otherwise, No.
`APPD_LOGCOL_TOKEN_URL`	Defines the URL the collector retrieves Oauth2 tokens from.	Yes, if you deploy the Log Collector to send logs directly to the Common Ingestion Service; otherwise, No.
`APPD_LOGCOL_SELF_LOGGING_LEVEL`	Defines the logging level for the Log Collector's self-logging. Default: `info`.	No
`APPD_LOGCOL_SSL_ENABLED`	Enables or disables SSL communication on the export of application logs and the Log Collector's self-logs and self-telemetry to the Cisco AppDynamics Distribution of OpenTelemetry Collector or Common Ingestion Service. Valid values: `true`, `false`. Default: `false`.	No
`APPD_LOGCOL_SSL_CERTIFICATE_AUTHORITIES`	List of your root CA certificates.	No
`APPD_LOGCOL_SSL_CERTIFICATE`	Full pathname of your certificate for SSL client authentication.	No
`APPD_LOGCOL_SSL_KEY`	Full pathname of your private client certificate SSL key.	No

Install the Log Collector on the ECS Cluster

On the AWS console, go to the Amazon ECS dashboard and navigate to Task definitions.
Click Create new task definition > Create new task definition with JSON.

Create a task definition with following content, replacing all placeholders with your values:

{
    "family": "<your-log-collector-task-definition-name>",
    "containerDefinitions": [
        {
            "name": "lca",
            "image": "appdynamics/appdynamics-cloud-log-collector-agent:24.1.0-970",
            "essential": true,
            "entryPoint": [
                "appd-filebeat-executor"
            ],
            "command": [
                "-f=/usr/share/filebeat/filebeat",
                "-b=/usr/share/filebeat/appd-filebeat-ecs.yml",
                "-a=-c,/usr/share/filebeat/appd-filebeat-ecs.yml,-strict.perms=false,-e"
            ],
            "environment": [
                {
                    "name": "APPD_LOGCOL_CLIENT_SECRET",
                    "value": "<your-client-secret>"
                },
                {
                    "name": "APPD_LOGCOL_BASE_URL",
                    "value": "<your-appd-base-url-without-trailing-slash>"
                },
                {
                    "name": "APPD_LOGCOL_CLIENT_ID",
                    "value": "<your-client-id>"
                },
                {
                    "name": "APPD_LOGCOL_SELF_LOGGING_LEVEL",
                    "value": "<log-collector-self-logging-level>"
                },
                {
                    "name": "APPD_LOGCOL_TOKEN_URL",
                    "value": "<your-appd-token-url>"
                }
            ],
            "mountPoints": [
                {
                    "sourceVolume": "var_lib_docker_containers",
                    "containerPath": "/var/lib/docker/containers/",
                    "readOnly": true
                },
                {
                    "sourceVolume": "var_run_dockersock",
                    "containerPath": "/var/run/docker.sock",
                    "readOnly": false
                },
                {
                    "sourceVolume": "filebeat_data",
                    "containerPath": "/opt/appdynamics/logcollector-agent/data/",
                    "readOnly": false
                },
                {
                    "sourceVolume": "system-release",
                    "containerPath": "/etc/amazon-linux-release",
                    "readOnly": true
                }
              ],
            "user": "0",
            "privileged": true,
            "logConfiguration": {
                "logDriver": "json-file",
                "options": {}
            }
        }
    ],
    "executionRoleArn": "<your-task-execution-role-arn>",
    "networkMode": "host",
    "Volumes": [
        {
            "name": "var_lib_docker_containers",
            "host": {
                "sourcePath": "/var/lib/docker/containers/"
            }
        },
        {
            "name": "var_run_dockersock",
            "host": {
                "sourcePath": "/var/run/docker.sock"
            }
        },
        {
            "name": "filebeat_data",
            "host": {
                "sourcePath": "/opt/appdynamics/logcollector-agent/data/"
            }
        },
        {
            "name": "system-release",
            "host": {
                "sourcePath": "/usr/lib/system-release"
            }
        }
      ],
    "requiresCompatibilities": [
        "EC2"
    ],
    "cpu": "400",
    "memory": "512",
    "runtimePlatform": {
        "cpuArchitecture": "X86_64",
        "operatingSystemFamily": "LINUX"
    }
}

JSON

Deploy the Log Collector as an Amazon ECS service with the following settings:
- Launch Type: EC2
- Application Type: Service
- Service Type: Daemon
- Service Name: Any name you choose as the service name for the Log Collector. We suggest appdlogcol-service.

Configure Default Log Collection

By default, the Log Collector collects logs generated by all containers running on each of the Amazon EC2 instances inside the Amazon ECS cluster. You don't need to modify the task definition for default log collection.

Configure Custom Log Collection

You can configure custom log collection using an override file in YAML format. The override file can contain the following settings for Docker container logs:

defaultConfig
- mulitLinePattern
- mulitLineNegate
- multiLineMatch
- messagePaser
  - <each message parser config>
- logFormat
- enabled
conditionalConfigs
- condition
  The conditions you can match on inside conditionalConfig are:
  - host
  - port
  - docker.container.id
  - docker.container.image
  - docker.container.name
  - docker.container.labels

- config
  - <similar structure as defaultConfig above>
excludeCondition

For example, an override file with the following content disables the default log collection and configure custom collection from container images named *log-gen-app*. For containers matching this condition, the overrides apply a log4J message parser and corresponding pattern:

logCollectorConfig:
  container:
    defaultConfig:
      enabled: false
    conditionalConfigs:
      - condition:
          contains:
            docker.container.image: log-gen-app
        config:
          messageParser:
            log4J:
              enabled: true
              pattern: "%d{yyyy-MM-dd HH:mm:ss} %-5p %m%n"

YML

To configure the Log Collector using an override file, you need to:

Add the override file into an Amazon Elastic File System (Amazon EFS) volume. See Amazon EFS volumes.
On your AWS console, select to the Amazon ECS dashboard and navigate to Task definitions.
Select the log collection task definition you created previously, and click Create new revision with JSON.

Edit the task definition:

In volumes, add your Amazon EFS volume:

"volumes": [
        {"..."},
        {
            "name": "ecs-override",
            "efsVolumeConfiguration": {
		        "fileSystemId": "<your-efs-id>",
		        "rootDirectory": "/"
	        }
        }
]

JSON

In containerDefinitions, add an entry to the mountPoints array with the mount point for your Amazon EFS volume. You must map your Amazon EFS volume to a directory such as /efs/<your-override-file-path> inside the container:

{
	"sourceVolume": "ecs-override",
	"containerPath": "/efs/",
	"readOnly": true
}

{
    "...",
    "containerDefinitions": [
        {
            "...",
            "mountPoints": [
                {"..."},
                {
                    "sourceVolume": "ecs-override",
                    "containerPath": "/efs/",
                    "readOnly": true
                }
            ],
        }
    ],
    ...
}

JSON

Add the following option (keeping the other options as-is) inside commands under containerDefinitions:
```
"-o=/efs/<your-override-file-path>",
```
CODE

Confirm that your task definition now looks like this:

{
    "family": "<your-log-collector-task-definition-name>",
    "containerDefinitions": [
        {
            ...
            "command": [
                "-f=/usr/share/filebeat/filebeat",
                "-b=/usr/share/filebeat/appd-filebeat-ecs.yml",
                "-a=-c,/usr/share/filebeat/appd-filebeat-ecs.yml,-strict.perms=false,-e",
                "-o=/efs/<your-override-file-path>",
            ],
            ...
            "mountPoints": [
                {
                    ...
                },
                {
                    ...
                },
                {
                    ...
                },
                {
                    "sourceVolume": "ecs-override",
                    "containerPath": "/efs/",
                    "readOnly": true
                }
            ],
            ...
        }
    ],
    ...
    "volumes": [
        {
            ...
        },
        {
            ...
        },
        {
            ...
        },
        {
            "name": "ecs-override",
            "efsVolumeConfiguration": {
                "fileSystemId": "<your-efs-id>",
                "rootDirectory": "/"
            }
        }
    ],
    ...
}

JSON

View Logs

Normally, you can view logs from applications running on Amazon ECS on the detail view of any of the following entities:

ECS Cluster
ECS Task Definition
ECS Service (Note: the Logs widget is not yet on the detaill view of this entity)
ECS Task
ECS Container

However, at the moment, there's a known issue which affects the detail view of some entities: the histogram does not work or does not connect to the Logs page. Workaround: navigate to Explore > Logs and search for log messages from source whose attributes match these values:

Filter Attrribute	Description
`aws.ecs_container.ec2.id`	EC2 instance ID where container is running
`aws.ecs_task.cluster_name`	ARN of the ECS Cluster
`aws.ecs_task.task_definition_name`	Your ECS task definition name
`container.id`	Container ID of the application

View Diagnostics

View Log Collector Status

On the AWS console, verify the collector deployment on ECS cluster page under the Services tab. A successful deployment has the following characteristics:

Deployment and Tasks are green.
Revision displays the correct deployed task definition revision.
Last deployment is Completed.

View Log Collector Logs

The Log Collector's logs are also collected by default unless you set defaultConfig.enabled to false in the override file. To view these on the Logs page, filter log messages with the value of the attribute aws.ecs_task.task_definition_name set to your Log Collector task definition name.

Set Self-Logging Levels

Set the APPD_LOGCOL_SELF_LOGGING_LEVEL environment variable in the task definition.
For example:

...
"environment": [
                {...},
                {
                    "name": "APPD_LOGCOL_SELF_LOGGING_LEVEL",
                    "value": "<log-collector-self-logging-level>"
                }
            ],
...

CODE

Restart the Log Collector:
1. On the AWS console, navigate to the service update or deploy page.
2. Re-deploy the service by clicking Update Service and selecting Force new deployment.

Amazon Web Services, the AWS logo, AWS, and any other AWS Marks used in these materials are trademarks of Amazon.com, Inc. or its affiliates in the United States and/or other countries.