The Filelog Receiver is a component within the open-source OpenTelemetry™ Collector and the Cisco AppDynamics Distribution of OpenTelemetry Collector. It scrapes logs from the path you specify. The Filelog Receiver is the only option for log collection via OpenTelemetry for applications written in languages other than Java.

  • If your application is running on a supported Kubernetes cluster, we recommend that you use the Cisco AppDynamics Distribution of OpenTelemetry Collector.
  • If your application is running on a baremetal machine (but not Amazon ECS), you can use our Cisco AppDynamics Distribution of OpenTelemetry Collector or your own OpenTelemetry collector.

To enable and configure the Filelog Receiver, preform the following steps for your environment:

Prerequisites

  1. Ensure that your application writes logs to a physical file, not to STDERR or STDOUT.
  2. Ensure that an OpenTelemetry Collector is already running on your machine and has been configured to send the MELT data to Cisco Cloud Observability. If you're using Cisco AppDynamics Distribution of OpenTelemetry Collector, see Auto-Instrument Your Applications using OpenTelemetry Operator for Kubernetes.

Configure the Filelog Receiver of Cisco AppDynamics Distribution of OpenTelemetry Collector on Kubernetes

On you machine, add these settings under the appdynamics-otel-collector section in your collectors-values.yaml.

Syntax: 

Whenever messageParserType  is json or infra, the messageParserPattern must be an empty string ("") since these two parser types do not require any pattern.

appdynamics-otel-collector:
	enableFileLog: true
	filelogReceiverConfig:
		includeLogsPath: [ "<path>" ]
		excludeLogsPath: [ "<optional-path>" ]
		messageParserType: "<log4j, logback, json, grok, timestamp, or infra>"
 		messageParserPattern: "<string>"
 
	service:
		pipelines:
			logs/filelog:
				exporters: [otlphttp, logging]
YML

Example for JSON logs: 

appdynamics-otel-collector:
	enableFileLog: true
	filelogReceiverConfig:
		includeLogsPath: [ "/var/log/*/*app-json-*/*/*log" ]
        excludeLogsPath: [ "/var/log/pods/*/otel-collector/*.log" ]
		messageParserType: "json"
 		messageParserPattern: ""
 
	service:
		pipelines:
			logs/filelog:
				exporters: [otlphttp, logging]
YML

Example for log4j logs:

appdynamics-otel-collector:
	enableFileLog: true
	filelogReceiverConfig:
		includeLogsPath: [ "/var/log/*/*app-log4j-*/*/*log" ]
		excludeLogsPath: [ "/var/log/pods/*/otel-collector/*.log" ]
 		messageParserType: "log4j"
 		messageParserPattern: "%d{yyyY-MM-dd 'T'HH:mm:ss} %p %C{1.} [st] &m%n"

	service:
		pipelines:
			logs/filelog:
				exporters: [otlphttp, logging]
YML

Configure the Filelog Receiver of Cisco AppDynamics Distribution of OpenTelemetry Collector on Baremetal (Linux)

Since you're already using Cisco AppDynamics Distribution of OpenTelemetry Collector, you should have already configured it as explained in Deploy the Cisco AppDynamics Distribution of OpenTelemetry Collector in Amazon EC2 (Linux). The only update you need to make is to add an override configuration file and specify its location in the environment variable APPD_OTELCOL_EXTRA_CONFIG in /opt/appdynamics/appdynamics.conf. For example: 

APPD_OTELCOL_EXTRA_CONFIG=--config=file:/opt/appdynamics/extra.yaml
YML

Syntax of the override configuration file: 

receivers:
  filelog:
    include: ["<path>"]
    exclude: ["<optional-path>"]
    resource:
      - service.namespace: "<string>"
      - service.name: "<string>"
      - service.instance.id: "<string>"
    multiline:
      line_start_pattern: '<multiline-pattern>'
    attributes:
      _message_parser.type: "<log4j, logback, json, grok, timestamp, or infra>"
      _message_parser.pattern: "<string>"

service:
  pipelines:
    logs/filelog:
      receivers: [ filelog ]
      processors: [ memory_limiter ]
      exporters: [ otlphttp ]
YML

In the receivers section, specify service.namespace, service.name, and service.instance.id in order to correlate your logs and traces.


Sample override configuration file for log4j logs: 

receivers:
  filelog:
    include: [/var/log/sam-app/logs.log]
    resource:
      - service.namespace: sam-EC2
      - service.name: sam-app
      - service.instance.id: i-0a000589fd7f5fbd3
    multiline:
      line_start_pattern: '^\d{4}\-\d{2}\-\d{2}'
    attributes:
      _message_parser.type: log4j
      _message_parser.pattern: "%d{yyyy-MM-dd HH:mm:ss} %-4p %m%n"

service:
  pipelines:
    logs/filelog:
      receivers: [ filelog ]
      processors: [ memory_limiter ]
      exporters: [ otlphttp ]
YML

Sample override configuration file for multiple log sources with different service names but same namespace:

receivers:
  filelog/app-1:
    include: [/var/log/app-1/logs.log]
    resource:
      - service.namespace: sam-EC2
      - service.name: app-1
      - service.instance.id: i-0a000589fd7f5fbd3
    multiline:
      line_start_pattern: '^\d{4}\d{2}\d{2}'
    attributes:
      _message_parser.type: log4j
      _message_parser.pattern: "%d{yyyy-MM-dd HH:mm:ss} %-4p %m%n"
  filelog/app-2:
    include: [/var/log/app-2/logs.log]
    resource:
      - service.namespace: sam-EC2
      - service.name: app-2
      - service.instance.id: i-0a000589fd7f5fbd3
    attributes:
      _message_parser.type: json
      _message_parser.pattern: ""
 
service:
  pipelines:
    logs/filelog:
      receivers: [ filelog/app-1, filelog/app-2 ]
      processors: [ memory_limiter ]
      exporters: [ otlphttp ]
YML

Configure the Filelog Receiver of an Open-Source OpenTelemetry Collector on Baremetal (Linux)

Add an override configuration file and specify its location by using the --config flag while starting up your collector. The contents of the configuration file should be the same as the override configuration file for Cisco AppDynamics Distribution of OpenTelemetry Collector, shown above.

In the receivers section, specify service.namespace, service.name, and service.instance.id in order to correlate your logs and traces.

For information on Filelog Receiver  advanced configuration options, see the Filelog Receiver README on GitHub.


OpenTelemetry™ is a trademark of The Linux Foundation®.