AWS Lambda Service Logs

Using our AWS CloudFormation template, you can deploy a CloudFormation stack to forward AWS Lambda service logs to the Splunk AppDynamics Common Ingestion Service, where they are associated with the right entities in your observability domains. This page provides instructions for using our CloudFormation template to create a CloudFormation stack.

Prerequisites

You must have the aws command line interface (CLI) connected to the AWS account you want to set up log collection on.

Plan Your Setup

In the following table, determine the parameters you will pass to the CloudFormation stack:

When providing multiple comma-separated values, you need to escape commas, like this: ParameterKey=AppDEC2LogGroupPrefixes,ParameterValue="appd/ec2/\,appd/lambda/"

ParameterKey	Description	Required?
AppDCredentialsSecretName	Name of secret you created in Create a Secret.	Yes
AppDEC2LogGroupPrefixes	This parameter applies only to Amazon EC2 service logs. Comma-separated list of EC2 log group prefixes. If your EC2 logs are not coming from a log group having the prefix /aws/ec2 (default), then you must  define your log group prefixes to identify logs as coming from an EC2 service.	Yes in some circumstances
FirehoseForwarderLambdaLoggingLevel	Logging level of the Firehose forwarder Lambda function (AppDFirehoseLambda). Default: "DEBUG".	No
FailedLogForwarderLambdaLoggingLevel	Logging level of the S3 forwarder Lambda function (AppDFailedLogForwarderLambda). Default: "DEBUG".	No
S3ProcessorLambdaLoggingLevel	Logging level of the S3 processor Lambda function (AppDS3ProcessorLambda). Default: "DEBUG".	No
S3ServicesLogsBucketArn	ARN of the existing S3 bucket which has logs from from AWS services. Default: AppDS3ServicesLogsBucket.	No
TLSMinVersion	Specifies the minimum TLS version to use. Default: "TLSv1.3" (TLS version 1.3). Valid values: "TLSv1" or "TLSv1.0" for TLS version 1.0 "TLSv1.1" for TLS version 1.1 "TLSv1.2" for TLS version 1.2 "TLSv1.3" for TLS version 1.3 If you specify an invalid value, the Lambdas return an error and no logs are processed.	No
TokenRotatorLambdaLoggingLevel	Logging level of the token rotator Lambda function (AppDTokenRotatorLambda). Default: "DEBUG".	No

Components of the appd-aws-service-log-collector CloudFormation Stack 

Our CloudFormation template creates the following Lambda functions in your AWS account:

• AppDDownloaderLambda - This is an inline Lambda function which downloads the other Splunk AppDynamicsLambda functions .zip files and saves them to an S3 bucket.  
• AppDFirehoseLambda - This is the Firehose processor forwarder created by the CloudFormation template to process and forward logs to the  back end. In case of failure, it forwards logs to a backup S3 bucket.
• AppDFailedLogForwarderLambda - This is the S3 forwarder Lambda function. It sends the failed logs stored in the S3 bucket to the  back end. This Lambda function is scheduled to run every 20 minutes by default, provided by the CloudFormation template.
• AppDS3ProcessorLambda - This is S3 Processor Lambda. It processes logs stored in the s3 bucket coming from services that log directly to the s3 bucket.
• AppDTokenRotatorLambda - This is used to rotate the Splunk AppDynamics token (on expiry) that is required to send the logs to Cisco Cloud Observability.
•  AppDFailedLogsBucket - The S3 bucket stores failed logs for retry.
• AppDS3ServicesLogsBucket - This new bucket stores logs coming from services that log to s3.
• AppDTokenSecret - this is a secret value which stores the Splunk AppDynamics token.

Create a Secret

1. Create a secret in AWS Secret Manager with the Cisco Cloud Observability credentials (tenantId, clientId, clientSecret, endpoint):

   1. Get your client ID, client secret, Tenant ID, and Tenant endpoint from the Cisco Cloud Observability Account Management portal. If you're using a "Service Principal", make sure the Authentication Type to Basic.

   2. On the AWS console, click Secrets Manager.

   3. Click Store a new secret.                                                                    

   4. On the Choose secret type page, do the following:

      1. For Secret Type, select Other type of secret.

      2. In Key/value pairs, either enter your secret values in JSON Key/value pairs, or select the Plaintext tab and enter the secrets as given below.

         Secret Structure

         {"endpoint":"<tenant-endpoint>","tenantId":"<tenant-id>","clientId":"<client-id>","clientSecret":"<client-secret>"}

         JSON
         

         Make sure <tenant-endpoint> is using Hypertext Transfer Protocol Secure (HTTPS).

      3. For Encryption key, select aws/secretsmanager.

      4. Click Next.
   5. On the Configure secret page, do the following:
      1. Enter a descriptive Secret name and Description. This Secret name will be passed to the create-stack command as a parameter.
      2. Skip all other sections on this page.
      3. Click Next.
   6. Skip everything on Configure rotation page.
   7. On the Review page, review your secret details, and then click Store.

Create the CloudFormation Stack

1. Download the CloudFormation template, aws-service-log-collector-template-<latest-version>.zip, from the Cisco Cloud Observability artifactory: 

   curl https://appdynamics.jfrog.io/artifactory/zip-hosted/appdcloud/collectors/aws-services-log-collector-linux-amd64/<latest-version>/aws-services-log-collector-linux-amd64-<latest-version>.zip \
   --output ./appd-aws-service-log-collector-template.zip

   BASH

   where <latest-version> is 23.7.0-268.

2. Unzip the downloaded file.

3. Run the aws cloudformation create-stack command with the template that was in the .zip file that you downloaded (template.yaml), replacing all placeholders with the values you planned in your planning step.

   

   If you already have a CloudFormation stack named appd-aws-service-log-collector, run the update-stack command instead.

   For example: 

   aws cloudformation create-stack \
     --stack-name appd-aws-service-log-collector \
     --template-body file://./template.yaml \
     --parameters \
     ParameterKey=AppDCredentialsSecretName,ParameterValue=[SECRET-NAME] \
     --capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_IAM \
     --region [STACK-REGION]

   BASH

   This command creates a CloudFormation stack with all the right resources.

   
   

4. On the AWS console, navigate to https://<aws-region>.console.aws.amazon.com/cloudformation/home?region=<stack-region>, where <stack-region> is the AWS region you have deployed the stack in, and verify that the CloudFormation stack has been created by validating that the status of the stack is CREATE_COMPLETE. This might take 5-10 minutes.
5. If the CloudFormation stack creation failed, check the Events tab for errors.

Update a Running CloudFormation Stack

Update a Secret

1. Create a new secret with a different name than your current secret.

2. Run the update-stack command with the new secret.

3. When the update is complete, trigger the AppDTokenRotatorLambda function manually:

   1. Open the AppDTokenRotatorLambda function page.
   2. Select the Test tab.
   3. In the Event name field, enter a value.
   4. Click Test.
   5. Verify that the status is Executing function: succeeded, which means that the Lambda function was triggered successfully.

Upgrade to New Template or Release

1. Download the new template.
2. Update the parameter keys and values based on the latest template changes, if any.

3. Run the update-stack  command with the updated LambdaVersion in the parameter list. 
   For example, 

   aws cloudformation update-stack \
     --stack-name appd-aws-service-log-collector \
     --template-body file://./template.yaml \
     --parameters \
     ParameterKey=AppDCredentialsSecretName,ParameterValue=[SECRET-NAME] \
     ParameterKey=AppDEC2LogGroupPrefixes,ParameterValue="[LOGGROUP-PREFIX-1]\,[LOGGROUP-PREFIX-2]" \
     --capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_IAM \
     --region [STACK-REGION]

   BASH
4. Monitor the CloudFormation console and wait for the update to complete. Confirm that there are no errors.
5. Verify that the  APPD_LAMBDA_FUNCTION_VERSION environment variable in all the Lambda functions (AppDFirehoseLambda, FailedLogForwarderLambda, AppDTokenRotatorLambda) matches the latest updated LambdaVersion.

Upgrade or Downgrade a Lambda Version

1. Run the update-stack  command with the updated LambdaVersion  in the parameter list.
   For example, 

   aws cloudformation update-stack \
     --stack-name appd-aws-service-log-collector \
     --template-body file://./template.yaml \
     --parameters \
     ParameterKey=AppDCredentialsSecretName,ParameterValue=[SECRET-NAME] \
     ParameterKey=AppDEC2LogGroupPrefixes,ParameterValue="[LOGGROUP-PREFIX-1]\,[LOGGROUP-PREFIX-2]" \
     ParameterKey=LambdaVersion,ParameterValue="[NEW-LAMBDA-VERSION]" \
     --capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_IAM \
     --region [STACK-REGION]

   BASH
2. Monitor the CloudFormation console and wait for the action to complete. Confirm that there are no errors.
3. Verify that the APPD_LAMBDA_FUNCTION_VERSION  environment variable in all the Lambda functions (AppDFirehoseLambda, FailedLogForwarderLambda, AppDTokenRotatorLambda) matches the latest updated LambdaVersion.

Amazon Web Services, the AWS logo, AWS, and any other AWS Marks used in these materials are trademarks of Amazon.com, Inc. or its affiliates in the United States and/or other countries.

Subscribe to AWS Lambda Service Logs

After you've deployed the CloudFormation stack, subscribe to logs from AWS Services as follows.

The following steps assume that you're already sending logs to CloudWatch.

1. Select the CloudWatch log group from which you want logs to be sent to Cisco Cloud Observability.

   

   If you are subscribing to a log group having EC2 service logs, it should either have the prefix "/aws/ec2" (default), or you must define your log group prefixes using the ParameterKey AppDEC2LogGroupPrefixes to identify logs as coming from an EC2 service.

2. Find out the ARN of the Cisco Cloud Observability Firehose, using the command:

   aws cloudformation describe-stacks --stack-name appd-aws-service-log-collector --region <stack-region> \
    --query "Stacks[0].Outputs[?OutputKey=='AppDLogsFirehoseArn']" --output text

   CODE

3. Find out the ARN of the Cloudwatch-to-Firehose role, using the command:

   aws cloudformation describe-stacks --stack-name appd-aws-service-log-collector --region <stack-region> \
   --query "Stacks[0].Outputs[?OutputKey=='CloudWatchLogsToAppDFirehoseRoleArn']" --output text

   CODE

4. Create a subscription filter using the following command:

   aws logs put-subscription-filter \
         --log-group-name "<log-group-name>" \
         --filter-name "<filter-name>" \
         --filter-pattern " " \
         --destination-arn "<AppDLogsFirehose-Arn>" \
         --role-arn "<CloudWatchLogsToAppDLogsFirehoseRole-Arn>"

   BASH

View AWS Lambda Service Logs

To see the log messages related to an AWS Lambda services, navigate to Observe > Function Services > Lambda, select a specific Lambda function, and click View all logs. An alternate way to view logs is to navigate to Explore > Logs and use the filter attributes(cloud.platform) = aws_lambda. See Explore Logs.