Amazon S3 Access Logs

Using our AWS CloudFormation template, you can deploy a CloudFormation stack to forward Amazon S3 access logs to the Splunk AppDynamics Common Ingestion Service, where they are associated with the right entities in your observability domains. This page provides instructions for using our CloudFormation template to create a CloudFormation stack.

Prerequisites

You must have the aws command line interface (CLI) connected to the AWS account you want to set up log collection on.

Plan Your Setup

In the following table, determine the parameters you will pass to the CloudFormation stack:

When providing multiple comma-separated values, you need to escape commas, like this: ParameterKey=AppDEC2LogGroupPrefixes,ParameterValue="appd/ec2/\,appd/lambda/"

ParameterKey	Description	Required?
AppDCredentialsSecretName	Name of secret you created in Create a Secret.	Yes
AppDEC2LogGroupPrefixes	This parameter applies only to Amazon EC2 service logs. Comma-separated list of EC2 log group prefixes. If your EC2 logs are not coming from a log group having the prefix /aws/ec2 (default), then you must  define your log group prefixes to identify logs as coming from an EC2 service.	Yes in some circumstances
FirehoseForwarderLambdaLoggingLevel	Logging level of the Firehose forwarder Lambda function (AppDFirehoseLambda). Default: "DEBUG".	No
FailedLogForwarderLambdaLoggingLevel	Logging level of the S3 forwarder Lambda function (AppDFailedLogForwarderLambda). Default: "DEBUG".	No
S3ProcessorLambdaLoggingLevel	Logging level of the S3 processor Lambda function (AppDS3ProcessorLambda). Default: "DEBUG".	No
S3ServicesLogsBucketArn	ARN of the existing S3 bucket which has logs from from AWS services. Default: AppDS3ServicesLogsBucket.	No
TLSMinVersion	Specifies the minimum TLS version to use. Default: "TLSv1.3" (TLS version 1.3). Valid values: "TLSv1" or "TLSv1.0" for TLS version 1.0 "TLSv1.1" for TLS version 1.1 "TLSv1.2" for TLS version 1.2 "TLSv1.3" for TLS version 1.3 If you specify an invalid value, the Lambdas return an error and no logs are processed.	No
TokenRotatorLambdaLoggingLevel	Logging level of the token rotator Lambda function (AppDTokenRotatorLambda). Default: "DEBUG".	No

Components of the appd-aws-service-log-collector CloudFormation Stack 

Our CloudFormation template creates the following Lambda functions in your AWS account:

• AppDDownloaderLambda - This is an inline Lambda function which downloads the other Splunk AppDynamicsLambda functions .zip files and saves them to an S3 bucket.  
• AppDFirehoseLambda - This is the Firehose processor forwarder created by the CloudFormation template to process and forward logs to the  back end. In case of failure, it forwards logs to a backup S3 bucket.
• AppDFailedLogForwarderLambda - This is the S3 forwarder Lambda function. It sends the failed logs stored in the S3 bucket to the  back end. This Lambda function is scheduled to run every 20 minutes by default, provided by the CloudFormation template.
• AppDS3ProcessorLambda - This is S3 Processor Lambda. It processes logs stored in the s3 bucket coming from services that log directly to the s3 bucket.
• AppDTokenRotatorLambda - This is used to rotate the Splunk AppDynamics token (on expiry) that is required to send the logs to Cisco Cloud Observability.
•  AppDFailedLogsBucket - The S3 bucket stores failed logs for retry.
• AppDS3ServicesLogsBucket - This new bucket stores logs coming from services that log to s3.
• AppDTokenSecret - this is a secret value which stores the Splunk AppDynamics token.

Create a Secret

1. Create a secret in AWS Secret Manager with the Cisco Cloud Observability credentials (tenantId, clientId, clientSecret, endpoint):

   1. Get your client ID, client secret, Tenant ID, and Tenant endpoint from the Cisco Cloud Observability Account Management portal. If you're using a "Service Principal", make sure the Authentication Type to Basic.

   2. On the AWS console, click Secrets Manager.

   3. Click Store a new secret.                                                                    

   4. On the Choose secret type page, do the following:

      1. For Secret Type, select Other type of secret.

      2. In Key/value pairs, either enter your secret values in JSON Key/value pairs, or select the Plaintext tab and enter the secrets as given below.

         Secret Structure

         {"endpoint":"<tenant-endpoint>","tenantId":"<tenant-id>","clientId":"<client-id>","clientSecret":"<client-secret>"}

         JSON
         

         Make sure <tenant-endpoint> is using Hypertext Transfer Protocol Secure (HTTPS).

      3. For Encryption key, select aws/secretsmanager.

      4. Click Next.
   5. On the Configure secret page, do the following:
      1. Enter a descriptive Secret name and Description. This Secret name will be passed to the create-stack command as a parameter.
      2. Skip all other sections on this page.
      3. Click Next.
   6. Skip everything on Configure rotation page.
   7. On the Review page, review your secret details, and then click Store.

Create the CloudFormation Stack

1. Download the CloudFormation template, aws-service-log-collector-template-<latest-version>.zip, from the Cisco Cloud Observability artifactory: 

   curl https://appdynamics.jfrog.io/artifactory/zip-hosted/appdcloud/collectors/aws-services-log-collector-linux-amd64/<latest-version>/aws-services-log-collector-linux-amd64-<latest-version>.zip \
   --output ./appd-aws-service-log-collector-template.zip

   BASH

   where <latest-version> is 23.7.0-268.

2. Unzip the downloaded file.

3. Run the aws cloudformation create-stack command with the template that was in the .zip file that you downloaded (template.yaml), replacing all placeholders with the values you planned in your planning step.

   

   If you already have a CloudFormation stack named appd-aws-service-log-collector, run the update-stack command instead.

   For example: 

   aws cloudformation create-stack \
     --stack-name appd-aws-service-log-collector \
     --template-body file://./template.yaml \
     --parameters \
     ParameterKey=AppDCredentialsSecretName,ParameterValue=[SECRET-NAME] \
     --capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_IAM \
     --region [STACK-REGION]

   BASH

   This command creates a CloudFormation stack with all the right resources.

   
   

4. On the AWS console, navigate to https://<aws-region>.console.aws.amazon.com/cloudformation/home?region=<stack-region>, where <stack-region> is the AWS region you have deployed the stack in, and verify that the CloudFormation stack has been created by validating that the status of the stack is CREATE_COMPLETE. This might take 5-10 minutes.
5. If the CloudFormation stack creation failed, check the Events tab for errors.

Update a Running CloudFormation Stack

Update a Secret

1. Create a new secret with a different name than your current secret.

2. Run the update-stack command with the new secret.

3. When the update is complete, trigger the AppDTokenRotatorLambda function manually:

   1. Open the AppDTokenRotatorLambda function page.
   2. Select the Test tab.
   3. In the Event name field, enter a value.
   4. Click Test.
   5. Verify that the status is Executing function: succeeded, which means that the Lambda function was triggered successfully.

Upgrade to New Template or Release

1. Download the new template.
2. Update the parameter keys and values based on the latest template changes, if any.

3. Run the update-stack  command with the updated LambdaVersion in the parameter list. 
   For example, 

   aws cloudformation update-stack \
     --stack-name appd-aws-service-log-collector \
     --template-body file://./template.yaml \
     --parameters \
     ParameterKey=AppDCredentialsSecretName,ParameterValue=[SECRET-NAME] \
     ParameterKey=AppDEC2LogGroupPrefixes,ParameterValue="[LOGGROUP-PREFIX-1]\,[LOGGROUP-PREFIX-2]" \
     --capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_IAM \
     --region [STACK-REGION]

   BASH
4. Monitor the CloudFormation console and wait for the update to complete. Confirm that there are no errors.
5. Verify that the  APPD_LAMBDA_FUNCTION_VERSION environment variable in all the Lambda functions (AppDFirehoseLambda, FailedLogForwarderLambda, AppDTokenRotatorLambda) matches the latest updated LambdaVersion.

Upgrade or Downgrade a Lambda Version

1. Run the update-stack  command with the updated LambdaVersion  in the parameter list.
   For example, 

   aws cloudformation update-stack \
     --stack-name appd-aws-service-log-collector \
     --template-body file://./template.yaml \
     --parameters \
     ParameterKey=AppDCredentialsSecretName,ParameterValue=[SECRET-NAME] \
     ParameterKey=AppDEC2LogGroupPrefixes,ParameterValue="[LOGGROUP-PREFIX-1]\,[LOGGROUP-PREFIX-2]" \
     ParameterKey=LambdaVersion,ParameterValue="[NEW-LAMBDA-VERSION]" \
     --capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_IAM \
     --region [STACK-REGION]

   BASH
2. Monitor the CloudFormation console and wait for the action to complete. Confirm that there are no errors.
3. Verify that the APPD_LAMBDA_FUNCTION_VERSION  environment variable in all the Lambda functions (AppDFirehoseLambda, FailedLogForwarderLambda, AppDTokenRotatorLambda) matches the latest updated LambdaVersion.

Amazon Web Services, the AWS logo, AWS, and any other AWS Marks used in these materials are trademarks of Amazon.com, Inc. or its affiliates in the United States and/or other countries.

Enable Amazon S3 Access Logs

By default, Amazon S3 doesn't collect server access logs. To enable server access logging:

1. Sign in to the AWS Management console and open the Amazon S3 console at https://console.aws.amazon.com/s3/.
2. In the Buckets list, select the name of the bucket that you want to enable server access logging for.
3. Click Properties.
4. In the Server access logging section, click Edit.
5. Under Server access logging, click Enable.
6. Under Destination bucket, specify a bucket and the mandatory prefix s3-access-logs/ after the bucket name. Be sure to include a forward slash (/) after the prefix to make it easier to find your logs.
7. Under Log object key format, select [DestinationPrefix][SourceAccountId]/[SourceRegion]/[SourceBucket]/[YYYY]/[MM]/[DD]/[YYYY]-[MM]-[DD]-[hh]-[mm]-[ss]-[UniqueString], then select either S3 event time or Log file delivery time.
8. Click Save changes.

When you enable server access logging on a bucket, AWS does two things: it enables logging on the source bucket and updates the bucket policy for the destination bucket to grant the s3:PutObject permission to the logging service principal (logging.s3.amazonaws.com). For more information about this bucket policy, see Grant permissions to the logging service principal by using a bucket policy.

To confirm that server access logging is enabled, view the logs in the destination bucket. It might take a few hours before the logs are delivered to the destination bucket. For more information about how and when logs are delivered, see How are logs delivered?

View Amazon S3 Access Logs

To see the access logs related to a specific S3 bucket, navigate to Observe > AWS S3 Buckets, select the target S3 bucket, and click View all logs. See Explore Logs.