Using our AWS CloudFormation template, you can deploy a CloudFormation stack to forward Amazon Elastic Load Balancing (Amazon ELB) service logs (includes ALB, CLB, NLB) to the Splunk AppDynamics Common Ingestion Service, where they are associated with the right entities in your observability domains. This page provides instructions for using our CloudFormation template to create a CloudFormation stack.

Prerequisites

You must have the aws command line interface (CLI) connected to the AWS account you want to set up log collection on.

Plan Your Setup

In the following table, determine the parameters you will pass to the CloudFormation stack:

When providing multiple comma-separated values, you need to escape commas, like this: ParameterKey=AppDEC2LogGroupPrefixes,ParameterValue="appd/ec2/\,appd/lambda/"

ParameterKeyDescriptionRequired?
AppDCredentialsSecretNameName of secret you created in Create a Secret.Yes
AppDEC2LogGroupPrefixes
This parameter applies only to Amazon EC2 service logs.

Comma-separated list of EC2 log group prefixes. If your EC2 logs are not coming from a log group having the prefix /aws/ec2 (default), then you must  define your log group prefixes to identify logs as coming from an EC2 service.

Yes in some circumstances

FirehoseForwarderLambdaLoggingLevel

Logging level of the Firehose forwarder Lambda function (AppDFirehoseLambda). Default: "DEBUG".No

FailedLogForwarderLambdaLoggingLevel

Logging level of the S3 forwarder Lambda function (AppDFailedLogForwarderLambda). Default: "DEBUG".

No

S3ProcessorLambdaLoggingLevel

Logging level of the S3 processor Lambda function (AppDS3ProcessorLambda). Default: "DEBUG".No

S3ServicesLogsBucketArn

ARN of the existing S3 bucket which has logs from from AWS services. Default: AppDS3ServicesLogsBucket.

No

TLSMinVersion

Specifies the minimum TLS version to use. Default: "TLSv1.3" (TLS version 1.3). Valid values:

  • "TLSv1" or "TLSv1.0" for TLS version 1.0
  • "TLSv1.1" for TLS version 1.1
  • "TLSv1.2" for TLS version 1.2
  • "TLSv1.3" for TLS version 1.3

If you specify an invalid value, the Lambdas return an error and no logs are processed. 

No

TokenRotatorLambdaLoggingLevel

Logging level of the token rotator Lambda function (AppDTokenRotatorLambda). Default: "DEBUG".No

Components of the appd-aws-service-log-collector CloudFormation Stack 

Our CloudFormation template creates the following Lambda functions in your AWS account:

  • AppDDownloaderLambda - This is an inline Lambda function which downloads the other Splunk AppDynamicsLambda functions .zip files and saves them to an S3 bucket.  
  • AppDFirehoseLambda - This is the Firehose processor forwarder created by the CloudFormation template to process and forward logs to the  back end. In case of failure, it forwards logs to a backup S3 bucket.
  • AppDFailedLogForwarderLambda - This is the S3 forwarder Lambda function. It sends the failed logs stored in the S3 bucket to the  back end. This Lambda function is scheduled to run every 20 minutes by default, provided by the CloudFormation template.
  • AppDS3ProcessorLambda - This is S3 Processor Lambda. It processes logs stored in the s3 bucket coming from services that log directly to the s3 bucket.
  • AppDTokenRotatorLambda - This is used to rotate the Splunk AppDynamics token (on expiry) that is required to send the logs to Cisco Cloud Observability.
  •  AppDFailedLogsBucket - The S3 bucket stores failed logs for retry.
  • AppDS3ServicesLogsBucket - This new bucket stores logs coming from services that log to s3.
  • AppDTokenSecret - this is a secret value which stores the Splunk AppDynamics token.

Create a Secret

  1. Create a secret in AWS Secret Manager with the Cisco Cloud Observability credentials (tenantId, clientId, clientSecret, endpoint):

    1. Get your client ID, client secret, Tenant ID, and Tenant endpoint from the Cisco Cloud Observability Account Management portal. If you're using a "Service Principal", make sure the Authentication Type to Basic.

    2. On the AWS console, click Secrets Manager.

    3. Click Store a new secret.                                                                    

    4. On the Choose secret type page, do the following:

      1. For Secret Type, select Other type of secret.

      2. In Key/value pairs, either enter your secret values in JSON Key/value pairs, or select the Plaintext tab and enter the secrets as given below.

        Secret Structure

        {"endpoint":"<tenant-endpoint>","tenantId":"<tenant-id>","clientId":"<client-id>","clientSecret":"<client-secret>"}
        JSON

        Make sure <tenant-endpoint> is using Hypertext Transfer Protocol Secure (HTTPS).

      3. For Encryption key, select aws/secretsmanager.

      4. Click Next.
    5. On the Configure secret page, do the following:
      1. Enter a descriptive Secret name and Description. This Secret name will be passed to the create-stack command as a parameter.
      2. Skip all other sections on this page.
      3. Click Next.
    6. Skip everything on Configure rotation page.
    7. On the Review page, review your secret details, and then click Store.

Create the CloudFormation Stack

  1. Download the CloudFormation template, aws-service-log-collector-template-<latest-version>.zip, from the Cisco Cloud Observability artifactory: 

    curl https://appdynamics.jfrog.io/artifactory/zip-hosted/appdcloud/collectors/aws-services-log-collector-linux-amd64/<latest-version>/aws-services-log-collector-linux-amd64-<latest-version>.zip \
--output ./appd-aws-service-log-collector-template.zip
    BASH

    where <latest-version> is 23.7.0-268.

  2. Unzip the downloaded file.

  3. Run the aws cloudformation create-stack command with the template that was in the .zip file that you downloaded (template.yaml), replacing all placeholders with the values you planned in your planning step.

    If you already have a CloudFormation stack named appd-aws-service-log-collector, run the update-stack command instead.

    For example: 

    aws cloudformation create-stack \
  --stack-name appd-aws-service-log-collector \
  --template-body file://./template.yaml \
  --parameters \
  ParameterKey=AppDCredentialsSecretName,ParameterValue=[SECRET-NAME] \
  --capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_IAM \
  --region [STACK-REGION]
    BASH

    This command creates a CloudFormation stack with all the right resources.


  4. On the AWS console, navigate to https://<aws-region>.console.aws.amazon.com/cloudformation/home?region=<stack-region>, where <stack-region> is the AWS region you have deployed the stack in, and verify that the CloudFormation stack has been created by validating that the status of the stack is CREATE_COMPLETE. This might take 5-10 minutes.
  5. If the CloudFormation stack creation failed, check the Events tab for errors.

Update a Running CloudFormation Stack

Update a Secret

  1. Create a new secret with a different name than your current secret.

  2. Run the update-stack command with the new secret.

  3. When the update is complete, trigger the AppDTokenRotatorLambda function manually:

    1. Open the AppDTokenRotatorLambda function page.
    2. Select the Test tab.
    3. In the Event name field, enter a value.
    4. Click Test.
    5. Verify that the status is Executing function: succeeded, which means that the Lambda function was triggered successfully.

Upgrade to New Template or Release

  1. Download the new template.
  2. Update the parameter keys and values based on the latest template changes, if any.

  3. Run the update-stack  command with the updated LambdaVersion in the parameter list. 
    For example, 

    aws cloudformation update-stack \
  --stack-name appd-aws-service-log-collector \
  --template-body file://./template.yaml \
  --parameters \
  ParameterKey=AppDCredentialsSecretName,ParameterValue=[SECRET-NAME] \
  ParameterKey=AppDEC2LogGroupPrefixes,ParameterValue="[LOGGROUP-PREFIX-1]\,[LOGGROUP-PREFIX-2]" \
  --capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_IAM \
  --region [STACK-REGION]
    BASH
  4. Monitor the CloudFormation console and wait for the update to complete. Confirm that there are no errors.
  5. Verify that the  APPD_LAMBDA_FUNCTION_VERSION environment variable in all the Lambda functions (AppDFirehoseLambda, FailedLogForwarderLambda, AppDTokenRotatorLambda) matches the latest updated LambdaVersion.

Upgrade or Downgrade a Lambda Version

  1. Run the update-stack  command with the updated LambdaVersion  in the parameter list.
    For example, 

    aws cloudformation update-stack \
  --stack-name appd-aws-service-log-collector \
  --template-body file://./template.yaml \
  --parameters \
  ParameterKey=AppDCredentialsSecretName,ParameterValue=[SECRET-NAME] \
  ParameterKey=AppDEC2LogGroupPrefixes,ParameterValue="[LOGGROUP-PREFIX-1]\,[LOGGROUP-PREFIX-2]" \
  ParameterKey=LambdaVersion,ParameterValue="[NEW-LAMBDA-VERSION]" \
  --capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_IAM \
  --region [STACK-REGION]
    BASH
  2. Monitor the CloudFormation console and wait for the action to complete. Confirm that there are no errors.
  3. Verify that the APPD_LAMBDA_FUNCTION_VERSION  environment variable in all the Lambda functions (AppDFirehoseLambda, FailedLogForwarderLambda, AppDTokenRotatorLambda) matches the latest updated LambdaVersion.



Amazon Web Services, the AWS logo, AWS, and any other AWS Marks used in these materials are trademarks of Amazon.com, Inc. or its affiliates in the United States and/or other countries.

Subscribe to Amazon ELB Service Logs

  1. Attach a policy to your S3 bucket.

    Your S3 bucket must have a bucket policy that grants Elastic Load Balancing permission to write the access logs to the bucket. For help with attaching the policy, see Step 2: Attach a policy to your S3 bucket.

  2. If you specified the S3ServicesLogsBucketArn parameter for the CloudFormation template, add a trigger for the S3 bucket:
    1. In the Function overview pane of your function’s console page, choose Add trigger.
    2. Select S3.
    3. Under Bucket, select the bucket you created earlier.
    4. Under Event types, select All object create events.
    5. Under Recursive invocation, select the check box to acknowledge that using the same Amazon S3 bucket for input and output is not recommended. You can learn more about recursive invocation patterns in Lambda by reading Recursive patterns that cause run-away Lambda functions in Serverless Land.
    6. Click Add.
  3. Enable access logs for your load balancer:
    1. Open the Amazon EC2 console.
    2. In the navigation pane, select Load Balancers.
    3. Select the name of your load balancer to open its details page.
    4. On the Attributes tab, click Edit.
    5. For Monitoring, enable Access logs.
    6. For S3 URI, enter the S3 URI for your log files.
      The URI that you specify depends on whether you're using a prefix:
      • URI with a prefix: s3://<bucket-name>/<prefix>
      • URI without a prefix: s3://<bucket-name>
    7. Click Save changes.

View Amazon ELB Service Logs

Navigate to the entity centric page for that service and click View all logs. See Explore Logs.