Related pages:

The Controller creates a secure credential keystore that holds a secret key used to encrypt credentials.

Stored Credentials

The secure credential store manages the following credentials: 

  • LDAP authentication user password. See LDAP Authentication.

  • Database collector credentials, including database user password and the machine user password. 
  • SMTP server/Email passwords.
  • Splunk AppDynamics On-Premises account access keys.

Back up the credential store as part of your normal backup procedures for the Controller, as described in the following section.

Secure Credential Store Backup

Make sure your Controller backup plan includes the secure credential keystore file .appd.scskeystore. In the case that the secure credential keystore file should become corrupted, restore the .appd.scskeystore file from backup.

If you run the Controller in high availability mode, both the primary Controller and the secondary Controller must use the same secure credential keystore file. If you use an HA deployment strategy, verify that it propagates the secure credential keystore file from the primary to the secondary.

Replace a Compromised Secure Credential Store

The following steps describe how to replace a secure credential store. It assumes the following:

  • You have a single-tenant Controller installation.
  • You know the plain-text value of your Account Access Key. You can view the access key in the Controller under Settings > License.

As detailed in the sections that follow, the steps are broken into these parts: 

  1. Create a new secure credential store.
  2. Update the Controller with the password of the new secure credential store.
  3. Update the account access key.
  4. Update the account access key for the system account.
  5. Restart the Controller and update passwords.

Create a new Secure Credential Store

  1. Rename the existing secure credential keystore file.

  2. Initialize a new secure credential keystore using the secure credential store utility.

    By default the utility installs to: <controller_home>/tools/lib/scs-tool.jar

    For example: 

    /controller/jre8/bin/java -jar ./scs-tool.jar generate_ks -filename '<controller_home>/.appd.scskeystore' -storepass 'MyCredentialStorePassword'

    The secure credential store utility confirms it created and initialized the keystore: 

    Successfully created and initialized new KeyStore file: /opt/appdynamics/Controller/.appd.scskeystore
Verification - New KeyStore file: /opt/appdynamics/Controller/.appd.scskeystore is properly initialized.

Update the Controller with the new Secure Credential Store Password

  1. Shut down the Controller.

  2. Obfuscate the password you used to initialize the secure credential keystore:

    /controller/jre8/bin/java -jar <controller_home>/tools/lib/scs-tool.jar obfuscate -plaintext '<Secure_Credential_Store_Password>'

    For example:

    /controller/jre8/bin/java -jar /opt/appdynamics/Platform/controller/tools/lib/scs-tool.jar obfuscate -plaintext 'MyCredentialStorePassword'

    The secure credential store utility writes out an obfuscated password for use in the Controller configuration. For example: 

    s_gsnwR6+LDch8JBf1RamiBoWfMvjjipkrtJMZXAYEkw8=

  3. Log in as the root user:

    <controller_home>/bin/controller.sh login-db

    On Windows, use controller.bat.

  4. Update the secure credential keystore password to the newly obfuscated password: 

    UPDATE global_configuration_cluster
SET value = '<obfuscated_secure_credential_keystore_password>'
WHERE name = 'scs.keystore.password';

Update the Account Access Key

  1. Log in as the root user:

    <controller_home>/bin/controller.sh login-db

    On Windows, use controller.bat.

  2. Update the account access key for the account to the plain text string. When the Controller starts, it will encrypt the account access key: 

    UPDATE account
SET access_key = '<plain_text_account_access_key>',
    encryption_scheme = NULL
WHERE id = <account_id>;

    You can get the account id by running the following query: select id account_id,name account_name,access_key,encryption_scheme from account;

  3. Only if you changed the plain text value of the account access key. Update the account access key for the agent users:

    UPDATE user
SET encrypted_password = SHA1('<plain_text_account_access_key>')
WHERE account_id = <account_id>
AND name = 'singularity-agent';

    If you changed the plain text value of the account access key, you need to update the access key for all the agents.

    The access key belongs to the "customer1" account in a single-tenant Controller and the "default" account in a multi-tenant Controller. In addition, account_id is the account id of the "customer1" account in a single-tenant Controller and the "default" account in a multi-tenant Controller.

  4. If you have default license rules, update the account access key using v1_license_rules API. 

    For earlier Controller versions, you must use browser tools to migrate license rules.

Update the Account Access Key for the System Account

  1. Generate the new access key for the system account: 

    ../jre/1.8.0_152/bin/java -jar ./tools/lib/scs-tool.jar encrypt -filename ./.appd.scskeystore -storepass 'REPLACE_TO_NOT_OBFUSCATED_STOREPASS_VALUE' -plaintext 'NEW_SYSTEM_ACCOUNT_ACCESS_KEY'
    CODE

     

  2. Once you have generated the system account access key:

    1. Edit the controller-info.xml file to add your specific information:

      <controller-dir>/appserver/jetty/appagent/verX.X.X.X/conf/controller-info.xml
      CODE
    2. Edit the credential-store-password value with the obfuscated storepass value.
    3. Edit the account-access-key with new encrypted access key value.

    4. Run SQL:

      update account set access_key='ENCRYPTED_SYSTEM_ACCOUNT_ACCESS_KEY' where id=1; update mds_auth.account set access_key='ENCRYPTED_SYSTEM_ACCOUNT_ACCESS_KEY' where id='00000000-0000-0000-0000-000000000001'; update mds_auth.account set access_key='ENCRYPTED_SYSTEM_ACCOUNT_ACCESS_KEY' where id='00000000-0000-0000-0000-000000000002';
      CODE
    5. Update the account access key for the agent users.
      UPDATE user SET encrypted_password = SHA1('<plain_text_system_account_access_key>') WHERE account_id = <system_account_id> AND name = 'singularity-agent';
      CODE

    6. Stop appserver.

    7. Start appserver.

If you use LDAP, DBmon, or HTTP Request Actions and Templates, then you must also reconfigure those components with the same passwords to ensure that they are encrypted with new SCS key.

Restart the Controller and Update Passwords

  1. Restart the Controller.
  2. Log in to the Controller as a user with the following permissions:
    • Administer users, groups, roles, authentication, etc.
    • Configure Email / SMS.
  3. As necessary, re-enter the following passwords: