The time required to migrate from Events Service 4.5.x to 23.x may vary depending on the data volume. Hence, we recommend that you migrate delta data and metadata in multiple iterations to ensure minimal post cutover data on Events Service 23.x.

Prepare Events Service 4.5.x Data for Migration

1. Go to $APPDYNAMICS_HOME/platform/events-service/processor/conf/event-service-api-store.properties and enable the following property in the Events Service 4.5.x platform:

   ad.es.node.http.enabled=true

   CODE
   

   Repeat this step on each node of the Events Service cluster.

2. Restart the Events Service 4.5 platform from Enterprise Console.

3. Disable the job compaction for INDEX_COMPACTION_PARENT_JOB on Events Service 23.x.

   curl -H 'Content-type: application/json' -XPOST 'http://<ELB_hostname_of_Events_Service_23.x>:<ElasticSearch_Port>/job_framework_job_details_v1/_update_by_query' -d '{
    "query": {
     "term": {
      "name": "INDEX_COMPACTION_PARENT_JOB"
     }
    },
    "script": {
     "inline": "ctx._source.jobDataMap.enabled = false",
     "lang": "painless"
    }
   }'

   CODE
   

   You can find the HTTP port of Elasticsearch from the ad.es.node.http.port property in $APPDYNAMICS_HOME/platform/events-service/processor/conf/events-service-api-store.properties file for the corresponding Events Service instance.

4. In the Events Service 23.x platform, edit the configuration file, $APPDYNAMICS_HOME/platform/events-service/processor/conf/events-service-api-store.yml to add the property reindex.remote.whitelist with the comma separated list. This list consists of:
   
   
   • Host Name and port of Elasticsearch 2.x.

   • Host Name and port of Elasticsearch 8.x.

     Syntax to Update reindex.remote.whitelist

     - className: com.appdynamics.analytics.processor.elasticsearch.configuration.ElasticsearchConfigManagerModule
       properties:
         nodeSettings:
           cluster.name: ${ad.es.cluster.name}
           reindex.remote.whitelist: "<ELB_hostname_of_Events_Service_4.5.x>:<Port>,<ELB_hostname_of_Events_Service_23.x>:<Port>,<node1_hostname_of_Events_Service_4.5.x>:<Port>,<node2_hostname_of_Events_Service_4.5.x>:<Port>"

     CODE

     Example

     reindex.remote.whitelist: "es2.elb.amazonaws.com:9200,es8.elb.amazonaws.com:10200,es2node1.amazonaws.com:9200,es2node2.amazonaws.com:9200,10.20.30.40:9200,10.20.30.41:9200,10.20.30.42:9200"

     CODE
5. Restart the Events Service 23.x platform from Enterprise Console.

6. Create the application.yml file in the folder where you have downloaded the migration utility:

   Folder Structure

   /home/username/
   	- analytics-on-prem-es2-es8-migration-LATESTVERSION-exec.jar
   	- config
   		- application.yml

   CODE
   Click here to expand...

   Example

   es:
     # Target Elasticsearch hostname and port for the data migration
     #(Get this from $APPDYNAMICS_HOME/platform/events-service/processor/conf/events-service-api-store.properties ad.es.node.http.port from events-service 24.x)
     targetHostName: poc-3tb-console-1177482667.us-west-2.elb.amazonaws.com
     targetPort: 10200
     username: admin
     password: password
   
     # Source Elasticsearch hostname and port for the data migration
     #(Get this from $APPDYNAMICS_HOME/platform/events-service/processor/conf/events-service-api-store.properties ad.es.node.http.port from events-service 4.x)
     sourceHostName: poc-3tb-console-1177482667.us-west-2.elb.amazonaws.com
     sourcePort: 9200
   
     # Maximum number of retries per index during the migration 
     maxRetriesPerIndex: 3
   
     # Delay in milliseconds between each retry attempt
     delayBetweenRetry: 2000
   
     # Set this to true if you want to use an external version for iteration 2 and subsequent,
     # or false if you want to use an internal version for migration.
     useExternalVersion: true 
   
   clusters:
     source:
       api:
         keys:
           # Controller API key for the source Elasticsearch v2.x cluster
           #(Get this from $APPDYNAMICS_HOME/platform/events-service/processor/conf/events-service-api-store.properties ad.accountmanager.key.controller  from events-service 4.x)
           CONTROLLER: ba1140cb-ad2b-4a74-8f2f-2ba3648a99c5
   
           # OPS API key for the source Elasticsearch v2.x cluster
           #(Get this from $APPDYNAMICS_HOME/platform/events-service/processor/conf/events-service-api-store.properties ad.accountmanager.key.ops  from events-service 4.x)
           OPS: "ba339ff3-4244-4aa5-a458-acb66332aeb6"
   		
         # API hostname and port of the source events-service cluster
         # Get the port from the events-service-api-store.properties file ad.dw.http.port from events-service 4.x
   	  api: http://poc-3tb-console-1177482667.us-west-2.elb.amazonaws.com:9080               
   
         # Provide the absolute path of the certificate (if needed) for the source events-service cluster 
         cert: ""
         # Set this to true if you want to verify the hostname when creating the SSL context
         # or false to skip the hostname verification.
         check: false
   
       elasticsearch:
   		# URL and internal URL of the source Elasticsearch v2.x cluster
   		#(Get this from $APPDYNAMICS_HOME/appdynamics/platform/events-service/processor/conf/events-service-api-store.properties ad.es.node.http.port from events-service 4.x)
   		url: http://poc-3tb-console-1177482667.us-west-2.elb.amazonaws.com:9200
   		internal: http://poc-3tb-console-1177482667.us-west-2.elb.amazonaws.com:9200
   
   		# Elasticsearch version of the source cluster (v2.x in this case)
   		version: 2
   		protocol: http #Provide the scheme to connect to the elasticsearch Valid option : {http}
     
     destination:
       # URL and internal URL of the destination Elasticsearch v8.x cluster
       api:
        keys:
   		# URL and internal URL of the source Elasticsearch v2.x cluster
   		#(Get this from $APPDYNAMICS_HOME/appdynamics/platform/events-service/processor/conf/events-service-api-store.properties ad.es.node.http.port from events-service 4.x)
           CONTROLLER: ba1140cb-ad2b-4a74-8f2f-2ba3648a99c5
   
   		# OPS API key for the destination Elasticsearch v8.x cluster
           # Get this from $APPDYNAMICS_HOME/platform/events-service/processor/conf/events-service-api-store.properties ad.accountmanager.key.ops from events-service 24.x
           OPS: "ba339ff3-4244-4aa5-a458-acb66332aeb6"
   
   	  # API hostname and port of the destination events-service cluster (not Elasticsearch v8.x cluster)
   	  # Get the port from the events-service-api-store.properties file ad.dw.http.port from events-service 4.x 
   	  	api: "http://poc-3tb-console-1177482667.us-west-2.elb.amazonaws.com:10080"
   
         # Provide the absolute path of the certificate (if needed) for the source events-service cluster
         cert: ""
         # Set this to true if you want to verify the hostname when creating the SSL context
         # or false to skip the hostname verification.
         check: false
   
       elasticsearch:
         # URL and internal URL of the source Elasticsearch v2.x cluster
         #(Get this from $APPDYNAMICS_HOME/appdynamics/platform/events-service/processor/conf/events-service-api-store.properties ad.es.node.http.port from events-service 4.x)
         url: http://poc-3tb-console-1177482667.us-west-2.elb.amazonaws.com:10200
         internal: http://poc-3tb-console-1177482667.us-west-2.elb.amazonaws.com:10200
   
         # Elasticsearch version of the source cluster (v2.x in this case)
         version: 8
         protocol: http #Provide the scheme to connect to the elasticsearch Valid options : {http, https}
         trustStoreFile: "client-ca.cer"
         elasticsearchUsername: ""
         elasticsearchPassword: ""
   
   
   migration:
     # Number of search hits per request during migration,
     search_hits: 5000
   
     # Number of concurrent reindex requests during migration (based on CPU cores), maximum value recommended is 8.
     reindex_concurrency: 4
   
     # Batch size for scrolling through search results during migration, reduce this if you hit the exception : Remote responded with chunk size
     reindex_scroll_batch_size: 5000
   
     # Number of reindex requests per second during migration
     reindex_requests_per_second: 8000
   
     # Threshold size in bytes for large rollovers during migration (60GB in this case)
     # TO DO: Detailed property explanation
     large_rollover_threshold_size_bytes: 60000000000 # 60 GB
   
     # Threshold size in bytes for small rollovers during migration (20GB in this case)
     small_rollover_threshold_size_bytes: 20000000000 # 20 GB
   
     # Required low disk size watermark during migration (85% in this case)
     required_low_disk_size_watermark: 85
   
   #SQLite3 configuration for status monitoring
   spring:
     jpa:
       properties:
         hibernate:
           # Hibernate dialect for SQLite database
           dialect: org.hibernate.community.dialect.SQLiteDialect

   YML

Migrate Events Service Data Using Cutover Approach

1. Run the migration utility by keeping Events Service 4.5.x platform as the source and Events Service 23.x as the destination.
   

   1. Rollover the Events Service 4.5.x indices.

      nohup java -jar analytics-on-prem-es2-es8-migration-23.7.0-243-exec.jar rollover > rollover_output.log 2>&1 &

      CODE

   2. Migrate data and metadata from Events Service 4.5.x to 23.x.

      nohup java -jar analytics-on-prem-es2-es8-migration-LATESTVERSION-exec.jar data > data_output.log 2>&1 &
      nohup java -jar analytics-on-prem-es2-es8-migration-LATESTVERSION-exec.jar metadata > metadata_output.log 2>&1 &

      CODE

2. Migrate delta from Events Service 4.5.x to 23.x platform:

   nohup java -jar analytics-on-prem-es2-es8-migration-LATESTVERSION-exec.jar data > delta_data_output.log 2>&1 &
   nohup java -jar analytics-on-prem-es2-es8-migration-LATESTVERSION-exec.jar metadata > delta_metadata_output.log 2>&1 &

   CODE

   

   

   • We recommend that you repeat this step until delta in Events Service 4.5.x is minimum. Use logs to determine how much time each iteration takes to migrate the delta. If the total time taken for the iteration is significantly lesser than the previous iterations, it indicates that the delta in Events Service 4.5.x cluster has reduced. You can stop the iterations if the last two iterations take nearly the same time to migrate.
     For example, if the iteration takes 30 seconds where the previous iteration also took nearly 30 seconds, you can consider that the delta is at minimum.
   • Reduce the time gap between delta migration and post cutover migration. It ensures the data to migrate after cutover is minimum.

   2023-08-09T21:52:43.116+05:30  INFO 87453 --- [           main] c.a.analytics.onprem.MigrationTool       : Total time taken: 30 seconds

   CODE

   When the delta in your Events Service 4.5.x cluster is minimal, you can update the load balancer ports. Change only the instance port numbers in the load balancer configuration such that the Events Service 4.5.x traffic flows to Events Service 23.x. This way the incoming traffic for the Events Service 4.5.x will be redirected to Events Service 23.x platform. This starts the cutover phase.
   The following is an example configuration:
   

   Platform	Ports	Load Balancer Configuration
   		Load Balancer Port	Instance Port
   Events Service 23.x	10080, 10081	10080, 10081	10080, 10081
   Event Service 4.5.x	9080, 9081	9080, 9081	10080, 10081

   

   You may lose some data if you have active sessions or transactions.

3. Update the application.yml file by keeping the host name of any Events Service node as source and destination.
   The following is an example file:
   
   

   Click here to expand...

   es:
     targetHostName: <Node 1 IP Address>
     targetPort: 10200
     username: admin
     password: password
     sourceHostName: <Node 1 IP Address>
     sourcePort: 9200
     maxRetriesPerIndex: 3
     delayBetweenRetry: 2000
     useExternalVersion: true # Set this to true, if you want to use external version to be used for iteration 2 and subsequent. False, if you want to use internal
   
   clusters:
     source:
       api:
         keys:
           CONTROLLER: ba1140cb-ad2b-4a74-8f2f-2ba3648a99c5
           OPS: "ba339ff3-4244-4aa5-a458-acb66332aeb6"
         api: http://<Node 1 IP Address>:9080
         cert: ""
         check: false
       elasticsearch:
         url: http://<Node 1 IP Address>:9200
         internal: http://<Node 1 IP Address>:9200
         version: 2
         protocol: http #Provide the scheme to connect to the elasticsearch Valid option : {http}
     destination:
       api:
         keys:
           CONTROLLER: ba1140cb-ad2b-4a74-8f2f-2ba3648a99c5
           OPS: "ba339ff3-4244-4aa5-a458-acb66332aeb6"
         api: "http://<Node 1 IP Address>:10080"
         cert: ""
         check: false
       elasticsearch:
         url: http://<Node 1 IP Address>:10200
         internal: http://<Node 1 IP Address>:10200
         version: 8
         protocol: http #Provide the scheme to connect to the elasticsearch Valid options : {http, https}
         trustStoreFile: "client-ca.cer"
         elasticsearchUsername: ""
         elasticsearchPassword: ""
   
   migration:
     search_hits: 5000
     reindex_concurrency: 4
     reindex_scroll_batch_size: 5000
     reindex_requests_per_second: 8000
     large_rollover_threshold_size_bytes: 60000000000 # 60 GB
     small_rollover_threshold_size_bytes: 20000000000 # 20 GB
     required_low_disk_size_watermark: 85
   
   spring:
     jpa:
       properties:
         hibernate:
           dialect: org.hibernate.community.dialect.SQLiteDialect

   CODE

4. Run the following commands to migrate the post cutover data from Events Service 4.5.x to 23.x.

   nohup java -jar analytics-on-prem-es2-es8-migration-LATESTVERSION-exec.jar post_cutover_metadata > post_cutover_metadata_output.log 2>&1 &
   nohup java -jar analytics-on-prem-es2-es8-migration-LATESTVERSION-exec.jar post_cutover_data > post_cutover_data_output.log 2>&1 &

   CODE

5. Enable the the INDEX_COMPACTION_PARENT_JOB on Events Service 23.x

   curl -H 'Content-type: application/json' -XPOST 'http://localhost:9200/job_framework_job_details_v1/_update_by_query' -d '{
    "query": {
     "term": {
      "name": "INDEX_COMPACTION_PARENT_JOB"
     }
    },
    "script": {
     "inline": "ctx._source.jobDataMap.enabled = true",
     "lang": "painless"
    }
   }'

   CODE

Restore the events-service-api-store.properties file that you have backed up during the preparation phase.

• Prepare the Events Service for Migration using node
• Prepare the Events Service for Migration using Load Balancer

The Events Service 23.x platform is now synchronized. Verify the Migration status and if the migration is successful, you may delete the Events Service 4.5.x platform.