This page explains the dot field errors that occurs during the Events Service migration from 4.5.x to 24.x or later.

Analyze the migration logs to determine the reasons for migration failure. While you troubleshoot this issue, you may experience partial data loss. However, follow the recommended troubleshooting steps to select the important data for migration.

Issue

Elasticsearch 2.x and 8.x have the different styles of mapping the indexes that have dot fields. The mapping style is changed from the Elasticsearch 5.x release. For more information, see Upgrading fields with dots to 5.x. Therefore, the migration utility fails to reindex the dot fields.

When you run the migration utility, the migration fails with the following error:

Sample Error Message

2024-04-18T19:51:00.177+05:30 ERROR 71165 --- [           main] c.a.a.o.m.m.t.DotFieldIndicesHandlerTask : Execution of DotFieldIndicesHandlerTask has failed.
2024-04-18T19:51:00.177+05:30 ERROR 71165 --- [           main] c.a.a.o.m.s.r.RemoteReindexerServiceImpl : Error while reindexing
 
java.lang.IllegalStateException: Not all accounts & eventTypes have been resolved/rectified, Un-rectified accountEventTypes : [customer1_846e8755-1ace-4087-9c72-1ed30749ada6___biz_txn_v1]
  at com.appdynamics.analytics.onprem.migration.model.tasks.DotFieldIndicesHandlerTask.execute(DotFieldIndicesHandlerTask.java:88) ~[classes!/:24.4.0-18]
  at com.appdynamics.analytics.onprem.migration.service.reindex.RemoteReindexerServiceImpl.reindex(RemoteReindexerServiceImpl.java:116) ~[classes!/:24.4.0-18]
  at com.appdynamics.analytics.onprem.MigrationTool.run(MigrationTool.java:105) ~[classes!/:24.4.0-18]
  at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:771) ~[spring-boot-3.1.1.jar!/:3.1.1]
  at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:755) ~[spring-boot-3.1.1.jar!/:3.1.1]
  at org.springframework.boot.SpringApplication.run(SpringApplication.java:319) ~[spring-boot-3.1.1.jar!/:3.1.1]
  at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:150) ~[spring-boot-3.1.1.jar!/:3.1.1]
  at com.appdynamics.analytics.onprem.MigrationTool.main(MigrationTool.java:59) ~[classes!/:24.4.0-18]
  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[na:na]
  at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
  at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[na:na]
  at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) ~[analytics-on-prem-es2-es8-migration-24.4.0-18-exec.jar:24.4.0-18]
  at org.springframework.boot.loader.Launcher.launch(Launcher.java:95) ~[analytics-on-prem-es2-es8-migration-24.4.0-18-exec.jar:24.4.0-18]
  at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) ~[analytics-on-prem-es2-es8-migration-24.4.0-18-exec.jar:24.4.0-18]
  at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65) ~[analytics-on-prem-es2-es8-migration-24.4.0-18-exec.jar:24.4.0-18]
JSON

This error indicates that some indexes fail to migrate from Elasticsearch 2.x to 8.x.

Cause

The index mappings in Elasticsearch 2.x is different  from the Elasticsearch 8.x. The following is an example mapping in the Elasticsearch 2.x:

Example: Elasticsearch 2.x Index Mapping

{
    "customer1_846e8755-1ace-4087-9c72-1ed30749ada6___biz_txn_v1___2023-12-09_09-50-05": {
        "mappings": {
            "customer1_846e8755-1ace-4087-9c72-1ed30749ada6___biz_txn_v1": {
                "properties": {
                    "segments": {
                        "type": "nested",
                        "properties": {
                            "userData": {
                                "properties": {
                                    "RequestApplicationSecurityToken": {
                                        "type": "string",
                                        "index": "not_analyzed"
                                    },
                                    "RequestApplicationSecurityToken.ApplicationContext": {
                                        "type": "string",
                                        "index": "not_analyzed"
                                    },
                                    "RequestApplicationSecurityToken.AppliesTo": {
                                        "type": "string",
                                        "index": "not_analyzed"
                                    },
                                    "RequestApplicationSecurityToken.NameId": {
                                        "type": "string",
                                        "index": "not_analyzed"
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}
JSON

In this example, Elasticsearch 2.x compiles the mapping and identifies each of the following properties as an independent field:

  • RequestApplicationSecurityToken

  • RequestApplicationSecurityToken.ApplicationContext

  • RequestApplicationSecurityToken.AppliesTo

  • RequestApplicationSecurityToken.NameId

But, the Elasticsearch 8.x version does not consider the mapping in the same way. This version identifies the RequestApplicationSecurityToken field as the parent. And, it identifies the RequestApplicationSecurityToken.* fields as the children. In the given example, RequestApplicationSecurityToken.ApplicationContext, RequestApplicationSecurityToken.AppliesTo, RequestApplicationSecurityToken.NameId are considered child fields. Therefore, the reindexing of these fields fails during migration.

In Elasticsearch 8.x, the parent field cannot have any value. It can be either nested or object or properties. However, the Elasticsearch 2.x version allowed the same field to have strings. 

Workaround

To troubleshoot the dot field errors:

  1. Locate the file that list the problematic fields in the out directory.

    Example Files

    - out/ProblematicFieldsReport_2024-04-18_19-51-00.json
- out/customer1_846e8755-1ace-4087-9c72-1ed30749ada6___biz_txn_v1___2023-12-09_09-50-05.json
    CODE
  2. Inspect the /ProblematicFieldsReport_2024 file for the failed indexes:

    Example File

    {
  "customer1_846e8755-1ace-4087-9c72-1ed30749ada6___biz_txn_v1" : {
    "relatedIndices" : [
      "customer1_846e8755-1ace-4087-9c72-1ed30749ada6___biz_txn_v1___2023-12-09_09-50-05"
    ],
    "problematicFields" : [
      "segments.userData.RequestApplicationSecurityToken",
      "segments.userData.RequestApplicationSecurityToken.ApplicationContext",
      "segments.userData.RequestApplicationSecurityToken.AppliesTo",
      "segments.userData.RequestApplicationSecurityToken.NameId"
    ]
  }
}
    JSON

    This JSON file lists the accountName___eventType. Each list has the following entries:
    • relatedIndices - The list of indexes that have the Elasticsearch 2.x mapping style.
    • problematicFields - The list of problematic fields for the index.

      The migration utility cannot cover all the reserved keywords. So, you may experience issues when the utility fails to process the unknown reserved keywords. To resolve this issue, see Ignore the Reserved Keywords.

  3. Update the mapping to match the Elasticsearch 8.x style.

    Example update for the Elasticsearch 8.x

    {
  "mappings" : {
    "properties" : {
      "segments" : {
        "type" : "nested",
        "properties" : {
          "userData" : {
            "properties" : {
              "RequestApplicationSecurityToken" : {
                "type" : "keyword",
                "index" : true
              },
              "RequestApplicationSecurityToken.ApplicationContext" : {
                "type" : "keyword",
                "index" : true 
              },
              "RequestApplicationSecurityToken.AppliesTo" : {
                "type" : "keyword",
                "index" : true
              },
              "RequestApplicationSecurityToken.NameId" : {
                "type" : "keyword",
                "index" : true
              }
            }
          }
        }
      }
    }
  }
}
    JSON
  4. Review and decide on how to fix the mapping file.

    Ensure to keep either parent or child fields only. Because, the migration utility does not support the combination of parent and child fields.

    For example, if the Elasticsearch 2.x has the data in following structure:

    Example Source File

    "_source": {
    "segments": [
        {
            "userData": {
                "RequestApplicationSecurityToken": "example_token_value1",
                "RequestApplicationSecurityToken.ApplicationContext": "example_application_context1",
                "RequestApplicationSecurityToken.AppliesTo": "example_applies_to1",
                "RequestApplicationSecurityToken.NameId": "example_name_id1"
            }
        }
    ]
}
    JSON
    The data in Elasticsearch 8.x appears after fixing the dot field errors.

    If you choose to remove parent fields, the data in Elasticsearch 8.x appears as follows:

    Sample Output

    "_source": {
    "segments": [
        {
            "userData": {
                "RequestApplicationSecurityToken.ApplicationContext": "example_application_context1",
                "RequestApplicationSecurityToken.AppliesTo": "example_applies_to1",
                "RequestApplicationSecurityToken.NameId": "example_name_id1"
            }
        }
    ]
}
    JSON

    If you choose to remove child fields, the data in Elasticsearch 8.x appears as follows:

    Sample Output

    "_source": {
    "segments": [
        {
            "userData": {
                "RequestApplicationSecurityToken": "example_token_value1"
            }
        }
    ]
}
    JSON



     Before you decide the approach to fix the mapping file, analyze the data.
  5. Analyze the parent and child fields to restore them based on your business requirements:

    Consider the following points to analyze the significance of parent and child fields for your business:

    • What type of data is important for the business after the migration?
    • What type of data you require in the future?
    • How many documents contain the parent field?
    • How many documents contain the child fields?

    Based on your analysis, decide which field to keep or lose after migration.



    GET /customer1_846e8755-1ace-4087-9c72-1ed30749ada6___biz_txn_v1___2023-12-09_09-50-05/_search
{
  "_source": ["segments.userData.RequestApplicationSecurityToken"]
}
    CODE 


    GET /customer1_846e8755-1ace-4087-9c72-1ed30749ada6___biz_txn_v1___2023-12-09_09-50-05/_search
{
    "_source": ["segments.userData.RequestApplicationSecurityToken.ApplicationContext",
      "segments.userData.RequestApplicationSecurityToken.AppliesTo",
      "segments.userData.RequestApplicationSecurityToken.NameId"]
}
    CODE


  6. Create the excludeFieldsConfig.json file in the migration config folder. In this file, specify the fields that you decided to ignore.

    Sample excludeFieldsConfig.json

    {
    "customer1_846e8755-1ace-4087-9c72-1ed30749ada6___biz_txn_v1": [
        "segments.userData.RequestApplicationSecurityToken"
    ],
    "accountName___eventType2": [
      "excludefield1"
    ]
}
    JSON
  7. In the migration config/application.yml file, add this property under the migration section:
    migration:
  search_hits: 5000
  reindex_concurrency: 4
  reindex_scroll_batch_size: 5000
  reindex_requests_per_second: 8000
  large_rollover_threshold_size_bytes: 60000000000 # 60 GB
  small_rollover_threshold_size_bytes: 20000000000 # 20 GB
  required_low_disk_size_watermark: 85

  # Exclude Parent of Child fields during migration
  exclude_fields_config_file_path: "config/excludeFieldsConfig.json"
    CODE
  8. Empty the /out directory and run the migration utility tool.

Remove the Reserved Keywords

The migration utility cannot cover all the reserved keywords. If you have reserved keywords that are missed by the utility, analyze the fields in the problematic fields report. Identify such missing keywords and update the application.yml file to skip them during migration.

The following is an example file that has the reserved keywords that are missed in the migration utility.

Example Mapping File

"type": "nested",
"properties": {
    "userData": {
        "properties": {
            "RequestApplicationSecurityToken": {
                "type": "string",
                "index": "not_analyzed"
            },
            "RequestApplicationSecurityToken.ApplicationContext": {
                "type": "string",
                "index": "not_analyzed"
            },
            "RequestApplicationSecurityToken.AppliesTo": {
                "type": "string",
                "index": "not_analyzed"
            },
            "RequestApplicationSecurityToken.NameId": {
                "type": "string",
                "index": "not_analyzed"
            }
        }
    }
JSON


  1. To exclude the reserved keywords, locate the problematic fields file in the out directory.

    Example Files

    - out/ProblematicFieldsReport_2024-04-18_19-51-00.json
- out/customer1_846e8755-1ace-4087-9c72-1ed30749ada6___biz_txn_v1___2023-12-09_09-50-05.json
    CODE
  2. Inspect the /ProblematicFieldsReport_2024 file for the failed indexes.
    The report includes the following fields:

    Missing Reserved Keywords

    RequestApplicationSecurityToken.type.ApplicationContext.type
RequestApplicationSecurityToken.type.ApplicationContext.index
RequestApplicationSecurityToken.type.ApplicationContext.norms
    JSON
    This example report has the reserved keywords such as type, index, and norms.
  3. Update the application.yml file to skip the reserved keywords:
    1. Create a CSV file in the migration config folder. For example, skipFields.csv.
    2. Include the reserved keywords in the CSV format:

      skipFields.csv

      type,index
      CODE
    3. In the migration config/application.yml file, add this property under the migration section:
      migration:
  search_hits: 5000
  reindex_concurrency: 4
  reindex_scroll_batch_size: 5000
  reindex_requests_per_second: 8000
  large_rollover_threshold_size_bytes: 60000000000 # 60 GB
  small_rollover_threshold_size_bytes: 20000000000 # 20 GB
  required_low_disk_size_watermark: 85

  # Exclude Parent of Child fields during migration
  exclude_fields_config_file_path: "config/excludeFieldsConfig.json"
  skip_fields_config_file_path: "config/skipFields.csv"
      CODE

If the issue still persists, the excludeFieldsConfig.json has the invalid configuration. Complete the following steps to update the configuration:

  1. Stop the process by using the kill command.
  2. Inspect the mapping file for errors.
  3. Update the excludeFieldsConfig.json file.
  4. Empty the /out directory and run the migration utility tool again.