How to rebuild an index after node failure due to failed Import

This article offers guidance on troubleshooting failed migrations that result in index corruption.

Configuration Manager for Jira (CMJ) causes index corruption due to failed import.

Rebuilding an index without performing a full restore from backup following node failure largely depends on the specific database or search engine being utilized. However, there are general steps that are typically followed:

1. Identify the failed node: Determine which node has failed and verify the status of the index.

2. Bring the node back Online: If recoverable, return the node online, ensuring access to necessary data and configuration files.

3. Check the replication status: If your system uses replication or sharding, ensure proper data replication across other nodes; manually sync data if replication is inactive.

4. Perform index recovery: Some databases or search engines have built-in mechanisms for automatic shard recovery upon node restart (e.g., Elasticsearch).

5. Rebuild index segments: If needed, manually intervene to rebuild index segments by reindexing data or merging existing segments.
   In Jira, rebuilding index segments is usually necessary when you encounter issues related to indexing, such as search problems or inconsistencies in data retrieval. Here's a general guide on how to rebuild index segments in Jira:

   1. Access Jira Administration: Log in to your Jira instance with administrative privileges.

   2. Navigate to Indexing: Go to the Jira Administration area. Depending on your Jira version, the exact location of the indexing settings may vary slightly, but typically, you'll find it under "System" or "System Administration."

   3. Find Indexing Options: Look for options related to indexing. In older versions of Jira, this might be found under "Indexing" or "Advanced Settings." It might be under "System > Indexing" or similar in newer versions.

   4. Trigger Index Rebuild: Look for an option to trigger a full index rebuild or to rebuild specific index segments. This option may be labeled differently depending on your Jira version. For example, it might be called "Rebuild Index" or "Reindex."

   5. Choose Segments to Rebuild (if applicable): If your Jira instance allows you to rebuild specific index segments, choose the segments you want to rebuild. Otherwise, proceed with rebuilding the entire index.

   6. Start Rebuilding: Once you've selected the appropriate options, start the index rebuilding process. Depending on the size of your Jira instance and the complexity of your data, this process may take some time.

   7. Monitor Progress: Monitor the progress of the index rebuild. Some Jira instances provide a progress indicator or log messages to help you track the process.

   8. Verify Completion: Once the index rebuild process is complete, verify that it finished successfully. Check for any error messages or warnings that might indicate issues during the rebuild process.

   9. Test Search Functionality: After rebuilding the index, it's a good idea to perform some searches in Jira to ensure that the search functionality works as expected and that all data is retrieved correctly.

   10. Regular Maintenance: Implement regular index maintenance tasks to keep your Jira instance running smoothly. This might include periodic index rebuilds or optimizations to improve performance and reliability.

   It's worth noting that the exact steps and options for rebuilding index segments may vary depending on your specific Jira version and configuration. Always refer to the official Jira documentation or consult with your system administrator for the most accurate instructions tailored to your environment.

6. Monitor and verify: Once rebuilt or recovered, monitor its status to ensure full functionality; conduct validation checks or queries against the index's integrity as required.

7. Update cluster metadata: If applicable, update cluster metadata to reflect the recovered index and ensure proper coordination within the cluster.
   Updating cluster metadata in Jira typically involves modifying the cluster.properties file, which contains configuration settings for Jira clustering. Here's a general outline of the steps to update cluster metadata in Jira:

   1. Access the Jira server: Log in to the server where Jira is installed using appropriate credentials.

   2. Locate the cluster.properties file: The cluster.properties file is in the Jira home directory under the 'shared' folder. Navigate to the appropriate directory where Jira is installed.

   3. Backup the cluster.properties file: Before making any changes, it is always a good practice to create a backup of the file. This ensures you can revert to the previous state if anything goes wrong.

   4. Edit the cluster.properties file: Open the cluster.properties file in a text editor. You can use tools like vi, nano, or Notepad, depending on your operating system.

      cluster.properties # This ID must be unique across the cluster jira.node.id = node1 # The location of the shared home directory for all Jira nodes jira.shared.home = /var/atlassian/jira/home

   5. Update cluster metadata: In Jira Data Center, cluster metadata typically refers to information about the nodes within the cluster and their configurations. Here's an example of what cluster metadata might look like:

      Cluster Name: MyJiraCluster Node 1: - Node ID: 123456789 - IP Address: 192.168.1.101 - Status: Active - Role: Master - Hardware: 8 CPU cores, 16GB RAM Node 2: - Node ID: 987654321 - IP Address: 192.168.1.102 - Status: Active - Role: Worker - Hardware: 4 CPU cores, 8GB RAM Node 3: - Node ID: 246813579 - IP Address: 192.168.1.103 - Status: Standby - Role: Worker - Hardware: 4 CPU cores, 8GB RAM Load Balancer: - Type: Round Robin - IP Address: 192.168.1.100 - Active Nodes: Node 1, Node 2 Database: - Type: PostgreSQL - Version: 12 - Connection String: jdbc:postgresql://db.example.com:5432/jira - Replication: Active-Passive with Node 1 as the primary Shared File System: - Type: NFS - Mount Point: /mnt/jira/shared - Status: Connected

      
      Look for the section where cluster metadata is defined. This section usually contains properties related to the cluster name, node ID, IP addresses, and other relevant information. Modify the values according to your requirements. This metadata provides an overview of the cluster setup, including node details, roles, hardware specifications, load balancer configuration, database setup, and shared file system information. It helps administrators understand the architecture and health of the Jira Data Center deployment.

   6. Save the changes: After updating the cluster metadata, save the changes to the cluster.properties file.

   7. Restart Jira: Restart the Jira service to apply the changes. This ensures that Jira reads the updated cluster metadata from the cluster.properties file.

   8. Verify the changes: After restarting Jira, verify that the cluster metadata has been updated correctly. You can access the Jira application and check the cluster settings.

   It's important to note that modifying configuration files directly can cause issues if not done correctly. Always ensure that you have a backup of the files and double-check your changes before restarting Jira. Additionally, it's recommended that such operations be performed during scheduled maintenance windows to minimize disruption to users.

8. Optimize performance: Consider optimizing performance after rebuilding by tuning settings such as shard allocation, replica settings, or index mappings.