How to fix corrupted indexes in Neo4j

In this article, we will explore how to fix problems related to a corrupted index and provide step-by-step solutions to resolve it in Neo4j 4.4.x.

Below is an error message related to a corrupted index: 

java.io.IOException: Failed to flush index updates at org.neo4j.internal.recordstorage.BatchContextImpl.applyPendingIndexUpdates(BatchContextImpl.java:101) ~[neo4j-record-storage-engine-4.4.6.jar:4.4.6] at org.neo4j.internal.recordstorage.BatchContextImpl.close(BatchContextImpl.java:77) ~[neo4j-record-storage-engine-4.4.6.jar:4.4.6] at org.neo4j.internal.recordstorage.RecordStorageEngine.apply(RecordStorageEngine.java:511) ~[neo4j-record-storage-engine-4.4.6.jar:4.4.6] ... 33 more Caused by: java.util.concurrent.ExecutionException: org.neo4j.index.internal.gbptree.TreeInconsistencyException: GSPP READ failure Pointer[type=CHILD_-1, offset=60, nodeId=67804] Pointer state A: CRASH GSP[generation=147783680, pointer=9685151252480, checksum=0(OK)] Pointer state B: BROKEN GSP[generation=146735104, pointer=9616431775744, checksum=642(NOT OK)] stableGeneration=2253, unstableGeneration=2255 Generations: A > B | GB+Tree[file:/data/databases/neo4j/schema/index/native-btree-1.0/442/index-442"

This error suggests that the index with ID 442 is corrupted and should be recreated or re-populated.

You can show detailed index information using

SHOW INDEXES YIELD *

This will display the full list of indexes from which you can find which index is related to the corrupted index. Search for the index ID shown in the error:

Below is an example on how to find the name corresponding to an index with ID 7:

What are the causes of index corruption?

Index corruption in Neo4j can occur due to various reasons, including:

1. Abrupt Termination: If the Neo4j Database process is forcefully terminated or crashes unexpectedly during heavy write operations, it can lead to index corruption. When the process is abruptly halted, index updates may not be properly flushed to disk, resulting in inconsistencies and potential corruption.

2. Disk or Hardware Failure: Physical issues with the disk where the index data is stored or hardware failures can also contribute to index corruption. Disk errors, bad sectors, or faulty hardware components can disrupt the integrity of the index files, causing them to become corrupted.

3. Other reasons such as: Software Bugs, Operating System or File System Errors, could also cause index corruption.

How do we fix index corruption?

To resolve this problem, we will discuss two options: re-populating the index and recreating the index.

Option 1: Re-populate the Index:

To begin, it's necessary to stop the Neo4j server and delete the index directory associated with the corrupted index. The index directory path usually looks like "/data/databases/<dbname>/schema/index/native-btree-1.0/442".

Deleting this directory ensures that the server will repopulate the index when it restarts.

During the repopulation process, it's important to note that the system will be unable to handle requests. You can monitor the progress of the index repopulation by checking the size of the index in the debug.log file. Search for the index name 442, and the log will display information such as size and timestamp.

Option 2: Recreate the Index:

In this scenario, you will need to drop the corrupted index using the index name obtained from the "SHOW INDEXES" command. Execute the command "DROP INDEX index_name" while replacing "index_name" with the actual name of the index.

After dropping the index, you can recreate it using the "createStatement" obtained from the "SHOW INDEXES YIELD *". The "createStatement" will provide a Cypher statement used to create the index.

It's important to truncate the statement from the keyword "OPTIONS" onwards:

CREATE BTREE INDEX `index_5c0603ad` FOR (n:`Person`) ON (n.`name`)

During the process of recreating the index, the Neo4j server will remain functional and capable of handling requests. However, the performance may be affected because Cypher queries will no longer benefit from index lookups and will instead require a node scan, which can be slower.

After successfully re-creating the index, you should verify its state by using the "show indexes" command. The index state should be displayed as "ONLINE," indicating that it is active and operational.

To check the overall state of your database, you can use the "show databases" command. If the database is still in quarantine mode, you can exit this mode by executing the command "CALL dbms.quarantineDatabase("database_name", false)" against the system database. To access the system database, use the command ":use system" before executing the quarantine command.