During a rolling upgrade to Neo4j 5.25, one of the nodes (core1) went into a "dirty" state and failed to join the cluster due to a mismatch in block storage format between nodes. 

Error

2024-12-16 05:40:49.001+0000 INFO  [c.n.d.r.DbmsReconciler] Database 'neo4j' is requested to transition from DIRTY{db=neo4j/ab04cb8b} to NOT_HERE{db=neo4j/bb43a13d} by SystemGraph:StoreReplaced:101 (requestCount:1)
2024-12-16 05:40:49.003+0000 ERROR [c.n.d.r.DbmsReconciler] Encountered error when attempting to reconcile database neo4j (request by SystemGraph:StoreReplaced:101 (requestCount:1)) to state 'not allocated here', database remains in state 'dirty'
java.util.NoSuchElementException: No value present

 

Cause

The issue was caused by running rpm -ivh instead of rpm -uvh, which resulted in installing a new Neo4j server with a different block storage format. This created a mismatch between nodes, causing core1 to go into a "dirty" state and fail to join the cluster.

Fix

To resolve the issue:

1. Stop all cores.
2. Upgrade all instances.
3. Take a dump of the system and neo4j(user) database from another node (core2 or core3).
4. Restore the dump on core1.
5. Start all cores.
6. Enable the new server core1.
7. Drop the old server core1.

Prevention

To prevent similar issues in the future, ensure that rpm -uvh is used for upgrades to avoid installing a new Neo4j server with a different block storage format during the rolling upgrade