Article DetailsDetailsHelp Details This document aims to describe segment files used in Lucene indexes, some regularly seen exceptions arising due to spurious segment files and how to fix or rebuild both implicit and explicit Lucene indexes. Active segments in the index are stored in the segment info file, segments_N. There may be one or more segments_N files in the index, however, the one with the largest generation is the active one (when older segments_N files are present it’s because they temporarily cannot be deleted, or, a writer is in the process of committing, or a custom IndexDeletionPolicy is in use). This file lists each segment by name, has details about the separate norms and deletion files, and also contains the size of each segment. Read further here: (https://lucene.apache.org/core/3_0_3/fileformats.html#Segments File) Lets explore a couple of examples where segment file formats have halted upgrades and Neo4j server initialisation. Exception 1: Occassionally, some old or unrequired segment.id files present within the Lucene index directory, may prevent an upgrade to proceed, throwing the error similar to the following: ERROR [o.n.k.i.s.p.LegacyIndexMigrator] Migration of legacy indexes failed. Index: idx19 can't be migrated. Migration of legacy index at path:/opt/maaii/data/neo4j/upgrade/index/lucene/node/idx19 failed. org.neo4j.upgrade.lucene.LegacyIndexMigrationException: Migration of legacy index at path:/opt/maaii/data/neo4j/upgrade/index/lucene/node/idx19 failed. Caused by: org.apache.lucene.index.IndexFormatTooOldException: Format version is not supported (resource BufferedChecksumIndexInput(MMapIndexInput(path="/opt/maaii/data/neo4j/upgrade/index/lucene/node/idx19/segments_1"))): -11 (needs to be between 1071082519 and 1071082519). This version of Lucene only supports indexes created with release 4.0 and later In this case, the segment files were left over from some spurious (broken, deleted) indexes which were never used and rather only encountered as a result of an upgrade. Such files may not be required at present and can safely be deleted from e.g. /opt/<user>/data/neo4j/upgrade/index/lucene/node/<manual index name>/segments* file. Whilst experiencing an instance failure to initialise following an upgrade due to this issue, recovery would simply be to validate that your current indexes defined are as expected and that the query performance has not degraded by deleting old corrupt indexes and replacing them by the automatically generated schema indexes or manually creating explicit indexes ([] graph.db/index is the explicit indexes, and they cannot be automatically rebuilt). Deletion of explicit indexes is only advised if you know the content of those indexes and can confidently recreate them later manually. So the way to go about this would be to take a backup of indexes elsewhere, delete the index files, and then proceed with the upgrade/migrations without them, which should now run smoothly. Such indexes will have only "segment_*" . To rebuild, you need to know what that index is based on, e.g. which property. You can then create an explicit index manually using the procedure described in the Neo4j Operations Manual below: Restart migration/upgrade. If you encounter a similar error but for a different index, remove the /opt/<user>/data/neo4j/upgrade/index/lucene/node/<manual index name>/segments* file. Record the name of the <manual index name> for it will need to be rebuilt post successful migration. Repeat steps 2-3 Once migration is clean, rebuild the manual indexes which resulted in error for steps 1-3. https://neo4j.com/docs/cypher-manual/current/indexes/semantic-indexes/ https://neo4j.com/docs/java-reference/current/java-embedded/indexes/ Exception 2: Some unrequired segment.id files present within the Lucene index directory may prevent the Neo4j process from starting up, throwing exceptions similar to the following: INFO [org.neo4j]: Startup failed: Component 'org.neo4j.kernel.extension.KernelExtensions@3b9771a5' was successfully initialized, but failed to start. Please see attached cause exception.: Component 'org.neo4j.index.lucene.LuceneKernelExtension@76c2058b' was successfully initialized, but failed to start. Please see attached cause exception.: Component 'org.neo4j.index.impl.lucene.LuceneDataSource@2c4f00de' was successfully initialized, but failed to start. Please see attached cause exception.: org.apache.lucene.index.CorruptIndexException: doc counts differ for segment _ps: fieldsReader shows 995 but segmentInfo shows 745: doc counts differ for segment _ps: fieldsReader shows 995 but segmentInfo shows 745 In this instance, the explicit Lucene index encounters an excepion. The indexes are auxiliary Lucene indexes that were usually populated with the same contents as the nodes, but there’s no actual enforced relation between them, so they cannot be rebuilt automatically and hence will either have to be fixed by attempting to use Lucene’s own repair tool which is explained here: https://solr.pl/en/2011/01/17/checkindex-for-the-rescue/ A sample command would be: java -ea:org.apache.lucene... org.apache.lucene.index.CheckIndex pathToIndex [-fix] [-verbose] [-segment X] [-segment Y] Else an easier way is usually to rebuild that lucene index manually. The following articles provide procedures for rebuilding explicit indexes and running a CheckIndex operation to verify their integrity: References: https://neo4j.com/docs/java-reference/current/java-embedded/indexes/https://neo4j.com/docs/cypher-manual/current/indexes/semantic-indexes/https://solr.pl/en/2011/01/17/checkindex-for-the-rescue/ https://lucene.apache.org/core/3_0_3/fileformats.html#SegmentsFileOther Lucene Index Articles InformationTitleFixing Lucene Segment File ExceptionsArticle Created Date3/15/2024, 10:56 PMURL Name360000710547-Fixing-Lucene-Segment-File-ExceptionsLast Modified Date6/17/2026, 3:40 PM