Article DetailsDetailsHelp DetailsThis article aims to describes two types of property record consistency issues and discuss options for subsequent recovery. Upon running a consistency check on a database, the inconsistency report may report an inconsistent property chain, such as the one below: ERROR The next property record does not have this record as its previous record. Property[27560311,used=true,prev=-1,next=27560310, (blocks not loaded)] Inconsistent with: Property[27560310,used=true,prev=-1,next=27560309, (blocks not loaded)] This indicates a semi applied transaction, without subsequent recovery, i.e. a first property was inserted in the chain contained within the data block and that record was valid, but the next property insertion attempt failed the recovery part and was never fully updated. Later updates to this chain (additional property updates) had to reference of the first property being the previous object in the chain, thus resulting in an inconsistency. Such inconsistencies may result from the Neo4j instance running completely out of disk space, or, OS snapshots performed whilst write transactions may still be running in the background. Inconsistencies such as the above can normally be recovered from by creating a copy of the database, using the neo4j-admin copy tool. Upon successful completion of the copy process, a subsequent consistency check of the database should not report inconsistent property chains as above. Highlighted below is another category of a somewhat "inconsistent" property record, which in fact is a duplication rather than an inconsistency, but nonetheless, is reported by the current versions of the consistency checker tool, in the inconsistency report. Whilst updates to the neo4j-admin copy tool are currently underway to address such duplicates (possibly removing first of the duplicate, followed by re-running store validation), it may be useful to be able to identify the reported error as well as having a way to manually addressing it. The error in question is reported as: ERROR The property chain contains multiple properties that have the same property key id, which means that the entity has at least one duplicate property. Node[7046288,used=true,created=false,rel=54890131,prop=107473277,labels=Inline(0x720a418820:[0, 1, 2, 3, 4, 5, 8]),light] or in case of duplicate relationship property keys and/or values, we might see a report such as: ERROR The property chain contains multiple properties that have the same property key id, which means that the entity has at least one duplicate property. Relationship[6144062,used=true,source=1057197,target=1056917,type=4,sPrev=6144192,sNext=6136281,tPrev=6145946,tNext=6144014,prop=62805816,!sFirst,!tFirst] Above are similar errors, indicating that for a single node (e.g. Node ID 7046288 in the above example), there are multiple values for the same property key(as properties are key value pairs), for example node.prop = 2 AND node.prop=3. Same description holds in the second of the above highlighted examples, with a property of the Relationship ID 6144062. One way to manually identify, count and remove such duplicate objects is by using the parent Node or Relationship ID (as the reported case may be) in a cypher query to match the node e.g. as: match (p) where id(p) in [1,2,3,…] return properties(p), id(p) where [1,2,3] in the above example, is a list of Node IDs extracted from the inconsistency report. In cases where numerous Node IDs are reported, it may help to parse the inconsistency report e.g. as: awk -F'[' '/ *Node\[/ {print $2}' /Users/user/Desktop/inconsistency.report where "/Users/user/Desktop/inconsistency.report" is the full path to the report file. Similarly, relationships with duplicate properties can be matched as: match (a)-[r]-(b) where id(r) in [1,2,3,…] return properties(r), id(r), a, b and parsed as above to extract a list of IDs for later use in a cypher deletion query. If removing/updating the property rather than the parent Node/Relationship, is preferred, it can be achieved in the above example below to remove the property from all nodes reporting duplicates, as: match (p) where id(p) in [1,2,3] REMOVE p.property return p limit 1 A uniqueness constraint should be added to the node/relationship property at this point to avoid future duplications. Properties removed as above may then be re-inserted as: MERGE (p:Person {property: 'value'}) ON CREATE SET p.property = <value> RETURN p.property Mapping previously deleted property values to the relevant Node labels may pose a challenge with this approach, especially for large number of duplicates. One option is to set such properties using a parameter. This is less of a challenge if deleting the parent Nodes/Relationships is an option e.g. one can get a count of such Nodes as: MATCH (p:Person) USING INDEX p:Person(property) WHERE p.property IS NOT NULL WITH p.property as property, count(p) as count, collect(p) as nodes WHERE count > 1 RETURN count(nodes) as total_nodes A similar statement can then be executed to delete the above nodes in batched transactions e.g. as: MATCH (p:Person) USING INDEX p:Person(property) WHERE p.property IS NOT NULL WITH p.property as property, count(p) as count, collect(p) as nodes WHERE count > 1 CALL { WITH nodes UNWIND nodes as node DETACH DELETE node } IN TRANSACTIONS OF 10 ROWS RETURN count(nodes) as total_nodes Upon removing/updating the duplicate properties, or deleting the parent Nodes/Relationships, a constraint on these (and any other properties that see updates) should be created, e.g. as: CREATE CONSTRAINT FOR (p:Person) REQUIRE p.name IS UNIQUE Note that creating a constraint automatically creates an index on the property. This is a good stage to confirm store consistency by performing another consistency check. Assuming this check does not report further inconsistencies, the consistent store (on the now running Neo4j instance) should be fully backed up or dumped and restored/loaded back to the Neo4j instance or seeded to the cluster.InformationTitleIdentify and remove duplicate properties or their parent objectsArticle Created Date3/15/2024, 10:55 PMURL Name6021530752915-Identify-and-remove-duplicate-properties-or-their-parent-objectsLast Modified Date3/24/2024, 10:20 PM