Article DetailsDetailsHelp DetailsSpecifying upper bounds to variable path length (number of hops) patterns is best practice to prevent excessive use or exhaustion of heap memory, cpu, and JVM resources.This applies to both the latest Quantified Path Patterns in Neo4j 5.x, as well as the older variable-length pattern matching in Neo4j 4.x.In general, the idea to keep in mind is that the higher the upper bound, the more distinct paths can be found, and this correlates directly with the work that needs to be done, since all found paths may need to be returned or filtered, depending on the query.Cypher by default uses relationship isomorphism: within a single path, a relationship may only be traversed once. This still allows the same node to be revisited within a single path, provided no relationship is repeated, and it still allows the same nodes and relationships to be traversed and returned multiple times across the multiple paths found by a MATCH, provided only that each path is distinct.The point is that even in simple graphs, the number of distinct paths that can be found may be far far larger than expected, possibly into a count of billions or larger, and you may see this grow quickly as the upper bound in a quantified or variable-length pattern is increased, and especially if the upper bound is omitted completely. Consider the following query on the Neo4j movies graph, using the variable-length pattern syntax: PROFILE MATCH(bacon:Person {name:"Kevin Bacon"})-[*1..4]-(hollywood) RETURN DISTINCT hollywood The query above returns nodes up to 4 hops away from the actor Kevin Bacon. Examining query.log ($NEO4J_HOME/logs/query.log) for this query, shows the total resource usage as follows: INFO id:11622 - 45 ms: 30240 B - bolt-session bolt neo4j-browser/v4.4.2 client/127.0.0.1:56496 server/127.0.0.1:7687> neo4j - neo4j - PROFILE MATCH (bacon:Person {name:"Kevin Bacon"})-[*1..4]-(hollywood) RETURN DISTINCT hollywood - {} - runtime=pipelined - {type: 'user-direct', app: 'neo4j-browser_v4.4.2'} Above query completed in around 45ms and used around 30KB of memory, producing the following profile: We can see from the plan above, that the variable length expansion was the most expensive part of the query, responsible for bulk of the memory and db hits, and 135 distinct end nodes were found as a result.Note that this is using a more efficient VarLengthExpand(Pruning) operator, because we requested DISTINCT nodes in the query. If we had omitted the DISTINCT in our RETURN, this would have instead generated 297 rows, each row being a distinct path to its respective end node. And of course many paths would have the same end node, as each would be found via a distinct path. Work and costs increase as we raise the upper bound The nodes in this dataset however were not dense, the pattern length was relatively low and the total node count wasn't high (171 total nodes). This data model would be a contrast to most production datasets however and therefore we would expect the variable length pattern match to consume significantly more resources, possibly resulting in resource contention, high GC pauses, or even a JVM out of memory/cpu. This is further proven by increasing the pattern length by moving the pattern bounds further apart e.g. as: PROFILE MATCH(bacon:Person {name:"Kevin Bacon"})-[*1..6]-(hollywood) RETURN DISTINCT hollywood which is logged as: INFO id:11859 - 40 ms: 44960 B - bolt-session bolt neo4j-browser/v4.4.2 client/127.0.0.1:56496 server/127.0.0.1:7687> neo4j - neo4j - PROFILE MATCH (bacon:Person {name:"Kevin Bacon"})-[*1..6]-(hollywood) RETURN DISTINCT hollywood - {} - runtime=pipelined - {type: 'user-direct', app: 'neo4j-browser_v4.4.2'} With an increased pattern length from 4 before, to 6 in this case, notice the increased total memory consumption to 45KB (compared to 30KB before). The variable pattern expansion shows a significant rise in db hits as well (from 275 before, to 622 in this case): Note the distinct nodes found has increased to 171, because this is the total number of nodes in the movies graph. Sharply rising costs as the upper bound increases This does not seem like much of an increase, from 135 nodes to 171 nodes, but the true costs and danger can be better observed if we omit the DISTINCT, to show how the number of rows can grow even with just gradual increases to the upper bound. When we omit DISTINCT, the number of rows (distinct paths, instead of distinct nodes) increases to 3259. This jump from 297 to 3259 rows only by raising the upper bound by 2 should make it clear that the number of rows found can explode as the upper bound is raised, and the number of rows found that may need subsequent filtering can raise the work required to untenable levels, as matched rows rocket into the billions, trillions, and higher.This table further illustrates how the number of rows matched skyrockets as the upper bound is raised. Upper boundMatched path count, without DISTINCT4297632598466861084466121642857414307851229 Going from 10 to 12 for the upper bound has jumped us from 10s of thousands to 10s of millions of rows, and it will only get more untenable from there.Remember...this is only being run on a very simple 171 node graph. Nearly all production graphs are far larger than this, and so the costs may grow higher far earlier, depending on the data model. It does not take much complexity at all for higher-bounded path patterns to become problematic, provided the data model and the pattern does not naturally curtail and prune the paths to expand. DISTINCT nodes and pruning expansion as mitigation The pruning expansion used when we have DISTINCT in the query may save us, in the case that we really want distinct reachable nodes, but only in later versions. In Neo4j 5.x, this can cleverly avoid the path explosion that would otherwise happen, as once all distinct reachable nodes have been found, further paths will not be explored.Of course, not all queries are seeking only distinct nodes, and so this mitigation cannot always be used. Omitting upper bound can result in OOM and other memory issues In Neo4j 4.x, however, even when we use DISTINCT to get the pruning expansion behavior, continuing to raise the upper bound can become untenable. This scenario also can show what can happen, even in the latest versions of Neo4j, when we are not looking for DISTINCT nodes and must instead consider every possible distinct path. In 4.x, we could see the impact (an OOM of the JVM) if we attempted to run the query without an upper bound: PROFILE MATCH(bacon:Person {name:"Kevin Bacon"})-[*1..]-(hollywood) RETURN DISTINCT hollywood The query above runs indefinitely, eventually leading to a JVM OOM (assuming no transaction size limits are placed on the db or overall dbms) and a significant rise in CPU as evident by the screenshot showing 100% cpu utilisation by the Neo4j JVM below: In conclusion, it is advised to identify the specific pattern length absolutely required for a desired query outcome and accordingly place finite bounds on variable relationship patterns. Further, we recommend that if there are filterings that can be performed during expansion to prune away paths early, those can be most easily expressed using Neo4j 5.x's quantified path patterns.InformationTitleSpecifying Upper Bounds On Relationship Pattern Matching in Cypher QueriesArticle Created Date3/15/2024, 10:54 PMURL Name12667263111955-Specifying-Upper-Bounds-On-Relationship-Pattern-Matching-in-Cypher-QueriesLast Modified Date6/28/2024, 9:53 AM