Learn best practices for batched loading and transactions

While you may use neo4j-admin import for your initial database load, you will need to use other means of data import once your graph has already been created.

LOAD CSV is often used as one of the primary means of data import, and includes the ability to perform built in batching when you add USING PERIODIC COMMIT.

LOAD CSV documentation

However, not all data to be imported is in CSV form, and there are some approaches to batching that are far more efficient than others. Michael Hunger, one of our Developer Relations engineers, has authored some batching tips and tricks that can make all the difference for batched data loading.

5 Tips & Tricks for Fast Batched Updates

Sometimes you need to make large changes to the graph outside of the context of data loading, such as mass refactoring changes or just large transactions in general. To avoid blowing the heap, you'll want to batch these transactions. Currently the easiest means of doing so is via APOC Procedures, notably through the use of apoc.periodic.iterate() .

apoc.periodic.iterate()

Please be aware that best practices when using apoc.periodic.iterate() (to avoid memory leaks) include passing nodes and relationships by id() only between the outer and inner queries of the iterate() call, and re-matching in the inner query by the passed id() or ids.

 

Indexes are critical to ensure your batch processing is performant. You can review EXPLAIN plans to verify that indexes are being correctly used, or to check for inefficient lookups that might be dragging down batch performance.

Use indexes for batch processing

Large DELETEs

Deleting a large amount of data can be very resource intensive and cause performance issues or worse. Follow our best practices for large deletes to avoid common pitfalls:

Large DELETE transaction best practices

How to avoid using excessive memory on deletes involving dense nodes