This article, presenting an overview of how to use Google Cloud Dataflow with Neo4j Aura, was originally based on the blog post Loading Data into Neo4j With Google Cloud Dataflow co-authored by Anthony Krinsky who is a Senior Engineer at Neo4j & Kathryn Eeds who is a Partner Engineer at Google Cloud.

We have now published official documentation regarding the use of Google Cloud Dataflow with Neo4j, located in the connectors section of Neo4j Docs.

Data Ingestion to Neo4j

There are multiple methods for ingesting data into Neo4j, the most popular approach for bulk loading is the LOAD CSV Cypher command from any client connection such as Java, Python, Go, .NET, Node, Spring, and others.

For Data scientists, there are the Neo4j Spark connector and Data Warehouse connector, which both run on DataProc and are easily incorporated into Python notebooks.

For individual users, the graphical ETL import tool is very convenient and for enterprises needing lifecycle management, Apache Hop, a project co-sponsored by Neo4j, is a great option.

The Dataflow approach is interesting and different for a few reasons. Although it requires a customized JSON configuration file, that’s all that is required. No notebooks, no Spark environment, no code, no cost when the system is idle.
 

What is GCP Dataflow?

Google Dataflow is a fast, serverless, no-ops platform for running Apache Beam pipelines. Most Google customers will run Dataflow as a hosted service which autoscales compute. When idle, Dataflow costs nothing. Full details can be found in the Dataflow documentation.

Dataflow users can coordinate flows using the Cloud Dataflow operator in the Google Cloud Composer service which is based on Apache Airflow. Together, these tools allow orchestrated data movement across GCP native and partner solutions such as BigQuery, Spanner, Cloud Storage and now also Neo4j along with many more.

Also, Dataflow runs within the context of GCP security, so if a resource is accessible to the project and service account there is no need to track and secure another resource locator and set of credentials. Finally, the Neo4j flex template implements Neo4j java API best practices.

Overview Diagram

Features and Best Practices

The GCP to Neo4j template was designed to allow these capabilities and best practices:

• No code: Customization through configuration not code. Job optimization and run-time are opinionated, taking into account best practices.
• Merge and append: Append is useful when the database is empty, otherwise use merge for incremental updates. All Cypher statements are generated automatically, internally.
• Optimized orchestration: There will be errors if edges are created before nodes are committed. The framework commits nodes in parallel and then single threads edges to improve performance and eliminate deadlock errors.
• Multiple sources: One single table or view from one source is often insufficient to populate a useful Neo4j model.
• Multiple targets: Multiple sources can flow to multiple target node and edge sets to facilitate ambitious projects.
• Aggregations and groupings: For sources that support SQL push down and those that do not, targets at multiple levels of grain can be populated. In addition, calculations may be defined using ANSI SQL.
• Support for run-time variables: Incremental batch processing require run-time parameterization of queries or file paths.
• Support for pre- and post-load actions:  It may be necessary to log actions or initiate in-database operations after loading or refreshing data.  Through declarative dependencies, complex DAGs are configurable.  Soon, it will be possible to write back from Cypher to BigQuery as well.
• Throughput throttling: Batch size and parallelism values can be controlled by configuration, experiment with these to help achieve optimal performance.

Creating a Job

To create a Dataflow job, you can either use the GCloud CLI or the Google Cloud console user interface.

Google Cloud Command Line Interface

Google makes it easy to test Dataflow jobs from the command line with the local Dataflow runner. All pre-processing happens locally which makes it easy to see errors, often from configuration file validation. Please note, to run a job this way you must clone the repository locally.

It is more likely that you will choose to script execution of Dataflow jobs using the Gcloud CLI.  A prerequisite is to enable the following APIs: Dataflow, Compute Engine, Logging, Cloud Storage, Cloud Storage JSON, BigQuery, Pub/Sub, Resource Manager, App Engine, Artifact Registry, Cloud Scheduler, and Cloud Build APIs.

There are several scripts available in the test resources folder that show how this is possible.

Google Cloud Console User Interface

Creating a job from the user interface is very easy. From the Google cloud console Dataflow user interface, we choose “Custom Template” in the job template user interface and supply this URL:

gs://neo4j-dataflow/flex-templates/images/googlecloud-to-neo4j-image-spec.json

The template JSON file describes arguments that the template expects and has a pointer to the location of the docker image that runs the pipeline.

After selecting the template, you will be prompted for two required parameters which point to files on Google Cloud Storage as shown in the screenshot below: Job configuration file, and Neo4j connection metadata file.

•