Customers use the Neo4j provided neo4j-admin helm chart  for database backups in Kubernetes installs of Neo4j. Starting in Neo4j v5, the cluster architecture has decoupled servers from databases, improving scalability and cloud-readiness. As such, all databases are not required to be present on all cluster members. 

For example, if I have a 5 member cluster having 3 Primaries and 2 Secondaries, I may define a database named mydb1 to have the topology  of 1 PRIMARY. As such, mydb1 would only be present on 1 of the 5 members.  Similar, I may have defined mydb2 to have a topology of 3 PRIMARIES it will be on the 3 Primaries, but not on the Secondaries. Or I could define the topology as 1 Primary and 1 Secondary and so on. 

When configuring the neo4j-admin helm chart for backups, one would typically specify the databaseAdminServiceName to connect and the configure the database(s) to backup. 

In one earlier 5.x case, the customer site attempted to configure the databaseAdminServiceName to point to the loadbalancer service for the cluster.  In addition, the site used "" for the database parameter which defaults to all databases in the backup job.

 #name of the database to backup ex: neo4j or neo4j,system (You can provide command separated database names)
 # In case of comma separated databases failure of any single database will lead to failure of complete operation
 database: ""

It is important to note the backup job does not route to all servers if using the loadbalancer service name for databaseAdminServiceName. In this case, the backup job will enumerate all the databases in the cluster and attempt to back them up on the server it connects to. If a particular database is not present on the server which it is connected to, it will fail. You may see an error below if checking the Kubernetes pod logs for the backup job. 

Caused by: com.neo4j.backuprestore.backup.BackupExecutionException: Database 'mydb1' was not found or not running

To add more context, some customers have 100+ databases and therefore do not use the list of database names but rather the option of all mentioned above. In addition, they do not want to create multiple backup jobs - one for each server - as this would cause a lot of duplication of backups for databases that do exist on multiple cluster members.

To help overcome this challenge, the helm charts were updated starting in 5.26.7 to include the ability to use multiple databaseBackupEndpoints so that it works in the same manner to the non-k8s backups --from parameter. This produces one backup per database and contacts all endpoints in one job to overcome the challenge. You can use a list of IP addresses or preferably a list of services as IP addresses change. 

backup:
  # Specify multiple backup endpoints as comma-separated string
  # e.g. "10.3.3.2:6362,10.3.3.3:6362,10.3.3.4:6362"
  databaseBackupEndpoints: "primary1-admin.default.svc.cluster.local:6362,primary2-admin.default.svc.cluster.local:6362,primary3-admin.default.svc.cluster.local:6362"

This change has also been implemented for the Calendar Version releases starting in 2025.05.0.

Please note that helm chart behavior and configurations may change over time so it is always best to use the latest helm charts based on the Neo4j major version in use and to check the documentation based on this version.