Background:

Customer site began receiving authentication and connection errors on a particular node in the Neo4j cluster like below:

"Could not perform discovery for database 'dbnamehere'. No routing server available."

{"time":"2022-08-16 09:35:48.436+0000","level":"WARN","category":"o.n.b.r.DefaultBoltConnection","message":"The client is unauthorized due to authenticationfailure."}
{"time":"2022-08-16 09:35:49.172+0000","level":"WARN","category":"o.n.b.r.DefaultBoltConnection","message":"The client has provided incorrect authenticationdetails too many times in a row."}

The same user and password could successfully connect via the Neo4j browser and cypher-shell.

Resolution:

A special character, %, was used within the password configured for the Neo4j Ops Manager Agent causing the bolt connection failures. The % sign is used within systemd as a specifier and should be escaped using %% for a single percent sign.

Neo4j Ops Manager and Agents do not read the service file directly but rather receives the configured values from the OS systemd. This means that if the % sign has not been escaped in the service file, Neo4j Ops Manager receives values that have been substituted.

For the example of the case of a password using a %, escaping the % using %% would be done within the service file configuration to resolve the issue.

Environment="CONFIG_INSTANCE_1_BOLT_PASSWORD=a%ecret"

becomes

Environment="CONFIG_INSTANCE_1_BOLT_PASSWORD=a%%ecret"