How to monitor if a follower, read-replica or secondary is in sync with Leader / Primary (Causal Cluster)Mar 18, 2024Knowledge
Article DetailsDetailsHelp Details To monitor if a Follower is in sync with its Leader, or know how much it is lagging behind, it is possible to check the Last Committed Transaction Id from Leader and Follower. Last Committed Transaction Id can be assessed in one of the following ways: From the Neo4j BrowserVia the Neo4j MetricsVia the JMX MBeans 1. Checking Last Transaction Id from the Neo4j Web Interface From the Neo4j Browser: type :sysinfo and hit enterfrom the "Transactions" frame, identify the parameter "Last Tx Id" Note : For 4.x and 5.x, Select the database from "Use Database" header and then run :sysinfo. Running sysinfo is not supported on System and Composite databases. You can also call the dbms.queryJmx procedure in the following way: Neo4j 3.5 call dbms.queryJmx("org.neo4j:instance=kernel#0,name=Transactions") yield attributes return attributes["LastCommittedTxId"] Neo4j 4.x CALL dbms.queryJmx("neo4j.metrics:name=neo4j.neo4j.transaction.last_committed_tx_id") yield attributes return attributes["Count"]["value"] as `Last Committed Tx Id`; Note Neo4j 4.x supports multiple databases hosted on the same DBMS instance. In the above example, the database name is neo4j. This will need to be changed to match the database being monitored Neo4j 5.x CALL dbms.queryJmx("neo4j.metrics:name=neo4j.database.neo4j.transaction.last_committed_tx_id") yield attributes return attributes["Count"]["value"] as `Last Committed Tx Id`; Note Neo4j 5.x supports multiple databases hosted on the same DBMS instance. In the above example, the database name is "neo4j" . Database metrics format has been changed with 5.x, the new format is <user-configured-prefix>.database.<database-name>.<metric-name>, where the <user-configured-prefix> can be configured with the server.metrics.prefix configuration setting . <database-name> will need to be changed to match the database being monitored 2. Checking Last Committed Transaction Id via the Neo4j Metrics Assuming Neo4j’s csv metrics are enabled already, you can analyse the following csv file: neo4j.transaction.last_committed_tx_id.csv. Note From 3.4 onwards metrics are enabled by default. If you are running any version prior to 3.4, you need to enable the metrics in your neo4j.conf file on all instances. Please refer to https://neo4j.com/docs/operations-manual/current/monitoring/metrics/#metrics-enable in order to do so. 3. Checking Last Committed Transaction Id via the JMX MBeans Please check: LastCommittedTxId You can do this using curl if you prefer: $ curl -v http://localhost:7474/db/manage/server/jmx/domain/org.neo4j/instance%3Dkernel%230%2Cname%3DTransactions Note For more information on the supported Neo4j JMX MBeans and how to connect to the JMX monitoring programmatically or via JConsole, please refer to https://neo4j.com/docs/java-reference/current/jmx-metrics/ Determining how much a Follower is lagging behind its Leader To determine how much a Follower is lagging behind its Leader, you can compare the Last Commited Transaction Id (assessed in any of the methods described above) from Leader and Follower: (Last Commited Transaction Id)_leader - (Last Commited Transaction Id)_follower The higher the difference, the more the Follower is lagging behind (in terms of commited transactions). Because data propagation depends on a combination of different factors such as size of transactions, concurrency, hardware, network latency, etc, it’s virtually impossible to correlate all of this into a time unit. Note Important Note: One of Neo4j’s Causal Cluster requirement is to safeguard data. The Core Servers do so by replicating all transactions using the Raft protocol. This ensures that the data is safely durable before confirming transaction commit to the end user application. In practice this means once a majority of Core Servers in a cluster (N/2+1) have accepted the transaction, it is safe to acknowledge the commit to the end user application. This safety requirement has an impact on write latency so make sure to take this into consideration when monitoring and determining whether there is an issue or not. You can read more about Neo4j’s Causal Clustering here: https://neo4j.com/docs/operations-manual/current/clustering/introduction/ InformationTitleHow to monitor if a follower, read-replica or secondary is in sync with Leader / Primary (Causal Cluster)Article Created Date3/15/2024, 10:56 PMURL Name360005861574-How-to-monitor-if-a-follower-read-replica-or-secondary-is-in-sync-with-Leader-Primary-Causal-ClusterLast Modified Date3/18/2024, 11:00 AM