Native Memory Tracking (NMT) is a JVM feature that allows us to track the "Native memory" allocation of the Neo4j application, which is useful when we want to understand in detail how Neo4j is allocating the memory.

The exported NMT information contains the details of the "Native memory" allocation on that specific instant of time, so we need to collect a sequence of this information in order to track the memory behavior over time.
The most effective way to process the NMT information is to collect it into files.

To enable the NMT collection, we need to add the following configuration to Neo4j (this requires a restart):

dbms.jvm.additional=-XX:NativeMemoryTracking=summary

With Neo4j 5.x, this configuration setting is renamed as "server.jvm.additional".

The above option enables NMT at a summary level, but it can also be configured at a detail level to observe the memory allocation at a method level.

At this point, we can collect the NMT information by executing the following command from the terminal:

jcmd <pid> VM.native_memory summary

 

We can also take advantage of the option to print the difference in memory allocation over time, this is useful when we need to collect several NMTs to observe the behaviour over a period of time. To do so, follow these steps:

1. Set up an NMT baseline, usually after the restart of Neo4j: 

   jcmd <pid> VM.native_memory baseline

2. Collect the NMT diff file every X minutes, by using the following command (the example also saves the output of the command into a file called "nmt.txt"): 

   jcmd <PID> VM.native_memory summary.diff > nmt.txt

A sample output of the NMT diff file will be as follows:

Native Memory Tracking:

Total: reserved=47604792KB +11329643KB, committed=47114816KB +12575067KB

- Java Heap (reserved=32514048KB, committed=32505856KB)
(mmap: reserved=32514048KB, committed=32505856KB)

- Class (reserved=10096176KB +8914996KB, committed=10093936KB +9936756KB)
(classes #423768 +398945)
( instance classes #420731 +396911, array classes #3037 +2034)
(malloc=658992KB +651316KB #14427513 +14326372)
(mmap: reserved=9437184KB +8263680KB, committed=9434944KB +9285440KB)
( Metadata: )
( reserved=8388608KB +8263680KB, committed=8386368KB +8262208KB)
( used=3063743KB +2944792KB)
( free=5322625KB +5317416KB)
( waste=0KB =0.00%)
( Class space:)
( reserved=1048576KB, committed=1048576KB +1023232KB)
( used=531947KB +511419KB)
( free=516629KB +511813KB)
( waste=0KB =0.00%)

- Thread (reserved=508990KB -68206KB, committed=53890KB +3878KB)
(thread #493 -66)
(stack: reserved=506632KB -67884KB, committed=51532KB +4200KB)
(malloc=1782KB -244KB #2960 -396)
(arena=576KB -77 #984 -132)

- Code (reserved=262276KB +9600KB, committed=237832KB +161180KB)
(malloc=14588KB +9600KB #39355 +15368)
(mmap: reserved=247688KB, committed=223244KB +151580KB)
...

You can take advantage of the following utility called NMT-parser to convert all the collected NMT files into a chart image, to quickly identify unexpected behaviors. You can find the NMT-parser utility here: https://github.com/dam-mond/nmt-parser.

Additionally, it's also possible to configure Neo4j to print the NMT information at application exit, which is very useful if we're trying to analyze the memory behavior in scenarios where the OS is terminating the application for an out-of-memory error. To enable this, add the following configuration on Neo4j:

dbms.jvm.additional=-XX:+UnlockDiagnosticVMOptions
dbms.jvm.additional=-XX:+PrintNMTStatistics

 

NMT reference documentation:
- https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr007.html
- https://docs.oracle.com/en/java/javase/11/vm/native-memory-tracking.html