Skip to content

Commit 829a7fb

Browse files
Add missing options of the copy and incremental import commands (neo4j#2068)
Document changes for the 2025.01 release: * new option of the copy command * new option of the incremental import command The `neo4j-admin database restore` command will be updated in the 2025.02 release.
1 parent 64cbc03 commit 829a7fb

File tree

3 files changed

+82
-23
lines changed

3 files changed

+82
-23
lines changed

modules/ROOT/pages/backup-restore/copy-database.adoc

Lines changed: 46 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,8 @@ For more information, see <<copy-estimating-iops, Estimating the processing time
2828
[source,role=noheader]
2929
----
3030
neo4j-admin database copy [-h] [--copy-schema] [--expand-commands] [--force] [--verbose] [--compact-node-store[=true|false]]
31-
[--additional-config=<file>] [--from-pagecache=<size>] [--temp-path=<path>] [--to-format=<format>]
31+
[--additional-config=<file>] [--from-pagecache, --max-off-heap-memory=<size>]
32+
[--temp-path=<path>] [--to-format=<format>]
3233
[--to-path-schema=<path>] [--copy-only-node-properties=<label.property>[,<label.property>...]]...
3334
[--copy-only-nodes-with-labels=<label>[,<label>...]]... [--copy-only-relationship-properties=<relationship.
3435
property>[,<relationship.property>...]]... [--copy-only-relationships-with-types=<type>[,<type>...]]...
@@ -123,13 +124,6 @@ The indexes will be built the first time the database is started.
123124
|Force the command to run even if the integrity of the database cannot be verified.
124125
|
125126

126-
|--from-pagecache=<size>
127-
|The size of the page cache to use for reading.
128-
[TIP]
129-
You can use the `--from-pagecache` option to speed up the copy operation by specifying how much cache to allocate when reading the source.
130-
The `--from-pagecache` should be assigned whatever memory you can spare since Neo4j does random reads from the source.
131-
|8m
132-
133127
|--from-path-data=<path>
134128
|Path to the databases directory, containing the database directory to source from.
135129
It can be used to target databases outside of the installation.
@@ -155,6 +149,12 @@ relationship types will not be included in the copy.
155149
Cannot be combined with `--copy-only-relationships-with-types`.
156150
|
157151

152+
|--from-pagecache, --max-off-heap-memory=<size>
153+
|label:new[Introduced in 2025.01] Maximum memory that neo4j-admin can use for various data structures and caching to improve performance.
154+
Values can be plain numbers, such as 10000000, or 20G for 20 gigabytes.
155+
It can also be specified as a percentage of the available memory, for example 70%.
156+
|90%
157+
158158
|--skip-labels=<label>[,<label>...]
159159
|A comma-separated list of labels to ignore.
160160
|
@@ -210,6 +210,18 @@ The block format is the default format for all newly-created databases as long a
210210
For more information on the block format, see xref:database-internals/store-formats.adoc[Store formats].
211211
====
212212

213+
[NOTE]
214+
====
215+
Neo4j 2025.01 introduces a dual-named option `--from-pagecache, --max-off-heap-memory=<size>`, which enhances the functionality of the `--from-pagecache=<size>` option.
216+
217+
The new option determines the maximum amount of off-heap memory available during the copy process for reading and writing, instead of specifying how much cache to allocate when reading the source.
218+
219+
For details, see <<off-heap-memory-control,Improving the performance>>.
220+
221+
Note that the `--from-pagecache` option may be removed in future versions.
222+
====
223+
224+
213225
[[copy-database-examples]]
214226
== Examples
215227

@@ -363,3 +375,29 @@ Therefore, with an additional 144 GB of both read and write, the best-case scena
363375

364376
Finally, it is also important to consider that in almost all Cloud environments, the published IOPS value may not be the same as the actual value, or be able to continuously maintain the maximum possible IOPS.
365377
The real processing time for this example _could_ be well above that estimation of 3 hours.
378+
379+
[[off-heap-memory-control]]
380+
=== Improving the performance
381+
382+
You can improve the performance of the copy process by specifying the memory limit.
383+
Neo4j 2025.01 introduces an option `--from-pagecache, --max-off-heap-memory=<size>` to replace the old `--from-pagecache=<size>` option.
384+
385+
The new option controls how much off-heap memory the copy process may use in addition to the heap size the JVM is given.
386+
Values can be plain numbers, such as 10000000, or 20G for 20 gigabytes.
387+
It can also be specified as a percentage of the available memory, for example 70%.
388+
389+
Starting from 2025.01, to configure the off-heap memory value, you can use either the old name `--from-pagecache=<size>` or the new one `--max-off-heap-memory=<size>`.
390+
391+
392+
.Using `--from-pagecache=<size>`
393+
[source, shell]
394+
----
395+
bin/neo4j-admin database copy neo4j copy --from-pagecache=95%
396+
----
397+
398+
.Using `--max-off-heap-memory=<size>`
399+
[source, shell]
400+
----
401+
bin/neo4j-admin database copy neo4j copy --max-off-heap-memory=95%
402+
----
403+

modules/ROOT/pages/changes-deprecations-removals.adoc

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -529,4 +529,20 @@ Replaced by xref:backup-restore/aggregate.adoc[`neo4j-admin backup aggregate`]
529529
For more information, see xref:clustering/databases.adoc#s3-seed-provider[Seed from URI].
530530

531531

532+
== Changes in Neo4j 2025.x
533+
534+
The section covers changes to Neo4j server functionality across different areas.
535+
536+
=== Neo4j-admin tool
537+
538+
Neo4j 2025.01::
539+
540+
* The `neo4j-admin database copy` command.
541+
+
542+
The functionality of the `--from-pagecache=<size>` option is changed. +
543+
Instead of specifying how much cache to allocate when reading the source, now you can control the maximum amount of off-heap memory used for the copy operation, both for reading and writing.
544+
By configuring the off-heap memory value, you can impact the cache allocation as well. +
545+
To reflect this change, a new name was added to the option: `--max-off-heap-memory=<size>`.
546+
+
547+
For details, refer to the xref:backup-restore/copy-database.adoc#off-heap-memory-control[Improving the performance].
532548

modules/ROOT/pages/tools/neo4j-admin/neo4j-admin-import.adoc

Lines changed: 20 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -595,21 +595,22 @@ The syntax for importing a set of CSV files incrementally is:
595595

596596
[source, syntax, role="nocopy"]
597597
----
598-
neo4j-admin database import incremental [-h] [--expand-commands] --force [--verbose] [--auto-skip-subsequent-headers
599-
[=true|false]] [--ignore-empty-strings[=true|false]] [--ignore-extra-columns
600-
[=true|false]] [--legacy-style-quoting[=true|false]] [--normalize-types
601-
[=true|false]] [--skip-bad-entries-logging[=true|false]]
602-
[--skip-bad-relationships[=true|false]] [--skip-duplicate-nodes[=true|false]]
603-
[--strict[=true|false]] [--trim-strings[=true|false]]
604-
[--additional-config=<file>] [--array-delimiter=<char>] [--bad-tolerance=<num>]
605-
[--delimiter=<char>] [--high-parallel-io=on|off|auto]
606-
[--id-type=string|integer|actual] [--input-encoding=<character-set>]
607-
[--input-type=csv|parquet] [--max-off-heap-memory=<size>] [--quote=<char>]
608-
[--read-buffer-size=<size>] [--report-file=<path>] [--schema=<path>]
609-
[--stage=all|prepare|build|merge] [--threads=<num>] --nodes=[<label>[:
610-
<label>]...=]<files>... [--nodes=[<label>[:<label>]...=]<files>...]...
611-
[--relationships=[<type>=]<files>...]... [--multiline-fields=true|false|<path>[,
612-
<path>] [--multiline-fields-format=v1|v2]] <database>
598+
neo4j-admin database import incremental [-h] [--expand-commands] --force [--update-all-matching-relationships]
599+
[--verbose] [--auto-skip-subsequent-headers[=true|false]]
600+
[--ignore-empty-strings[=true|false]] [--ignore-extra-columns[=true|false]]
601+
[--legacy-style-quoting[=true|false]] [--normalize-types[=true|false]]
602+
[--skip-bad-entries-logging[=true|false]] [--skip-bad-relationships
603+
[=true|false]] [--skip-duplicate-nodes[=true|false]] [--strict[=true|false]]
604+
[--trim-strings[=true|false]] [--additional-config=<file>]
605+
[--array-delimiter=<char>] [--bad-tolerance=<num>] [--delimiter=<char>]
606+
[--high-parallel-io=on|off|auto] [--id-type=string|integer|actual]
607+
[--input-encoding=<character-set>] [--input-type=csv|parquet]
608+
[--max-off-heap-memory=<size>] [--quote=<char>] [--read-buffer-size=<size>]
609+
[--report-file=<path>] [--schema=<path>] [--stage=all|prepare|build|merge]
610+
[--threads=<num>] --nodes=[<label>[:<label>]...=]<files>... [--nodes=[<label>[:
611+
<label>]...=]<files>...]... [--relationships=[<type>=]<files>...]...
612+
[--multiline-fields=true|false|<path>[,<path>]
613+
[--multiline-fields-format=v1|v2]] <database>
613614
----
614615

615616
=== Description
@@ -878,6 +879,10 @@ performance, this value should not be greater than the number of available proce
878879
|Whether or not strings should be trimmed for whitespaces.
879880
|false
880881

882+
|--update-all-matching-relationships
883+
|label:new[Introduced in 2025.01] If one relationship data entry matches multiple existing relationships, this decides whether to update all matching, or to instead log as error.
884+
|false
885+
881886
|--verbose
882887
|Enable verbose output.
883888
|

0 commit comments

Comments
 (0)