Skip to content

Vectors in importer. #2381

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: dev
Choose a base branch
from
Open

Vectors in importer. #2381

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
90 changes: 63 additions & 27 deletions modules/ROOT/pages/import.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,20 +83,24 @@ The syntax for importing a set of CSV files is:

[source, syntax, role="nocopy"]
----
neo4j-admin database import full [-h] [--expand-commands] [--verbose] [--auto-skip-subsequent-headers[=true|false]]
[--ignore-empty-strings[=true|false]] [--ignore-extra-columns[=true|false]]
[--legacy-style-quoting[=true|false]] [--normalize-types[=true|false]]
[--overwrite-destination[=true|false]] [--skip-bad-entries-logging[=true|false]]
[--skip-bad-relationships[=true|false]] [--skip-duplicate-nodes[=true|false]] [--strict
[=true|false]] [--trim-strings[=true|false]] [--additional-config=<file>]
[--array-delimiter=<char>] [--bad-tolerance=<num>] [--delimiter=<char>]
[--format=<format>] [--high-parallel-io=on|off|auto] [--id-type=string|integer|actual]
neo4j-admin database import full [-h] [--expand-commands] [--verbose] [--auto-skip-subsequent-headers
[=true|false]] [--ignore-empty-strings[=true|false]]
[--ignore-extra-columns[=true|false]] [--legacy-style-quoting
[=true|false]] [--normalize-types[=true|false]] [--overwrite-destination
[=true|false]] [--skip-bad-entries-logging[=true|false]]
[--skip-bad-relationships[=true|false]] [--skip-duplicate-nodes
[=true|false]] [--strict[=true|false]] [--trim-strings[=true|false]]
[--additional-config=<file>] [--array-delimiter=<char>]
[--bad-tolerance=<num>] [--delimiter=<char>] [--format=<format>]
[--high-parallel-io=on|off|auto] [--id-type=string|integer|actual]
[--input-encoding=<character-set>] [--input-type=csv|parquet]
[--max-off-heap-memory=<size>] [--quote=<char>] [--read-buffer-size=<size>]
[--report-file=<path>] [--schema=<path>] [--temp-path=<path>] [--threads=<num>]
--nodes=[<label>[:<label>]...=]<files>... [--nodes=[<label>[:<label>]...=]<files>...]...
[--relationships=[<type>=]<files>...]... [--multiline-fields=true|false|<path>[,
<path>] [--multiline-fields-format=v1|v2]] <database>
[--max-off-heap-memory=<size>] [--quote=<char>]
[--read-buffer-size=<size>] [--report-file=<path>] [--schema=<path>]
[--temp-path=<path>] [--threads=<num>] [--vector-delimiter=<char>] --nodes=
[<label>[:<label>]...=]<files>... [--nodes=[<label>[:<label>]...=]
<files>...]... [--relationships=[<type>=]<files>...]...
[--multiline-fields=true|false|<path>[,<path>]
[--multiline-fields-format=v1|v2]] <database>
----

=== Description
Expand Down Expand Up @@ -166,6 +170,12 @@ Unicode character ID can be used if prepended by `\`.
| {check-mark}
| {check-mark}

|--vector-delimiter=<char>
|Delimiter character between vector coordinates within a value in CSV data. Accepts the same values as `--array-delimiter`.
|;
| {check-mark}
| {check-mark}

| --auto-skip-subsequent-headers[=true\|false]
|Automatically skip accidental header lines in subsequent files in file groups with more than one file.
|false
Expand Down Expand Up @@ -639,21 +649,26 @@ The syntax for importing a set of CSV files incrementally is:

[source, syntax, role="nocopy"]
----
neo4j-admin database import incremental [-h] [--expand-commands] --force [--update-all-matching-relationships]
[--verbose] [--auto-skip-subsequent-headers[=true|false]]
[--ignore-empty-strings[=true|false]] [--ignore-extra-columns[=true|false]]
[--legacy-style-quoting[=true|false]] [--normalize-types[=true|false]]
[--skip-bad-entries-logging[=true|false]] [--skip-bad-relationships
[=true|false]] [--skip-duplicate-nodes[=true|false]] [--strict[=true|false]]
neo4j-admin database import incremental [-h] [--expand-commands] --force
[--update-all-matching-relationships] [--verbose]
[--auto-skip-subsequent-headers[=true|false]]
[--ignore-empty-strings[=true|false]] [--ignore-extra-columns
[=true|false]] [--legacy-style-quoting[=true|false]]
[--normalize-types[=true|false]] [--skip-bad-entries-logging
[=true|false]] [--skip-bad-relationships[=true|false]]
[--skip-duplicate-nodes[=true|false]] [--strict[=true|false]]
[--trim-strings[=true|false]] [--additional-config=<file>]
[--array-delimiter=<char>] [--bad-tolerance=<num>] [--delimiter=<char>]
[--high-parallel-io=on|off|auto] [--id-type=string|integer|actual]
[--array-delimiter=<char>] [--bad-tolerance=<num>]
[--delimiter=<char>] [--high-parallel-io=on|off|auto]
[--id-type=string|integer|actual]
[--input-encoding=<character-set>] [--input-type=csv|parquet]
[--max-off-heap-memory=<size>] [--quote=<char>] [--read-buffer-size=<size>]
[--report-file=<path>] [--schema=<path>] [--stage=all|prepare|build|merge]
[--temp-path=<path>] [--threads=<num>] --nodes=[<label>[:<label>]...=]<files>...
[--nodes=[<label>[: <label>]...=]<files>...]... [--relationships=[<type>=]<files>...]...
[--multiline-fields=true|false|<path>[,<path>]
[--max-off-heap-memory=<size>] [--quote=<char>]
[--read-buffer-size=<size>] [--report-file=<path>]
[--schema=<path>] [--stage=all|prepare|build|merge]
[--temp-path=<path>] [--threads=<num>] [--vector-delimiter=<char>]
--nodes=[<label>[:<label>]...=]<files>... [--nodes=[<label>[:
<label>]...=]<files>...]... [--relationships=[<type>=]
<files>...]... [--multiline-fields=true|false|<path>[,<path>]
[--multiline-fields-format=v1|v2]] <database>
----

Expand Down Expand Up @@ -1274,7 +1289,7 @@ For properties, the `<name>` part of the field designates the property key, whil
You can have properties in both node data files and relationship data files.
The max length of property keys for block format is 16,383 characters.

Use one of `int`, `long`, `float`, `double`, `boolean`, `byte`, `short`, `char`, `string`, `point`, `date`, `localtime`, `time`, `localdatetime`, `datetime`, and `duration` to designate the data type for properties.
Use one of `int`, `long`, `float`, `double`, `boolean`, `byte`, `short`, `char`, `string`, `point`, `date`, `localtime`, `time`, `localdatetime`, `datetime`, `duration`, and `vector` to designate the data type for properties.
By default, types (except arrays) are converted to Cypher types.
See link:{neo4j-docs-base-uri}/cypher-manual/current/values-and-types/property-structural-constructed/#_property_types[Cypher Manual -> Property, structural, and constructed values].

Expand Down Expand Up @@ -1391,6 +1406,27 @@ This overrides the header definition for `date1`, as well as the configured defa

====

Special considerations for vector data types::
+
--
A vector is specified using the Cypher syntax for maps.
The map must specify both the `coordinateType` and the `dimensions` of the vector.
The `coordinateType` can be one of `byte`, `short`, `int`, `long`, `float`, or `double`.
The `dimensions` must be between 1 and 4096, inclusive.

By default, vector values are separated by `;`.
A different delimiter can be specified with `--vector-delimiter`.

The dimensions of each vector in the data must match what is specified in the header.

A vector in a header could for example look like this:
[source, csv]
----
:ID,"vector1:vector{coordinateType:byte,dimensions:4096}"
----

Note that quotation marks are necessary, since the `,` inside the map would otherwise be interpreted as a column delimiter.
--

[[import-tool-id-spaces]]
== Using ID spaces
Expand Down