Vector test tools #128934

benwtrent · 2025-06-04T20:58:01Z

This adds some testing tools for verifying vector recall and latency directly without having to spin up an entire ES node and running a rally track.

Its pretty barebones and takes inspiration from lucene-util, but I wanted access to our own formats and tooling to make our lives easier.

Here is an example config file. This will build the initial index, run queries at num_candidates: 50, then again at num_candidates 100 (without reindexing, and re-using the cached nearest neighbors).

[{
  "doc_vectors" : "path",
  "query_vectors" : "path",
  "num_docs" : 10000,
  "num_queries" : 10,
  "index_type" : "hnsw",
  "num_candidates" : 50,
  "k" : 10,
  "hnsw_m" : 16,
  "hnsw_ef_construction" : 200,
  "index_threads" : 4,
  "reindex" : true,
  "force_merge" : false,
  "vector_space" : "maximum_inner_product",
  "dimensions" : 768
},
{
"doc_vectors" : "path",
"query_vectors" : "path",
"num_docs" : 10000,
"num_queries" : 10,
"index_type" : "hnsw",
"num_candidates" : 100,
"k" : 10,
"hnsw_m" : 16,
"hnsw_ef_construction" : 200,
"vector_space" : "maximum_inner_product",
"dimensions" : 768
}
]

To execute:

./gradlew :qa:vector:checkVec --args="/Path/to/knn_tester_config.json"

Calling ./gradlew :qa:vector:checkVecHelp gives some guidance on how to use it, additionally providing a way to run it via java directly (useful to bypass gradlew guff).

elasticsearchmachine · 2025-06-05T13:57:06Z

Pinging @elastic/es-search-relevance (Team:Search Relevance)

elasticsearchmachine · 2025-06-05T13:57:07Z

Pinging @elastic/es-delivery (Team:Delivery)

john-wagster

I reviewed this quickly and didn't have any immediate concerns. Agree this would be super valuable. +1 to get it in and iterate / refine.

...main/java/org/elasticsearch/index/codec/vectors/es818/ES818BinaryQuantizedVectorsWriter.java

pmpailis · 2025-06-06T13:26:56Z

qa/vector/src/main/java/org/elasticsearch/test/knn/KnnIndexer.java

+        if (Files.exists(indexPath)) {
+            logger.debug("KnnIndexer: existing index at %s", indexPath);
+        } else {
+            Files.createDirectory(indexPath);


Maybe we should change this to Files.createDirectories to ensure that all parent directories would be created if needed? (I was met with a NoSuchFileException when I first ran this)

pmpailis · 2025-06-06T14:03:22Z

qa/vector/src/main/java/org/elasticsearch/test/knn/KnnIndexTester.java

+                    cmdLineArgs.vectorSpace(),
+                    cmdLineArgs.numDocs()
+                );
+                if (cmdLineArgs.reindex()) {


minor: We could update this to check if the directory already exists. If for example I fail to specify "reindex=true" for an unseen vec file, I would be met with an exception when trying to open the dir

seems sane, adding it

pmpailis · 2025-06-06T14:40:46Z

qa/vector/src/main/java/org/elasticsearch/test/knn/KnnIndexTester.java

+                    Integer.toString(result.numDocs),
+                    Long.toString(result.indexTimeMS),
+                    Long.toString(result.forceMergeTimeMS),
+                    Integer.toString(result.numSegments),


Minor again:

index_type num_docs index_time(ms) force_merge_time(ms) num_segments latency(ms) QPS recall visited ---------- -------- -------------- -------------------- ------------ ----------- ------- ------ ------- hnsw 10000 421 0 4 0.40 2500.00 1.00 174.00 hnsw 10000 0 0 0 0.50 2000.00 1.00 174.00

The num_docs and num_segments could follow previous values if reindex=false

we can add whatever we want. Keeping it simple.

pmpailis · 2025-06-06T14:42:08Z

qa/vector/src/main/java/org/elasticsearch/test/knn/KnnIndexTester.java

+            // printout an example configuration formatted file and indicate that it is required
+            System.out.println("Usage: java -cp <your-classpath> org.elasticsearch.test.knn.KnnIndexTester <config-file>");
+            System.out.println("Where <config-file> is a JSON file containing one or more configurations for the KNN index tester.");
+            System.out.println("An example configuration object: ");


Minor: we could have just a sample of what the doc/query files look like

pmpailis

Really awesome ❤️ Some minor comments only but mainly for next iterations. Nothing blocking.

…arch

carlosdelest

So nice! 🥳

This adds some testing tools for verifying vector recall and latency directly without having to spin up an entire ES node and running a rally track. Its pretty barebones and takes inspiration from lucene-util, but I wanted access to our own formats and tooling to make our lives easier. Here is an example config file. This will build the initial index, run queries at num_candidates: 50, then again at num_candidates 100 (without reindexing, and re-using the cached nearest neighbors). ``` [{ "doc_vectors" : "path", "query_vectors" : "path", "num_docs" : 10000, "num_queries" : 10, "index_type" : "hnsw", "num_candidates" : 50, "k" : 10, "hnsw_m" : 16, "hnsw_ef_construction" : 200, "index_threads" : 4, "reindex" : true, "force_merge" : false, "vector_space" : "maximum_inner_product", "dimensions" : 768 }, { "doc_vectors" : "path", "query_vectors" : "path", "num_docs" : 10000, "num_queries" : 10, "index_type" : "hnsw", "num_candidates" : 100, "k" : 10, "hnsw_m" : 16, "hnsw_ef_construction" : 200, "vector_space" : "maximum_inner_product", "dimensions" : 768 } ] ``` To execute: ``` ./gradlew :qa:vector:checkVec --args="/Path/to/knn_tester_config.json" ``` Calling `./gradlew :qa:vector:checkVecHelp` gives some guidance on how to use it, additionally providing a way to run it via java directly (useful to bypass gradlew guff).

benwtrent added 3 commits June 4, 2025 15:32

Adding vector knn testing tools

e1d7abd

iter

672cb1e

adjusting vector reclal testing

291d4bd

benwtrent added >non-issue :Delivery/Build Build or test infrastructure :Search Relevance/Vectors Vector search v9.1.0 labels Jun 4, 2025

benwtrent added 2 commits June 5, 2025 09:27

fixing things adding help

3ca7ef2

iter

11af34e

benwtrent marked this pull request as ready for review June 5, 2025 13:56

elasticsearchmachine added Team:Delivery Meta label for Delivery team Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch labels Jun 5, 2025

john-wagster approved these changes Jun 5, 2025

View reviewed changes

benwtrent added 3 commits June 5, 2025 12:54

adding license

a5864db

iter

9bc24c3

iter

31954c2

pmpailis reviewed Jun 6, 2025

View reviewed changes

...main/java/org/elasticsearch/index/codec/vectors/es818/ES818BinaryQuantizedVectorsWriter.java Show resolved Hide resolved

pmpailis reviewed Jun 6, 2025

View reviewed changes

Merge remote-tracking branch 'upstream/main' into vector-test-tools

e0bc391

pmpailis reviewed Jun 6, 2025

View reviewed changes

pmpailis approved these changes Jun 6, 2025

View reviewed changes

addressing PR comments, adding cpu time tracking for multithreaded se…

5a95173

…arch

benwtrent added the auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) label Jun 6, 2025

carlosdelest approved these changes Jun 6, 2025

View reviewed changes

elasticsearchmachine merged commit 155c0da into elastic:main Jun 6, 2025
18 checks passed

benwtrent deleted the vector-test-tools branch June 6, 2025 16:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Vector test tools #128934

Vector test tools #128934

benwtrent commented Jun 4, 2025 •

edited

Loading

Uh oh!

elasticsearchmachine commented Jun 5, 2025

Uh oh!

elasticsearchmachine commented Jun 5, 2025

Uh oh!

john-wagster left a comment

Uh oh!

Uh oh!

pmpailis Jun 6, 2025

Uh oh!

pmpailis Jun 6, 2025

Uh oh!

benwtrent Jun 6, 2025

Uh oh!

pmpailis Jun 6, 2025

Uh oh!

benwtrent Jun 6, 2025

Uh oh!

pmpailis Jun 6, 2025

Uh oh!

pmpailis left a comment

Uh oh!

carlosdelest left a comment

Uh oh!

Uh oh!

Uh oh!

Vector test tools #128934

Vector test tools #128934

Conversation

benwtrent commented Jun 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Jun 5, 2025

Uh oh!

elasticsearchmachine commented Jun 5, 2025

Uh oh!

john-wagster left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pmpailis Jun 6, 2025

Choose a reason for hiding this comment

Uh oh!

pmpailis Jun 6, 2025

Choose a reason for hiding this comment

Uh oh!

benwtrent Jun 6, 2025

Choose a reason for hiding this comment

Uh oh!

pmpailis Jun 6, 2025

Choose a reason for hiding this comment

Uh oh!

benwtrent Jun 6, 2025

Choose a reason for hiding this comment

Uh oh!

pmpailis Jun 6, 2025

Choose a reason for hiding this comment

Uh oh!

pmpailis left a comment

Choose a reason for hiding this comment

Uh oh!

carlosdelest left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

benwtrent commented Jun 4, 2025 •

edited

Loading