Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RDoc-3184 Create a dedicated directory for Vector Search #1989

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,32 +1,7 @@
[
{
"Path": "ravendb-as-vector-database.markdown",
"Name": "RavenDB as a Vector Database",
"DiscussionId": "3a848621-110f-4bbd-9147-58f743b0a950",
"Mappings": []
},
{
"Path": "vector-search-using-dynamic-query.markdown",
"Name": "Vector Search using a Dynamic Query",
"DiscussionId": "2b55d124-a9ff-474c-8171-d065daf938a3",
"Mappings": []
},
{
"Path": "vector-search-using-static-index.markdown",
"Name": "Vector Search using a Static Index",
"DiscussionId": "a25c7cca-e662-401f-9b66-8ce1102a2a09",
"Mappings": []
},
{
"Path": "indexing-attachments-for-vector-search.markdown",
"Name": "Indexing Attachments for Vector Search",
"DiscussionId": "7510d989-4925-4e7d-8c48-c3608c09506c",
"Mappings": []
},
{
"Path": "data-types-for-vector-search.markdown",
"Name": "Data Types for Vector Search",
"DiscussionId": "251b7f9e-06a8-4a67-8707-4a87ee9e5d7d",
"Path": "/vector-search",
"Name": "Vector Search",
"Mappings": []
}
]
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
[
{
"Path": "ravendb-as-vector-database.markdown",
"Name": "RavenDB as a Vector Database",
"DiscussionId": "3a848621-110f-4bbd-9147-58f743b0a950",
"Mappings": []
},
{
"Path": "vector-search-using-dynamic-query.markdown",
"Name": "Vector Search using a Dynamic Query",
"DiscussionId": "2b55d124-a9ff-474c-8171-d065daf938a3",
"Mappings": []
},
{
"Path": "vector-search-using-static-index.markdown",
"Name": "Vector Search using a Static Index",
"DiscussionId": "a25c7cca-e662-401f-9b66-8ce1102a2a09",
"Mappings": []
},
{
"Path": "indexing-attachments-for-vector-search.markdown",
"Name": "Indexing Attachments for Vector Search",
"DiscussionId": "7510d989-4925-4e7d-8c48-c3608c09506c",
"Mappings": []
},
{
"Path": "data-types-for-vector-search.markdown",
"Name": "Data Types for Vector Search",
"DiscussionId": "251b7f9e-06a8-4a67-8707-4a87ee9e5d7d",
"Mappings": []
}
]
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,13 @@
as outlined below.

* Text and numerical data that is not pre-quantized can be further quantized in the generated embeddings.
Learn more in [Quantization options](../ai-integration/vector-search-using-dynamic-query#quantization-options).
Learn more in [Quantization options](../../ai-integration/vector-search/vector-search-using-dynamic-query#quantization-options).

* In this page:
* [Supported data types for vector search](../ai-integration/data-types-for-vector-search#supported-data-types-for-vector-search)
* [Textual data](../ai-integration/data-types-for-vector-search#textual-data)
* [Numerical data](../ai-integration/data-types-for-vector-search#numerical-data)
* [RavenVector](../ai-integration/data-types-for-vector-search#ravenvector)
* [Supported data types for vector search](../../ai-integration/vector-search/data-types-for-vector-search#supported-data-types-for-vector-search)
* [Textual data](../../ai-integration/vector-search/data-types-for-vector-search#textual-data)
* [Numerical data](../../ai-integration/vector-search/data-types-for-vector-search#numerical-data)
* [RavenVector](../../ai-integration/vector-search/data-types-for-vector-search#ravenvector)

{NOTE/}

Expand Down Expand Up @@ -41,7 +41,7 @@
* Ensure that all vectors within this field across all documents in the collection are generated by the **same model** and model version and have the **same dimensions**.
* Consistency in both dimensionality and model source is crucial for meaningful comparisons in the vector space.

* In addition to the native types described below, we highly recommended using [RavenVector](../ai-integration/data-types-for-vector-search#ravenvector)
* In addition to the native types described below, we highly recommended using [RavenVector](../ai-integration/vector-search/data-types-for-vector-search#ravenvector)
for efficient storage and fast queries when working with numerical embeddings.

{CONTENT-FRAME: }
Expand All @@ -61,7 +61,7 @@ Use when you prioritize storage efficiency and query speed.
`byte[] / sbyte[]` - A single pre-quantized embedding vector in the _Int8_ or _Binary_ quantization format.
`byte[][] / sbyte[][]` - An array of pre-quantized embedding vectors.

When storing data in these formats in your documents, you should use [RavenDB’s vector quantizer methods](../ai-integration/vector-search-using-dynamic-query#section-1).
When storing data in these formats in your documents, you should use [RavenDB’s vector quantizer methods](../../ai-integration/vector-search/vector-search-using-dynamic-query#section-1).

{CONTENT-FRAME/}
{CONTENT-FRAME: }
Expand Down Expand Up @@ -106,7 +106,7 @@ When a class property is stored as a `RavenVector`, the vector's content will ap
For example:

{CODE-TABS}
{CODE-TAB:csharp:CSharp_class class@AiIntegration\DataTypes.cs /}
{CODE-TAB:csharp:CSharp_class class@AiIntegration\VectorSearch\DataTypes.cs /}
{CODE-TABS/}

![json document](images/json-document.png "RavenVector in a JSON document")
Expand All @@ -117,14 +117,14 @@ For example:

### Vector Search

- [RavenDB as a vector database](../ai-integration/ravendb-as-vector-database)
- [Vector search using a static index](../ai-integration/vector-search-using-static-index)
- [Vector search using a dynamic query](../ai-integration/vector-search-using-dynamic-query)
- [RavenDB as a vector database](../../ai-integration/vector-search/ravendb-as-vector-database)
- [Vector search using a static index](../../ai-integration/vector-search/vector-search-using-static-index)
- [Vector search using a dynamic query](../../ai-integration/vector-search/vector-search-using-dynamic-query)

### Querying

- [Query overview](../client-api/session/querying/how-to-query)
- [Full-text search](../client-api/session/querying/text-search/full-text-search)
- [Query overview](../../client-api/session/querying/how-to-query)
- [Full-text search](../../client-api/session/querying/text-search/full-text-search)



Original file line number Diff line number Diff line change
Expand Up @@ -4,18 +4,18 @@
{NOTE: }

* This article explains how to index attachments using a **static-index** to enable vector search on their content.
Note: Vector search on attachment content is not available when making a [dynamic query](../ai-integration/vector-search-using-dynamic-query).
Note: Vector search on attachment content is not available when making a [dynamic query](../../ai-integration/vector-search/vector-search-using-dynamic-query).

* **Prior to this article**, refer to the [Vector search using a static index](../ai-integration/vector-search-using-static-index) article for general knowledge about
* **Prior to this article**, refer to the [Vector search using a static index](../../ai-integration/vector-search/vector-search-using-static-index) article for general knowledge about
indexing a vector field.

* In this page:
* [Overview](../ai-integration/indexing-attachments-for-vector-search#overview)
* [Indexing TEXT attachments](../ai-integration/indexing-attachments-for-vector-search#indexing-text-attachments)
* [Indexing NUMERICAL attachments](../ai-integration/indexing-attachments-for-vector-search#indexing-numerical-attachments)
* [LINQ index](../ai-integration/indexing-attachments-for-vector-search#linq-index)
* [JS index](../ai-integration/indexing-attachments-for-vector-search#js-index)
* [Indexing ALL attachments](../ai-integration/indexing-attachments-for-vector-search#indexing-all-attachments)
* [Overview](../../ai-integration/vector-search/indexing-attachments-for-vector-search#overview)
* [Indexing TEXT attachments](../../ai-integration/vector-search/indexing-attachments-for-vector-search#indexing-text-attachments)
* [Indexing NUMERICAL attachments](../../ai-integration/vector-search/indexing-attachments-for-vector-search#indexing-numerical-attachments)
* [LINQ index](../../ai-integration/vector-search/indexing-attachments-for-vector-search#linq-index)
* [JS index](../../ai-integration/vector-search/indexing-attachments-for-vector-search#js-index)
* [Indexing ALL attachments](../../ai-integration/vector-search/indexing-attachments-for-vector-search#indexing-all-attachments)

{NOTE/}

Expand All @@ -36,7 +36,7 @@
They are stored as **binary data**, regardless of content type.

* Attachments are handled as streams, allowing efficient upload and retrieval.
Learn more in: [What are attachments](../document-extensions/attachments/what-are-attachments).
Learn more in: [What are attachments](../../document-extensions/attachments/what-are-attachments).

{CONTENT-FRAME/}
{CONTENT-FRAME: }
Expand All @@ -55,7 +55,7 @@ enabling vector search on text or numerical data that is stored in the attachmen
* **Attachments with NUMERICAL data**:
* While attachments can store any file type, RavenDB does Not generate embeddings from images, videos, or other non-textual content.
Each attachment must contain a **single** precomputed embedding vector, generated externally.
* RavenDB indexes the embedding vector from the attachment in and can apply [quantization](../ai-integration/vector-search-using-dynamic-query#quantization-options)
* RavenDB indexes the embedding vector from the attachment in and can apply [quantization](../../ai-integration/vector-search/vector-search-using-dynamic-query#quantization-options)
(e.g., index it in _Int8_ format) if this is configured.
* All embeddings indexed within the same vector-field in the static-index must be vectors of the **same dimension** to ensure consistency in indexing and search.
They must also be created using the **same model**.
Expand All @@ -71,22 +71,22 @@ enabling vector search on text or numerical data that is stored in the attachmen
This applies to all _Company_ documents that contain an attachment with that name.

{CODE-TABS}
{CODE-TAB:csharp:LINQ_index index_1@AiIntegration\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:JS_index index_2@AiIntegration\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:IndexDefinition index_3@AiIntegration\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:Storing_text_attachments store_attachments_1@AiIntegration\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:LINQ_index index_1@AiIntegration\VectorSearch\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:JS_index index_2@AiIntegration\VectorSearch\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:IndexDefinition index_3@AiIntegration\VectorSearch\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:Storing_text_attachments store_attachments_1@AiIntegration\VectorSearch\VectorSearchWithAttachments.cs /}
{CODE-TABS/}

Execute a vector search using the index:
Results will include _Company_ documents whose attachment contains text similar to `"chinese food"`.

{CODE-TABS}
{CODE-TAB:csharp:Query query_1@AiIntegration\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:Query_async query_1_async@AiIntegration\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:DocumentQuery query_2@AiIntegration\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:DocumentQuery_async query_2_async@AiIntegration\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:RawQuery query_3@AiIntegration\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:RawQuery_async query_3_async@AiIntegration\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:Query query_1@AiIntegration\VectorSearch\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:Query_async query_1_async@AiIntegration\VectorSearch\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:DocumentQuery query_2@AiIntegration\VectorSearch\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:DocumentQuery_async query_2_async@AiIntegration\VectorSearch\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:RawQuery query_3@AiIntegration\VectorSearch\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:RawQuery_async query_3_async@AiIntegration\VectorSearch\VectorSearchWithAttachments.cs /}
{CODE-TAB-BLOCK:sql:RQL}
from index "Companies/ByVector/FromTextAttachment"
where vector.search(VectorFromAttachment, $searchTerm, 0.8)
Expand All @@ -96,7 +96,7 @@ where vector.search(VectorFromAttachment, $searchTerm, 0.8)

You can now extract the text from the attachments of the resulting documents:

{CODE:csharp extract_attachment_content@AiIntegration\VectorSearchWithAttachments.cs /}
{CODE:csharp extract_attachment_content@AiIntegration\VectorSearch\VectorSearchWithAttachments.cs /}

{PANEL/}

Expand All @@ -112,21 +112,21 @@ You can now extract the text from the attachments of the resulting documents:
* Each attachment contains raw numerical data in 32-bit floating-point format.

{CODE-TABS}
{CODE-TAB:csharp:LINQ_index index_4@AiIntegration\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:IndexDefinition index_5@AiIntegration\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:Storing_numerical_attachments store_attachments_2@AiIntegration\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:LINQ_index index_4@AiIntegration\VectorSearch\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:IndexDefinition index_5@AiIntegration\VectorSearch\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:Storing_numerical_attachments store_attachments_2@AiIntegration\VectorSearch\VectorSearchWithAttachments.cs /}
{CODE-TABS/}

Execute a vector search using the index:
Results will include _Company_ documents whose attachment contains vectors similar to the query vector.

{CODE-TABS}
{CODE-TAB:csharp:Query query_4@AiIntegration\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:Query_async query_4_async@AiIntegration\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:DocumentQuery query_5@AiIntegration\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:DocumentQuery_async query_5_async@AiIntegration\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:RawQuery query_6@AiIntegration\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:RawQuery_async query_6_async@AiIntegration\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:Query query_4@AiIntegration\VectorSearch\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:Query_async query_4_async@AiIntegration\VectorSearch\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:DocumentQuery query_5@AiIntegration\VectorSearch\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:DocumentQuery_async query_5_async@AiIntegration\VectorSearch\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:RawQuery query_6@AiIntegration\VectorSearch\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:RawQuery_async query_6_async@AiIntegration\VectorSearch\VectorSearchWithAttachments.cs /}
{CODE-TAB-BLOCK:sql:RQL}
from index "Companies/ByVector/FromNumericalAttachment"
where vector.search(VectorFromAttachment, $queryVector)
Expand All @@ -138,23 +138,23 @@ where vector.search(VectorFromAttachment, $queryVector)

### JS index

* The following is the JavaScript index format equivalent to the [LINQ index](../ai-integration/indexing-attachments-for-vector-search#linq-index) shown above.
* The following is the JavaScript index format equivalent to the [LINQ index](../../ai-integration/vector-search/indexing-attachments-for-vector-search#linq-index) shown above.

* The main difference is that JavaScript indexes do Not support `getContentAsStream()` on attachment objects:
* Because of this, embedding vectors must be stored in attachments as **Base64-encoded strings**.
* Use `getContentAsString()` to retrieve the attachment content as a string, as shown in this example.

{CODE-TABS}
{CODE-TAB:csharp:JS_index index_6@AiIntegration\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:Storing_numerical_attachments_as_base64 store_attachments_3@AiIntegration\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:JS_index index_6@AiIntegration\VectorSearch\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:Storing_numerical_attachments_as_base64 store_attachments_3@AiIntegration\VectorSearch\VectorSearchWithAttachments.cs /}
{CODE-TABS/}

Execute a vector search using the index:
Results will include _Company_ documents whose attachment contains vectors similar to the query vector.

{CODE-TABS}
{CODE-TAB:csharp:RawQuery query_7@AiIntegration\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:RawQuery_async query_7_async@AiIntegration\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:RawQuery query_7@AiIntegration\VectorSearch\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:RawQuery_async query_7_async@AiIntegration\VectorSearch\VectorSearchWithAttachments.cs /}
{CODE-TAB-BLOCK:sql:RQL}
from index "Companies/ByVector/FromNumericalAttachment/JS"
where vector.search(VectorFromAttachment, $queryVector)
Expand All @@ -171,21 +171,21 @@ where vector.search(VectorFromAttachment, $queryVector)
* It indexes embeddings generated from the numerical data stored in ALL attachments of all _Company_ documents.

{CODE-TABS}
{CODE-TAB:csharp:LINQ_index index_7@AiIntegration\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:IndexDefinition index_8@AiIntegration\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:Storing_numerical_attachments store_attachments_4@AiIntegration\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:LINQ_index index_7@AiIntegration\VectorSearch\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:IndexDefinition index_8@AiIntegration\VectorSearch\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:Storing_numerical_attachments store_attachments_4@AiIntegration\VectorSearch\VectorSearchWithAttachments.cs /}
{CODE-TABS/}

Execute a vector search using the index:
Results will include Company documents whose attachments contains vectors similar to the query vector.

{CODE-TABS}
{CODE-TAB:csharp:Query query_8@AiIntegration\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:Query_async query_8_async@AiIntegration\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:DocumentQuery query_9@AiIntegration\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:DocumentQuery_async query_9_async@AiIntegration\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:RawQuery query_10@AiIntegration\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:RawQuery_async query_10_async@AiIntegration\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:Query query_8@AiIntegration\VectorSearch\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:Query_async query_8_async@AiIntegration\VectorSearch\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:DocumentQuery query_9@AiIntegration\VectorSearch\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:DocumentQuery_async query_9_async@AiIntegration\VectorSearch\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:RawQuery query_10@AiIntegration\VectorSearch\VectorSearchWithAttachments.cs /}
{CODE-TAB:csharp:RawQuery_async query_10_async@AiIntegration\VectorSearch\VectorSearchWithAttachments.cs /}
{CODE-TAB-BLOCK:sql:RQL}
from index "Companies/ByVector/AllAttachments"
where vector.search(VectorFromAttachment, $queryVector)
Expand All @@ -199,11 +199,11 @@ where vector.search(VectorFromAttachment, $queryVector)

### Vector Search

- [RavenDB as a vector database](../ai-integration/ravendb-as-vector-database)
- [Vector search using a dynamic query](../ai-integration/vector-search-using-dynamic-query)
- [Vector search using a static index](../ai-integration/vector-search-using-static-index)
- [RavenDB as a vector database](../../ai-integration/vector-search/ravendb-as-vector-database)
- [Vector search using a dynamic query](../../ai-integration/vector-search/vector-search-using-dynamic-query)
- [Vector search using a static index](../../ai-integration/vector-search/vector-search-using-static-index)

### Querying

- [Query overview](../client-api/session/querying/how-to-query)
- [Full-text search](../client-api/session/querying/text-search/full-text-search)
- [Query overview](../../client-api/session/querying/how-to-query)
- [Full-text search](../../client-api/session/querying/text-search/full-text-search)
Loading
Loading