Merge pull request #1972 from Danielle9897/RDoc-3122-vectorSearch

RDoc-3122 Vector Search

Documentation/7.0/Raven.Documentation.Pages/.docs.json

@@ -39,6 +39,11 @@
         "Name": "Integrations",
         "Mappings": []
     },
+    {
+        "Path": "/ai-integration",
+        "Name": "AI Integration",
+        "Mappings": []
+    },
     {
         "Path": "/glossary",
         "Name": "Glossary",

Documentation/7.0/Raven.Documentation.Pages/ai-integration/.docs.json

@@ -0,0 +1,26 @@
+[
+    {
+        "Path": "ravendb-as-vector-database.markdown",
+        "Name": "RavenDB as a Vector Database",
+        "DiscussionId": "3a848621-110f-4bbd-9147-58f743b0a950",
+        "Mappings": []
+    },
+    {
+        "Path": "vector-search-using-dynamic-query.markdown",
+        "Name": "Vector Search using a Dynamic Query",
+        "DiscussionId": "2b55d124-a9ff-474c-8171-d065daf938a3",
+        "Mappings": []
+    },
+    {
+        "Path": "vector-search-using-static-index.markdown",
+        "Name": "Vector Search using a Static Index",
+        "DiscussionId": "a25c7cca-e662-401f-9b66-8ce1102a2a09",
+        "Mappings": []
+    },
+    {
+        "Path": "data-types-for-vector-search.markdown",
+        "Name": "Data Types for Vector Search",
+        "DiscussionId": "251b7f9e-06a8-4a67-8707-4a87ee9e5d7d",
+        "Mappings": []
+    }
+]

Documentation/7.0/Raven.Documentation.Pages/ai-integration/data-types-for-vector-search.dotnet.markdown

@@ -0,0 +1,130 @@
+# Data Types for Vector Search
+---
+
+{NOTE: }
+
+* Data for vector search can be stored in **raw** or **pre-quantized** formats using several data types,  
+  as outlined below.
+
+* Text and numerical data that is not pre-quantized can be further quantized in the generated embeddings.  
+  Learn more in [Quantization options](../ai-integration/vector-search-using-dynamic-query#quantization-options).
+
+* In this page:
+  * [Supported data types for vector search](../ai-integration/data-types-for-vector-search#supported-data-types-for-vector-search)
+      * [Textual data](../ai-integration/data-types-for-vector-search#textual-data)
+      * [Numerical data](../ai-integration/data-types-for-vector-search#numerical-data) 
+  * [RavenVector](../ai-integration/data-types-for-vector-search#ravenvector)
+
+{NOTE/}
+
+---
+
+{PANEL: Supported data types for vector search}
+
+### Textual data
+
+{CONTENT-FRAME: }
+
+`string` - A single text entry.  
+`string[]` - An array of text entries.
+
+{CONTENT-FRAME/}
+
+---
+
+### Numerical data
+
+* You can store **pre-generated** embedding vectors in your documents,  
+  typically created by machine-learning models from text, images, or other sources.
+
+* When storing numerical embeddings in a document field:  
+  * Ensure that all vectors within this field across all documents in the collection are generated by the **same model** and model version and have the **same dimensions**.
+  * Consistency in both dimensionality and model source is crucial for meaningful comparisons in the vector space.
+
+* In addition to the native types described below, we highly recommended using [RavenVector](../ai-integration/data-types-for-vector-search#ravenvector)  
+  for efficient storage and fast queries when working with numerical embeddings.
+
+{CONTENT-FRAME: }
+
+**Raw embedding data**:  
+Use when precision is critical.  
+
+`float[]` - A single vector of numerical values representing raw embedding data.  
+`float[][]`- An array of vectors, where each entry is a separate embedding vector.  
+
+{CONTENT-FRAME/}
+{CONTENT-FRAME: }
+
+**Pre-quantized data**:   
+Use when you prioritize storage efficiency and query speed.  
+
+`byte[] / sbyte[]` - A single pre-quantized embedding vector in the _Int8_ or _Binary_ quantization format.  
+`byte[][] / sbyte[][]` - An array of pre-quantized embedding vectors.  
+
+When storing data in these formats in your documents, you should use [RavenDB’s vector quantizer methods](../ai-integration/vector-search-using-dynamic-query#section-1).
+
+{CONTENT-FRAME/}
+{CONTENT-FRAME: }
+
+**Base64-encoded data**:  
+Use when embedding data needs to be represented as a compact and easily serializable string format.
+
+`string` - A single vector encoded as a Base64 string.    
+`string[]` - An array of Base64-encoded vectors.    
+
+{CONTENT-FRAME/}
+{CONTENT-FRAME: }
+
+**Using lists**:  
+
+While arrays (`float[]`) are the most direct representation of numerical embeddings,  
+you can also use lists (for example, `List<float>` or `List<float[]>`) for dynamic sizing in your application code.
+
+{CONTENT-FRAME/}
+{PANEL/}
+
+{PANEL: RavenVector}
+
+RavenVector is RavenDB's dedicated data type for storing and querying **numerical embeddings**.   
+It is highly optimized to minimize storage space and improve the speed of reading arrays from disk,  
+making it ideal for vector search.
+
+For example, you can define:
+
+{CODE-BLOCK:csharp}
+RavenVector<float>;       // A single vector of floating-point values.
+List<RavenVector<float>>; // A collection of float-based vectors.
+RavenVector<sbyte>;       // A single pre-quantized vector in Int8 format (8-bit signed integer).
+List<RavenVector<sbyte>>; // A collection of sbyte-based vectors.
+RavenVector<byte>;        // A single pre-quantized vector in Binary format (8-bit unsigned integer).
+List<RavenVector<byte>>;  // A collection of byte-based vectors.
+{CODE-BLOCK/}
+
+---
+
+When a class property is stored as a `RavenVector`, the vector's content will appear under the `@vector` field in the JSON document stored in the database.
+For example:
+
+{CODE-TABS}
+{CODE-TAB:csharp:CSharp_class class@AiIntegration\DataTypes.cs /}
+{CODE-TABS/}
+
+![json document](images/json-document.png "RavenVector in a JSON document")
+
+{PANEL/}
+
+## Related Articles
+
+### Vector Search
+
+- [RavenDB as a vector database](../ai-integration/ravendb-as-vector-database)
+- [Vector search using a static index](../ai-integration/vector-search-using-static-index)
+- [Vector search using a dynamic query](../ai-integration/vector-search-using-dynamic-query)
+
+### Querying
+
+- [Query overview](../client-api/session/querying/how-to-query)
+- [Full-text search](../client-api/session/querying/text-search/full-text-search)
+
+
+

Documentation/7.0/Raven.Documentation.Pages/ai-integration/ravendb-as-vector-database.markdown

@@ -0,0 +1,117 @@
+# RavenDB as a Vector Database
+---
+
+{NOTE: }
+
+* In this page:
+    * [What is a vector database](../ai-integration/ravendb-as-vector-database#what-is-a-vector-database)
+    * [Why choose RavenDB as your vector database](../ai-integration/ravendb-as-vector-database#why-choose-ravendb-as-your-vector-database)
+
+{NOTE/}
+
+---
+
+{PANEL: What is a vector database}
+
+* A vector database stores data as high-dimensional numerical representations (embedding vectors),  
+  enabling searches based on contextual meaning and vector similarity rather than exact keyword matches.  
+  Instead of relying on traditional indexing, it retrieves relevant results by measuring how close vectors are in a multi-dimensional space.
+
+* Vector databases are widely used in applications such as:  
+
+   * Semantic search – Finding documents based on meaning rather than exact words.
+   * Recommendation engines – Suggesting products, media, or content based on similarity.
+   * AI and machine learning – Powering LLMs, multi-modal search, and object detection.
+
+**Embeddings**:  
+
+* A vector database stores data as high-dimensional vectors in a high-dimensional space.  
+  These vectors, known as **embeddings**, are mathematical representations of your data.
+
+* Each embedding is an array of numbers (e.g. [0.45, 3.6, 1.25, 0.7, ...]), where each dimension represents specific characteristics of the data, capturing its contextual meaning.  
+  Words, phrases, entire documents, images, audio, and other types of data can all be vectorized.
+
+* The raw data is converted into embeddings using [transformers](https://huggingface.co/docs/transformers).  
+  To optimize storage and computation, transformers can encode embeddings with lower-precision data types, such as 8-bit integers, through a technique called [quantization](../ai-integration/vector-search-using-dynamic-query#quantization-options).
+
+**Indexing embeddings and searching**:  
+
+* The embedding vectors are indexed and stored in a vector space.
+  Their positions reflect relationships and characteristics of the data as determined by the model that generated them.
+  The distance between two embeddings in the vector space correlates with the similarity of their original inputs within that model's context.
+
+* Vectors representing similar data are positioned close to each other in the vector space.  
+  This is achieved using algorithms such as [HNSW](https://en.wikipedia.org/wiki/Hierarchical_navigable_small_world), which is designed for indexing and querying embeddings.
+  HNSW constructs a graph-based structure that efficiently retrieves approximate nearest neighbors in high-dimensional spaces.
+
+* This architecture enables **similarity searches**. Instead of conventional keyword-based queries,  
+  a vector database lets you find relevant data based on semantic and contextual meaning.
+
+{PANEL/}
+
+{PANEL: Why choose RavenDB as your vector database}
+
+##### An integrated solution:  
+
+* RavenDB provides an integrated solution that combines high-performance NoSQL capabilities with advanced vector indexing and querying features,
+  enabling efficient storage and management of high-dimensional vector data.
+
+##### Reduced infrastructure complexity:
+
+* RavenDB's built-in vector search eliminates the need for external vector databases,  
+  simplifying your infrastructure and reducing maintenance overhead.
+
+##### AI integration:  
+
+* You can use RavenDB as the vector database for your AI-powered applications, including large language models (LLMs).
+  This eliminates the need to transfer data to expensive external services for vector similarity search,
+  providing a cost-effective and efficient solution for vector-based operations.
+
+##### Multiple field types in indexes:  
+
+* An index can consist of multiple index-fields, each having a distinct type, such as a standard field, a spatial field, a full-text search field, or a **vector field**.
+  This flexibility allows you to work with complex documents containing various data types and retrieve meaningful insights by querying the index across all these fields.  
+  An example is available in [Indexing multiple field types](../ai-integration/vector-search-using-static-index#indexing-multiple-field-types).
+
+* Document attachments can also be indexed as vector fields, and Map-Reduce indexes can incorporate vector fields in their reduce phase, 
+  further extending the versatility of your data processing and search capabilities.
+
+##### Built-in embedding support:
+
+* **Textual input**:  
+  RavenDB enables vector search on **textual input**, allowing you to retrieve results based on contextual similarity, rather than exact word matches. 
+  It does this by automatically converting raw text into dense numerical embeddings using the built-in [bge-micro-v2](https://huggingface.co/TaylorAI/bge-micro-v2) model,
+  which transforms text into 384-dimensional vectors that capture semantic meaning.
+  When querying with a phrase, RavenDB generates an embedding for the search term using that same model and compares it against indexed embeddings.
+
+* **Numerical arrays input**:  
+  Documents in RavenDB can also contain numerical arrays with **pre-made embeddings** created elsewhere.  
+  Use RavenDB's dedicated data type, [RavenVector](../ai-integration/data-types-for-vector-search#ravenvector), to store these embeddings in your document entities.  
+  This type is highly optimized to reduce storage space and enhance the speed of reading arrays from disk.
+
+* **HNSW algorithm usage**:  
+  All embeddings, whether generated from textual input or pre-made numerical arrays,  
+  are indexed and searched for using the [HNSW](https://en.wikipedia.org/wiki/Hierarchical_navigable_small_world) algorithm.
+
+* **Optimize storage via quantization**:  
+  RavenDB allows you to select the quantization format for the generated embeddings when creating the index.  
+  Learn more in [Quantization options](../ai-integration/vector-search-using-dynamic-query#quantization-options).
+
+{PANEL/}
+
+## Related Articles
+
+### Client API
+
+- [RQL](../client-api/session/querying/what-is-rql) 
+- [Query overview](../client-api/session/querying/how-to-query)
+
+### Vector Search
+
+- [Vector search using a dynamic query](../ai-integration/vector-search-using-dynamic-query.markdown)
+- [Vector search using a static index](../ai-integration/vector-search-using-static-index.markdown)
+- [Data types for vector search](../ai-integration/data-tuypes-for-vector-search)
+
+### Server
+
+- [indexing configuration](../server/configuration/indexing-configuration)