本页尚未翻译

本页内容仍为英文原文，源自官方文档 integrations/embedding/colpali。中文翻译正在分批进行中。

ColPali

We support ColPali model embeddings for multimodal multi-vector retrieval. ColPali produces multiple embedding vectors per input (multi-vector), enabling more nuanced similarity matching between text queries and image documents.

Using ColPali requires the colpali-engine package, which can be installed using pip install colpali-engine.

说明

ColPali produces multi-vector embeddings, meaning each input generates multiple embedding vectors rather than a single vector. Use MultiVector(func.ndims()) instead of Vector(func.ndims()) when defining your schema.

Supported models are:

• Metric-AI/ColQwen2.5-3b-multilingual-v1.0 (default)
• vidore/colpali-v1.3
• vidore/colqwen2-v1.0
• vidore/colSmol-256M

Supported parameters (to be passed in create method) are:

Parameter	Type	Default Value	Description
model_name	str	"Metric-AI/ColQwen2.5-3b-multilingual-v1.0"	The name of the model to use.
device	str	"auto"	The device for inference. Can be "auto", "cpu", "cuda", or "mps".
dtype	str	"bfloat16"	Data type for model weights (bfloat16, float16, float32, float64).
pooling_strategy	str	"hierarchical"	Token pooling strategy: "hierarchical", "lambda", or None.
pool_factor	int	2	Factor to reduce sequence length when pooling is enabled.
batch_size	int	2	Batch size for processing inputs.
quantization_config	Optional[BitsAndBytesConfig]	None	Quantization configuration for the model (requires bitsandbytes).

This embedding function supports ingesting images as both bytes and URLs. You can query them using text.

import tempfile
from pathlib import Path

import lancedb
import pandas as pd
import requests
from lancedb.embeddings import get_registry
from lancedb.pydantic import LanceModel, MultiVector

db = lancedb.connect(str(Path(tempfile.mkdtemp()) / "colpali-demo"))
func = get_registry().get("colpali").create()

class Images(LanceModel):
    label: str
    image_uri: str = func.SourceField()
    image_bytes: bytes = func.SourceField()
    vector: MultiVector(func.ndims()) = func.VectorField()
    vec_from_bytes: MultiVector(func.ndims()) = func.VectorField()

table = db.create_table("images", schema=Images)
labels = ["cat", "dog", "horse"]
uris = [
    "http://farm1.staticflickr.com/53/167798175_7c7845bbbd_z.jpg",
    "http://farm9.staticflickr.com/8387/8602747737_2e5c2a45d4_z.jpg",
    "http://farm9.staticflickr.com/8216/8434969557_d37882c42d_z.jpg",
]
image_bytes = [requests.get(uri).content for uri in uris]
table.add(
    pd.DataFrame({"label": labels, "image_uri": uris, "image_bytes": image_bytes})
)

Now we can search using text queries:

actual = (
    table.search("a furry pet", vector_column_name="vector")
    .limit(1)
    .to_pydantic(Images)[0]
)
print(actual.label)