本页尚未翻译

本页内容仍为英文原文，源自官方文档 integrations/embedding/imagebind。中文翻译正在分批进行中。

ImageBind

We have support for imagebind model embeddings. You can download our version of the packaged model via - pip install imagebind-packaged==0.1.2.

This function is registered as imagebind and supports Audio, Video and Text modalities(extending to Thermal,Depth,IMU data):

Parameter	Type	Default Value	Description
name	str	"imagebind_huge"	Name of the model.
device	str	"cpu"	The device to run the model on. Can be "cpu" or "gpu".
normalize	bool	False	set to True to normalize your inputs before model ingestion.

Below is an example demonstrating how the API works:

import lancedb
from lancedb.embeddings import get_registry
from lancedb.pydantic import LanceModel, Vector

db = lancedb.connect("/tmp/imagebind-db")
func = get_registry().get("imagebind").create()

class ImageBindModel(LanceModel):
    text: str
    image_uri: str = func.SourceField()
    audio_path: str
    vector: Vector(func.ndims()) = func.VectorField()

text_list = ["A dog.", "A car", "A bird"]
image_paths = [
    "./assets/dog_image.jpg",
    "./assets/car_image.jpg",
    "./assets/bird_image.jpg",
]
audio_paths = [
    "./assets/dog_audio.wav",
    "./assets/car_audio.wav",
    "./assets/bird_audio.wav",
]

inputs = [
    {"text": a, "audio_path": b, "image_uri": c}
    for a, b, c in zip(text_list, audio_paths, image_paths)
]

table = db.create_table("img_bind", schema=ImageBindModel)
table.add(inputs)

Now, we can search using any modality:

image search

query_image = "./assets/dog_image2.jpg"
actual = table.search(query_image).limit(1).to_pydantic(ImageBindModel)[0]
print(actual.text == "dog")

audio search

query_audio = "./assets/car_audio2.wav"
actual = table.search(query_audio).limit(1).to_pydantic(ImageBindModel)[0]
print(actual.text == "car")

Text search

You can add any input query and fetch the result as follows:

query = "an animal which flies and tweets"
actual = table.search(query).limit(1).to_pydantic(ImageBindModel)[0]
print(actual.text == "bird")

If you have any questions about the embeddings API, supported models, or see a relevant model missing, please raise an issue on GitHub.