本页尚未翻译

本页内容仍为英文原文，源自官方文档 curation/clustering。中文翻译正在分批进行中。

Cluster records by index partition

Group records by the partition an existing IVF vector index already assigns them to, without training a new clustering model.

Applies to: Embedded ❌, Enterprise ✅

ivf_partition groups records by the partition a vector falls into, using the centroids of an IVF-family vector index you already built for search -- no separate clustering model to train or maintain. Use it as a GROUP BY key to aggregate per cluster, or to sample across clusters instead of across rows.

Example data

Connect with an Enterprise connection.

Create the example table

Use the same curation_embeddings fixture as dedupe -- eight records, where rows 10 and 20 are near-identical, 30, 40 and 50 form one looser cluster, and 60, 70 and 80 are each distinct and increasingly far apart -- but build the index with more partitions. Clustering wants partitions fine-grained enough to tell the two clusters apart; deduplication only needs a distance threshold within one:

table.create_index("embedding", config=IvfFlat(num_partitions=5, distance_type="l2"))
table.wait_for_index(["embedding_idx"])

await table.createIndex("embedding", {
  config: lancedb.Index.ivfFlat({ numPartitions: 5, distanceType: "l2" }),
});
await table.waitForIndex(["embedding_idx"], 300);

use std::time::Duration;

use lancedb::index::vector::IvfFlatIndexBuilder;
use lancedb::index::Index;
use lancedb::DistanceType;

table
    .create_index(
        &["embedding"],
        Index::IvfFlat(IvfFlatIndexBuilder::default().num_partitions(5).distance_type(DistanceType::L2)),
    )
    .execute()
    .await?;
table.wait_for_index(&["embedding_idx"], Duration::from_secs(300)).await?;

ivf_partition reads an existing index's trained centroids; it does not train one. The column needs an up-to-date, single-segment IVF-family index before the function runs, the same requirement dedupe has.

Group records by partition

ivf_partition(column) returns the partition each row's vector falls into. Rows closer to the same centroid than to any other land in the same partition, so the near-identical cat photos (10, 20) group together, the three dog photos (30, 40, 50) group together, and the three increasingly distant photos (60, 70, 80) are more likely to each keep their own partition:

SELECT ivf_partition(embedding) AS cluster, array_agg(id) AS ids, count(*) AS n
FROM curation_embeddings
GROUP BY cluster
ORDER BY cluster

The exact partition numbers depend on how the index's centroids were trained, but rows that are close together in embedding space consistently land in the same group. Aggregate anything per cluster this way -- count(*) for cluster sizes, avg(score) for a per-cluster quality signal, or array_agg(id) to collect candidate rows for a downstream pass.

Sample across clusters instead of across rows

Pass a selection mode to group by only a subset of the index's partitions, instead of all of them:

-- The first 2 partitions, deterministically
SELECT ivf_partition(embedding, 'first', 2) AS cluster, array_agg(id) AS ids
FROM curation_embeddings
GROUP BY cluster;

-- 2 partitions chosen at random, reproducible via the seed
SELECT ivf_partition(embedding, 'random', 2, 42) AS cluster, array_agg(id) AS ids
FROM curation_embeddings
GROUP BY cluster;

'first' takes the first count partitions and accepts no seed, since the result is already deterministic. 'random' requires a seed, so the same call reproduces the same partitions. Either way, the result is a spatially-stratified sample across the embedding space: one that covers each selected region of the data, rather than a plain row sample that can easily pick many near-duplicate rows from a dense cluster and miss a sparse one entirely.

Keep a clustering current with a materialized view

Wrap a GROUP BY ivf_partition(...) query in a materialized view so the grouping stays current as new records arrive:

CREATE MATERIALIZED VIEW curation_embeddings_clustered AS
SELECT ivf_partition(embedding) AS cluster, array_agg(id) AS ids
FROM curation_embeddings
GROUP BY cluster

A refresh reassigns every row's partition from the index's current centroids, so retraining the index with replace=True regroups the view on the next refresh. See Refresh a materialized view for manual and automatic refresh options.

A typical use case: deduplication

Near-duplicate records are close together in embedding space, so they consistently land in the same or a neighboring partition. Grouping by ivf_partition first, then comparing distances only within each group, finds near-duplicate candidates without a full pairwise scan across the whole table.

This is exactly the approach dedupe and duplicate_pairs use internally, packaged as a table function with a distance_threshold instead of a GROUP BY clause to write yourself. Reach for those directly for deduplication; use ivf_partition when the task is a different kind of per-cluster grouping or sampling, not deduplication specifically.

Related guides

IVF\_PARTITIONThe full function signature: arguments, sampling modes, and positional syntax.Deduplicate near-duplicatesThe packaged table function for the deduplication use case above.Statistical SamplingRow-level and fragment-level sampling, without clustering.