本页尚未翻译

本页内容仍为英文原文，源自官方文档 sql/dql/dedupe。中文翻译正在分批进行中。

DEDUPE

The dedupe() table function: one representative row per near-duplicate group, scanned over the index.

Applies to: Embedded ❌, Enterprise ✅

dedupe scans a vector column's index partitions for rows closer than a distance threshold, resolves the overlapping pairs it finds into duplicate groups, and returns one representative row per group -- without a full pairwise scan or a shuffle. See duplicate_pairs to inspect the pairs themselves before committing to this resolution.

SELECT * FROM dedupe(
    <table>,
    <vector_column>,
    distance_threshold => <float>
    [, resolution => 'connected_components' | 'direct_representative']
)

SELECT * FROM dedupe(
    'frames',
    'vec',
    distance_threshold => 0.02
)

说明

Earlier releases spelled this vector_dedup; that name is rejected on current deployments.

Input

Argument	Type	Required	Meaning
table name	string literal	yes	the table to scan
vector column	string literal	yes	the indexed vector column to compare
distance_threshold	float	yes	maximum distance for a pair to count as close
resolution	'connected_components' (default) | 'direct_representative'	no	how overlapping pairs group together -- see Resolution

dedupe always scans every partition and returns original rows -- sampling a subset of partitions and id_column are pair-query-only options, accepted by duplicate_pairs but refused here.

Output

One row per duplicate group, in the table's own schema -- every column of <table>, including a BlobV2 column, comes through unchanged on the representative row Lance selected for that group. Unlike duplicate_pairs, there is no fixed dedupe-specific schema to document: the output schema is the source table's schema.

Because the result is already shaped like the table, it composes directly into a new table or a materialized view:

CREATE MATERIALIZED VIEW frames_deduped AS
SELECT * FROM dedupe(
    'frames',
    'vec',
    distance_threshold => 0.02
)

Resolution

resolution chooses how overlapping pairs group together:

resolution value	How pairs group
connected_components (default)	groups transitively: if ab and bc are both close, all three resolve together
direct_representative	only pairs within the threshold of each other resolve together

Estimating on a sample first

dedupe itself always scans every partition. To estimate a duplicate rate on a large table before committing to a full scan, sample partitions with duplicate_pairs instead, which accepts selection, count and seed for exactly this.

What to read next

DUPLICATE\_PAIRSInspect the close pairs themselves instead of the resolved groups.Deduplicate near-duplicatesWhen to reach for this instead of exact-key deduplication.