本页尚未翻译

本页内容仍为英文原文，源自官方文档 datasets/laion-1m。中文翻译正在分批进行中。

LAION-1M

A Lance-formatted slice of the LAION image-text corpus (~1M rows) with inline JPEG bytes, CLIP image embeddings (img_emb), full metadata, and a pre-built ANN index — all available directly from the Hub at hf://datasets/lance-format/laion-1m/data/train.lance.

View on Hugging FaceSource dataset card and downloadable files for `lance-format/laion-1m`.

Key features

• Inline JPEG bytes in the image column — no sidecar files, no image folders.
• Pre-computed CLIP image embeddings (img_emb, 768-dim) with a bundled IVF_PQ index for similarity search.
• Full LAION metadata — captions, URLs, NSFW flags, EXIF, dimensions, similarity scores.
• One columnar dataset — scan metadata cheaply, then fetch image bytes only for the rows you want.

Splits

train.lance

Schema

Column	Type	Notes
key	int64	Row key (natural join key for merges)
image	large_binary	Inline JPEG bytes
image_path	string	Original filename
caption	string	Image caption
url	string	Source URL
NSFW	int64	0 = safe, 1 = NSFW
LICENSE	string	Per-row license tag
similarity	float64	CLIP image–text cosine similarity
width, height	int64	Image dimensions
original_width, original_height	int64	Original dimensions before resize
exif, md5, status, error_message	string	Provenance / metadata
img_emb	fixed_size_list<float32, 768>	CLIP image embedding

Pre-built indices

• IVF_PQ on img_emb — vector similarity search (L2)

Why Lance?

1. Blazing Fast Random Access: Optimized for fetching scattered rows, making it ideal for random sampling, real-time ML serving, and interactive applications without performance degradation.
2. Native Multimodal Support: Store text, embeddings, and other data types together in a single file. Large binary objects are loaded lazily, and vectors are optimized for fast similarity search.
3. Native Index Support: Lance comes with fast, on-disk, scalable vector and FTS indexes that sit right alongside the dataset on the Hub, so you can share not only your data but also your embeddings and indexes without your users needing to recompute them.
4. Efficient Data Evolution: Add new columns and backfill data without rewriting the entire dataset. This is perfect for evolving ML features, adding new embeddings, or introducing moderation tags over time.
5. Versatile Querying: Supports combining vector similarity search, full-text search, and SQL-style filtering in a single query, accelerated by on-disk indexes.
6. Data Versioning: Every mutation commits a new version; previous versions remain intact on disk. Tags pin a snapshot by name, so retrieval systems and training runs can reproduce against an exact slice of history.

Load with datasets.load_dataset

You can load Lance datasets via the standard HuggingFace datasets interface, suitable if your pipeline already speaks Dataset / IterableDataset or you want a quick streaming sample without installing anything Lance-specific.

import datasets

hf_ds = datasets.load_dataset("lance-format/laion-1m", split="train", streaming=True)
for row in hf_ds.take(3):
    print(row["caption"])

Load with LanceDB

LanceDB is the embedded retrieval library built on top of the Lance format (docs), and is the interface most users interact with. It wraps the dataset as a queryable table with search and filter builders, and is the entry point used by the Search, Curate, Evolve, and Versioning examples below.

import lancedb

db = lancedb.connect("hf://datasets/lance-format/laion-1m/data")
tbl = db.open_table("train")
print(len(tbl))

Tip — for production use, download locally first. Streaming from the Hub works for exploration, but heavy random access and ANN search are far faster against a local copy:

hf download lance-format/laion-1m --repo-type dataset --local-dir ./laion-1m

Then point LanceDB at ./laion-1m/data.

Search

The bundled IVF_PQ index on img_emb makes approximate-nearest-neighbor search a single call. In production you would encode a user prompt or query image through CLIP at runtime and pass the resulting 768-d vector to tbl.search(...). The example below uses the embedding from row 42 as a runnable stand-in.

import lancedb

db = lancedb.connect("hf://datasets/lance-format/laion-1m/data")
tbl = db.open_table("train")

query = (
    tbl.search()
    .select(["img_emb"])
    .limit(1)
    .offset(42)
    .to_list()[0]["img_emb"]
)

hits = (
    tbl.search(query)
    .metric("L2")
    .select(["caption", "url", "similarity"])
    .limit(10)
    .to_list()
)
for r in hits:
    print(f"{r['similarity']:.3f} | {r['caption'][:80]}")

Tune metric, nprobes, and refine_factor to trade recall against latency for your workload.

Curate

Building a focused subset usually means combining similarity with metadata filters. Lance evaluates both inside a single query, so the candidate set comes back already filtered. The example below finds images visually similar to a seed row and restricts the result to safe-rated, high-resolution rows in one call. The bounded .limit(500) keeps the output small enough to inspect or hand off.

import lancedb

db = lancedb.connect("hf://datasets/lance-format/laion-1m/data")
tbl = db.open_table("train")

seed = (
    tbl.search()
    .select(["img_emb", "caption"])
    .limit(1)
    .offset(42)
    .to_list()[0]
)

candidates = (
    tbl.search(seed["img_emb"])
    .where('"NSFW" = 0 AND similarity > 0.3 AND width >= 512', prefilter=True)
    .select(["key", "url", "caption", "similarity"])
    .limit(500)
    .to_list()
)
print(f"{len(candidates)} candidates around: {seed['caption'][:60]}")

The result is a plain list of dictionaries, ready to inspect, persist as a manifest of row keys, or feed into the Evolve and Train workflows below.

Evolve

Lance stores each column independently, so a new column can be appended without rewriting the existing data. The lightest form is a SQL expression: derive the new column from columns that already exist, and Lance computes it once and persists it. The example below adds a precomputed aspect_ratio and an is_high_res flag, either of which can then be used directly in where clauses without recomputing the predicate on every query.

Note: Mutations require a local copy of the dataset, since the Hub mount is read-only. See the Materialize-a-subset section at the end of this card for a streaming pattern that downloads only the rows and columns you need, or use hf download to pull the full corpus.

import lancedb

db = lancedb.connect("./laion-1m/data")  # local copy required for writes
tbl = db.open_table("train")

tbl.add_columns({
    "aspect_ratio": "CAST(width AS DOUBLE) / CAST(height AS DOUBLE)",
    "is_high_res": "width >= 512 AND height >= 512",
})

The original columns and indices are untouched, so existing code that does not reference the new columns continues to work unchanged.

If the values you want to attach already live in another table (offline labels, classifier predictions, aesthetic scores), add the new fields and update matching rows by key:

import pyarrow as pa

labels = pa.table({
    "key": pa.array([0, 1, 2]),
    "aesthetic_score": pa.array([7.1, 6.4, 8.9]),
})
tbl.add_columns([f for f in labels.schema if f.name != "key"])
tbl.merge_insert("key").when_matched_update_all().execute(labels)

Matched rows receive the new values while unmatched rows stay null, and the update commits as a new table version. For column values that require a Python computation (e.g., running an embedding model over the image bytes), Lance provides a batch-UDF API in the underlying library — see the Lance data evolution docs for that pattern.

Train

Projection lets a training loop read only the columns each step actually needs. LanceDB tables expose this through Permutation.identity(tbl).select_columns([...]), which plugs straight into the standard torch.utils.data.DataLoader so prefetch, shuffling, and batching behave as in any PyTorch pipeline. Columns added in the Evolve section above cost nothing per batch until they are explicitly projected.

import lancedb
from lancedb.permutation import Permutation
from torch.utils.data import DataLoader

db = lancedb.connect("hf://datasets/lance-format/laion-1m/data")
tbl = db.open_table("train")

train_ds = Permutation.identity(tbl).select_columns(["image", "caption"])
loader = DataLoader(train_ds, batch_size=256, shuffle=True, num_workers=4)

for batch in loader:
    # batch carries only the projected columns; img_emb / img_emb_dinov3 stay on disk.
    # decode the JPEG bytes, tokenize the captions, forward, backward...
    ...

Switching feature sets is a configuration change: passing ["img_emb_dinov3", "caption"] to select_columns(...) on the next run reads only those columns, with no data movement or shard reorganization.

Versioning

Every mutation to a Lance dataset, whether it adds a column, merges labels, or builds an index, commits a new version. Previous versions remain intact on disk. You can list versions and inspect the history directly from the Hub copy; creating new tags requires a local copy since tags are writes.

import lancedb

db = lancedb.connect("hf://datasets/lance-format/laion-1m/data")
tbl = db.open_table("train")

print("Current version:", tbl.version)
print("History:", tbl.list_versions())
print("Tags:", tbl.tags.list())

Once you have a local copy, tag a version for reproducibility:

local_db = lancedb.connect("./laion-1m/data")
local_tbl = local_db.open_table("train")
local_tbl.tags.create("aesthetic-v1", local_tbl.version)

A tagged version can be opened by name, or any version reopened by its number, against either the Hub copy or a local one:

tbl_v1 = db.open_table("train", version="aesthetic-v1")
tbl_v5 = db.open_table("train", version=5)

Pinning supports two workflows. A retrieval system locked to aesthetic-v1 keeps returning stable results while the dataset evolves in parallel; newly added columns or labels do not change what the tag resolves to. A training experiment pinned to the same tag can be rerun later against the exact same data, so changes in metrics reflect model changes rather than data drift. Neither workflow needs shadow copies or external manifest tracking.

Materialize a subset

Reads from the Hub are lazy, so exploratory queries only transfer the columns and row groups they touch. Mutating operations (Evolve, tag creation) need a writable backing store, and a training loop benefits from a local copy with fast random access. Both can be served by a subset of the dataset rather than the full corpus. The pattern is to stream a filtered query through .to_batches() into a new local table; only the projected columns and matching row groups cross the wire, and the bytes never fully materialize in Python memory.

import lancedb

remote_db = lancedb.connect("hf://datasets/lance-format/laion-1m/data")
remote_tbl = remote_db.open_table("train")

batches = (
    remote_tbl.search()
    .where('"NSFW" = 0 AND similarity > 0.35 AND width >= 512')
    .select(["key", "image", "caption", "url", "img_emb"])
    .to_batches()
)

local_db = lancedb.connect("./laion-subset")
local_db.create_table("train", batches)

The resulting ./laion-subset is a first-class LanceDB database. Every snippet in the Evolve, Train, and Versioning sections above works against it by swapping hf://datasets/lance-format/laion-1m/data for ./laion-subset.

Citation

@article{schuhmann2022laion5b,
  title={LAION-5B: An open large-scale dataset for training next generation image-text models},
  author={Schuhmann, Christoph and others},
  journal={NeurIPS Datasets and Benchmarks Track},
  year={2022}
}

License

Content inherits LAION's original licensing and safety guidelines. Review LAION policy before downstream use.