本页尚未翻译

本页内容仍为英文原文，源自官方文档 curation/inspect-datasets。中文翻译正在分批进行中。

Inspect Datasets

Query a table, a HuggingFace dataset, or a dataset in object storage the same way, with plain SELECT, WHERE, projection and LIMIT.

Applies to: Embedded ❌, Enterprise ✅

The same SELECT statement inspects three kinds of source: a table already in your database, a HuggingFace dataset, or a dataset in object storage -- once a storage integration covers the latter two. Nothing is downloaded or imported until you choose to import it.

The basics: project, filter, limit

A table name, a HuggingFace URI, and an object-storage URI all work the same way after FROM:

SELECT id, text
FROM documents
WHERE text IS NOT NULL
LIMIT 100

• Project -- name the columns you need (id, text) instead of SELECT *, especially for a source with large or many columns.
• Filter -- WHERE narrows rows before they come back; combine predicates with AND/OR the same as any other table.
• Limit -- LIMIT n caps how many rows return. It bounds the result, not the read: a remote source is still fetched at file or row-group granularity, so a small LIMIT on a source with large files or row groups can still touch more data than the row count suggests.

The rest of this page swaps documents for a HuggingFace or object-storage source -- the SELECT/WHERE/LIMIT syntax above doesn't change.

Inspect a HuggingFace dataset

A single-quoted HuggingFace dataset URI works directly in a SQL FROM clause, the same as a table name -- no separate download, no intermediate file on disk:

SELECT id, text
FROM 'hf://datasets/owner/repo/data/train-*.parquet'
WHERE text IS NOT NULL
LIMIT 100

reader = db.execute_query("""
    SELECT id, text
    FROM 'hf://datasets/owner/repo/data/train-*.parquet'
    WHERE text IS NOT NULL
    LIMIT 100
""")
sample = reader.read_all().to_pylist()

The URI is hf://datasets/<owner>/<repo>[@<revision>]/<path>. <path> can be one Parquet file, a directory, or a glob: * matches within one directory, ** matches across directories, and a repository root selects every Parquet file beneath it. Select a dataset configuration or split by including it in the path. The default revision is main; each statement resolves the revision to a commit once, so every worker reads the same snapshot. This source reads Parquet files -- it does not run a dataset's own generation scripts, and it does not compute embeddings or build indexes, so chunk, embed, or index the result as a separate step, the same as data from any other source.

说明

Files matched by one statement must have compatible schemas, and an unmatched path is an error rather than a silent fallback to a different split. Nested Parquet values come through as-is; a floating-point list is not automatically converted into a fixed-size vector column.

Inspect a dataset in object storage

An s3://, gs:// or az:// prefix covered by a storage integration reads an existing Lance dataset at that prefix, the same way as a managed table:

SELECT id, text
FROM 's3://customer-data/raw/documents.lance'
WHERE text IS NOT NULL
LIMIT 100

This reads a dataset another process already wrote in Lance format, not an arbitrary file format -- there is no Parquet-style reader for cloud storage the way there is for hf://. See Storage Integration for every provider's setup.

Access a private repository or bucket

A private HuggingFace repository, one subject to HuggingFace's authenticated rate limits, or any non-public bucket needs a storage integration covering it. For HuggingFace that integration is bound to a token Secret; for a non-public S3 bucket it can instead be an IAM role integration, which needs no Secret at all -- see AWS S3.

db.create_secret("hf_token", "<your-hugging-face-token>")

# create_storage_integration is not in a released SDK yet; use the SQL
# form below until it ships.
db.execute_query("""
    CREATE STORAGE INTEGRATION private_hub
        STORAGE_PROVIDER = 'HUGGINGFACE'
        STORAGE_PREFIX = 'hf://datasets/owner/private/data/'
        HUGGINGFACE_TOKEN_SECRET = hf_token
""").read_all()

reader = db.execute_query(
    "SELECT * FROM 'hf://datasets/owner/private/data/*.parquet' LIMIT 10"
)

CREATE SECRET hf_token TYPE = GENERIC_STRING SECRET_STRING = '<your-hugging-face-token>'

CREATE STORAGE INTEGRATION private_hub
    STORAGE_PROVIDER = 'HUGGINGFACE'
    STORAGE_PREFIX = 'hf://datasets/owner/private/data/'
    HUGGINGFACE_TOKEN_SECRET = hf_token

SELECT * FROM 'hf://datasets/owner/private/data/*.parquet' LIMIT 10

A query never names the integration: the planner matches the most specific STORAGE_PREFIX that covers the URI being read, across every integration the caller has USAGE on. A HuggingFace integration always needs HUGGINGFACE_TOKEN_SECRET; creating one without it is rejected. For public access, register no integration for that prefix at all -- an hf:// read that matches no integration is anonymous, with no environment-variable fallback. See Storage Integration for every provider and the full credential model.

Import into a table

Once you've inspected a source and it looks right, CREATE TABLE ... AS SELECT imports it:

CREATE TABLE imported AS
SELECT *
FROM 'hf://datasets/owner/repo/data/train-*.parquet'

db.execute_query("""
    CREATE TABLE imported AS
    SELECT * FROM 'hf://datasets/owner/repo/data/train-*.parquet'
""").read_all()
table = db.open_table("imported")

This CREATE TABLE AS SELECT runs as a distributed import when the deployment has an available plan-executor fleet: the planner assigns disjoint file and row-group ranges to workers, each worker writes uncommitted Lance files directly to the destination, and a coordinator commits the completed table from their transaction metadata once every worker finishes. A query with a global limit, aggregation or join keeps its usual coordination; a small import can run as one local task without a fleet. The table becomes queryable only after that final commit -- nothing partial is ever visible, and a failed or cancelled import can resume the same durable job from its retained attempt rather than restarting from nothing. To start over after a failed one-off import instead, drop the partial table and rerun the statement.

说明

Importing requires a native database with database_manifest_enabled. Combining a HuggingFace source with a WAL-backed destination table in one CTAS is rejected, since the import cannot publish a base-only read that omits the destination's unflushed data.

What to read next

Storage IntegrationThe credential object behind private repository access.Create a FunctionAdd embeddings or derived columns to the imported data.Chunk records into multiple rowsSplit long text or media fields after importing.Table`CREATE TABLE AS SELECT` syntax and options in full.