本页尚未翻译

本页内容仍为英文原文，源自官方文档 sql/ddl/storage-integration。中文翻译正在分批进行中。

Storage Integration

CREATE, DESCRIBE, SHOW and DROP STORAGE INTEGRATION: a namespace-scoped, reusable credential for reading or importing from external storage.

Applies to: Embedded ❌, Enterprise ✅

A storage integration is a namespace-scoped catalog object that lets a query read from, or import from, a storage location outside the deployment's own tables -- an S3, GCS or Azure prefix, or a HuggingFace dataset repository -- without embedding a credential in the query itself. It is addressed and scoped the same way a Secret is: a 1-3 part identifier, name, namespace.name or database.namespace.name, with a nested namespace path $-joined into that middle part.

CREATE STORAGE INTEGRATION

CREATE STORAGE INTEGRATION <name>
    STORAGE_PROVIDER = 'S3' | 'AZURE' | 'GCP' | 'HUGGINGFACE'
    STORAGE_PREFIX = '<uri-prefix>'
    [ <provider option> = <value> [...] ]

STORAGE_PROVIDER and STORAGE_PREFIX are required for every provider. STORAGE_PREFIX is the URI prefix the integration covers -- a query only matches an integration whose prefix is a prefix of the URI it reads. Options ending in _SECRET take a Secret identifier, not a quoted string: name, namespace.name or database.namespace.name, with a nested namespace path $-joined in the middle part (db.`team$prod`.key). Omitted parts default to the session's current database and namespace. Creating the integration requires USAGE on each referenced Secret and on its database and namespace. Passing an option a provider does not accept, such as an Azure option on an S3 integration, is refused.

S3

CREATE STORAGE INTEGRATION raw
    STORAGE_PROVIDER = 'S3'
    STORAGE_PREFIX = 's3://customer-data/raw/'
    STORAGE_AWS_ROLE_ARN = 'arn:aws:iam::123456789012:role/lancedb-access'

CREATE STORAGE INTEGRATION raw
    STORAGE_PROVIDER = 'S3'
    STORAGE_PREFIX = 's3://customer-data/raw/'
    STORAGE_AWS_ACCESS_KEY_ID_SECRET = bootstrap.credentials.aws_key_id
    STORAGE_AWS_SECRET_ACCESS_KEY_SECRET = bootstrap.credentials.aws_key_secret

Option	Type	Meaning
STORAGE_AWS_ROLE_ARN	string	an IAM role to assume, by cross-account trust (see below)
STORAGE_AWS_ACCESS_KEY_ID_SECRET	Secret ref	a static access key ID, paired with the secret key below
STORAGE_AWS_SECRET_ACCESS_KEY_SECRET	Secret ref	the matching static secret access key
STORAGE_AWS_REGION	string, optional	the bucket's region
STORAGE_AWS_ENDPOINT	string, optional	an S3-compatible endpoint URL (MinIO, Tigris, ...) instead of AWS itself

Give either STORAGE_AWS_ROLE_ARN alone, or both key-pair options together -- mixing the two, or supplying only one half of the key pair, is refused.

Azure

CREATE STORAGE INTEGRATION blobs
    STORAGE_PROVIDER = 'AZURE'
    STORAGE_PREFIX = 'az://customer-data/raw/'
    STORAGE_AZURE_STORAGE_ACCOUNT = 'customeraccount'
    STORAGE_AZURE_TENANT_ID_SECRET = bootstrap.credentials.tenant
    STORAGE_AZURE_CLIENT_ID_SECRET = bootstrap.credentials.client
    STORAGE_AZURE_CLIENT_SECRET_SECRET = bootstrap.credentials.credential

Option	Type	Meaning
STORAGE_AZURE_STORAGE_ACCOUNT	string	the storage account name
STORAGE_AZURE_TENANT_ID_SECRET	Secret ref	Entra ID tenant
STORAGE_AZURE_CLIENT_ID_SECRET	Secret ref	service principal client ID
STORAGE_AZURE_CLIENT_SECRET_SECRET	Secret ref	service principal client secret

All four are required together; there is no role-based alternative for Azure.

GCP

CREATE STORAGE INTEGRATION gcs
    STORAGE_PROVIDER = 'GCP'
    STORAGE_PREFIX = 'gs://customer-data/raw/'
    GCP_CREDENTIALS_JSON_SECRET = bootstrap.credentials.gcp

GCP_CREDENTIALS_JSON_SECRET -- note the option name has no STORAGE_ prefix, unlike every other provider's options -- names a Secret holding the full service-account JSON key. It is required; GCP has no role-based alternative here.

HuggingFace

CREATE STORAGE INTEGRATION hub
    STORAGE_PROVIDER = 'HUGGINGFACE'
    STORAGE_PREFIX = 'hf://datasets/owner/private/data/'
    HUGGINGFACE_TOKEN_SECRET = bootstrap.credentials.hf_token

HUGGINGFACE_TOKEN_SECRET is the only HuggingFace option, and it is required: creating an integration without it is refused. For anonymous access, at HuggingFace's default anonymous rate limits, register no integration for that prefix at all. See Inspect Datasets for the full read and import workflow this integration unlocks.

The S3 trust policy

Creating an STORAGE_AWS_ROLE_ARN integration does not finish the AWS side of the setup. The result row's trust_policy, permissions_policy and configuration_instructions columns -- also visible from DESCRIBE -- are the IAM policy documents your role needs: attach trust_policy as the role's trust relationship, scoped to aws_account_id, aws_iam_principal_arn and aws_external_id from the same row, and attach permissions_policy to grant that role read access to the bucket. access_verified reports whether the deployment already confirmed it can assume the role and read the prefix; false typically means the trust policy has not been attached yet.

Reading and importing through the integration

A query or CREATE TABLE ... AS SELECT can reference a matching URI directly, once an integration covers it -- the integration is never named in the statement. What the URI reads depends on the provider:

• HuggingFace (hf://datasets/...) reads Parquet files from a dataset repository. See Inspect Datasets for the full workflow, including how a CTAS import distributes across plan executors.
• S3, GCS and Azure (s3://, gs://, az://) read an existing Lance dataset at that prefix, the same way as a managed table:

SELECT * FROM 's3://customer-data/raw/events.lance' LIMIT 10

CREATE TABLE events_copy AS
SELECT * FROM 's3://customer-data/raw/events.lance'

A CTAS import distributes the same way regardless of provider: disjoint file and row-group ranges go to plan-executor workers, each writes uncommitted Lance files, and a coordinator commits the table once every worker finishes. See CREATE TABLE for CREATE TABLE AS SELECT syntax in general.

DESCRIBE STORAGE INTEGRATION

DESCRIBE STORAGE INTEGRATION raw
DESC STORAGE INTEGRATION platform.hub

Returns one row with every column CREATE returns: the provider, prefix, the Secret references and plain-string options that apply to its provider (the rest are null), created_at_millis, access_verified, and -- for an STORAGE_AWS_ROLE_ARN integration -- the generated AWS policy documents from above. No call ever returns a Secret's value, only the reference naming it.

SHOW STORAGE INTEGRATIONS

SHOW STORAGE INTEGRATIONS
SHOW STORAGE INTEGRATIONS IN platform
LIST STORAGE INTEGRATIONS

LIST is an alias for SHOW here. Returns one name column; IN narrows the listing to one namespace.

DROP STORAGE INTEGRATION

DROP STORAGE INTEGRATION raw
DROP STORAGE INTEGRATION IF EXISTS raw

Returns dropped, true or false. There is no ALTER STORAGE INTEGRATION: to rotate a credential, rotate the Secret it references (see Secret) rather than dropping and recreating the integration. A storage integration an attached database still depends on blocks the drop; detach the database first.

GRANT and REVOKE

GRANT CREATE STORAGE INTEGRATION ON SCHEMA platform TO ROLE platform_admins
GRANT USAGE ON STORAGE INTEGRATION platform.hub TO ROLE research
GRANT OWNERSHIP ON STORAGE INTEGRATION platform.hub TO ROLE platform_admins
REVOKE USAGE ON STORAGE INTEGRATION platform.hub FROM ROLE research

CREATE_STORAGE_INTEGRATION on the namespace creates one; USAGE on the integration itself lets a query match and read through it; OWNERSHIP grants full control, including dropping it. See Privileges for the full statement syntax, every grantee form, and how a query picks a matching integration without naming one.

What to read next

Inspect DatasetsQuery and import HuggingFace or object-storage datasets, the most common use of a storage integration today.SecretThe credential storage integration options reference.PrivilegesEvery privilege, and the grant and revoke syntax that covers each one.