本页尚未翻译

本页内容仍为英文原文，源自官方文档 ingestion/define-storage-integration。中文翻译正在分批进行中。

Define Storage Integration

Set up a reusable credential for AWS, GCP, Azure or HuggingFace before importing data from it.

Applies to: Embedded ❌, Enterprise ✅

Before you can import data from external storage or attach an existing database, define a storage integration once per location. It is a namespace-scoped catalog object that stores the credential -- neither operation names the integration directly, LanceDB matches the URI you read against whichever integration's prefix covers it.

What each provider needs

Provider	Credential	Notes
AWS S3	An IAM role to assume, or a static access key pair	A role needs one extra step outside LanceDB: trust policy setup.
Google Cloud Storage	A service account's JSON key	No role-based alternative.
Azure Blob Storage	A service principal's tenant, client ID and client secret	All three required together.
HuggingFace	An access token	Always required to create an integration. For anonymous access at HuggingFace's default rate limits, register no integration for that prefix instead.

Every credential option takes a Secret reference, never the value directly -- create the Secret first.

Create one

AWS S3

Give either an IAM role to assume, or a static access key pair -- mixing the two, or supplying only one half of the key pair, is refused.

CREATE STORAGE INTEGRATION raw
    STORAGE_PROVIDER = 'S3'
    STORAGE_PREFIX = 's3://customer-data/raw/'
    STORAGE_AWS_ROLE_ARN = 'arn:aws:iam::123456789012:role/lancedb-access'

CREATE STORAGE INTEGRATION raw
    STORAGE_PROVIDER = 'S3'
    STORAGE_PREFIX = 's3://customer-data/raw/'
    STORAGE_AWS_ACCESS_KEY_ID_SECRET = aws_key_id
    STORAGE_AWS_SECRET_ACCESS_KEY_SECRET = aws_key_secret

db.create_storage_integration(
    "raw", "s3://customer-data/raw/",
    storage_provider="S3",
    aws_role_arn="arn:aws:iam::123456789012:role/lancedb-access",
)

db.create_storage_integration(
    "raw", "s3://customer-data/raw/",
    storage_provider="S3",
    aws_access_key_id_secret="aws_key_id",
    aws_secret_access_key_secret="aws_key_secret",
)

use lancedb::storage_integration::CreateStorageIntegrationRequest;

let info = db.create_storage_integration(
    "raw",
    CreateStorageIntegrationRequest::s3(
        "s3://customer-data/raw/",
        "arn:aws:iam::123456789012:role/lancedb-access",
    ),
    &[],
).await?;

Google Cloud Storage

CREATE STORAGE INTEGRATION gcs
    STORAGE_PROVIDER = 'GCP'
    STORAGE_PREFIX = 'gs://customer-data/raw/'
    GCP_CREDENTIALS_JSON_SECRET = gcp_service_account

db.create_storage_integration(
    "gcs", "gs://customer-data/raw/",
    storage_provider="GCP",
    gcp_credentials_json_secret="gcp_service_account",
)

The Secret holds the full service-account JSON key. GCP has no role-based alternative.

Azure Blob Storage

CREATE STORAGE INTEGRATION blobs
    STORAGE_PROVIDER = 'AZURE'
    STORAGE_PREFIX = 'az://customer-data/raw/'
    STORAGE_AZURE_STORAGE_ACCOUNT = 'customeraccount'
    STORAGE_AZURE_TENANT_ID_SECRET = azure_tenant
    STORAGE_AZURE_CLIENT_ID_SECRET = azure_client
    STORAGE_AZURE_CLIENT_SECRET_SECRET = azure_credential

db.create_storage_integration(
    "blobs", "az://customer-data/raw/",
    storage_provider="AZURE",
    azure_storage_account="customeraccount",
    azure_tenant_id_secret="azure_tenant",
    azure_client_id_secret="azure_client",
    azure_client_secret_secret="azure_credential",
)

All four options are required together; there is no role-based alternative for Azure.

HuggingFace

CREATE STORAGE INTEGRATION hub
    STORAGE_PROVIDER = 'HUGGINGFACE'
    STORAGE_PREFIX = 'hf://datasets/owner/private/data/'
    HUGGINGFACE_TOKEN_SECRET = hf_token

db.create_storage_integration(
    "hub", "hf://datasets/owner/private/data/",
    storage_provider="HUGGINGFACE",
    huggingface_token_secret="hf_token",
)

HUGGINGFACE_TOKEN_SECRET is required to create this integration; omitting it is rejected, not treated as anonymous. For anonymous access at HuggingFace's default rate limits, register no integration for that prefix at all.

Finish the AWS trust policy

An IAM role integration needs one step outside LanceDB: describing it returns the trust_policy and permissions_policy documents your role needs. Attach them in AWS, then describe the integration again and check access_verified to confirm LanceDB can assume the role. A static access key pair skips this step entirely.

See Storage Integration for every provider's complete option reference, S3's two credential modes in full, and the output schema, or Storage Integration for the Python and SQL CRUD operations at a glance.

What to read next

Import DatasetsRead or import through the integration once it exists.Attach DatabaseAttach an existing database at a location the integration covers.Storage IntegrationThe complete SQL reference: every provider's options and output columns.