本页尚未翻译

本页内容仍为英文原文，源自官方文档 basics/catalog。中文翻译正在分批进行中。

LanceDB Lakehouse

LanceDB Lakehouse objects are organized into a catalog of databases, namespaces and tables -- connect to a database to work with them.

Catalog Hierarchy

A catalog keeps track of named data objects, such as tables and views, so applications can discover and access them. In LanceDB Enterprise, the catalog is the top-level entry point: it organizes databases, which in turn organize namespaces and tables -- similar to a catalog in Postgres, Snowflake, or other database systems. The catalog itself has no name of its own -- it's just the root -- and an Enterprise deployment's catalog is identified by its endpoint, not a name like the objects beneath it.

LanceDB Embedded has no catalog at all. Connect directly to a database instead of going through a catalog.

A database contains namespaces that group related tables and views. Namespaces can also contain other namespaces, letting teams organize data by project, environment or purpose.

For example, a search application can keep production and staging data in separate namespaces within the same database, with prod further split into a nested search namespace:

mermaid

%%{init: {"theme": "base", "themeVariables": {
  "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
  "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
  "edgeLabelBackground": "#ffffff", "fontFamily": "Inter, sans-serif", "fontSize": "18px"
}}}%%
flowchart TD
    catalog["root catalog"] --> database["ai_data · database"]
    database --> prod["prod · namespace"]
    database --> staging["staging · namespace"]
    prod --> search["search · namespace"]
    search --> search_docs["documents · table"]
    search --> search_view["recent_documents · view"]
    search --> search_fn["embed · function"]
    staging --> staging_docs["documents · table"]
    classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19

Object	What it represents
Catalog	The unnamed root of an Enterprise deployment, identified by its endpoint; lists, creates and drops databases. Not present in Embedded.
Database	A container for namespaces and their data objects.
Namespace	A named group of catalog objects -- tables, views, secrets, functions, storage integrations, or nested namespaces -- addressed by its namespace path.
Table	Data stored as rows and columns.
View	A saved query, evaluated when read. Available in Enterprise.
Materialized view	The stored results of a query, updated by a refresh. Available in Enterprise.
Secret	A named credential that a Function binds by reference. Available in Enterprise.
Storage integration	A named credential for reading or importing from storage outside the deployment's own tables, such as an S3 prefix or a HuggingFace repository. Available in Enterprise.
Function	Registered code used to compute column values. Available in Enterprise.

Namespace name and path

A namespace's name is just its own component, such as search. Its path is the full sequence of names from the database down to it: in the diagram above, search sits inside prod, so its path is the sequence prod, search. A namespace path is never written by joining those names with . -- a dot is a valid character inside a name, so a dot-joined path couldn't always be split back into its components unambiguously. The two forms below are how SQL actually writes one.

Schema

SQL addresses a namespace path as a schema, the ANSI standard's term for the same concept: a nested path collapses into one identifier, its components joined by $ instead of .. The path's length changes what the schema looks like:

Path	Schema
`[]` (the database root)	`public` -- the reserved name for the empty path
`["prod"]`	`prod` -- one component needs no joining
`["prod", "search"]`	`prod$search` -- joined, and backtick-quoted because `$` isn't a bare-identifier character

$ can't appear in an ordinary name, so a schema can always be split back into its exact path, unlike a dot-joined one. (This "schema" is the namespace path as a SQL identifier -- distinct from a table's Arrow schema, its column names and types.)

Object identifier

An object -- a table, view, function, secret, or storage integration -- is addressed in SQL by the ANSI three-part identifier database.schema.name, where schema is exactly the $-joined path above. The documents table in search, nested inside prod, in database ai_data, is ai_data.`prod$search`.documents. One directly under the database root uses public: ai_data.public.documents.

See Namespace for the full CREATE/ALTER/DROP syntax.

Connecting

You always run table and query operations within a database; cross-database operations go through SQL instead. The same Python, TypeScript and Rust SDKs reach a database and the catalog separately.

Connecting to Catalog

connect_catalog(endpoint) connects to Enterprise's root catalog, which can list, create and drop databases across the whole deployment. Its connect_database(name) and create_database(name) then return the same kind of database connection lancedb.connect() does.

This connection is for database administration, not table or query work: list, create and drop databases through it, then switch to a database connection for everything else -- tables, queries, and SQL all run within a database, never directly against the catalog. Local filesystem and object-store deployments have no catalog to connect to -- each is already scoped to one database.

Connecting to Database

lancedb.connect(uri) opens one database in a single call -- a local directory, an object-storage path, or (with a db:// URI) a specific Enterprise database. The table below shows which URI reaches which kind of database directly.

URI	Mode	Where queries run
A local path, such as `./data`	Embedded	In your process
`s3://`, `gs://` or `az://`	Embedded	In your process, reading object storage
`db://`	Enterprise	On your cluster

A local directory

Embedded in your application, LanceDB reads and writes the tables in a directory on disk.

python

import lancedb

uri = "ex_lancedb"
db = lancedb.connect(uri)

python

import lancedb

uri = "ex_lancedb"
async_db = await lancedb.connect_async(uri)

typescript

import * as lancedb from "@lancedb/lancedb";

const db = await lancedb.connect("ex_lancedb");

rust

use lancedb::connect;

let db = connect("ex_lancedb").execute().await?;

Object storage

With an object storage URI the library is still embedded: your process runs every query and reads and writes the bucket directly, with credentials from its environment or its connection's storage_options.

python

import lancedb

uri = "s3://your-bucket/path"
# You can also use "gs://your-bucket/path" or "az://your-container/path".
db = lancedb.connect(uri)

python

import lancedb

uri = "s3://your-bucket/path"
# You can also use "gs://your-bucket/path" or "az://your-container/path".
async_db = await lancedb.connect_async(uri)

typescript

import * as lancedb from "@lancedb/lancedb";

const uri = "s3://your-bucket/path";
// You can also use "gs://your-bucket/path" or "az://your-container/path".
const db = await lancedb.connect(uri);

In Rust, pass the same URI to connect.

LanceDB supports these object storage backends:

Backend	URI scheme	Credentials
AWS S3, and S3-compatible stores (MinIO, Tigris, ...)	`s3://`	`AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY`, or `storage_options`. Compatible stores also need `endpoint` and `region`.
Google Cloud Storage	`gs://`	`GOOGLE_SERVICE_ACCOUNT` (path to a service account JSON), or `storage_options`
Azure Blob Storage	`az://`	`AZURE_STORAGE_ACCOUNT_NAME` + `AZURE_STORAGE_ACCOUNT_KEY` (or a SAS token), or `storage_options`
Tencent COS	`cos://`	`TENCENTCLOUD_SECRET_ID` / `TENCENTCLOUD_SECRET_KEY`, or `storage_options`
GooseFS (a cache layer over COS or S3)	`goosefs://`	Reads from the GooseFS client environment

Pass credentials as environment variables (uppercase), or as a storage_options dict at connection or table level (lowercase keys); a table's storage_options inherit the connection's and override on a per-key basis.

Choosing among these is a tradeoff between latency, scalability, cost and operational complexity, before you even pick a cloud-specific backend: object storage (S3, GCS, Azure Blob) has the highest latency but is the cheapest and scales storage effectively without limit; networked file storage (EFS, Filestore, Azure Files) and third-party stores (MinIO, WekaFS) sit in between; block storage (EBS, Persistent Disk, Managed Disk) and local NVMe are the fastest but tie data to one instance, so scaling means sharding or copying it yourself.

S3-compatible stores

python

import lancedb

db = lancedb.connect(
    "s3://bucket/path",
    storage_options={
        "region": "us-east-1",
        "endpoint": "http://minio:9000",
    },
)

typescript

async function storageS3Minio() {
  const db = await lancedb.connect("s3://bucket/path", {
    storageOptions: {
      region: "us-east-1",
      endpoint: "http://minio:9000",
    },
  });
  return db;
}

If the endpoint is http:// (common in local development), also set ALLOW_HTTP=true or pass allow_http=True in storage_options.

S3 Express

python

import lancedb

db = lancedb.connect(
    "s3://my-bucket--use1-az4--x-s3/path",
    storage_options={
        "region": "us-east-1",
        "s3_express": "true",
    },
)

typescript

async function storageS3Express() {
  const db = await lancedb.connect(
    "s3://my-bucket--use1-az4--x-s3/path",
    {
      storageOptions: {
        region: "us-east-1",
        s3Express: "true",
      },
    },
  );
  return db;
}

Consult AWS's networking requirements for S3 Express before enabling it.

Server-side encryption with KMS

Set aws_server_side_encryption to aws:kms and aws_sse_kms_key_id to the key ID or ARN to encrypt at rest with an AWS KMS key. The same options apply at connection or table level, and combine with bucket-level default encryption.

python

import lancedb

db = lancedb.connect(
    "s3://bucket/path",
    storage_options={
        "aws_server_side_encryption": "aws:kms",
        "aws_sse_kms_key_id": "<your-kms-key-id-or-arn>",
    },
)

typescript

async function storageS3SseKms() {
  const db = await lancedb.connect("s3://bucket/path", {
    storageOptions: {
      awsServerSideEncryption: "aws:kms",
      awsSseKmsKeyId: "<your-kms-key-id-or-arn>",
    },
  });
  return db;
}

The IAM principal needs kms:Encrypt, kms:Decrypt and kms:GenerateDataKey on the configured KMS key.

说明

Tencent COS and GooseFS build availability. Both are bundled by default in the Python wheel. In the Node binding, GooseFS is bundled by default but COS is not -- build with the cos Cargo feature enabled to use it there. In Rust, neither is in the default feature set; build with the cos or goosefs Cargo feature enabled to use either one.

LanceDB Enterprise

A db:// URI connects directly to one database in LanceDB Enterprise, where your cluster runs queries and background work. Get the database URI, API key, region and endpoint from your administrator, then replace the placeholders below. Use your cluster's region in place of us-east-1. In application code, load credentials from environment variables or a secrets manager.

python

import lancedb

db = lancedb.connect(
    uri="db://your-database",
    api_key="your-api-key",
    region="us-east-1",
    host_override="https://your-endpoint",
    sql_host_override="grpc+tls://your-sql-endpoint:10026",
)

typescript

import * as lancedb from "@lancedb/lancedb";

const db = await lancedb.connect("db://your-database", {
  apiKey: "your-api-key",
  region: "us-east-1",
  hostOverride: "https://your-endpoint",
});

In Rust, enable the crate's remote feature; the connection builder then takes the same settings through api_key, region and host_override. To use OAuth instead of an API key, see Authentication; for every connection option, see Database.

Organization and access control

Separate projects and environments. Keep production and staging tables in distinct namespaces, even when they use the same table names.
Find shared datasets. Give teams a consistent place to discover and access the data they work with.
Control access in Enterprise. Use permissions on databases, namespaces and tables to manage who can access shared data and what they can do with it.

Learn more

Catalog: initial access to a new deployment, and the superuser bootstrap.
Databases: connect to and manage a database.
Namespaces: create groups and address tables within them.
Tables: create, inspect and manage your data.
Views and materialized views: work with query-defined data.
Secrets: store a credential that a Function can bind by reference.
Storage integrations: read or import from storage outside your own tables.
Authorization: configure Enterprise access permissions.

providers

materializedview

namespace

table

transaction

LanceDB Lakehouse

Catalog Hierarchy

Namespace name and path

Schema

Object identifier

Connecting

Connecting to Catalog

Connecting to Database

A local directory

Object storage

S3-compatible stores

S3 Express

Server-side encryption with KMS

LanceDB Enterprise

Organization and access control

Learn more

LanceDB Lakehouse ​

Catalog Hierarchy ​

Namespace name and path ​

Schema ​

Object identifier ​

Connecting ​

Connecting to Catalog ​

Connecting to Database ​

A local directory ​

Object storage ​

S3-compatible stores ​

S3 Express ​

Server-side encryption with KMS ​

LanceDB Enterprise ​

Organization and access control ​

Learn more ​

LanceDB Lakehouse

Catalog Hierarchy

Namespace name and path

Schema

Object identifier

Connecting

Connecting to Catalog

Connecting to Database

A local directory

Object storage

S3-compatible stores

S3 Express

Server-side encryption with KMS

LanceDB Enterprise

Organization and access control

Learn more