LanceDB 中文文档

外观

本页尚未翻译

本页内容仍为英文原文，源自官方文档 enterprise/distributed-real-time-write。中文翻译正在分批进行中。

Distributed RTE Write ​

How Enterprise scales RTE writes horizontally across a dedicated serving tier, and the throughput limits that come with it.

Applies to: Embedded ❌, Enterprise ✅

RTE writes describes table.enable_rte(spec) as one call that works the same way on either connection. On Enterprise, that call does not run locally -- it posts to the server, which installs the write path on a dedicated, horizontally scalable serving tier instead of running it in your process. This page covers that tier: how it shards a table's writes, the one real limit sharding does not remove, and the throughput you can expect.

How writes are routed ​

Enabling RTE on a remote table sends the spec to the server, which installs the write path on the dataset instead of running it in your process. From then on, the query node routes every request for that table -- reads and writes alike -- to a single owning RTE node, a stateful component separate from the query nodes and plan executors that serve ordinary queries. A query node routes different tables to different RTE nodes, and a single RTE node can host shards from many different tables at once. RTE nodes share nothing but the underlying object storage; no node talks to another one directly.

mermaid
%%{init: {"theme": "base", "themeVariables": {
  "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
  "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
  "clusterBkg": "#f4ebe8", "clusterBorder": "#ad9e95", "edgeLabelBackground": "#ffffff",
  "fontFamily": "Inter, sans-serif", "fontSize": "18px"
}}}%%
flowchart LR
    Q["Query node<br/>routes each table's shards to an RTE node"]

    subgraph W0["RTE node A"]
        direction TB
        S0["Shard 0 — table 'frames'"]
        S1["Shard 0 — table 'events'"]
    end

    subgraph W1["RTE node B"]
        direction TB
        S2["Shard 0 — table 'metrics'"]
        S3["Shard 1 — table 'metrics'"]
    end

    Q -- "'frames'" --> W0
    Q -- "'events'" --> W0
    Q -- "'metrics'" --> W1
    classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19

Shards scale one table, replicas scale the cluster ​

RTEWriteSpec::bucket(column, num_buckets) splits a table into num_bucketstable shards, each a separate, independently-appending stream, giving the table that many parallel write paths instead of one. A shard's identity is derived from the table itself, not assigned by an operator, so a shard survives the table being claimed by a different RTE node later.

This is where the horizontal scaling actually happens: splitting frames into 8 shards gives it 8 independent append streams instead of 1, raising that table's own write ceiling. Adding RTE node replicas spreads shard ownership across more nodes, so the cluster can host more tables -- and more shards -- without any one node becoming a bottleneck for everything it serves.

RTEWriteSpec::identity(column) and RTEWriteSpec::unsharded() both resolve to a single table shard -- bucket is the sharding mode that actually parallelizes write throughput; the other two are about routing semantics, not scaling.

Throughput ​

A table shard is the unit of capacity: measured runs show it sustaining roughly 85-90 MB/s of durable append before acknowledgement, and throughput scaling nearly linearly with shard count up to at least 8 shards on one node -- the bottleneck is the durable write itself, not CPU or memory. This is observed capacity, not an enforced quota; nothing in the deployment caps a shard's or a table's throughput below it. Contact support if your workload needs more write throughput than this provides.

RTE Write (Upsert)Enable RTE and write to a table; the same call on Embedded or Enterprise.How the real time engine worksMemWAL, shards, memtables and SSTables -- the shared architecture.Distributed RTE ReadHow reads combine the WAL tier's fresh rows with the base table across the fleet.