LanceDB 中文文档

外观

本页尚未翻译

本页内容仍为英文原文，源自官方文档 deployment/slurm。中文翻译正在分批进行中。

Slurm ​

Run LanceDB Enterprise's own pods as Slurm-scheduled jobs, and reach a Neocloud's GPU capacity through Slurm in the multicloud case.

Applies to: Embedded ❌, Enterprise ✅

Slurm is the workload manager most GPU clusters run to schedule and account for jobs -- queueing work across shared nodes, enforcing fair-share and priority policies between teams, and tracking GPU-hour usage per job. That makes it the de facto standard in AI training infrastructure, where expensive GPU nodes are shared across many training and data jobs instead of dedicated to one team. GPU-focused cloud providers ("Neoclouds") such as CoreWeave and Nebius run Slurm-based scheduling across their Kubernetes clusters for exactly this reason.

Slurm shows up for LanceDB Enterprise in two related ways. Its own pods -- query nodes, plan executors, and background job workers alike -- can run as Slurm-scheduled jobs rather than plain Kubernetes pods, under the same scheduling and accounting as everything else on the cluster. And in the multicloud case, when the remote cloud is a Neocloud, Slurm is commonly how the dispatched GPU work actually gets scheduled there.

How it fits together ​

Scheduling LanceDB's pods through Slurm puts them under the same scheduler as GPU training jobs, so the cluster's capacity is accounted for consistently regardless of which one is using it -- Slurm's fair-share and priority policies apply to LanceDB the same way they apply to a training job, including preemption when training needs the capacity back.

mermaid
%%{init: {"theme": "base", "themeVariables": {
  "background": "#ffffff", "primaryColor": "#ffffff", "primaryTextColor": "#221d19",
  "primaryBorderColor": "#ad9e95", "lineColor": "#6b625c", "textColor": "#221d19",
  "clusterBkg": "#f4ebe8", "clusterBorder": "#ad9e95", "edgeLabelBackground": "#ffffff",
  "fontFamily": "Inter, sans-serif", "fontSize": "18px"
}}}%%
flowchart LR
    subgraph P["Your cloud"]
        direction TB
        Job["Job orchestration"]
    end

    subgraph N["Neocloud (e.g. CoreWeave, Nebius)"]
        direction TB
        Slurm["Slurm scheduler"]
        Train["GPU training jobs"]
        LDB["LanceDB Enterprise pods"]
        Slurm --> Train
        Slurm --> LDB
        Train -.->|"can preempt"| LDB
    end

    Job -- "dispatches GPU work" --> Slurm
    LDB -- "results" --> Job
    classDef default fill:#ffffff,stroke:#ad9e95,stroke-width:1px,color:#221d19
    classDef accent fill:#fff1ec,stroke:#e8582d,stroke-width:1.5px,color:#221d19
    class Slurm accent

Why it matters ​

Most GPU infrastructure already runs Slurm for training. Deploying LanceDB Enterprise into that same infrastructure, under the same scheduler, means it can share capacity with training workloads safely -- queued, accounted, and preemptible like everything else -- instead of requiring a separate, dedicated pool of nodes carved out just for LanceDB. Combined with multicloud dispatch, that makes it practical to keep your primary deployment wherever it already lives and still reach GPU capacity on a Neocloud built around Slurm, only for the work that needs it.

Contact us to learn more about running LanceDB Enterprise on a Slurm-scheduled cluster.

MulticloudThe cross-cloud dispatch model this page's Neocloud case builds on.ArchitectureThe query, executor, cache and jobs fleets this scheduling applies to.