外观
本页尚未翻译
本页内容仍为英文原文，源自官方文档 training/observability。中文翻译正在分批进行中。
Observability
Pipeline counters for diagnosing StreamingDataset bottlenecks.
Applies to: Embedded ✅, Enterprise ✅
Optimizing data loader performance can be tricky, as runtime bottlenecks are often difficult to locate across various stages of the I/O and CPU pipelines. To help distinguish bottlenecks, StreamingDataset exposes several pipeline counters: raw_queue_depth shows the number of loaded rows waiting to be transformed, and prefetch_queue_depth is the number of transformed rows ready to be consumed. unscanned_rows and consumed_rows show how far the current iterator has progressed.
If the prefetch_queue_depth is consistently zero but the raw_queue_depth is not, then you likely have a CPU transformation bottleneck. You should investigate GIL bottlenecks or look for ways to optimize your transformation. This can often be done by batching the compute work. If both prefetch_queue_depth and raw_queue_depth are consistently zero while the consumer is waiting, I/O is the likely bottleneck. A larger read batch size or clumped shuffling could help.
python
import threading, time
from lancedb.streaming import StreamingDataset
ds = StreamingDataset(table, shuffle_seed=42)
def log_pipeline_health():
while True:
print(
f"unscanned={ds.unscanned_rows} "
f"raw={ds.raw_queue_depth} "
f"cooked={ds.prefetch_queue_depth} "
f"consumed={ds.consumed_rows}"
)
time.sleep(1.0)
monitor = threading.Thread(target=log_pipeline_health, daemon=True)
monitor.start()
for sample in ds:
train_step(sample)
print(
f"bytes loaded: {ds.bytes_loaded} "
f"fetch time: {ds.fetch_time:.2f}s "
f"transform time: {ds.transform_time:.2f}s"
)bytes_loaded measures raw Arrow buffer bytes before transformation. bytes_loaded, fetch_time, and transform_time are cumulative across iterations of the same dataset instance; because work runs concurrently, the summed stage times can exceed wall-clock time. The queue depths report rows currently waiting in the pipeline, and the progress counters reflect the latest iterator snapshot.