本页尚未翻译

本页内容仍为英文原文，源自官方文档 geneva/deployment/dependency-verification。中文翻译正在分批进行中。

Dependency Verification

Diagnose and resolve package version mismatches between local and distributed worker environments.

When running Geneva UDFs on distributed workers, your code is serialized locally and executed on remote workers. If the worker environment differs from your local environment, you may encounter subtle and difficult-to-debug errors.

Example environment mismatch errors

Symptom	Likely Cause
`TypeError: Enum.__new__() missing 1 required positional argument`	`attrs` version mismatch
`TypeError: Can't instantiate abstract class`	Package structure differences
`ArrowInvalid: cannot cast` / serialization errors	NumPy 1.x vs 2.x mismatch
`ModuleNotFoundError` on workers	Package only installed locally
Model loading failures	PyTorch version mismatch
Permission denied errors	Missing API keys in envrionment variables

These issues are notoriously difficult to debug because the error messages often don't indicate the root cause.

The `compare_ray_environments` Tool

Geneva provides a diagnostic tool to compare your local environment against Ray workers.

If you are encountering a hang or exception you can use the following diagnosis worklflow to resolve the problem.

Step 1

Run the diagnostic tool programatically or via the CLI.

Step 2

Check PACKAGES and ENV VARS output sections for mismatches.

Step 3

Identify critical packages: numpy, torch, pyarrow, attrs, pydantic.

Step 4

Identify inconsistent environment variables: AWS_*, GOOGLE_APPLICATION_CREDENTIALS

Step 5

Fix with manifest for quick testing:

python

from geneva.manifest.builder import PipManifestBuilder

manifest = PipManifestBuilder.create("fix").pip(["numpy==1.26.4"]).build()

Step 6

OPTIONAL: Build custom image for production (if using KubeRay).

Programmatic Usage

python

from geneva.runners.ray.compare_env import compare_ray_environments

# Compare and print (requires Geneva context to be initialize via `with db.context(..)`)
result = compare_ray_environments()

# Compare environments, filtering environment variables with specified prefix
result = compare_ray_environments(env_prefix="PY")

CLI Usage

bash

# Connect to existing Ray cluster
python -m geneva.runners.ray.compare_env

# Start new local Ray cluster
python -m geneva.runners.ray.compare_env --address local

# Filter env vars by prefix
python -m geneva.runners.ray.compare_env --env-prefix RAY

# Show full JSON snapshots
python -m geneva.runners.ray.compare_env --show-all

# Skip sys.path comparison
python -m geneva.runners.ray.compare_env --no-sys-path

Understanding the Output

The tool outputs several sections to help you identify mismatches.

PYTHON / PLATFORM

Shows Python version and OS information for both environments:

=== PYTHON / PLATFORM ===
Local:
  Python: 3.11.9 (main, Apr 19 2024, 16:48:06) [GCC 11.2.0]
  Impl  : CPython
  Exec  : /home/user/.venv/bin/python
  OS    : Linux 5.15.0-generic (x86_64)

Remote:
  Python: 3.11.9 (main, Apr 19 2024, 16:48:06) [GCC 11.2.0]
  Impl  : CPython
  Exec  : /home/ray/anaconda3/bin/python
  OS    : Linux 5.4.0-aws (x86_64)

注意

Watch for different Python versions or different OS types (macOS local vs Linux remote).

Architecture Mismatch (macOS to Linux)

If you see different OS types (e.g., Darwin locally vs Linux remotely), compiled extensions may fail with ModuleNotFoundError or segfaults.

Solution: Run Geneva from the same OS/architecture as your cluster (typically Linux x86_64). Use a Linux VM, container, or remote development environment.

Environment Variables

Environment variables present in only one environment:

=== ENV VARS: keys only in LOCAL ===
  + CONDA_PREFIX
  + VIRTUAL_ENV

=== ENV VARS: keys only in REMOTE ===
  + RAY_ADDRESS
  + KUBERNETES_SERVICE_HOST

注意

Missing AWS_* or GOOGLE_APPLICATION_CREDENTIALS can cause storage authentication failures.

Passing Environment Variables to Workers

If critical environment variables are missing on workers, you can pass them via the manifest or cluster configuration. Set the values in the shell before starting the Python program:

bash

export AWS_ACCESS_KEY_ID="your-access-key-id"
export AWS_SECRET_ACCESS_KEY="your-secret-access-key"
export MY_API_KEY="your-api-key"

Option 1: Via Manifest

python

from geneva.manifest.builder import PipManifestBuilder
import os

manifest = (
    PipManifestBuilder.create("my-manifest")
    .env_vars({
        "AWS_ACCESS_KEY_ID": os.environ["AWS_ACCESS_KEY_ID"],
        "AWS_SECRET_ACCESS_KEY": os.environ["AWS_SECRET_ACCESS_KEY"],
        "MY_API_KEY": os.environ["MY_API_KEY"],
    })
    .build()
)

Option 2: Via Cluster Configuration

python

from geneva.cluster.builder import KubeRayClusterBuilder
import os

cluster = (
    KubeRayClusterBuilder.create("my-cluster")
    .ray_init_kwargs({
        "runtime_env": {
            "env_vars": {
                "AWS_ACCESS_KEY_ID": os.environ["AWS_ACCESS_KEY_ID"],
                "AWS_SECRET_ACCESS_KEY": os.environ["AWS_SECRET_ACCESS_KEY"],
            }
        }
    })
    .build()
)

注意

Avoid hardcoding secrets. Use os.environ to pass values from your local environment, or use a secrets manager in production.

Packages

The tool shows version mismatches and packages only present in one environment:

=== PACKAGES: version mismatches ===
  * numpy: local=1.26.4  remote=2.2.6
  * torch: local=2.0.1  remote=2.8.0+cpu
  * attrs: local=23.2.0  remote=24.2.0
  * pyarrow: local=14.0.1  remote=17.0.0

=== PACKAGES: only in LOCAL ===
  + my-custom-package
  + dev-tools

=== PACKAGES: only in REMOTE ===
  + kuberay-client

注意

Watch for major version differences (NumPy 1.x vs 2.x) and PyTorch version mismatches.

Common Package Issues

Issue	Symptoms	Fix
NumPy 1.x vs 2.x	`ArrowInvalid`, `ValueError: cannot convert`, serialization failures	Pin `numpy==1.26.4`
PyTorch mismatch	Model loading failures, CUDA errors, unexpected inference results	Pin to matching `torch` version
attrs mismatch	`TypeError: Enum.__new__() missing 1 required positional argument`	Pin `attrs` to local version
Missing package	`ModuleNotFoundError: No module named 'xyz'`	Add package to manifest

Fixing Package Mismatches

Option 1: Manifest pip Dependencies

Specify packages in a Geneva manifest for a quick fix:

python

import geneva
from geneva.manifest.builder import PipManifestBuilder

manifest = (
    PipManifestBuilder.create("my-manifest")
    .pip([
        "numpy==1.26.4",
        "torch==2.0.1",
        "attrs==23.2.0",
    ])
    .build()
)

conn = geneva.connect("s3://my-bucket/my-db")
conn.define_manifest("my-manifest", manifest)
with conn.context(cluster="my-cluster", manifest="my-manifest"):
    conn.open_table("my-table").backfill("my-column")

Pros: Quick, reusable across sessions, stored with your database.

Cons: Slower startup (downloads packages), may not work for complex dependencies.

Option 2: Custom Ray Worker Image

For KubeRay deployments, build a custom worker image:

dockerfile

# Dockerfile.ray-worker
FROM rayproject/ray:2.30.0-py311

# Install exact versions
RUN pip install \
    numpy==1.26.4 \
    torch==2.0.1 \
    attrs==23.2.0 \
    geneva==0.8.0

# Copy any custom packages
COPY ./my_udfs /app/my_udfs

Then reference in RayCluster spec:

yaml

spec:
  workerGroupSpecs:
    - template:
        spec:
          containers:
            - image: your-registry/ray-worker:latest

Pros: Fastest startup, reproducible.

Cons: Requires image build/push workflow.

Option 3: Conda Environment

Use a conda environment on workers via the cluster builder:

python

from geneva.cluster.builder import KubeRayClusterBuilder

cluster = (
    KubeRayClusterBuilder.create("my-cluster")
    .ray_init_kwargs({
        "runtime_env": {"conda": "environment.yml"}
    })
    .build()
)

Or specify conda channels and dependencies inline:

python

from geneva.cluster.builder import KubeRayClusterBuilder

cluster = (
    KubeRayClusterBuilder.create("my-cluster")
    .ray_init_kwargs({
        "runtime_env": {
            "conda": {
                "channels": ["conda-forge"],
                "dependencies": [
                    "python=3.10",
                    "ffmpeg<8",
                    "torchvision=0.22.1",
                ],
            },
            "config": {"eager_install": True},
        }
    })
    .build()
)

Pros: Best for complex dependencies with native libraries (ffmpeg, CUDA).

Cons: Slower environment creation, requires conda on workers.

API Reference

For detailed API documentation on the environment comparison functions, see the Geneva Diagnostics API Reference.

providers

materializedview

namespace

table

transaction

Dependency Verification

Example environment mismatch errors

The `compare_ray_environments` Tool

Programmatic Usage

CLI Usage

Understanding the Output

PYTHON / PLATFORM

Architecture Mismatch (macOS to Linux)

Environment Variables

Passing Environment Variables to Workers

Packages

Common Package Issues

Fixing Package Mismatches

API Reference

Dependency Verification ​

Example environment mismatch errors ​

The compare_ray_environments Tool ​

Programmatic Usage ​

CLI Usage ​

Understanding the Output ​

PYTHON / PLATFORM ​

Architecture Mismatch (macOS to Linux) ​

Environment Variables ​

Passing Environment Variables to Workers ​

Packages ​

Common Package Issues ​

Fixing Package Mismatches ​

API Reference ​

Dependency Verification

Example environment mismatch errors

The `compare_ray_environments` Tool

Programmatic Usage

CLI Usage

Understanding the Output

PYTHON / PLATFORM

Architecture Mismatch (macOS to Linux)

Environment Variables

Passing Environment Variables to Workers

Packages

Common Package Issues

Fixing Package Mismatches

API Reference