How to set CPU and memory size for a Ray job?

[hongbo-miao]

My Ray cluster has different size of nodes.

Here is my current code:

import daft
import ray
from daft import DataCatalogType
from daft.io.catalog import DataCatalogTable


def hm_daft_analysis() -> None:
    ray_address = "ray://hm-ray-cluster-head-svc.production-hm-ray-cluster.svc:10001"
    ray.init(
        ray_address,
        runtime_env={
            "pip": [
                "getdaft[aws,deltalake,ray]==0.4.6",
            ],
        },
    )
    daft.context.set_runner_ray(ray_address)

    table = DataCatalogTable(
        catalog=DataCatalogType.GLUE,
        database_name="production_motor_db",
        table_name="motor_data",
    )

    df = daft.read_deltalake(table)

    # Here do some computing
    # ...
    df = df.collect()
    print(f"{df = }")


if __name__ == "__main__":
    hm_daft_analysis()

I originally thought I can set memory and CPU by

    ray.init(
        ray_address,
        runtime_env={
            "pip": [
                "getdaft[aws,deltalake,ray]==0.4.6",
            ],
        },
        num_cpus=10,
        object_store_memory=100 * 10**9,  # 100 GB
    )

However, this gives me error

Call to ray.init() on the server failed with: When connecting to an existing cluster, num_cpus and num_gpus must not be provided.

In a standard Ray job, I can set

@ray.remote(num_cpus=10, memory=100 * 10**9)

How to set How to set CPU and memory size for a Ray job when use with Daft? Thanks!