Serverless compute spec

[abelanger5]

Overview

This proposal introduces a new "Managed Compute" feature to enable Hatchet to orchestrate required resources and compute environments for running functions on the queue. This feature aims to provide more granular control over the compute resources allocated to different parts of a Hatchet workflow, while simplifying the management of diverse compute requirements across functions.

Goals

1. Enable users to specify compute requirements for individual functions and durable tasks.
2. Allow Hatchet to automatically orchestrate the required resources and compute environments for running functions on the queue.
3. Provide a flexible way to define and reuse compute configurations.
4. Integrate managed compute capabilities with existing Hatchet function decorators and worker setup.

Proposed Changes

1. ComputeConfig Class

Introduce a new ComputeConfig class in the Hatchet SDK to define compute requirements:

from hatchet_sdk.v2.hatchet import ComputeConfig

cpu = ComputeConfig(
    cpu="2",
    memory="4Gi",
    gpu=None,
    image="hatchet/default-cpu:latest"
)

gpu = ComputeConfig(
    cpu="4",
    memory="16Gi",
    gpu="nvidia-tesla-t4",
    image="hatchet/default-gpu:latest"
)

2. Decorator Extensions

Extend the @hatchet.function and @hatchet.durable decorators to accept a managed_compute parameter:

@hatchet.function(managed_compute=gpu)
def my_func(context: Context) -> MyResultType:
    # Function implementation

@hatchet.durable(managed_compute=cpu)
async def my_durable_func(context: DurableContext):
    # Durable function implementation

3. Worker Configuration

Modify the hatchet.managed_worker function to handle managed compute:

worker = hatchet.managed_worker(
    "test-worker", 
    max_runs=5,
    default_compute=cpu
)

Implementation Details

1. Create a new module hatchet_sdk.v2.managed_compute to house the ComputeConfig class and related utilities.
2. Modify the hatchet.function and hatchet.durable decorators to incorporate managed compute information.
3. Update the ManagedWorker class to handle compute configurations.

Backwards Compatibility

This feature will be backwards compatible. Functions without a specified managed_compute will use the worker's default_compute configuration.

Future Considerations

1. Integration with cloud provider-specific compute offerings (e.g., AWS EC2 instance types, GCP machine types).
2. Dynamic compute allocation based on workload characteristics and queue backlog.
3. Cost estimation and optimization features based on compute configurations.

Open Questions

1. What's the best approach for the local development story, especially for GPU-dependent functions?
2. Autoscaling or cold starting (autoscale from 0)
3. Implications for desired-labels/worker labels
4. How do we discover compute environments to build?
5. How do we support self-hosted users and self-hosted worker pools i.e. K8S?