Kaito workspace with CPU-only node

[clarenceb]

Is it possible to create a workspace with CPU based nodes? I know KAITO is designed to provision GPU nodes and inference will be slow on CPU, but is it possible? What's a sample Workspace definition look like for this to work?

[ishaansehgal99]

Yes, it is possible to create a workspace with CPU-based nodes. While Kaito is optimized for GPU workloads to maximize performance, CPU-based inference can still work, albeit slower. For CPU-based setups, using the phi-2 or phi-3 presets would be your best bet performance-wise among the existing options. Below is a sample Workspace definition for a CPU-based setup:

apiVersion: kaito.sh/v1alpha1
kind: Workspace
metadata:
  name: workspace-falcon-7b
spec:
  resource:
    instanceType: <FALLBACK_GPU_SKU>
    labelSelector:
      matchLabels:
        app: falcon-7b
    preferredNodes: 
    - your_node_name # Here you define your already made CPU-Node 
  inference:
    preset:
      name: falcon-7b-instruct

Kaito is mainly built to handle GPU-based workloads since that’s where we can deliver the most value. That said, we get that some setups might need CPU-based nodes, and we’re happy to help and guide you through those if needed.
cc @Fei-Guo