Batch inference

Run batch inference using a trained model

POST /inference/batch

Batch Inference

cURL

curl --request POST \
  --url https://inference-service-433968519479.us-central1.run.app/inference/batch \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '
{
  "hf_token": "<string>",
  "model_source": "<string>",
  "model_type": "adapter",
  "base_model_id": "<string>",
  "messages": [
    [
      {}
    ]
  ],
  "use_vllm": false
}
'

200

{
  "results": [
    "<string>"
  ]
}

Authorizations

Authorization · string · header · required

Bearer authentication header of the form Bearer <token>, where <token> is your auth token.

Body

application/json

hf_token · string · required

model_source · string · required

model_type · enum<string> · required

Available options: adapter, merged, base

base_model_id · string · required

messages · object[][] · required

use_vllm · boolean | null · default: false

Response

200

application/json

Successful Response

results · string[] · required