Evaluation

Run evaluation of a fine-tuned model on a dataset

POST /evaluation

Evaluation

cURL

curl --request POST \
  --url https://inference-service-433968519479.us-central1.run.app/evaluation \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '
{
  "hf_token": "<string>",
  "model_source": "<string>",
  "model_type": "adapter",
  "base_model_id": "<string>",
  "dataset_id": "<string>",
  "task_type": "conversation",
  "metrics": [
    "rouge"
  ],
  "max_samples": 123,
  "num_sample_results": 3,
  "use_vllm": false
}
'

200

{
  "metrics": {},
  "num_samples": 123,
  "dataset_id": "<string>",
  "samples": [
    {
      "prediction": "<string>",
      "reference": "<string>",
      "sample_index": 123,
      "input": [
        {}
      ]
    }
  ]
}

Authorizations

Authorization · string · header · required

Bearer authentication header of the form Bearer <token>, where <token> is your auth token.

Body

application/json

hf_token · string · required

model_source · string · required

model_type · enum<string> · required

Available options: adapter, merged, base

base_model_id · string · required

dataset_id · string · required

task_type · enum<string> | null

Available options: conversation, qa, summarization, translation, classification, general

metrics · enum<string>[] | null

Available options: rouge, bertscore, accuracy, exact_match, bleu, meteor, recall, precision, f1

max_samples · integer | null

num_sample_results · integer | null · default: 3

use_vllm · boolean | null · default: false

Response

200

application/json

Successful Response

metrics · Metrics · object · required

Show child attributes

num_samples · integer · required

dataset_id · string · required

samples · SampleResult · object[] · required

Show child attributes