Process dataset

Process a dataset into ChatML format

The processing converts the dataset to ChatML format using the provided configuration. The configuration can include field mappings that specify either direct column mappings or template strings with column references.

Example field mappings:

{
    "system_field": {"type": "template", "value": "You are a helpful assistant."},
    "user_field": {"type": "column", "value": "question"},
    "assistant_field": {"type": "template", "value": "Answer: {answer}"}
}

POST /datasets/process

Process Dataset

cURL

curl --request POST \
  --url https://preprocessing-service-433968519479.us-central1.run.app/datasets/process \
  --header 'Content-Type: application/json' \
  --data '
{
  "dataset_name": "<string>",
  "dataset_source": "upload",
  "dataset_id": "<string>",
  "processing_mode": "language_modeling",
  "config": {
    "field_mappings": {},
    "normalize_whitespace": true,
    "augmentation_config": {
      "augmentation_factor": 1.5,
      "use_eda": false,
      "use_back_translation": false,
      "use_paraphrasing": false,
      "use_synthesis": false,
      "gemini_api_key": "<string>",
      "synthesis_ratio": 123,
      "custom_prompt": "<string>"
    },
    "split_config": {
      "train_split": "<string>",
      "test_split": "<string>",
      "type": "hf_split"
    }
  },
  "dataset_subset": "default"
}
'

200

{
  "dataset_name": "<string>",
  "dataset_subset": "<string>",
  "dataset_source": "upload",
  "dataset_id": "<string>",
  "processed_dataset_id": "<string>",
  "num_examples": 123,
  "created_at": "<string>",
  "splits": [
    "<string>"
  ],
  "modality": "text",
  "full_splits": []
}

Body

application/json

dataset_name · string · required

dataset_source · enum<string> · required

Available options: upload, huggingface

dataset_id · string · required

processing_mode · enum<string> · required

Specifies the preprocessing mode to format the dataset for a specific fine-tuning task.

Available options: language_modeling, prompt_only, preference

config · PreprocessingConfig · object · required

Show child attributes

dataset_subset · string · default: default

Response

200

application/json

Successful Response

dataset_name · string · required

dataset_subset · string · required

dataset_source · enum<string> · required

Available options: upload, huggingface

dataset_id · string · required

processed_dataset_id · string · required

num_examples · integer · required

created_at · string · required

splits · string[] · required

modality · enum<string> · default: text

Available options: text, vision

full_splits · Full Splits · object[]