> ## Documentation Index
> Fetch the complete documentation index at: https://facetai.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Inference and Export Guide

> Complete guide to testing your models and exporting them for deployment

Use this guide to run quick checks and batch evaluations, compare models side‑by‑side, and export the final model for deployment.

## Run inference and evaluation

<Steps>
  <Step title="Select model(s)">
    Choose a trained or base model. Select two models to enable side‑by‑side comparisons.
  </Step>

  <Step title="Choose a mode">
    On the next screen, pick either Batch Inference (quick, lightweight checks) or Evaluation (full metrics on labeled data).
  </Step>

  <Step title="Load data">
    For Batch Inference: load a dataset and optionally limit the number of samples. For Evaluation: use a labeled eval split from preprocessing.
  </Step>

  <Step title="Run and monitor">
    Start the job and track progress. Large datasets take longer.
  </Step>

  <Step title="Review results">
    Inspect predictions vs. references (when available), compare models, and review summary metrics.
  </Step>
</Steps>

### Running batch inference

1. Load a dataset.
2. Select how many samples to run.
3. Review predictions vs. references (if available) for quick “vibe checks.”

<Tip>
  Batch inference is ideal for sanity checks before committing to a full evaluation.
</Tip>

### Evaluation

**Available Task Types**:

* `conversation` → BERTScore, ROUGE
* `qa` → Exact match, BERTScore
* `summarization` → ROUGE, BERTScore
* `translation` → BLEU, METEOR
* `classification` → Accuracy, Precision, Recall, F1
* `general` → BERTScore, ROUGE

<Tips>
  <Tip title="Use your eval split">
    You should already have an evaluation split from preprocessing.
  </Tip>

  <Tip title="Spot-check samples">
    Evaluations include example predictions vs. references for quick inspection.
  </Tip>

  <Tip title="Control sample size">
    You can limit the number of samples for faster runs.
  </Tip>
</Tips>

#### Using Specific Metrics

For more control, specify exact metrics to compute:

**Available Metrics**:

* `bertscore`: Semantic similarity (⭐ **Recommended for LLMs**)
* `rouge`: Text overlap and summarization quality
* `exact_match`: Perfect string matching
* `accuracy`: Token-level accuracy
* `precision`, `recall`, `f1`: Classification metrics
* `bleu`, `meteor`: Translation metrics

## Model export

Export your fine-tuned models in various formats for different deployment scenarios.

### Export Formats

<Tabs>
  <Tab title="Adapter Format">
    **Best for**: LoRA/QLoRA models, experimentation, combining adapters

    **Characteristics**:

    * Small file size (few MB)
    * Requires base model to run
    * Easy to merge with other adapters
    * Good for A/B testing different fine-tunings

    **Configuration**:

    ```json theme={null}
    {
      "export_id": "exp_123",
      "job_id": "job_456",
      "type": "adapter",
      "destination": ["gcs", "hf_hub"],
      "hf_repo_id": "username/my-adapter"
    }
    ```
  </Tab>

  <Tab title="Merged Format">
    **Best for**: Standalone deployment, vLLM inference, production use

    **Characteristics**:

    * Self-contained model
    * Larger file size (several GB)
    * Ready for immediate deployment
    * Compatible with most inference engines
  </Tab>

  <Tab title="GGUF Format">
    **Best for**: CPU inference, local deployment, resource-constrained environments

    **Characteristics**:

    * Quantized for efficiency
    * CPU-optimized
    * Compatible with llama.cpp, ollama
    * Good for edge deployment
  </Tab>
</Tabs>

### Export Destinations

<CardGroup cols={2}>
  <Card title="Google Cloud Storage" icon="cloud">
    **Best for**: Downloading models, GCP deployments

    * Download as zip files
    * Direct integration with GCP services
    * Good for private model storage
  </Card>

  <Card title="Hugging Face Hub" icon="database">
    **Best for**: Sharing models, public deployment

    * Publish to HF Hub for sharing
    * Easy integration with HF ecosystem
    * Good for open-source projects
  </Card>
</CardGroup>

### GGUF Quantization Options

Choose the right quantization level for your needs:

<Tabs>
  <Tab title="q8_0 (Recommended)">
    **Best for**: Most use cases, good balance of quality and efficiency

    * 8-bit quantization
    * Good quality retention
    * Reasonable file size
    * Fast inference
  </Tab>

  <Tab title="q4_k_m">
    **Best for**: Maximum compression, resource‑constrained environments

    * 4-bit quantization
    * Smaller file size
    * Some quality loss
    * Very fast inference
  </Tab>

  <Tab title="f16">
    **Best for**: Maximum quality, when file size isn't critical

    * 16-bit floating point
    * Best quality
    * Larger file size
    * Slower inference
  </Tab>
</Tabs>

## Inference providers

When running inference and evaluation on Facet AI, you can select from multiple providers optimized for different use cases.

* Use the HF provider for non‑Unsloth models.
* Use the Unsloth provider for models trained with Unsloth.
* If you plan to deploy with vLLM, choose the vLLM provider to keep behavior consistent across environments.

## Best practices

### Inference Testing

<Tips>
  <Tip title="Start Simple">
    Begin with basic text prompts to verify your model works correctly.
  </Tip>

  <Tip title="Test Edge Cases">
    Try prompts that might be challenging or outside your training data.
  </Tip>

  <Tip title="Compare with Base Model">
    Test the same prompts with the base model to see improvements.
  </Tip>

  <Tip title="Use Diverse Prompts">
    Test with various prompt styles and formats to ensure robustness.
  </Tip>
</Tips>

### Evaluation Strategy

<Tips>
  <Tip title="Choose Appropriate Metrics">
    Select metrics that align with your task objectives and success criteria.
  </Tip>

  <Tip title="Use Multiple Metrics">
    Combine different types of metrics for a comprehensive assessment.
  </Tip>

  <Tip title="Analyze Sample Results">
    Look at individual predictions to understand model behavior.
  </Tip>

  <Tip title="Test on Held-out Data">
    Use data the model hasn't seen during training for unbiased evaluation.
  </Tip>
</Tips>

### Export Strategy

<Tips>
  <Tip title="Choose the Right Format">
    Select export format based on your deployment needs and constraints.
  </Tip>

  <Tip title="Consider File Size">
    Balance model quality with file size requirements for your deployment.
  </Tip>

  <Tip title="Test After Export">
    Verify your exported model works correctly in your target environment.
  </Tip>

  <Tip title="Document Your Model">
    Include model cards and documentation when sharing on HF Hub.
  </Tip>
</Tips>

## Troubleshooting

### Common Issues

<AccordionGroup>
  <Accordion title="Inference Fails">
    * Model not found or invalid model format
    * Insufficient resources available

    <Tip>
      Retry later or deploy on your cloud if you need guaranteed capacity.
    </Tip>
  </Accordion>
</AccordionGroup>

## Next Steps

After testing and exporting your model:

1. **Deploy Your Model**: Set up your model for production use
2. **Monitor Performance**: Track model performance in production
3. **Collect Feedback**: Gather user feedback to improve your model
4. **Iterate**: Use insights to refine your model with additional training

Ready to deploy? Head to the [Deployment guide](/deployment/deployment) to learn how to set up your model for production use.