---

This repo contains example notebooks and accompanying documentation for using Fiddler.

Examples

These example notebooks aim to give you a quick start on various Fiddler capabilities using different model tasks, data types, and use cases. They can also serve as a reference guide for setting up the monitoring of your own models in Fiddler.

Getting Started

Use the projects in this repo to onboard models and data to illustrate ML model and LLM application monitoring, analysis, and protection. This repo contains the example notebooks listed below. You can launch them in a Google Colab environment using the Colab links.

Note: This repository uses Git Large File Storage (Git LFS) for managing large files. Please make sure you have Git LFS installed before cloning this repository. You can find installation instructions at git-lfs.github.com. You can find the file types tracked via GitLFS at the .gitattributes file (currently only .csv files)

The Basics

• LLM - Comparison
• LLM - Simple Monitoring Quickstart
• ML - Simple Monitoring Quickstart
• Managing Model Versions with Fiddler
• User-defined Feature Impact Upload

Specific Use Cases

• Image/Computer Vision Model Monitoring
• NLP Model Monitoring - Multiclass Classification
• Class Imbalance Drift Detection
• Ranking Model - Monitoring
• Regression Model - Monitoring

Fiddler Utils Package - Admin Automation Library

The fiddler_utils package is an admin automation library designed to reduce code duplication across utility scripts and notebooks. While not part of the official Fiddler SDK, it is available for both Fiddler field engineers and customers to use and extend.

Key Capabilities

• Connection Management - Multi-instance support for working with multiple Fiddler deployments
• FQL Utilities - Parse, validate, and manipulate Fiddler Query Language expressions
• Schema Validation - Compare and validate model schemas across instances
• Asset Management - Export/import segments, custom metrics, alerts, baselines, charts, and dashboards
• Model Operations - Complete model export/import and comprehensive model comparison
• Environment Analysis - Project and model inventory with statistics and reporting

Installation

From the fiddler-examples repository root:

pip install -e .

Quick Example

from fiddler_utils import get_or_init, ModelComparator, SegmentManager

# Initialize connection
get_or_init(url='https://your-instance.fiddler.ai', token='your_token')

# Compare two models
comparator = ModelComparator(model_a, model_b)
result = comparator.compare_all()
print(result.to_markdown())

# Export/import segments
segment_mgr = SegmentManager()
segments = segment_mgr.export_assets(model_id=source_model.id)
segment_mgr.import_assets(target_model_id=target_model.id, assets=segments)

Documentation

See the fiddler_utils README for comprehensive documentation, API reference, and usage examples.

Integration with Utilities

Several notebooks in the misc-utils directory demonstrate fiddler_utils features:

• export_import_models.ipynb - Complete model export/import workflows
• export_import_model_assets.ipynb - Asset-level transfers between models
• compare_models.ipynb - Comprehensive model comparison
• env_stats.ipynb - Environment reporting and analysis

Miscellaneous Utilities

The misc-utils directory contains utility notebooks for customer success engineers, field AI engineers, and solution engineers. These notebooks provide tools for various administrative tasks and solutions to common challenges when working with Fiddler deployments. See the misc-utils README for a detailed catalog of available utilities.

License

This project is licensed under the MIT license. See the LICENSE file for more info.

Additional Resources

• Documentation
• Fiddler Blog
• Fiddler Resource Library