fix(cli): prevent AI examples from overriding human-specified OpenAPI examples #11137
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…I examples The AI example enhancement system was incorrectly processing endpoints that already had human-provided OpenAPI examples, potentially overriding carefully crafted examples. **Root Cause:** - System creates both human examples (in v2Examples.userSpecifiedExamples) and autogenerated examples (with generic placeholders) for the same endpoint - AI enhancement collection only checked for autogenerated examples without verifying if human examples already existed - Generic placeholder values triggered AI processing even when human examples were present **Solution:** - Enhanced collectWorkItems() to check for existing human examples before AI processing - Added hasUserSpecifiedV2Examples() helper to detect user examples in v2Examples structures - Skip endpoints that already have human-specified examples - Only process endpoints with purely autogenerated examples **Testing:** - Added comprehensive test suite with 3 scenarios: 1. Human examples protection (ai-examples-issue) 2. Autogenerated examples processing (autogen-examples-test) 3. Conflict resolution when both AI overrides and human examples exist (mixed-examples-test) - All tests validate proper precedence: Human examples > AI overrides > Autogenerated examples **Impact:** - ✅ Protects human OpenAPI examples from being overridden - ✅ Maintains AI enhancement for purely autogenerated endpoints - ✅ Backward compatible with existing behavior - ✅ Proper conflict resolution between AI overrides and human examples 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4 <[email protected]>
…n and human example validation Added test fixtures and cases to identify two critical issues with example handling: 1. **AI Examples Schema Drift**: AI-generated examples in ai_examples_override.yml become stale when OpenAPI schemas change. Examples may contain removed properties or miss new required properties. 2. **No Human Example Validation**: Discovered that Fern doesn't validate human-provided OpenAPI examples against current schemas. Invalid properties are preserved in IR. **New Test Cases:** - `schema-change-test`: Demonstrates AI examples with outdated schema properties - `invalid-human-example-test`: Tests whether human examples are validated against schemas - Enhanced existing test with comprehensive conflict analysis **Key Findings:** - AI examples don't auto-update with schema changes - Human examples with invalid properties pass through unvalidated - System needs universal example validation mechanism **Test Results Show:** - Schema mismatch detection working for identifying the problem - Both human and AI examples need validation improvements - Foundation for implementing universal example schema validation These tests provide the baseline for implementing comprehensive example validation and auto-refresh capabilities in future PRs. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4 <[email protected]>
Co-Authored-By: [email protected] <[email protected]>
…ithAI.ts Co-Authored-By: [email protected] <[email protected]>
devin-ai-integration
bot
changed the title
fern-support
approved these changes
Dec 10, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Fixes an issue where AI-generated examples would override human-specified examples in OpenAPI specs. When users provide explicit examples in their OpenAPI specification, those should take precedence over AI-enhanced examples.
Changes Made
hasUserSpecifiedV2Examples()function to detect when an endpoint already has human-specified examples in the v2Examples structurecollectWorkItems()to skip endpoints that have human-specified examples, preventing AI enhancement from overriding themanycasts when accessing v2Examples properties not in the type definitionUpdates Since Last Revision
schema-change-testfixture directory and its related test/snapshots as requestedTesting
Human Review Checklist
hasUserSpecifiedV2Examples()correctly identifies all cases of human-specified examples (checks bothv2Examples.userSpecifiedExamplesandv2Responses.responses[].body.v2Examples.userSpecifiedExamples)as anycasts - these are necessary because v2Examples properties aren't in the EndpointV3 type definitionLink to Devin run: https://app.devin.ai/sessions/8e40cb0c83ce40558d484a30b606d10c
Requested by: [email protected] ([email protected])