diff --git a/docs/dir/.index b/docs/dir/.index index 94744ee0..eb6d10ca 100644 --- a/docs/dir/.index +++ b/docs/dir/.index @@ -3,6 +3,7 @@ nav: - Getting Started: getting-started.md - Architecture: architecture.md - Records: ads-records.md + - Validation: validation.md - Trust Model: trust-model.md - Features and Usage: scenarios.md - Event Streaming: events.md diff --git a/docs/dir/getting-started.md b/docs/dir/getting-started.md index 45476d18..7718351d 100644 --- a/docs/dir/getting-started.md +++ b/docs/dir/getting-started.md @@ -100,7 +100,7 @@ Directory API services can be deployed either using the `Taskfile` or directly v This starts the necessary components such as storage and API services. ```bash -task server:start +DIRECTORY_SERVER_OASF_API_VALIDATION_SCHEMA_URL=https://schema.oasf.outshift.com task server:start ``` ### Using Helm chart diff --git a/docs/dir/validation.md b/docs/dir/validation.md new file mode 100644 index 00000000..ef0e87bd --- /dev/null +++ b/docs/dir/validation.md @@ -0,0 +1,188 @@ +# Validation + +The Directory enforces validation on all records before accepting them. +Validation is performed using the [OASF SDK](../oasf/oasf-sdk.md), which requires an OASF schema server URL for validation. + +## Required Configuration + +The Directory server requires an OASF schema URL to start. You must provide a valid OASF schema server URL for validation to work: + +```bash +DIRECTORY_SERVER_OASF_API_VALIDATION_SCHEMA_URL=https://schema.oasf.outshift.com task server:start +``` + +The configuration can also be set via YAML file: + +```yaml +# server.config.yml +oasf_api_validation: + schema_url: "https://schema.oasf.outshift.com" +listen_address: "0.0.0.0:8888" +``` + +## Validation Behavior + +The Directory server uses API-based validation against the configured OASF schema server: + +- **Validation Method**: HTTP requests to the OASF schema server API. +- **Errors vs Warnings**: Only errors cause validation to fail. Warnings are returned but do not affect the validation result. +- **Schema Version Detection**: The schema version is automatically detected from each record's `schema_version` field. +- **Supported Versions**: The OASF SDK decoder supports specific schema versions (currently: 0.3.1, 0.7.0, and 0.8.0). Records using unsupported versions are not validated. +- **Unknown Classes**: Classes (such as modules, skills, and domains) not defined in the OASF schema are rejected with an error. Records must only use classes that are defined in the configured OASF schema instance. + +## OASF Instance Configurations + +While there is only one validation method (API validation), you can configure the Directory server to use different OASF instances. +The choice of OASF instance affects which records are accepted and how compatible your directory instance is with other directory instances. + +The following table shows which OASF instance configurations can exchange records with each other: + +| Instance Type | Can Pull From | Can Be Pulled By | +|--------------|---------------|------------------| +| **Official OASF Instance** | Official OASF instance only | Official OASF instance and custom instances with additional taxonomy | +| **Custom OASF Instance (Additional Taxonomy)** | Official OASF instance, custom instances with same extended taxonomy | Custom instances with same extended taxonomy only | +| **Custom OASF Instance (Changed Taxonomy)** | Custom instances with same changed taxonomy only | Custom instances with same changed taxonomy only | + +### Official OASF Instance + +Using the official OASF instance (`https://schema.oasf.outshift.com`) provides the baseline for the official network. +Records validated here form the most strict, compatible set. + +**Configuration:** + +```yaml +oasf_api_validation: + schema_url: "https://schema.oasf.outshift.com" +``` + +### Custom OASF Instance (Additional Taxonomy) + +Using a custom OASF instance with extensions that add to the taxonomy (with modules, skills, and domains that extend the official taxonomy but doesn't change or remove). + +Records using the extended taxonomy can only be pulled by nodes using the exact same extended taxonomy. + +**Configuration:** + +```yaml +oasf_api_validation: + schema_url: "https://your-custom-oasf-instance.com" +``` + +### Custom OASF Instance (Changed Taxonomy) + +Using a custom OASF instance with extensions that modify the taxonomy (with modules, skills, and domains that change or remove from the official taxonomy). + +This approach is completely incompatible with all other options, can only work with nodes using the exact same changed taxonomy. + +**Configuration:** + +```yaml +oasf_api_validation: + schema_url: "https://your-custom-oasf-instance.com" +``` + +## Deploying a Local OASF Instance + +You can deploy a local OASF instance alongside the Directory server for testing or development purposes. + +### Testing with Local OASF Server + +To test with a local OASF instance deployed alongside the directory server: + +1. Enable OASF in Helm values + + Edit `install/charts/dir/values.yaml` with the following configuration: + + ```yaml + apiserver: + oasf: + enabled: true + ``` + +2. Set schema URL to use the deployed OASF instance + + In the same file, set the schema URL to use the deployed OASF instance: + + ```yaml + apiserver: + config: + oasf_api_validation: + schema_url: "http://dir-ingress-controller.dir-server.svc.cluster.local" + ``` + + Replace `dir` with your Helm release name and `dir-server` with your namespace if different. + +3. Deploy the local OASF instance + + ```bash + task build + task deploy:local + ``` + +The OASF instance is deployed as a subchart in the same namespace and automatically configured for multi-version routing via ingress. + +### Using a Locally Built OASF Image + +If you want to deploy with a locally built OASF image (e.g., containing `0.9.0-dev` schema files), you need to load the image into Kind before deploying. +The `task deploy:local` command automatically creates a cluster and loads images, but it doesn't load custom OASF images. + +Follow the steps below: + +1. Create the Kind cluster + + ```bash + task deploy:kubernetes:setup-cluster + ``` + + This creates the cluster and loads the Directory server images. + +2. Build your local OASF image with the `latest` tag + + ```bash + cd /path/to/oasf/server + task build + ``` + +3. Load the OASF image into Kind + + ```bash + kind load docker-image ghcr.io/agntcy/oasf-server:latest --name agntcy-cluster + ``` + +4. Configure `values.yaml` to use the local image: + + ```yaml + oasf: + enabled: true + image: + repository: ghcr.io/agntcy/oasf-server + versions: + - server: latest + schema: 0.9.0-dev + default: true + ``` + +5. Deploy the Directory + + Don't use `task deploy:local` as it will recreate the cluster. + + ```bash + task deploy:kubernetes:dir + ``` + +!!! note + + If you update the local OASF image, reload it into Kind and restart the deployment: + + ```bash + kind load docker-image ghcr.io/agntcy/oasf-server:latest --name agntcy-cluster + kubectl rollout restart deployment/dir-oasf-0-9-0-dev -n dir-server + ``` + +## Related Documentation + +For more information, see the following: + +- [OASF Validation Service](../oasf/validation.md) - Detailed validation service documentation +- [Validation Comparison](../oasf/validation-comparison.md) - Comparison between API validator and JSON Schema +- [OASF Extensions](https://github.com/agntcy/oasf/blob/main/CONTRIBUTING.md#oasf-extensions) - Information about creating OASF extensions diff --git a/docs/oasf/.index b/docs/oasf/.index index 8d8197ef..bcc1c6db 100644 --- a/docs/oasf/.index +++ b/docs/oasf/.index @@ -6,5 +6,6 @@ nav: - Decoding Service: decoding.md - Translation Service: translation.md - Validation Service: validation.md + - Validation Comparison: validation-comparison.md - OASF Record Guide: agent-record-guide.md - OASF Contribution Guide: contributing.md diff --git a/docs/oasf/validation-comparison.md b/docs/oasf/validation-comparison.md new file mode 100644 index 00000000..ecafd275 --- /dev/null +++ b/docs/oasf/validation-comparison.md @@ -0,0 +1,109 @@ +# Validation Comparison: API Validator vs JSON Schema Draft-07 + +This table compares validation outcomes between the OASF API validator and JSON Schema Draft-07 validator. + +## Legend + +- **API Error**: + API validator returns ERROR +- **API Warning**: + API validator returns WARNING +- **JSON Schema Pass**: + JSON Schema Draft-07 validation passes +- **JSON Schema Fail**: + JSON Schema Draft-07 validation fails + +## Comparison Table + +| Case | Example | API Validator | JSON Schema | Notes | +| ---------------------------------------- | --------------------------------------------------------------------- | -------------------------------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------ | +| 1. Version Incompatibility (Later) | `schema_version: "1.0.0"` when server is `0.8.0` | ERROR (`version_incompatible_later`) | PASS | JSON Schema validates format only, not semantic compatibility 1 | +| 2. Version Incompatibility (Initial Dev) | `schema_version: "0.1.0"` when server is `0.8.0` | ERROR (`version_incompatible_initial_development`) | PASS | JSON Schema validates format only, not SemVer rules 1 | +| 3. Version Incompatibility (Prerelease) | `schema_version: "0.8.0-alpha"` when server is `0.8.0` | ERROR (`version_incompatible_prerelease`) | PASS | JSON Schema validates format only, not prerelease compatibility 1 | +| 4. Version Earlier (Compatible) | `schema_version: "0.7.0"` when server is `0.8.0` | WARNING (`version_earlier`) | PASS | JSON Schema validates format only, not version comparison 1 | +| 5. Regex Pattern Mismatch | `previous_record_cid: "invalid"` (doesn't match CID regex) | WARNING (`attribute_value_regex_not_matched`) | PASS | JSON Schema may export regex but validation is lenient; API treats as warning | +| 6. Regex Pattern Mismatch (Super Type) | String doesn't match super type regex | WARNING (`attribute_value_super_type_regex_not_matched`) | PASS | Same as above for super types | +| 7. Using Base Class | `skills: [{"name": "base_skill"}]` or `skills: [{"id": 0}]` | ERROR (`base_class_used`) | FAIL | Both catch base classes (base_skill, base_domain, base_module); JSON Schema `oneOf` doesn't include parent classes | +| 8. Using Deprecated Class | Using a deprecated skill/domain/module | WARNING (`class_deprecated`) | PASS | JSON Schema doesn't track deprecation | +| 9. Using Deprecated Attribute | Using a deprecated attribute | WARNING (`attribute_deprecated`) | PASS | JSON Schema doesn't track deprecation | +| 10. Using Deprecated Object | Using a deprecated object | WARNING (`object_deprecated`) | PASS | JSON Schema doesn't track deprecation | +| 11. Using Deprecated Enum Value | Using a deprecated enum value | WARNING (`attribute_enum_value_deprecated`) | PASS | JSON Schema doesn't track deprecation | +| 12. Using Deprecated Enum Array Value | Using a deprecated enum array value | WARNING (`attribute_enum_array_value_deprecated`) | PASS | JSON Schema doesn't track deprecation | +| 13. Enum Sibling Mismatch | Enum value doesn't match sibling caption | WARNING (`attribute_enum_sibling_incorrect`) | PASS | JSON Schema doesn't validate enum siblings | +| 14. Enum Sibling Suspicious (Other) | Enum value 99 with matching sibling caption | WARNING (`attribute_enum_sibling_suspicious_other`) | PASS | JSON Schema doesn't validate enum siblings | +| 15. Missing Recommended Attribute | Required attribute missing (if `warn_on_missing_recommended` enabled) | WARNING (`attribute_recommended_missing`) | PASS | JSON Schema only validates `required`, not `recommended` | +| 16. Empty Required Array | `skills: []` when skills is required | ERROR (`attribute_required_empty`) | FAIL | JSON Schema now includes `minItems: 1` for required arrays (after fix) | +| 17. Wrong Type | `name: 123` when name should be string | ERROR (`attribute_wrong_type`) | FAIL | Both catch type mismatches | +| 18. Missing Required Attribute | Missing required `name` field | ERROR (`attribute_required_missing`) | FAIL | Both catch missing required fields | +| 19. Unknown Attribute | Extra attribute not in schema | ERROR (`attribute_unknown`) | FAIL | JSON Schema uses `additionalProperties: false` | +| 20. Unknown Enum Value | Enum value not in allowed list | ERROR (`attribute_enum_value_unknown`) | FAIL | JSON Schema uses `enum` constraint | +| 21. Unknown Enum Array Value | Enum array value not in allowed list | ERROR (`attribute_enum_array_value_unknown`) | FAIL | JSON Schema validates array items | +| 22. Value Outside Range | Number outside type's range | ERROR (`attribute_value_exceeds_range`) | FAIL | JSON Schema uses `minimum`/`maximum` | +| 23. Value Exceeds Max Length | String exceeds type's max length | ERROR (`attribute_value_exceeds_max_len`) | FAIL | JSON Schema uses `maxLength` | +| 24. Value Not in Type Values | Value not in type's allowed values list | ERROR (`attribute_value_not_in_type_values`) | FAIL | JSON Schema uses `enum` constraint | +| 25. Unknown Class ID | `id: 99999` doesn't exist | ERROR (`id_unknown`) | FAIL | JSON Schema uses `oneOf` with `const` values | +| 26. Unknown Class Name | `name: "nonexistent"` doesn't exist | ERROR (`name_unknown`) | FAIL | JSON Schema uses `oneOf` with `const` values | +| 27. ID/Name Mismatch | `id: 2` and `name: "different"` refer to different classes | ERROR (`id_name_mismatch`) | FAIL | JSON Schema `oneOf` with `const` prevents mismatches | +| 28. Constraint Failed (at_least_one) | None of the constraint fields present | ERROR (`constraint_failed`) | FAIL | JSON Schema uses `anyOf` for `at_least_one` | +| 29. Constraint Failed (just_one) | Both fields in `just_one` constraint present | ERROR (`constraint_failed`) | FAIL | JSON Schema `oneOf` pattern correctly prevents both fields from being present | +| 30. Enum Array Sibling Missing | Enum array sibling array too short | ERROR (`attribute_enum_array_sibling_missing`) | PASS | JSON Schema doesn't validate enum array siblings | +| 31. Enum Array Sibling Incorrect | Enum array sibling value doesn't match | ERROR (`attribute_enum_array_sibling_incorrect`) | PASS | JSON Schema doesn't validate enum array siblings | +| 32. Enum Object Not Matched | Enum object doesn't match any allowed objects | ERROR (`enum_object_not_matched`) | FAIL | JSON Schema uses `oneOf` for enum objects | +| 33. Unknown Profile | Profile in `metadata.profiles` doesn't exist | ERROR (`profile_unknown`) | PASS | JSON Schema validates structure but not profile existence (only for classes, not record) | +| 34. Invalid Version Format | `schema_version: "invalid"` | ERROR (`version_invalid_format`) | FAIL | JSON Schema validates semantic versioning format 1 | +| 35. Array Duplicates | Duplicate items in array (e.g., same skill twice) | ERROR (`attribute_array_duplicate`) | FAIL | Both catch duplicates; JSON Schema uses `uniqueItems: true` | +| 36. Locators Duplicate Types | Multiple locators with same type | ERROR (`attribute_locators_duplicate_type`) | FAIL | Both catch duplicate types in locators array; JSON Schema uses `uniqueItems: true` | + + +1 When using oasf-sdk, any schema version that is not explicitly supported by the decoder (currently: 0.3.1, 0.7.0, and 0.8.0) will result in a failure, regardless of whether the API validator returns an error or warning. The SDK uses a switch statement that only handles specific versions, and any non-supported version will return an error: `unsupported OASF version: `. + +## Summary by Category + +### API ERROR, JSON Schema PASSES (Gaps in JSON Schema) + +1. Version incompatibility (later, initial dev, prerelease) +2. Enum array sibling validation (missing, incorrect) +3. Unknown profile (for classes) + +### API WARNING, JSON Schema PASSES (Non-blocking Issues) + +1. Version earlier (compatible) +2. Regex pattern mismatches +3. All deprecation warnings (class, attribute, object, enum values) +4. Enum sibling mismatches +5. Missing recommended attributes + +### Both FAIL (Both Catch the Issue) + +1. Empty required arrays +2. Wrong types +3. Missing required attributes +4. Unknown attributes +5. Unknown enum values +6. Value range/length violations +7. Unknown class ID/name (including unknown modules, skills, domains) +8. ID/name mismatches +9. Constraint violations (`just_one`, `at_least_one`) +10. Enum object mismatches +11. Invalid version format +12. Using base classes (base_skill, base_domain, base_module) +13. Array duplicates (duplicate items in arrays) +14. Locators duplicate types (duplicate types in locators array) + +## Key Insights + +1. JSON Schema is comprehensive for structural and value validation +2. Main gaps in JSON Schema: + + - Semantic version compatibility (only validates format) + - Enum array sibling validation + - Deprecation tracking + - Regex validation (treated as warnings, not errors) + - Profile existence (for classes) + +3. Both validators catch most structural and type validation issues, including: + + - Base classes (now errors in API validator, matching JSON Schema behavior) + - Array duplicates (both use duplicate detection) + - Locators duplicate types (both catch duplicate types in locators array) + - Unknown classes (modules, skills, domains - all treated the same way) diff --git a/docs/oasf/validation.md b/docs/oasf/validation.md index fc7f8692..ea00dda0 100644 --- a/docs/oasf/validation.md +++ b/docs/oasf/validation.md @@ -1,29 +1,67 @@ # Validation Service -The validation service validates OASF records against JSON schemas. It supports both embedded schemas and custom schema URLs. The validation service supports the following features: +The validation service validates OASF records against an OASF schema server via API validation. +The validation service supports the following features: -* Validates records against embedded schema versions (0.3.1 and 0.7.0). -* Supports custom schema URLs for remote validation. -* Returns detailed validation errors. -* Automatic schema version detection from records. +- Validates records against any OASF schema server URL. +- Returns detailed validation errors and warnings separately. +- Automatic schema version detection from records. +- Warnings do not affect validation result (only errors cause validation to fail). -Use `ValidateRecord` to validate a record against a specified schema URL or its embedded schema version. +## Initialization + +Create a validator instance by providing a schema URL: + +```go +validator, err := validator.New("https://schema.oasf.outshift.com") +if err != nil { + // Handle error +} +``` + +The schema URL is required and must be provided when creating the validator. +The validator uses this URL for all validation operations. + +## Validation + +Use `ValidateRecord` to validate a record against the configured schema URL. **Parameters:** -* `record`: The OASF record to validate (as a Protocol Buffer Struct) -* `options`: Optional validation options +- `ctx`: Context for cancellation and timeout control +- `record`: The OASF record to validate (as a Protocol Buffer Struct) **Returns:** -* `bool`: Whether the record is valid -* `[]string`: List of validation errors (empty if valid) -* `error`: Any error that occurred during validation +- `bool`: Whether the record is valid (true if no errors, false if errors present) +- `[]string`: List of validation error messages (empty if valid) +- `[]string`: List of validation warning messages (may be non-empty even if valid) +- `error`: Any error that occurred during validation + +!!! note + Warnings do not affect the validation result. A record is considered valid if there are no errors, regardless of whether warnings are present. + +## Validation Response -**Options:** +The validation response includes errors and warnings. -* `WithSchemaURL(url string)`: Validate against a custom schema URL instead of embedded schemas +Errors are critical validation failures that must be fixed. If any errors are present, the record is invalid. Warnings are non-critical issues or deprecation notices. Warnings do not affect the validation result. + +Error messages include the following: + +- Clear descriptions of what failed validation. +- Attribute paths (e.g., `data.servers[0]`). +- Constraint details for `constraint_failed` errors. + +Warning messages include the following: + +- Descriptions of non-critical issues. +- Attribute paths where applicable. ## Example Usage -For detailed examples, see the [OASF SDK repository](https://github.com/agntcy/oasf-sdk/blob/main/USAGE.md). +For detailed examples, see the [OASF SDK repository](https://github.com/agntcy/oasf-sdk/blob/main/USAGE.md). + +## Validation Comparison + +For a detailed comparison between the API validator and JSON Schema Draft-07 validation, including differences in error handling, warnings, and validation coverage, see the [Validation Comparison](validation-comparison.md) page.