[Bug] Annotated Fields Ignored

[eric-forte-elastic]

Pull Request

Issue link(s):
Resolves #5080

Summary - What I changed

Tried revering the Annotated change to definitions.py See issue for greater detail as to the issue. TLDR, Marshmallow ignores the annotation when validating.

However Annotated is preferred method as NewType is deprecated so more work will need to be done to determine fix before addressing this.

Update

Upgrading to marshmallow-dataclass==8.7.1 resolves the bug and Annotated functions as expected.

How To Test

Run view rule on a known bad rule to detect issues.

Example

[metadata]
creation_date = "2025/09/01"
integration = ["endpoint", "windows", "system", "m365_defender", "sentinel_one_cloud_funnel", "crowdstrike"]
maturity = "production"
updated_date = "2025/09/02"


[rule]
author = ["Elastic"]
description = "Adversaries may modify file or directory ownership to evade access control lists (ACLs) and access protected files."
from = "now-9m"
index = [
    "endgame-*",
    "logs-crowdstrike.fdr*",
    "logs-endpoint.events.process-*",
    "logs-m365_defender.event-*",
    "logs-sentinel_one_cloud_funnel.*",
    "logs-system.security*",
    "logs-windows.forwarded*",
    "logs-windows.sysmon_operational-*",
    "winlogbeat-*",
]
language = "eql"
license = "Elastic License v2"
name = "System File Ownership Change"
note = """## Triage and analysis

### Investigating System File Ownership Change

Adversaries may modify file or directory ownership to evade access control lists (ACLs) and access protected files.

#### Possible investigation steps

- Assess the ownership target file or directory and identify if it's a system critical file.
- Identify the user account that performed the action and whether it should perform this kind of action.
- Investigate the process execution chain (parent process tree) for unknown processes. Examine their executable files for prevalence, whether they are located in expected locations, and if they are signed with valid digital signatures.
- Investigate other alerts associated with the user/host during the past 48 hours.

### False positive analysis

- System updates, backup software and uninstallers tend to modify files ownership.

### Response and remediation

- Initiate the incident response process based on the outcome of the triage.
- Isolate the involved host to prevent further post-compromise behavior.
- If the triage identified malware, search the environment for additional compromised hosts.
  - Implement temporary network rules, procedures, and segmentation to contain the malware.
  - Stop suspicious processes.
  - Immediately block the identified indicators of compromise (IoCs).
  - Inspect the affected systems for additional malware backdoors like reverse shells, reverse proxies, or droppers that attackers could use to reinfect the system.
- Remove and block malicious artifacts identified during triage.
- Run a full antimalware scan. This may reveal additional artifacts left in the system, persistence mechanisms, and malware components.
- Investigate credential exposure on systems compromised or used by the attacker to ensure all compromised accounts are identified. Reset passwords for these accounts and other potentially compromised credentials, such as email, business systems, and web services.
- Determine the initial vector abused by the attacker and take action to prevent reinfection through the same vector.
- Using the incident response data, update logging and audit policies to improve the mean time to detect (MTTD) and the mean time to respond (MTTR).
"""
risk_score = 147
rule_id = "7eb54028-ca72-4eb7-8185-b6864572347db123"
severity = "medium"
tags = [
    "Domain: Endpoint",
    "OS: Windows",
    "Use Case: Threat Detection",
    "Tactic: Defense Evasion",
    "Tactic: Persistence",
    "Data Source: Elastic Endgame",
    "Resources: Investigation Guide",
    "Data Source: Elastic Defend",
    "Data Source: Windows Security Event Logs",
    "Data Source: Microsoft Defender for Endpoint",
    "Data Source: Sysmon",
    "Data Source: SentinelOne",
    "Data Source: Crowdstrike",
]
timeline_id = "e70679c2-6cde-4510-9764-4823df18f7db"
timeline_title = "Comprehensive Process Timeline"
timestamp_override = "event.ingested"
type = "eql"

query = '''
process where host.os.type == "windows" and event.type == "start" and
  (
   (process.name : "icacls.exe" and process.args : "/reset") or
   (process.name : "takeown.exe" and process.args : "/f") or
   (process.name : "icacls.exe" and process.args : "/grant" and process.args : "Everyone:F")
   ) and
   process.command_line : "*.exe *C:\\Windows\\*"
'''


[[rule.threat]]
framework = "MITRE ATT&CK"
[[rule.threat.technique]]
id = "T1222"
name = "File and Directory Permissions Modification"
reference = "https://attack.mitre.org/techniques/T1222/"
[[rule.threat.technique.subtechnique]]
id = "T1222.001"
name = "Windows File and Directory Permissions Modification"
reference = "https://attack.mitre.org/techniques/T1222/001/"



[rule.threat.tactic]
id = "TA0005"
name = "Defense Evasion"
reference = "https://attack.mitre.org/tactics/TA0005/"

detection-rules on  5080-bug-uuid-definition-discrepancy [?] is  v1.4.5 via  v3.12.11 (detection-rules-build) on  eric.forte 
❯ python -m detection_rules view-rule test_rule.toml

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Error loading rule in /tmp/fix_uui/detection-rules/test_rule.toml
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/tmp/fix_uui/detection-rules/detection_rules/__main__.py", line 31, in <module>
    main()
  File "/tmp/fix_uui/detection-rules/detection_rules/__main__.py", line 28, in main
    root(prog_name="detection_rules")
  File "/tmp/fix_uui/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/fix_uui/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/tmp/fix_uui/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/fix_uui/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/fix_uui/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/fix_uui/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/fix_uui/detection-rules/detection_rules/main.py", line 452, in view_rule
    rule = RuleCollection().load_file(rule_file)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/fix_uui/detection-rules/detection_rules/rule_loader.py", line 507, in load_file
    return self.load_dict(obj, path=path)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/fix_uui/detection-rules/detection_rules/rule_loader.py", line 483, in load_dict
    contents = TOMLRuleContents.from_dict(obj)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/fix_uui/detection-rules/detection_rules/mixins.py", line 121, in from_dict
    return schema.load(obj)
           ^^^^^^^^^^^^^^^^
  File "/tmp/fix_uui/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/marshmallow_dataclass/__init__.py", line 910, in load
    all_loaded = super().load(data, many=many, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/fix_uui/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/marshmallow/schema.py", line 792, in load
    return self._do_load(
           ^^^^^^^^^^^^^^
  File "/tmp/fix_uui/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/marshmallow/schema.py", line 999, in _do_load
    raise exc
marshmallow.exceptions.ValidationError: {'rule': [ValidationError({'risk_score': ['Must be greater than or equal to 1 and less than or equal to 100.'], 'rule_id': ['String does not match expected pattern.']}), ValidationError({'risk_score': ['Must be greater than or equal to 1 and less than or equal to 100.'], 'rule_id': ['String does not match expected pattern.'], 'type': ['Must be equal to esql.'], 'language': ['Must be equal to esql.']}), ValidationError({'risk_score': ['Must be greater than or equal to 1 and less than or equal to 100.'], 'rule_id': ['String does not match expected pattern.'], 'type': ['Must be equal to threshold.'], 'threshold': ['Missing data for required field.']}), ValidationError({'risk_score': ['Must be greater than or equal to 1 and less than or equal to 100.'], 'rule_id': ['String does not match expected pattern.'], 'type': ['Must be equal to threat_match.'], 'threat_mapping': ['Missing data for required field.'], 'threat_index': ['Missing data for required field.']}), ValidationError({'risk_score': ['Must be greater than or equal to 1 and less than or equal to 100.'], 'rule_id': ['String does not match expected pattern.'], 'type': ['Must be equal to machine_learning.'], 'anomaly_threshold': ['Missing data for required field.'], 'machine_learning_job_id': ['Missing data for required field.'], 'index': ['Unknown field.'], 'query': ['Unknown field.'], 'language': ['Unknown field.']}), ValidationError({'risk_score': ['Must be greater than or equal to 1 and less than or equal to 100.'], 'rule_id': ['String does not match expected pattern.'], 'type': ['Must be equal to query.']}), ValidationError({'risk_score': ['Must be greater than or equal to 1 and less than or equal to 100.'], 'rule_id': ['String does not match expected pattern.'], 'type': ['Must be equal to new_terms.'], 'new_terms': ['Missing data for required field.']})]}

Checklist

• Added a label for the type of pr: bug, enhancement, schema, maintenance, Rule: New, Rule: Deprecation, Rule: Tuning, Hunt: New, or Hunt: Tuning so guidelines can be generated
• Added the meta:rapid-merge label if planning to merge within 24 hours
• Secret and sensitive material has been managed correctly
• Automated testing was updated or added to match the most common scenarios
• Documentation and comments were added for features that require explanation

Contributor checklist

• Have you signed the contributor license agreement?
• Have you followed the contributor guidelines?

[github-actions]

Bug - Guidelines

These guidelines serve as a reminder set of considerations when addressing a bug in the code.

Documentation and Context

• Provide detailed documentation (description, screenshots, reproducing the bug, etc.) of the bug if not already documented in an issue.
• Include additional context or details about the problem.
• Ensure the fix includes necessary updates to the release documentation and versioning.

Code Standards and Practices

• Code follows established design patterns within the repo and avoids duplication.
• Ensure that the code is modular and reusable where applicable.

Testing

• New unit tests have been added to cover the bug fix or edge cases.
• Existing unit tests have been updated to reflect the changes.
• Provide evidence of testing and detecting the bug fix (e.g., test logs, screenshots).
• Validate that any rules affected by the bug are correctly updated.
• Ensure that performance is not negatively impacted by the changes.
• Verify that any release artifacts are properly generated and tested.
• Conducted system testing, including fleet, import, and create APIs (e.g., run make test-cli, make test-remote-cli, make test-hunting-cli)

Additional Checks

• Verify that the bug fix works across all relevant environments (e.g., different OS versions).
• Confirm that the proper version label is applied to the PR patch, minor, major.

[tradebot-elastic]

⛔️ Test failed

Results

• ❌ System File Ownership Change (eql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ System File Ownership Change (eql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta

[tradebot-elastic]

⛔️ Test failed

Results

• ❌ System File Ownership Change (eql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ System File Ownership Change (eql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta

[tradebot-elastic]

⛔️ Test failed

Results

• ❌ System File Ownership Change (eql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta

[tradebot-elastic]

⛔️ Test failed

Results

• ❌ System File Ownership Change (eql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta