feat: ESQL query validation against Elastic cluster #4955
+908
−56
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
traut
commented
Aug 1, 2025
|
|
||
| rule_integrations = meta.get("integration", []) | ||
| if rule_integrations: | ||
| for integration in rule_integrations: |
There was a problem hiding this comment.
simple style fix, replacing if condition with a more robust default value condition via
rule_integrations = meta.get("integration") or []
traut
commented
Aug 1, 2025
| package = value | ||
|
|
||
| if package in list(package_manifest): | ||
| if package in package_manifest: |
traut
commented
Aug 1, 2025
traut
commented
Aug 1, 2025
detection_rules/rule_validators.py
Outdated
|
|
||
| log(f"Got query columns: {', '.join(query_column_names)}") | ||
|
|
||
| # FIXME: validate the dynamic columns |
There was a problem hiding this comment.
The columns returned from the cluster must be validated against the input mapping, and the dynamic fields checked for validity.
|
at the moment (before any field validation) the test marks 33 rules out of 75 as invalid. The tests were executed against a vanilla local The many errors are most probably because of the bugs in the code, so I expect the number of invalid rules to go down after those are fixed. full log |
|
Updated to include initial dynamic field validation. This will parse the schema(s) for dynamic fields and perform some initial formatting check. It checks if the field has a proper prefix as described in #4909, and if the field is based on a field that is present in the schema. However, additional validation will be needed if we want to validate the proper types for ES|QL function and operator return values. https://www.elastic.co/docs/reference/query-languages/esql/esql-functions-operators Additionally, a number of the errors seen in the above testing are due to schema updates that do not have the required fields. For instance. Next steps are:
Note after discussion with @Mikaayenson we determined that the sub-field of the dynamic query does not need to have ecs enforcement here. E.g. For |
eric-forte-elastic
added
test-suite
Mikaayenson
reviewed
Sep 9, 2025
|
|
||
| return nested_multifields # type: ignore[reportUnknownVariableType] | ||
|
|
||
| def get_ecs_schema_mappings(self, current_version: Version) -> dict[str, Any]: |
There was a problem hiding this comment.
should this go in ecs.py?
There was a problem hiding this comment.
It could, but it would be specific for loading an index mapping for ESQL. This function is primarily taking the ecs mapping and editing it to an index mapping format, with specific handling of scaled floats. I think this boils down to preference, fine with me either way.
Mikaayenson
reviewed
Sep 9, 2025
Comment on lines
+899
to
+912
| def remote_validate_rule_contents( | ||
| self, kibana_client: Kibana, elastic_client: Elasticsearch, contents: TOMLRuleContents, verbosity: int = 0 | ||
| ) -> ObjectApiResponse[Any]: | ||
| """Remote validate a rule's ES|QL query using an Elastic Stack.""" | ||
| return self.remote_validate_rule( | ||
| kibana_client=kibana_client, | ||
| elastic_client=elastic_client, | ||
| query=contents.data.query, # type: ignore[reportUnknownVariableType] | ||
| metadata=contents.metadata, | ||
| rule_id=contents.data.rule_id, | ||
| verbosity=verbosity, | ||
| ) | ||
|
|
||
| def remote_validate_rule( # noqa: PLR0913 |
There was a problem hiding this comment.
should these go in remote_validation.py?
There was a problem hiding this comment.
No, see https://github.com/elastic/detection-rules/pull/4955/files/9e1150cbd9962a13e9ce11fd564eaea3855030bf#r2334535612 and https://github.com/elastic/detection-rules/pull/4955/files/9e1150cbd9962a13e9ce11fd564eaea3855030bf#r2334527576 for more detail, but in short, this is remote syntax validation. The fact of it being remote will go away upon the presence of local ESQL syntax validation. The remote_validation.py worksflows are not for query language syntax validation, but to run the query against the stack and provide the response (not specifically implying valid or invalid, that is left to the calling function).
Mikaayenson
reviewed
Sep 9, 2025
| return nested_schema # type: ignore[reportUnknownVariableType] | ||
|
|
||
|
|
||
| def combine_dicts(dest: dict[Any, Any], src: dict[Any, Any]) -> None: |
There was a problem hiding this comment.
is this supposed to be dict.update(dict)
There was a problem hiding this comment.
No, dict.update is a replace/update function.
This is a recursive merge (in effect combining the dictionaries vs overwriting the keys).
Example:
>>> from typing import Any
>>> def combine_dicts(dest: dict[Any, Any], src: dict[Any, Any]) -> None:
... """Combine two dictionaries recursively."""
... for k, v in src.items():
... if k in dest and isinstance(dest[k], dict) and isinstance(v, dict):
... combine_dicts(dest[k], v) # type: ignore[reportUnknownVariableType]
... else:
... dest[k] = v
...
>>> dest = {'a': 1, 'b': {'x': 10, 'y': 20}}
>>> src = {'b': {'y': 30, 'z': 40}, 'c': 3}
>>> combine_dicts(dest, src)
>>> dest
{'a': 1, 'b': {'x': 10, 'y': 30, 'z': 40}, 'c': 3}
>>> dest = {'a': 1, 'b': {'x': 10, 'y': 20}}
>>> dest.update(src)
>>> dest
{'a': 1, 'b': {'y': 30, 'z': 40}, 'c': 3}
Mikaayenson
reviewed
Sep 9, 2025
…tection-rules into esql-field-validation
|
Considerations from discussion with @Mikaayenson :
|
|
Follow up:
|
Mikaayenson
reviewed
Sep 18, 2025
There was a problem hiding this comment.
Left some questions and some nits. The only thing I think we should sync on is the placement of some of the functions across the different modules. We may be able to create an esql.py and migrate some things there. In particular placement for functions in misc, rule_validators, and utils.
| name: ES|QL Validation | ||
| on: | ||
| push: | ||
| branches: [ "main", "7.*", "8.*", "9.*" ] |
There was a problem hiding this comment.
Suggested change
| branches: [ "main", "7.*", "8.*", "9.*" ] | |
| branches: [ "main", "8.*", "9.*" ] |
Comment on lines
+17
to
+18
| path: elastic-container | ||
| repository: eric-forte-elastic/elastic-container |
There was a problem hiding this comment.
Any reason we can't use the upstream ECP? Also what about a dedicated stack?
|
|
||
| steps: | ||
| - name: Check out repository | ||
| uses: actions/checkout@v4 |
There was a problem hiding this comment.
Suggested change
| uses: actions/checkout@v4 | |
| uses: actions/checkout@v5 |
Also better to use the hash
| - name: Setup Detection Rules | ||
| uses: actions/checkout@v4 |
| fetch-depth: 0 | ||
|
|
||
| - name: Set up Python 3.13 | ||
| uses: actions/setup-python@v5 |
|
|
||
|
|
||
| def convert_to_nested_schema(flat_schemas: dict[str, str]) -> dict[str, Any]: | ||
| """Convert a flat schema to a nested schema with 'properties' for each sub-key.""" |
There was a problem hiding this comment.
Can we add why we need this for our future selves. E.g. needed to help generate a proper schema for x.
There was a problem hiding this comment.
Sure 👍
Note for posterity, we need this to conform to Kibana Index mapping schema.
Comment on lines
+547
to
+549
| # NOTE This is done with a dataclass but could also be done with dict, etc. | ||
| # Also other places this is called an <integration.integration> instead. | ||
| # such as in integrations.py def parse_datasets |
There was a problem hiding this comment.
Do we want to be consistent then? Is this package/integrations?
There was a problem hiding this comment.
This is package/integrations but I think that naming convention is incorrect based on how the integrations repo names them: e.g. https://github.com/elastic/integrations/tree/main/packages/aws/data_stream/cloudtrail
So I had this to be consistent with integrations and avoided renaming, but could also have this be package/integrations and we can rename to match integrations in a different PR.
There was a problem hiding this comment.
Peer review, switching to package/integrations as this is more consistent with the UI.
| def validate(self, _: "QueryRuleData", __: RuleMeta) -> None: # type: ignore[reportIncompatibleMethodOverride] | ||
| def get_unique_field_type(self, field_name: str) -> str | None: # type: ignore[reportIncompatibleMethodOverride] | ||
| """Get the type of the unique field. Requires remote validation to have occurred.""" | ||
| for field in self.esql_unique_fields: |
There was a problem hiding this comment.
Suggested change
| for field in self.esql_unique_fields: | |
| for field in self.unique_fields: |
?
There was a problem hiding this comment.
Fine to rename 👍 Will wait to do so until https://github.com/elastic/detection-rules/pull/4955/files/d973bd1ec599f530d273dd6028cd03e634d37170#r2358157087 is resolved.
Comment on lines
+725
to
+730
| def __init__(self, query: str) -> None: | ||
| """Initialize the ESQLValidator with the given query.""" | ||
| super().__init__(query) | ||
| self.esql_unique_fields: list[dict[str, str]] = [] | ||
| self.related_integrations: list[dict[str, str]] = [] | ||
| self.stack_version: str = "" |
There was a problem hiding this comment.
Given this is a dataclass and precedence of the other validators, it seems unconventional to __init__.
There was a problem hiding this comment.
From the dataclass perspective, we have been using __init__ for others, but yes certainly this is a break in convention for the other query validators. Given that this is in effect a remote validator (where the others are not), I think it would break convention to some degree. There are other ways of storing the output, but they are needed in the validation pipeline for things like related integrations, etc. So we then may need to store it as metadata in the rule or add it somewhere else so it can be accessed on a per rule basis without using an init here.
| ) | ||
|
|
||
| def validate_integration( | ||
| def get_rule_integrations(self, metadata: RuleMeta) -> list[str]: |
There was a problem hiding this comment.
I like the introduction of this. It would be nice to double check where we could use this throughout the repo.
|
Move all applicable Kibana index mapping functions to a separate index_mappings.py |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pull Request
Issue link(s):
Summary - What I changed
As a note to reviewers, the entry point when validating a given rule is through
remote_validate_rule.Another note, in some integrations (specifically Okta) there are fields defined in the integration where the mapping is not directly supported in the stack. See details below for an example. Fleet handles these cases by removing the offending fields. As such, this PR proposes a similar process. See
find_nested_multifieldsfor the core logic for identifying these offending fields.Details
When using the Okta mapping as-is, one would receive the following error:
We can see in the integration YAML
(Relevant Snippet)
logOnlySecurityData is a keyword but has fields, behaviors is a field of logOnlySecurityData and is also a keyword, but is also has fields like New_City which is not allowed according to the error message.
When installing the integration through fleet, one can see that it strips the sub-fields under behaviors.
How To Test
.detection-rules-cfg.yml) or from the environment variablesOnce you have the environment variables setup and stack ready, you can test the remote validation with the following command:
python -m pytest tests/test_rules_remote.py::TestRemoteRules::test_esql_rules -s -vNote,
-vis optional but provides useful debugging information.Also, test remote validation with the rule loader through view-rule via the following:
export DR_REMOTE_ESQL_VALIDATION=True python -m detection_rules view-rule rules/linux/discovery_port_scanning_activity_from_compromised_host.tomlChecklist
bug,enhancement,schema,maintenance,Rule: New,Rule: Deprecation,Rule: Tuning,Hunt: New, orHunt: Tuningso guidelines can be generatedmeta:rapid-mergelabel if planning to merge within 24 hoursContributor checklist