test(assisted_query): add eval for organizationName tag filter on issues dataset#117802
Closed
DominikB2014 wants to merge 1 commit into
Closed
test(assisted_query): add eval for organizationName tag filter on issues dataset#117802DominikB2014 wants to merge 1 commit into
DominikB2014 wants to merge 1 commit into
Conversation
…ues dataset
Regression scenario: when asked to filter issues by an organizationName-style
custom tag, the AI search agent fell back to embedding the raw org name as free
text in the query (e.g., 'mtk_agent_workshop') instead of generating a proper
tag filter ('organizationName:mtk_agent_workshop'). User feedback captured as
'Organization filter not found'.
Adds TestOrganizationNameTagFilter with four evals against the issues dataset
tool layer:
- get_issue_filter_keys surfaces organizationName when events carry that tag
- get_filter_key_values returns the expected values for the tag
- execute_issues_query accepts an organizationName tag filter and returns matches
- get_filter_key_values returns [] (not None) when the tag is absent, giving the
agent the correct signal to report 'no values' rather than fall back to text
Co-Authored-By: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com>
Contributor
Author
|
Closing — wrong file. The eval belongs in getsentry/seer evals/assisted_query/fixtures/examples/issues.json. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a test class to
tests/sentry/seer/assisted_query/test_issues_tools.pywith four evals targeting the issues dataset tool layer.Why
User feedback captured in Sentry: "Organization filter not found". The AI search agent was asked to filter issues by an
organizationName-style custom tag but couldn't find the filter, so it fell back to embedding the raw org name (mtk_agent_workshop) as free text in the query instead of generating a proper tag filter (organizationName:mtk_agent_workshop).What's tested
test_get_issue_filter_keys_includes_organization_name_tagget_issue_filter_keyssurfacesorganizationNamein the tags list when events carry that custom tagtest_get_filter_key_values_returns_organization_name_valuesget_filter_key_values(attribute_key="organizationName")returns expected values so the agent can build a correct tag filtertest_execute_issues_query_with_organization_name_tag_filterexecute_issues_queryacceptsorganizationName:acme_corpand returns matching issues (not an error dict)test_get_filter_key_values_returns_empty_list_when_tag_absent[](notNone) — the signal the agent must use to decide "no values" vs "invalid field"All four tests exercise the issues dataset path (
Dataset.Events+Dataset.IssuePlatform) via real Snuba queries, consistent with the surroundingtest_issues_tools.pypatterns.View Session in Sentry