ci(botocore): fix flaky botocore tests

[mabdinur]

Botocore sensitive data redaction sets AWSPayloadTagging.request_redaction_paths and AWSPayloadTagging.response_redaction_paths the first time AWSPayloadTagging.expand_payload_as_tags(...) is called.

In tests overriding config.botocore.payload_tagging_request and config.botocore.payload_tagging_request using tests.utils.override_config(..) is not sufficient. We also need to set ddtrace._trace.utils_botocore.span_tags._PAYLOAD_TAGGER.validated to False. This ensures the overriden botocore configurations are actually used.

Note: This PR does not fix the underlying issue of AWSPayloadTagging configurations becoming out of sync with ddtrace.config.botocore. We will need to address this bug in a future PR.

Depending on the order the tests were run the following snapshot test would fail: https://app.circleci.com/pipelines/github/DataDog/dd-trace-py/80321/workflows/54bb5dd9-9c50-4e2a-afeb-e168c6c1e6d9/jobs/4394672

Checklist

• PR author has checked that all the criteria below are met
• The PR description includes an overview of the change
• The PR description articulates the motivation for the change
• The change includes tests OR the PR description describes a testing strategy
• The PR description notes risks associated with the change, if any
• Newly-added code is easy to change
• The change follows the library release note guidelines
• The change includes or references documentation updates if necessary
• Backport labels are set (if applicable)

Reviewer Checklist

• Reviewer has checked that all the criteria below are met
• Title is accurate
• All changes are related to the pull request's stated goal
• Avoids breaking API changes
• Testing strategy adequately addresses listed risks
• Newly-added code is easy to change
• Release note makes sense to a user of the library
• If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
• Backport labels are set in a manner that is consistent with the release branch maintenance policy

[github-actions]

CODEOWNERS have been resolved as:

tests/contrib/botocore/test.py                                          @DataDog/apm-core-python @DataDog/apm-idm-python

[datadog-dd-trace-py-rkomorn]

Datadog Report

Branch report: munir/make-botocore-test-less-flaky
Commit report: 909975c
Test service: dd-trace-py

✅ 0 Failed, 87 Passed, 1468 Skipped, 4m 13.7s Total duration (35m 22.71s time saved)

[pr-commenter]

Benchmarks

Benchmark execution time: 2025-01-08 18:39:01

Comparing candidate commit 909975c in PR branch munir/make-botocore-test-less-flaky with baseline commit 86a367a in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 394 metrics, 2 unstable metrics.

[brettlangdon]

Does this need to be set for all botocore tests? would an auto-use fixture be better? e.g. if someone adds a new test do they need to copy/pasta this setting as well?

example:

@pytest.fixture(autouse=True)
def reset_payload_tagger():    
    span_tags._PAYLOAD_TAGGER.validated = False

[bouwkast]

Does this need to be set for all botocore tests? would an auto-use fixture be better? e.g. if someone adds a new test do they need to copy/pasta this setting as well?

example:

@pytest.fixture(autouse=True)
def reset_payload_tagger():    
    span_tags._PAYLOAD_TAGGER.validated = False

This feature is disabled by default so most of these tests will not use this, but if it is as simple as that then I think that is reasonable.

[mabdinur]

@pytest.fixture(autouse=True)
def reset_payload_tagger():    
    span_tags._PAYLOAD_TAGGER.validated = False

Yup that's a cleaner approach, we can also add this logic to BotocoreTest.setUp(...). Ideally we also should address the underlying issue and not just re-write tests to avoid the broken edge case.

I just want to fix flakies 🔫