You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[autorevert] Configure root logger for AWS lambda (#7183)
Explanation from gpt-5-codex:
> Info-level lines from the new flow disappear in Lambda because of how
Python logging behaves once the runtime has already attached its own
handler. On every cold start AWS sets up the root logger with a handler
that streams to CloudWatch and leaves the logger at its default WARNING
level. Python’s logging.basicConfig(...) (which setup_logging calls
in pytorch_auto_revert/__main__.py:24) only configures the root logger
when it has no handlers; if one is already present, the call is a
no-op—no level change, no formatter change. Even though LOG_LEVEL is
DEBUG, we never actually lower the root logger’s threshold, so
logging.info(...) statements from testers/autorevert_v2.py respect the
still-
WARNING root level and get filtered out. The lone line you’re seeing
([WARNING] Workflow … already restarted …) comes from
logging.warning(...) in workflow_checker.py:121, which survives because
it meets the logger’s threshold.
> In the legacy flow (see pytorch_auto_revert/testers/autorevert.py),
most “logs” were literal print(...) calls. Lambda captures stdout/stderr
and labels it as INFO, so those showed up even without any logging
configuration. We shifted to structured logging in v2, but because the
runtime’s pre-existing handler prevented basicConfig from taking effect,
everything below WARNING was silently dropped. The patch we discussed
explicitly sets the root logger level (and adjusts handler levels when
they’re left at NOTSET), so the value from LOG_LEVEL truly applies and
the [v2] … info lines will appear once the Lambda is redeployed.
----
The fix verified by monkey-patching the logger setup in prod.
0 commit comments