Skip to content

Commit a97279b

Browse files
authored
[autorevert] Configure root logger for AWS lambda (#7183)
Explanation from gpt-5-codex: > Info-level lines from the new flow disappear in Lambda because of how Python logging behaves once the runtime has already attached its own handler. On every cold start AWS sets up the root logger with a handler that streams to CloudWatch and leaves the logger at its default WARNING level. Python’s logging.basicConfig(...) (which setup_logging calls in pytorch_auto_revert/__main__.py:24) only configures the root logger when it has no handlers; if one is already present, the call is a no-op—no level change, no formatter change. Even though LOG_LEVEL is DEBUG, we never actually lower the root logger’s threshold, so logging.info(...) statements from testers/autorevert_v2.py respect the still- WARNING root level and get filtered out. The lone line you’re seeing ([WARNING] Workflow … already restarted …) comes from logging.warning(...) in workflow_checker.py:121, which survives because it meets the logger’s threshold. > In the legacy flow (see pytorch_auto_revert/testers/autorevert.py), most “logs” were literal print(...) calls. Lambda captures stdout/stderr and labels it as INFO, so those showed up even without any logging configuration. We shifted to structured logging in v2, but because the runtime’s pre-existing handler prevented basicConfig from taking effect, everything below WARNING was silently dropped. The patch we discussed explicitly sets the root logger level (and adjusts handler levels when they’re left at NOTSET), so the value from LOG_LEVEL truly applies and the [v2] … info lines will appear once the Lambda is redeployed. ---- The fix verified by monkey-patching the logger setup in prod.
1 parent fb3d216 commit a97279b

File tree

1 file changed

+16
-1
lines changed
  • aws/lambda/pytorch-auto-revert/pytorch_auto_revert

1 file changed

+16
-1
lines changed

aws/lambda/pytorch-auto-revert/pytorch_auto_revert/__main__.py

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,22 @@ def setup_logging(log_level: str) -> None:
2323
numeric_level = getattr(logging, log_level.upper(), None)
2424
if not isinstance(numeric_level, int):
2525
raise ValueError(f"Invalid log level: {log_level}")
26-
logging.basicConfig(level=numeric_level)
26+
27+
root_logger = logging.getLogger()
28+
root_logger.setLevel(numeric_level)
29+
30+
if not root_logger.handlers:
31+
handler = logging.StreamHandler()
32+
handler.setLevel(numeric_level)
33+
formatter = logging.Formatter(
34+
"%(asctime)s %(levelname)s [%(name)s] %(message)s"
35+
)
36+
handler.setFormatter(formatter)
37+
root_logger.addHandler(handler)
38+
else:
39+
for handler in root_logger.handlers:
40+
if handler.level == logging.NOTSET:
41+
handler.setLevel(numeric_level)
2742

2843

2944
def get_opts() -> argparse.Namespace:

0 commit comments

Comments
 (0)