fix: include task execution errors in task logs#60
Merged
Conversation
Executor failures only reached the task's error field, never the task log stream users read via `flowmesh logs`: the failure handler logged a bare "Task failed" line and TaskLogEmitter dropped record exc_info. Emit the formatted traceback for unexpected exceptions and a clean message for controlled ExecutionError, so the reason is visible in the stream. Signed-off-by: Noppanat Wadlom <noppanat.wad@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Surface task execution failures in the task log stream that users read via
flowmesh logs. Today a failure only reaches the task'serrorstatus field; the log stream gets a bareTask <id> failedline with no reason. This is the "Task error logs" item from the 2026 Q2 roadmap RFC (#48) — debugging a failed task, especially one whose spec fails an executor-side validation check, currently requires inspecting task status separately from the logs.The gap had two causes: the worker's failure handler logged a non-substantive message, and
TaskLogEmitter.emit()built its payload fromrecord.getMessage()only, silently discardingrecord.exc_infoso no traceback ever reached the stream.Changes
src/worker/utils/logging.py—TaskLogEmitter.emit()now appends the formatted traceback to the emitted message wheneverrecord.exc_infois set, using a sharedlogging.Formatterand caching intorecord.exc_text(consistent with stdlibFormatter.format). Anylogger.exception(...)under the attached handler now carries its traceback into both the gRPC stream and the JSONL sink.src/worker/runner.py— the task failure handler distinguishes controlledExecutionError(logs a cleanTask <id> failed: <reason>line, no traceback) from unexpected exceptions (logs the full traceback).TaskCancelledErrorkeeps its own separate branch since it terminates asCANCELLED, notFAILED.tests/worker/test_task_log_emitter.py— new unit tests assertingemit()includes the traceback and exception text whenexc_infois present, and emits just the formatted message otherwise.Test Plan
Also verified end-to-end against a local stack: brought up the server with CPU workers, submitted a workflow whose spec passes the server parser but fails an executor-side validation check, and confirmed the executor's error reason now appears in both the task log stream (
flowmesh task logs show) and the workflow log stream.Test Result
pytest tests/worker/test_task_log_emitter.py tests/worker/test_connector_logging.py— 3 passed.pre-commiton the changed files — gitleaks, isort, black, ruff, codespell, mypy all passed.echo executor mapping item must contain either 'expr' ...) appeared in both the task and workflow log streams, where before this change the stream showed only a bareTask <id> failed. The task's terminalerrorstatus field still reports the dispatcher's genericmax_attempts_exceeded, which is exactly why surfacing the real reason in the logs matters.