Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Service logs do not cover immediate startup failures #36

Closed
ScottFreeCode opened this issue Oct 18, 2021 · 5 comments · Fixed by #42 or #43
Closed

Service logs do not cover immediate startup failures #36

ScottFreeCode opened this issue Oct 18, 2021 · 5 comments · Fixed by #42 or #43
Labels
bug Something isn't working
Milestone

Comments

@ScottFreeCode
Copy link
Contributor

Usually when a service fails to pass the healthcheck, --log-services provides useful info. However, sometimes the log file is empty and to see its output I have to catch the container with a Docker attach command in the fleeting window during which it is live. It seems as though Touchstone connects to the container after the initial docker run command to get the logs. I would expect it to direct the container output to a file as part of the initial container run/start command, guaranteeing all output is logged.

@shanejansen shanejansen modified the milestone: 1.4.1 Oct 22, 2021
@shanejansen
Copy link
Owner

Log files are retrieved using the docker container logs command just before the container is stopped. Not sure why that wouldn't be capturing all logs. May have to do some investigation

@ScottFreeCode
Copy link
Contributor Author

Possibly the container is already dead when Touchstone attempts the command; it really only happens when the app fails to start up in the first place (but if I can catch it while it's trying to start I do see that it's logging, it's just that it's already crashed by the time Touchstone tries to shut it down).

(On the other hand if it's stopped but the container should still have its logs retrievable, and Docker still isn't providing them, maybe it's a Docker bug!)

@ScottFreeCode
Copy link
Contributor Author

It occurred to me rereading this that it's possible Touchstone may be starting the container with the autoremove flag, in which case removing that flag and additing explicit container cleanup after/during shutdown could resolve this.

(Speaking of which, does Touchstone clean up built services' intermediate images/layers? Obviously calling prune and potentially catching non-Touchstone stuff would be inadvisable; but as regards its own service images, the more Touchstone can clean up after itself the more disk space will be!)

@shanejansen
Copy link
Owner

Touchstone does run containers with the --rm flag, so you're right the container could be removed before Touchstone has a change to get logs if a health check fails.

During shutdown, Touchstone will clean up images that it produced (like supplying a Dockerfile in touchstone.yml)

@shanejansen shanejansen added this to the 1.4.1 milestone Nov 2, 2021
@shanejansen shanejansen added the bug Something isn't working label Nov 2, 2021
ScottFreeCode added a commit to ScottFreeCode/touchstone that referenced this issue Feb 4, 2022
@ScottFreeCode
Copy link
Contributor Author

ScottFreeCode commented Feb 4, 2022

I have opened two PRs with different possible fixes, with different pros and cons. #42 and #43 EDIT: wow, they fix most of each others' cons if combined, and I didn't even think about that!

@shanejansen shanejansen modified the milestones: v1.4.2, v1.4.1 Feb 17, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
2 participants