Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dev: Switch to leveled logging for alerts #95

Merged
merged 2 commits into from
Sep 30, 2024
Merged

dev: Switch to leveled logging for alerts #95

merged 2 commits into from
Sep 30, 2024

Conversation

jdkaplan
Copy link
Collaborator

@jdkaplan jdkaplan commented Sep 29, 2024

Google Cloud supports log-based alerts, especially if we start adding interesting fields to our logs (such as severity levels). The Go standard library now provides log/slog for structured (and leveled) logging, so let's start using it!

By default, the slog default logger also handles unstructured logs and emits them at "INFO" level. That's exactly what I would have done for most of our logs anyway, so we don't immediately have to change any code we already have.

But I did choose to change one place: end-of-batch has a log.Fatal which is the same as log.Info + os.Exit(1). Rather than exiting the whole process, I'd rather have it emit an error-level log and alert us. I also made it return an HTTP 500 response, so it still shows up as a failed cron job run.

Google App Engine can ingest structured logs, which lets us use the new
structured logger in the Go standard library.

This configures log/slog with some fields renamed to fit Google Cloud's
field name expectations. This setup also catches plain unstructured
`log.Print` and `log.Printf` calls and emits them at "INFO" level.
@jdkaplan
Copy link
Collaborator Author

jdkaplan commented Sep 29, 2024

I tested this by putting some example logs in func main and pushing to dev. They show up in the logs with their levels marked by different colors and icons, just as I expected.

The extra annotations/fields only show up when the log line is expanded, but I think that's okay.

The diff for adding test log lines
diff --git a/main.go b/main.go
index 40241e4..7f0c347 100644
--- a/main.go
+++ b/main.go
@@ -39,6 +39,13 @@ func main() {
 		},
 	})))
 
+	log.Printf("Jeremy test slog formatting: %s", "basic log")
+	slog.Debug("Jeremy test slog formatting", "jdk_level", -4)
+	slog.Info("Jeremy test slog formatting", "jdk_level", 0)
+	slog.Warn("Jeremy test slog formatting", "jdk_level", 4)
+	slog.Error("Jeremy test slog formatting", "jdk_level", 8)
+	log.Fatalln("Jeremy test slog formatting", "jdk_level", "panic")
+
 	ctx := context.Background()
 
 	appVersion := os.Getenv("GAE_VERSION")
Screenshots

The five test log lines in the Logs Explorer

The warning log line expanded in the Logs Explorer

@jdkaplan jdkaplan linked an issue Sep 29, 2024 that may be closed by this pull request
@jdkaplan jdkaplan marked this pull request as ready for review September 29, 2024 18:00
@jdkaplan jdkaplan requested a review from cceckman September 29, 2024 18:00
@jdkaplan jdkaplan mentioned this pull request Sep 29, 2024
Copy link
Collaborator

@cceckman cceckman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

@jdkaplan jdkaplan merged commit 21a010b into main Sep 30, 2024
3 checks passed
@jdkaplan jdkaplan deleted the jdk-slog branch September 30, 2024 22:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Alert on error
2 participants