Update troubleshooting guides with debug service usage (#41396)

* refactor(docs): update docs to include debug service usage * docs: code review suggestions * docs: update debug service intro definitions * Apply suggestions from code review Co-authored-by: Paul Gottschling <[email protected]> * docs(diagnostics): move teleport-debug-config section up in the guide --------- Co-authored-by: Paul Gottschling <[email protected]>
gravitational · May 21, 2024 · 892e139 · 892e139
1 parent 737cbdd
commit 892e139
Show file tree

Hide file tree

Showing 6 changed files with 119 additions and 10 deletions.
diff --git a/docs/pages/includes/config-reference/debug-service.yaml b/docs/pages/includes/config-reference/debug-service.yaml
@@ -0,0 +1,3 @@
+# These settings apply to the Debug service:
+debug_service:
+    enabled: true
diff --git a/docs/pages/includes/diagnostics/teleport-debug-config.mdx b/docs/pages/includes/diagnostics/teleport-debug-config.mdx
@@ -0,0 +1,6 @@
+<Admonition type="tip" title="Specify your Teleport configuration path">
+If your Teleport configuration is not placed on the default path
+(`/var/lib/teleport`), you must to specify its location to the CLI command
+using the `-c/--config` flag.
+</Admonition>
+
diff --git a/docs/pages/management/admin/troubleshooting.mdx b/docs/pages/management/admin/troubleshooting.mdx
@@ -18,6 +18,39 @@ services such as the Application Service and Database Service.
 
 ## Step 1/3. Enable verbose logging
 
+To change log levels in Teleport, you can use either of the following methods:
+
+- Debug Service: Allows on-the-fly log level adjustments without restarting the
+  instance, which is ideal for troubleshooting sessions.
+- Updating configuration: Involves updating the Teleport configuration file and
+  restarting the instance.
+
+<Tabs>
+<TabItem label="Debug Service">
+The Teleport Debug Service allows administrators to dynamically manage log levels
+without restarting the instance. The service, enabled by default, ensures
+local-only access and must be consumed from inside the same instance.
+
+To change the instance log level use the `teleport debug set-log-level` command:
+
+```code
+$ teleport debug set-log-level DEBUG
+Changed log level from "INFO" to "DEBUG".
+
+$ kubectl -n teleport exec my-pod -- teleport set-log-level DEBUG 
+Changed log level from "INFO" to "DEBUG".
+```
+
+If you're unsure what is the current level you can retrieve it using
+`teleport debug get-log-level`.
+
+After troubleshooting, remember to turn the log level back to avoid generating
+unnecessary logs.
+
+(!docs/pages/includes/diagnostics/teleport-debug-config.mdx!)
+</TabItem>
+
+<TabItem label="Updating configuration">
 To diagnose problems, you can configure the `teleport` process to run with
 verbose logging enabled by passing it the `-d` flag. `teleport` will write logs
 to stderr.
@@ -33,6 +66,12 @@ teleport:
 Restart the `teleport` process to apply the modified log level. Logs will resemble
 the following (these logs were printed while joining a server to a cluster, then
 terminating the `teleport` process on the server):
+</TabItem>
+</Tabs>
+
+Debug logs include the file and line number of the code that emitted the log, so
+you can investigate (or report) what a `teleport` process was doing before it ran into
+problems. Here's an example:
 
 ```
 DEBU [NODE:PROX] Agent connected to proxy: [aee1241f-0f6f-460e-8149-23c38709e46d.tele.example.com aee1241f-0f6f-460e-8149-23c38709e46d teleport-proxy-us-west-2-6db8db844c-ftmg9.tele.example.com teleport-proxy-us-west-2-6db8db844c-ftmg9 localhost 127.0.0.1 ::1 tele.example.com 100.92.90.42 remote.kube.proxy.teleport.cluster.local]. leaseID:4 target:tele.example.com:11106 reversetunnel/agent.go:414
@@ -43,10 +82,6 @@ DEBU [NODE:PROX] Pool is closing agent. leaseID:2 target:tele.example.com:11106
 DEBU [NODE:PROX] Pool is closing agent. leaseID:3 target:tele.example.com:11106 reversetunnel/agentpool.go:238
 ```
 
-Debug logs include the file and line number of the code that emitted the log, so
-you can investigate (or report) what a `teleport` process was doing before it ran into
-problems.
-
 <Notice
   type="warning"
 >

diff --git a/docs/pages/management/diagnostics/profiles.mdx b/docs/pages/management/diagnostics/profiles.mdx
@@ -7,20 +7,74 @@ Teleport leverages Go's diagnostic capabilities to collect and export
 profiling data. Profiles can help identify the cause of spikes in CPU,
 the source of memory leaks, or the reason for a deadlock.
 
-## Enable profiling
+## Using the Debug Service
+
+The Teleport Debug Service enables administrators to collect diagnostic profiles
+without enabling pprof endpoints at startup. The service, enabled by default,
+ensures local-only access and must be consumed from inside the same instance.
+
+`teleport debug profile` collects a list of pprof profiles. It outputs a
+compressed tarball (`.tar.gz`) to STDOUT. You decompress it using `tar` or
+direct the result to a file.
+
+By default, it collects `goroutine`, `heap` and `profile` profiles.
+
+Each profile collected will have a correspondent file inside the tarball. For
+example, collecting `goroutine,trace,heap` will result in `goroutine.pprof`,
+`trace.pprof`, and `heap.pprof` files.
+
+```code
+# Collect default profiles and save to a file.
+$ teleport debug profile > pprof.tar.gz
+$ tar xvf pprof.tar.gz
+
+# Collect default profiles and decompress it.
+$ teleport debug profile | tar xzv -C ./
+
+# Collect "trace" and "mutex" profiles and save to a file.
+$ teleport debug profile trace,mutex > pprof.tar.gz
+
+# Collect profiles setting the profiling time in seconds
+$ teleport debug profile -s 20 trace > pprof.tar.gz
+```
+
+(!docs/pages/includes/diagnostics/teleport-debug-config.mdx!)
+
+If you're running Teleport on a Kubernetes cluster you can directly collect
+profiles to a local directory without an interactive session:
+
+```code
+$ kubectl -n teleport exec my-pod -- teleport debug profile > pprof.tar.gz 
+```
+
+After extracting the contents, you can use `go tool` commands to explore and
+visualize them:
+
+```code
+# Opens the terminal interactive explorer
+$ go tool pprof heap.pprof
+
+# Opens the web visualizer
+$ go tool pprof -http : heap.pprof
+
+# Visualize trace profiles
+$ go tool trace trace.pprof
+```
+
+## Using diagnostics endpoints
 
 The profiling endpoint is only enabled if the `--debug` flag is supplied.
 
 (!docs/pages/includes/diagnostics/diag-addr-prereqs-tabs.mdx flags="--debug" !)
 
-## Collecting profiles
+### Collecting profiles
 
 Go's standard profiling endpoints are served at `http://127.0.0.1:3000/debug/pprof/`.
 Retrieving a profile requires sending a request to the endpoint corresponding
 to the desired profile type. When debugging an issue it is helpful to collect
 a series of profiles over a period of time.
 
-### CPU
+#### CPU
 CPU profile shows execution statistics gathered over a user specified period:
 
 ``` code
@@ -31,7 +85,7 @@ $ curl -o cpu.profile http://127.0.0.1:3000/debug/pprof/profile?seconds=30
 $ go tool pprof -http : cpu.profile
 ```
 
-### Goroutine
+#### Goroutine
 
 Goroutine profiles show the stack traces for all running goroutines in the system:
 
@@ -43,7 +97,7 @@ $ curl -o goroutine.profile http://127.0.0.1:3000/debug/pprof/goroutine
 $ go tool pprof -http : goroutine.profile
 ```
 
-### Heap
+#### Heap
 
 Heap profiles show allocated objects in the system:
 
@@ -55,7 +109,7 @@ $ curl -o heap.profile http://127.0.0.1:3000/debug/pprof/heap
 $ go tool pprof  -http : heap.profile
 ```
 
-### Trace
+#### Trace
 
 Trace profiles capture scheduling, system calls, garbage collections, heap size, and other events that are collected by the Go runtime
 over a user specified period of time:

diff --git a/docs/pages/reference/cli/teleport.mdx b/docs/pages/reference/cli/teleport.mdx
@@ -32,6 +32,9 @@ The primary commands for the `teleport` CLI are as follows:
 | `teleport start` | Starts the `teleport` process in the foreground using the current shell session, including any services configured by the [configuration YAML file](../config.mdx). |
 | `teleport status` | Prints the status of the current active Teleport SSH session. |
 | `teleport version` | Prints the current release version of the Teleport binary installed on your system. |
+| `teleport debug set-log-level` | Changes instance log level. |
+| `teleport debug get-log-level` | Fetches instance current log level. |
+| `teleport debug debug profile` | Export the application profiles (pprof format). |
 
 <Notice type="tip">
 For more information on subcommands when working with the `teleport` cli, use the `--help` option or `teleport <subcommand> --help`.

diff --git a/docs/pages/reference/config.mdx b/docs/pages/reference/config.mdx
@@ -178,6 +178,14 @@ These settings apply to the Jamf Service:
 (!docs/pages/includes/config-reference/jamf-service.yaml!)
 ```
 
+### Debug Service
+
+These settings apply to the Debug Service
+
+```yaml
+(!docs/pages/includes/config-reference/debug-service.yaml!)
+```
+
 ## Configuration versions
 
 In order to avoid breaking existing configurations, Teleport's configuration is