debugging: slides: properly introduce flamegraphs and hotspot

Flamegraphs, while being a very valuable tool to get a graphical view of
some profiling data, are only lightly covered, very early, and somehow
in the middle of other concepts.

Rework flamegraphs introduction to really introduce how to generate and
read them:
- move the flamegraph content in the system-wide profiling part, after
  having introduced the general perf commands, needed to generate
  flamegraph data
- explain how to fetch the scripts and how to use them
- add instructions about how to read a flamegraph, and a custom CPU
  flamegraph to illustrate it
- add hotspot as an alternative/extension

The only downside is that, now that flamegraphs are introduced later, we
reach the heaptrack part without having introduced flamegraphs, but it
may be fine to just announce that "we will cover flamegraphs more in
details later"

Fix #267

Signed-off-by: Alexis Lothoré <[email protected]>

Author: Tropicao

slides/debugging-application-profiling/debugging-application-profiling.tex

@@ -50,37 +50,6 @@ \section{Application Profiling}
   \end{itemize}
 \end{frame}
 
-\begin{frame}
-  \frametitle{Visualizing data with flamegraphs}
-  \begin{itemize}
-    \item Visualization based on hierarchical stacks
-    \item Allows to quickly find bottlenecks and explore the call stack
-    \item Popularized by Brendan Gregg tools which allows to generate
-          flamegraphs from \code{perf} results.
-    \begin{itemize}
-      \item Scripts to generate flamegraphs are available at
-            \url{https://github.com/brendangregg/FlameGraph}
-    \end{itemize} 
-  \end{itemize}
-  \center \includegraphics[width=0.4\textwidth]{../slides/debugging-application-profiling/flamegraph.png}\\
-  \tiny Image credits: \url{https://www.brendangregg.com/flamegraphs.html}
-\end{frame}
-
-\begin{frame}[fragile]
-  \frametitle{Going further with Flamegraphs}
-  \begin{itemize}
-    \item Really nice technical presentation from Brendan Gregg explaining
-          the use of flamegraphs for various metrics.
-    \begin{itemize}
-      \item Video: \url{https://www.youtube.com/watch?v=D53T1Ejig1Q}
-      \item Slides: \url{https://www.slideshare.net/brendangregg/usenix-atc-2017-visualizing-performance-with-flame-graphs}
-    \end{itemize}
-  \end{itemize}
-  \begin{center}
-  \center\includegraphics[height=0.5\textheight]{slides/debugging-application-profiling/flamegraph_atc20.png}
-  \end{center}
-\end{frame}
-
 \subsection{Memory profiling}
 
 \begin{frame}

slides/debugging-application-profiling/flamegraph.png

@@ -387,6 +387,108 @@ \subsection{perf}
   \end{block}
 \end{frame}
 
+
+\begin{frame}[fragile]
+  \frametitle{Using a GUI to display perf data}
+  \begin{itemize}
+    \item \code{perf report} is the default way to display perf data,
+      directly in the console
+    \item There are also graphical tools to display perf data:
+      \begin{itemize}
+        \item
+          \href{http://www.brendangregg.com/flamegraphs.html}{Flamegraphs}
+          \begin{itemize}
+            \item Visualization based on hierarchical stacks
+            \item Allows to quickly find bottlenecks and explore the call stack
+            \item Popularized by Brendan Gregg tools which allows to generate
+              flamegraphs from \code{perf} results.
+          \end{itemize}
+        \item \href{https://github.com/KDAB/hotspot}{Hotspot} software
+          \begin{itemize}
+            \item Developed and maintained by KDAB
+            \item A larger tool able to generate various types of
+              visualizations from a \code{perf.data} file
+            \item Can also perform the actual perf recording
+          \end{itemize}
+      \end{itemize}
+  \end{itemize}
+\end{frame} 
+
+\begin{frame}[fragile]
+  \frametitle{Visualizing data with flamegraphs (1/2)}
+  \begin{itemize}
+    \item Get the flamegraph scripts:
+  \begin{block}{}
+    \begin{minted}[fontsize=\small]{console}
+git clone https://github.com/brendangregg/FlameGraph fl
+    \end{minted}
+  \end{block}
+    \item Capture data:
+  \begin{block}{}
+    \begin{minted}[fontsize=\small]{console}
+perf record -g -- sleep 30
+    \end{minted}
+  \end{block}
+      \begin{itemize}
+        \item The \code{-g} option records call stacks for each sample
+      \end{itemize}
+    \item Format the data:
+      \begin{block}{}
+        \begin{minted}[fontsize=\small]{console}
+perf script|./fl/stackcollapse-perf.pl > out.perf-folded
+        \end{minted}
+      \end{block}
+      \begin{itemize}
+        \item \code{stackcollapse-perf.pl} is part of the
+          \href{https://github.com/brendangregg/FlameGraph}{Flamegraph
+          repository}
+        \item Other data sources are supported
+      \end{itemize}
+    \item Generate the Flamegraph:
+      \begin{block}{}
+        \begin{minted}[fontsize=\small]{console}
+./fl/flamegraph.pl out.perf-folded > flamegraph.svg
+        \end{minted}
+      \end{block}
+    \item The flamegraph can then be opened in  a web browser
+  \end{itemize}
+\end{frame}
+
+\begin{frame}[fragile]
+  \frametitle{Visualizing data with flamegraphs (2/2)}
+  \center
+	\includegraphics[width=1.0\textwidth]{slides/debugging-system-wide-profiling/flamegraph.png}\\
+  \begin{itemize}
+    \item The plates width represents how often a function has been
+      sampled
+    \item The "spikes" height represents the call stacks depth
+    \item Flamegraphs are interactive: clicking a plate will put focus on the
+      corresponding callstack
+    \item Colors can be tuned at flamegraph generation, for example to have
+      a clear split between kernel and userspace functions
+  \end{itemize}
+\end{frame}
+
+\begin{frame}[fragile]
+  \frametitle{Visualizing data with hotspot (1/2)}
+  \begin{itemize}
+	  \item Designed to provide a frontend to perf data files
+    \item Can generate flamegraphs on the fly, but not only:
+      \begin{itemize}
+        \item CPU/tasks timelines
+        \item Interactive callstacks navigation
+        \item Code disassembly
+      \end{itemize}
+    \item Configurable (eg: allows to set paths to find all needed debug
+      informations)
+  \end{itemize}
+\end{frame}
+
+\begin{frame}[fragile]
+  \frametitle{Visualizing data with hotspot (2/2)}
+  \center \includegraphics[width=0.9\textwidth]{slides/debugging-system-wide-profiling/hotspot.png}\\
+\end{frame}
+
 \subsection{ftrace and trace-cmd}
 
 \begin{frame}