virtual environment and pycharm and requirements.txt

Author: thomasWeise

draft.sh

@@ -0,0 +1,48 @@
+#!/bin/bash
+
+# Make a draft of the book.
+
+# strict error handling
+set -o pipefail  # trace ERR through pipes
+set -o errtrace  # trace ERR through 'time command' and other functions
+set -o nounset   # set -u : exit the script if you try to use an uninitialized variable
+set -o errexit   # set -e : exit the script if any statement returns a non-true return value
+
+echo "$(date +'%0Y-%0m-%0d %0R:%0S'): Welcome to the draft building script."
+
+echo "$(date +'%0Y-%0m-%0d %0R:%0S'): First deleting the git cache."
+rm -rf __git__ || true
+
+currentDir="$(pwd)"
+echo "$(date +'%0Y-%0m-%0d %0R:%0S'): We are working in directory: '$currentDir'."
+scriptDir="$currentDir/scripts"
+echo "$(date +'%0Y-%0m-%0d %0R:%0S'): The script directory is '$scriptDir'."
+
+echo "$(date +'%0Y-%0m-%0d %0R:%0S'): We delete all left over data."
+rm "book.pdf" || true
+rm -rf "$currentDir/website" || true
+
+echo "$(date +'%0Y-%0m-%0d %0R:%0S'): We setup a virtual environment in a temp directory."
+venvDir="$(mktemp -d)"
+echo "$(date +'%0Y-%0m-%0d %0R:%0S'): Got temp dir '$venvDir', now creating environment in it."
+python3 -m venv --upgrade-deps --copies "$venvDir"
+echo "$(date +'%0Y-%0m-%0d %0R:%0S'): Activating virtual environment in '$venvDir'."
+source "$venvDir/bin/activate"
+export PYTHON_INTERPRETER="$venvDir/bin/python3"
+echo "$(date +'%0Y-%0m-%0d %0R:%0S'): Setting python interpreter to '$PYTHON_INTERPRETER'."
+echo "$(date +'%0Y-%0m-%0d %0R:%0S'): We install all required Python packages from requirements.txt to virtual environment in '$venvDir'."
+"$PYTHON_INTERPRETER" -m pip install --no-input --timeout 360 --retries 100 -r requirements.txt
+echo "$(date +'%0Y-%0m-%0d %0R:%0S'): Finished installing the requirements, now printing all installed packages."
+pip freeze
+echo "$(date +'%0Y-%0m-%0d %0R:%0S'): Finished printing all installed packages."
+
+echo "$(date +'%0Y-%0m-%0d %0R:%0S'): We now execute the pdflatex compiler script."
+"$scriptDir/pdflatex.sh" "book.tex" quick
+
+echo "$(date +'%0Y-%0m-%0d %0R:%0S'): Deactivating virtual environment."
+deactivate
+echo "$(date +'%0Y-%0m-%0d %0R:%0S'): Deleting virtual environment."
+rm -rf "$venvDir"
+
+echo "$(date +'%0Y-%0m-%0d %0R:%0S'): We have finished the book building process."
+evince book.pdf

scripts/pdflatex.sh

@@ -2,6 +2,7 @@
 
 ## pdflatex Compiler Script
 ## $1 the document to compile
+## $2 == "quick" for incomplete compile
 
 # strict error handling
 set -o pipefail  # trace ERR through pipes
@@ -120,6 +121,7 @@ echo "$(date +'%0Y-%0m-%0d %0R:%0S'): We will now perform runs of the tool chain
 
 watchFileContents=""
 oldWatchFileContents="old"
+quickArg="${2:-}"
 cycle=0
 additional=1
 
@@ -221,7 +223,7 @@ while (("$additional" >= 0))  ; do
       echo "$(date +'%0Y-%0m-%0d %0R:%0S'): '$auxFile' does not exist, so we do not need to apply makeglossaries."
   fi
 
-  echo "$(date +'%0Y-%0m-%0d %0R:%0S'): Now loading the contents should no longer change when the built is complete."
+  echo "$(date +'%0Y-%0m-%0d %0R:%0S'): Now loading the contents that should no longer change when the built is complete."
   for suffix in "acn" "acr" "alg" "bbl" "bcf" "glg" "glo" "gls" "idx" "ind" "ist""slg" "slo" "sls" "toc"; do
     echo "$(date +'%0Y-%0m-%0d %0R:%0S'): Now looking for '$suffix' files."
     for theFile in *.$suffix; do
@@ -235,6 +237,13 @@ while (("$additional" >= 0))  ; do
 
   echo "$(date +'%0Y-%0m-%0d %0R:%0S'): Finished build cycle $cycle."
 
+  if [ "$quickArg" == "quick" ] ; then
+    if (("$cycle" > 1)) ; then
+      echo "$(date +'%0Y-%0m-%0d %0R:%0S'): Quick and incomplete compilation was selected, so we stop after now (cycle $cycle)."
+      break
+    fi
+  fi
+
   if (("$cycle" > 640)) ; then
     echo "$(date +'%0Y-%0m-%0d %0R:%0S'): Something odd is happening: We have performed $cycle cycles. That's too many. Let's quit."
     break
@@ -254,7 +263,7 @@ echo "$(date +'%0Y-%0m-%0d %0R:%0S'): The tool chain has been applied until noth
 latexWarningsCount=0
 latexWarningString=""
 logFile="$documentName.log"
-if [ -f "$logFile" ]; then
+if [ -f "$logFile" ] && [ "$quickArg" != "quick" ]; then
   echo "$(date +'%0Y-%0m-%0d %0R:%0S'): We found the log file '$logFile' and check its contents."
   fileContents="$(<$logFile)"
 

styles/listing.sty

@@ -247,6 +247,16 @@ breaklines=true%
 \gitCode{#1}{#2}{}{#3}{#4}{,style=bat_style}%
 }%
 %
+%%
+%% Use latexgit to place a plain text listing.
+%% #1 the git repository
+%% #2 the local path
+%% #3 the label (lst: will be pre-pended)
+%% #4 the caption
+\protected\gdef\gitText#1#2#3#4{%
+\gitCode{#1}{#2}{}{#3}{#4}{,style=text_style}%
+}%
+%
 %% Use latexgit to place a listing with program output.
 %% #1 the git repository
 %% #2 the local path

text/main/packages/packages.tex

@@ -1,5 +1,5 @@
 %
-\hsection{Packages}%
+\hsection{Using Packages}%
 %
 \begin{figure}%
 \centering%
@@ -15,5 +15,7 @@
 In this section of the book, we will focus on how we can obtain and use packages.%
 %
 \hinput{pipAndVenv}{pipAndVenv.tex}%
+\hinput{requirementsFiles}{requirementsFiles.tex}%
+\hinput{venvAndPycharm}{venvAndPycharm.tex}%
 \endhsection%
 %

text/main/packages/pipAndVenv/pipAndVenv.tex

@@ -36,26 +36,22 @@
 %
 \usefulTool{venv}{The module \venv\ is used for creating and managing \pglspl{virtualEnvironment} under \python.}%
 %
-%
 \gitPython{\programmingWithPythonCodeRepo}{packages/numpy_user.py}{--args format}{packages:numpy_user}{%
 A small \python\ script using the \numpy\ library.}%
 %
 For demonstration purposes, let us assume that we have written a program using the \pgls{package} \numpy~\cite{HMvdWGVCWTBSKPHvKBHFdRWPGMSRWAGO2020APWN,DBvR2024ITN,J2018NPSCADSAWNSAM}.
 The small program in \cref{lst:packages:numpy_user} only creates and prints a \numpy\ array, but for this, obviously, \numpy\ is needed.
 \numpy\ is not available in fresh \python\ installations and needs to be installed so that we can run our program.
-This will be the example that we will use to demonstrate the use of \pglspl{virtualEnvironment} in the following sections.%
-%
-%
-\hsection{pip and Virtual Environments in the Command Line}%
-%
+This will be the example that we will use to demonstrate the use of \pglspl{virtualEnvironment} in the following sections.
+
 As prerequisites to install \pglspl{package} in \pglspl{virtualEnvironment}, we need to make sure that both \pip\ and \venv\ are installed on our system.
 The procedures for both differ under \linux\ and \windows.
 Using \pip\ and \venv\ is often done in the \pgls{terminal}, i.e., by typing commands or executing shell scripts.
 This, naturally, too works differently under \linux\ and \windows.
 We therefore will briefly explore how to achieve these things under both operating systems.%
 %
 \FloatBarrier%
-\hsection{pip and Virtual Environments in \ubuntu\ \linux}%
+\hsection{pip and Virtual Environments in Ubuntu Linux}%
 %
 \begin{figure}%
 \centering%
@@ -206,7 +202,7 @@
 \FloatBarrier%
 \endhsection%
 %
-\hsection{pip and Virtual Environments under \windows}%
+\hsection{pip and Virtual Environments under Windows}%
 %
 \begin{figure}%
 \centering%
@@ -351,8 +347,7 @@
 %
 \endhsection%
 %
-\FloatBarrier%
-\endhsection%
+\FloatBarrier
 %
 \endhsection%
 %

text/main/packages/requirementsFiles/requirementsFiles.tex

@@ -0,0 +1,51 @@
+%
+\hsection{Requirements Files}
+%
+\gitPython{\programmingWithPythonCodeRepo}{packages/requirements.txt}{--args format}{packages:requirementstxt}{%
+A requirements file demanding that version~\textil{1.26.4} of \numpy\ be installed.}%
+%
+\gitBashAndOutput{\programmingWithPythonCodeRepo}{packages}{numpy_user_venv_req.sh}{packages:numpy_user_venv_req_sh}{%
+A script that runs our example program using a \pgls{virtualEnvironment} created by using the \textil{requirements.txt} file given in \cref{lst:packages:requirementstxt}.}%
+%
+It is very normal that our projects require multiple different \pglspl{package}.
+Sometimes, they require specific versions of specific \pglspl{package}.
+Instead of manually installing these dependencies with \pip\ every time we want to use our program, it makes sense to \emph{write them down}.
+Indeed, the dependencies of an application are a very important part of the documentation.%
+%
+\begin{sloppypar}%
+Requirements files offer a very simple format for this purpose.
+They are usually called \textil{requirements.txt} and reside in the root folder of a project.
+As their name implies, they are simple text files.
+Each of their lines lists one \pgls{package} that is required, optionally with version constraints.%
+\end{sloppypar}%
+%
+\Cref{lst:packages:requirementstxt} gives a trivial example of such a file.
+It contains the line \textil{numpy==1.26.4}.
+This means that \numpy\ of exactly the version~1.26.4 is required for our application.
+Had we written \textil{numpy>=1.26.4}, then a larger version of \numpy\ would also have been OK.
+The other operators, \textil{>}, \textil{<}, and~\textil{<=} can be used as well and have their natural corresponding meaning.
+If we had wanted, we could have written \textil{matplotlib} in the second line of our \textil{requirements.txt} file, which would then have meant that, besides \numpy\ of version~\textil{1.26.4}, the package \matplotlib\ is needed as well.
+Writing only \textil{matplotlib} would be interpreted as \inQuotes{find the highest version of \matplotlib\ that is compatible with \numpy\ version~1.26.4.}
+But we only need one package, \numpy, so our file just has a single line.
+
+In \cref{lst:packages:numpy_user_venv_req_sh}, we present a version of the script that we used to set up a virtual environment and run our two-line \numpy\ program under \ubuntu\ \linux~(see \cref{lst:packages:numpy_user_venv_req_sh}.)
+The only difference is that we now do not write \bashil{pip install numpy} but instead write \bashil{pip install -r requirements.txt}.
+This tells \pip\ to install \emph{all} the requirements from the requirements file \textil{requirements.txt}.
+Under \windows, it works exactly the same.
+
+Having a \textil{requirements.txt} file in the root directory of your project is very useful.
+It automatically informs anybody who works with your code or who uses your program which versions of which \pglspl{package} they need.
+Therefore, it makes sense to specify \pgls{package} versions as precisely as possible in \textil{requirements.txt}.%
+%
+\bestPractice{requirementsTxt}{%
+The \pglspl{package} that a \python\ project requires or depends on should be listed in a file called \textil{requirements.txt} in the root folder of the project.}%
+%
+\bestPractice{requirementsTxtExact}{%
+All required \pglspl{package} in a \textil{requirements.txt} file should be specified with the exact version, i.e., in the form of~\textil{package==version}. %
+This ensures that the behavior of the project can be exactly replicated on other machines. %
+It rules out that errors can be caused due to incompatible versions of dependencies, because it allows the users to exactly replicate the set up of the machine on which the project was developed.%
+}%
+%
+\FloatBarrier%
+\endhsection%
+%

text/main/packages/venvAndPycharm/venvAndPycharm.tex

@@ -0,0 +1,125 @@
+%
+\hsection{Virtual Environments in PyCharm}%
+%
+\begin{figure}%
+\centering%
+%
+\subfloat[][%
+We create a new project in \pycharm\ and select a \pgls{virtualEnvironment} as \menu{Custom Environment}. %
+We click~\menu{Create}.%
+\label{fig:venvPycharm1}%
+]{\tightbox{\includegraphics[width=0.48\linewidth]{\currentDir/venvPycharm1}}}%
+%
+\strut\hfill\strut%
+%
+\subfloat[][%
+Since I selected the folder where the other files (\textil{requirements.txt}, \textil{numpy_user.py}, \dots) are already located, \pycharm\ asks whether this is OK. %
+We select \menu{Create from Existing Sources} if asked.%
+\label{fig:venvPycharm2}%
+]{\tightbox{\includegraphics[width=0.48\linewidth]{\currentDir/venvPycharm2}}}%
+%
+\\[10pt]%
+%
+\subfloat[][%
+Let's take a look at \textil{requirements.txt}. %
+We double-click on this file.%
+\label{fig:venvPycharm3}%
+]{\tightbox{\includegraphics[width=0.48\linewidth]{\currentDir/venvPycharm3}}}%
+%
+\strut\hfill\strut%
+%
+\subfloat[][%
+It contains the one line \textil{numpy==1.26.4}, which means that our project requires \numpy\ in version~1.26.4.%
+\label{fig:venvPycharm4}%
+]{\tightbox{\includegraphics[width=0.48\linewidth]{\currentDir/venvPycharm4}}}%
+%
+%
+\caption{Using \textil{requirements.txt} and \pglspl{virtualEnvironment} in \pycharm~(part~1).}%
+\label{fig:venvPycharmA}%
+\end{figure}%
+%
+\begin{figure}%
+\ContinuedFloat%
+\centering%
+%
+\subfloat[][%
+We now click on \textil{numpy_user.py} and get informed that required \pglspl{package} are missing. %
+We can install them into our project's \pgls{virtualEnvironment} by clicking \menu{Install requirement}. %
+Notice:~This question could also have popped up when we opened \textil{requirements.txt}, in which case we would have installed the requirements then.%
+\label{fig:venvPycharm5}%
+]{\tightbox{\includegraphics[width=0.48\linewidth]{\currentDir/venvPycharm5}}}%
+%
+\strut\hfill\strut%
+%
+\subfloat[][%
+We get informed that the required packages were successfully installed.%
+\label{fig:venvPycharm6}%
+]{\tightbox{\includegraphics[width=0.48\linewidth]{\currentDir/venvPycharm6}}}%
+%
+\\[10pt]%
+%
+\subfloat[][%
+We right-click on \textil{numpy_user.py} and click on~\menu{Run \inSQuotes{numpy\_user.py}}.%
+\label{fig:venvPycharm7}%
+]{\tightbox{\includegraphics[width=0.48\linewidth]{\currentDir/venvPycharm7}}}%
+%
+\strut\hfill\strut%
+%
+\subfloat[][%
+Our program is executed without error and produces the expected output.%
+\label{fig:venvPycharm8}%
+]{\tightbox{\includegraphics[width=0.48\linewidth]{\currentDir/venvPycharm8}}}%
+%
+%
+\caption{Using \textil{requirements.txt} and \pglspl{virtualEnvironment} in \pycharm~(part~2).}%
+\label{fig:venvPycharmB}%
+\end{figure}%
+%
+The \pgls{IDE} \pycharm\ also supports using \pglspl{virtualEnvironment} and \textil{requirements.txt} files.
+Let us here demonstrate this on the example program that requires \numpy\ which was given as \cref{lst:packages:numpy_user} and \textil{requirements.txt} file given in \cref{lst:packages:requirementstxt}.
+In \cref{fig:venvPycharmA}, we work through the steps to create and manage a project with a \pgls{virtualEnvironment} in \pycharm\ on their basis.
+
+First, we need to create a new project~(\cref{fig:venvPycharm1}).
+For this purpose, we already copied the files \pglspl{virtualEnvironment} and \textil{requirements.txt} into a folder~(here:~\textil{/tmp/packages}).
+In the \menu{New Project} dialog of \pycharm, we select this folder as \menu{Location:}.
+As \menu{Interpreter type:}, we choose \menu{Custom Environment} and as \menu{Environment:}, we pick \menu{Generate new}.
+As \menu{Type:}, we choose \menu{Virtualenv}.
+The default \menu{Location} is \textil{.venv} inside our project directory and we keep this setting.
+After clicking \menu{Create}, a new project is created.
+
+Well, almost:
+\Cref{fig:venvPycharm2} reminds me that I copied some files into the project folder before creating the project.
+\pycharm\ wants to make sure that this is right and asks me so.
+As answer, I click~\menu{Create from Existing Sources}.
+
+We now are in the normal \pycharm\ project view in~\cref{fig:venvPycharm3}.
+All the files that I placed into the folder are there and also a \textil{.venv} directory has been created.
+Notice that you could as well create an empty project and copy the files there.
+
+We click on \textil{requirements.txt} and it opens in~\cref{fig:venvPycharm4}.
+Indeed, the file prescribes a single requirement, \numpy\ in version~1.26.4.
+
+When we click on the file \textil{numpy_user.py} to open it in~\cref{fig:venvPycharm5}, we notice a yellow bar at the top of the file's contents.
+This bar tells us that the required package \numpy\ is missing.
+It offers us the two choices to either \menu{Install requirement} or to \menu{Ignore requirement}.
+We select the first option and click it.
+Notice that this same yellow bar could also have appeared when we opened \textil{requirements.txt}.
+I am not sure why it appears only now.
+Either way, we accept the choice to install the requirement.
+
+In \cref{fig:venvPycharm6}, we are informed that the installation was successful.
+The small overlay also tells us that \numpy\ was installed in version~1.26.4, as prescribed by the \textil{requirements.txt} file.
+At the time of this writing, \numpy\ is already out at version~2.2.1, which would have been used if we had just installed it without version specification.
+So this confirms that, indeed, our \textil{requirements.txt} is used.
+We also can see that the red underline under \pythonilIdx{numpy} that was present in \cref{fig:venvPycharm5} is now gone in \cref{fig:venvPycharm6}, because now the package and corresponding modules can be loaded without error.
+
+As final check that everything went well we run our program \textil{numpy_user.py} in \cref{fig:venvPycharm7}.
+We right-click the file in three view on the left-hand side of the window.
+A popup menu appears, in which we click on \menu{Run \inSQuotes{numpy\_user.py}}.
+We could just as well have pressed \keys{\ctrl+\shift+F10}.
+Our program is executed and the expected output appears (\cref{fig:venvPycharm8}).
+All is well.%
+%
+\FloatBarrier%
+\endhsection%
+%