Changes for el9-based cmssw versions

hackathon_instructions.md

@@ -4,7 +4,7 @@
 First login to glados, and then run the following commands. 
 The general directory structure is: 
 ```
-lobster-python3
+lobster-python3-run3
     lobster
 ``` 
 If you'd like to use a different setup, just adjust the paths accordingly.
@@ -14,42 +14,41 @@ unset PYTHONPATH
 mkdir lobster-python3
 cd lobster-python3
 
-git clone https://github.com/NDCMS/lobster.git
+git clone https://github.com/anpicci/lobster.git
 cd lobster
-git checkout lobster-python3
+git checkout lobster-python3-run3
 
 conda env create -f lobster_env.yaml -n lobster
 conda activate lobster
 
-cd ..  # back to lobster-python3/
+cd ..  # back to lobster-python3-run3/
 
 git clone [email protected]:dmwm/WMCore.git --branch 2.3.5
 cd WMCore
 sed -i -E '/^(gfal2|htcondor|kkmysqlclient|rucio-clients|Sphinx|coverage|memory-profiler|mox3|nose|nose2|pycodestyle|pylint|pymongo)/d' requirements.txt
 pip install -r requirements.txt .
 
-cd ../lobster   # to lobster-python3/lobster
+cd ../lobster   # to lobster-python3-run3/lobster
 ```
 
-**Note:** The yaml file and the above command creates an environment named `lobster`,  
-if you'd like to change the name of the environment, change the value after the `-n` flag  
-e.g. `conda env create -f lobster_env.yaml -n lobster-python3`
+**Note:** The yaml file and the above command create an environment named `lobster`. If you'd like to change the name of the environment, change the value after the `-n` flag, e.g., `conda env create -f lobster_env.yaml -n lobster-python3-run3`
 
 
 Then, still in the cloned lobster directory, run the following command to install lobster as an editable package: 
 ```
 pip install -e .
 ```
 
-Now that the `lobster` env is setup, in the future all you need to do is run the following: 
+Now that the `lobster` env is set up, in the future all you need to do is run the following: 
 ```
 unset PYTHONPATH
 unset PERL5LIB
 conda activate lobster
+export PATH=/afs/crc.nd.edu/group/ccl/software/x86_64/RedHat9/cctools/7.11.1/bin:$PATH ##needed to use parrot_run when sandboxing CMSSW
 ```
 
 # Running a Simple Config
-In the lobster repository, there is a python script called "simple.py". This has been updated to work with `lobster-python3` and can be run in the following way: 
+In the lobster repository, there is a Python script called "simple.py". This has been updated to work with `lobster-python3` and can be run in the following way: 
 
 1. Set up the necessary CMSSW release in the same directory as where you're running the config file (see directions below).
 2. unset the pythonpath and start the `lobster` environment 
@@ -68,20 +67,20 @@ After the jobs are completed, check the output. In general, lobster output is st
 
 # Setting up a CMSSW environment for the simple example
 For the simple.py script, we're using CMSSW_10_6_26. There are two options: 
-1. install CMSSW_10_6_26 inside the same directory where simple.py is located 
+1. Install CMSSW_10_6_26 inside the same directory where simple.py is located 
     - `lobster/examples/`
-    - inside the examples directory run `unset PERL5LIB` and `cmsrel CMSSW_10_6_26` (NOTE: this has to be done outside of lobster conda environment)
-    - reminder: DO NO do cmsenv
+    - inside the examples directory, run `unset PERL5LIB` and `cmsrel CMSSW_10_6_26` (NOTE: this has to be done outside of lobster conda environment)
+    - reminder: DO NOT do cmsenv
 2. install CMSSW_10_6_26 somewhere else, and edit the path in simple.py on line 45: `release='<your-path-to-CMSSW_10_6_26>'`
 
 # Possible Errors
-After submitting a lobster process and starting a work_queue_factory, if there are no errors in `process.err` or `process_debug.log` but workers are never assigned to the job, try the follwing: 
+After submitting a lobster process and starting a work_queue_factory, if there are no errors in `process.err` or `process_debug.log` but workers are never assigned to the job, try the following: 
 - Kill the current work_queue_factory. 
 - Try running a worker directly with the following command:
 - `apptainer exec --bind /cvmfs:/cvmfs --bind $CONDA_PREFIX:/conda_env /afs/crc.nd.edu/group/ccl/software/runos/images/cc7-wq-7.11.1.img /conda_env/bin/work_queue_worker -M "lobster_${USER}.*" -dall --cores 1 --disk 10000 -t 150`
-- If the manual worker succeeds and not the work_queue_factory then it may indicate a bug in lobster, or your environment.  Save your logs and contact the development team for help.
+- If the manual worker succeeds and not the work_queue_factory, then it may indicate a bug in lobster or your environment.  Save your logs and contact the development team for help.
 
 # Other Troubleshooting
-- Did you remember to re-new your proxy?
+- Did you remember to renew your proxy?
 - Unset PYTHONPATH and PERL5LIB?
-- In the right conda env?  Run `work_queue_factory --version` and check the output for 7.11.1.
+- In the right conda env?  Run `work_queue_factory --version` and check the output for 7.11.1.

lobster/cmssw/sandbox.py

@@ -56,7 +56,7 @@ def __dontpack(self, fn):
         return False
 
     def _recycle(self, outdir):
-        release_and_arch = re.compile(r'sandbox-(.*)-(slc.*)-[A-Fa-f0-9]*.tar.bz2$')
+        release_and_arch = re.compile(r'sandbox-(.*?)-(slc\d+_\S+|el\d+_\S+)-[A-Fa-f0-9]+\.tar\.bz2$')
         shutil.copy2(self.recycle, outdir)
         m = release_and_arch.search(self.recycle)
         if not m:
@@ -65,9 +65,14 @@ def _recycle(self, outdir):
         return rtname, rtarch, os.path.join(outdir, os.path.split(self.recycle)[-1])
 
     def _get_cmssw_arch(self, dirname):
+        # First, search for 'slc*'
         candidates = glob.glob('{}/.SCRAM/slc*'.format(dirname))
+        # If no 'slc*' is found, fallback to searching for 'el*'
+        if not candidates:
+            candidates = glob.glob('{}/.SCRAM/el*'.format(dirname))
+
         if len(candidates) != 1:
-            raise AttributeError("Can't determine SCRAM arch!")
+            raise AttributeError("Can't determine SCRAM arch! in {0}".format(dirname))
         return os.path.basename(candidates[0])
 
     def _get_cmssw_version(self, dirname):
@@ -102,7 +107,7 @@ def ignore_file(fn):
         tarball = tarfile.open(outfile, "w|bz2")
 
         # package bin, etc
-        subdirs = ['bin', 'cfipython', 'external', 'lib', 'python']
+        subdirs = ['bin', 'cfipython', 'external', 'lib', 'python', 'biglib']
         subdirs += [os.path.join('src', incl) for incl in self.include]
 
         for (path, dirs, files) in os.walk(os.path.join(indir, 'src')):

lobster/commands/plot.py

@@ -139,19 +139,26 @@ def mp_pie(vals, labels, name, plotdir=None, **kwargs):
         ax.set_prop_cycle(monochrome)
 
     newlabels = []
-    total = sum(vals)  # total == 0 causes division error
-    if total != 0:
+    total = sum(vals)
+    if total > 0:
         for label, val in zip(labels, vals):
             if float(val) / total < .025:
                 newlabels.append('')
             else:
                 newlabels.append(label)
+    else:
+        newlabels = [''] * len(labels)
 
     with open(os.path.join(plotdir, name + '.dat'), 'w') as f:
         for l, v in zip(labels, vals):
             f.write('{0}\t{1}\n'.format(l, v))
 
-    patches, texts = ax.pie([max(0, val) for val in vals], labels=newlabels, **kwargs)
+    if total > 0:
+        patches, texts = ax.pie([max(0, val) for val in vals], labels=newlabels, **kwargs)
+    else:
+        patches, texts = [], []
+        ax.text(0.5, 0.5, 'No data', ha='center', va='center')
+        ax.set_axis_off()
 
     if paper:
         for p, h in zip(patches, itertools.cycle(hatching)):
@@ -287,9 +294,24 @@ def mp_plot(a, xlabel, stub=None, ylabel='tasks', bins=50, modes=None, ymax=None
                 if '/' not in ylabel:
                     ax.set_ylabel('{} / {:.0f} min'.format(ylabel,
                                                            (bins[1] - bins[0]) * 24 * 60.))
-            elif float('inf') not in a:  # error passing inf to hist
-                ax.hist([y for (x, y) in a], bins=bins,
-                        histtype='stepfilled', stacked=True, **kwargs)
+            else:
+                cleaned = []
+                has_data = False
+                for _, y in a:
+                    arr = np.asarray(y)
+                    finite = arr[np.isfinite(arr)]
+                    if len(finite) > 0:
+                        has_data = True
+                        cleaned.append(finite)
+                    else:
+                        cleaned.append(np.array([]))
+
+                if has_data:
+                    ax.hist(cleaned, bins=bins,
+                            histtype='stepfilled', stacked=True, **kwargs)
+                else:
+                    ax.text(0.5, 0.5, 'No finite data', ha='center', va='center')
+                    ax.set_axis_off()
         elif mode & Plotter.PROF:
             filename += '-prof'
             data['data'] = []

lobster/commands/process.py

@@ -17,6 +17,7 @@
 from lobster.core.source import TaskProvider
 
 import work_queue as wq
+print('\n\n\nUsing WQ version:', wq.__version__, '\n\n\n')
 
 logger = logging.getLogger('lobster.core')
 
@@ -136,7 +137,7 @@ def localkill(num, frame):
         process = psutil.Process()
         preserved = [f.name for f in args.preserve]
         preserved += [os.path.realpath(os.path.abspath(f)) for f in preserved]
-        openfiles = [f for f in process.open_files() if f.path not in preserved]
+        openfiles = [f for f in process.open_files() if f.path not in preserved and "vscode" not in f.path]
         openconns = process.connections()
 
         for c in openconns:
@@ -147,7 +148,7 @@ def localkill(num, frame):
             logger.error("cannot daemonize due to open files")
             for f in openfiles:
                 logger.error("open file: {}".format(f.path))
-            raise RuntimeError("open files or connections")
+            raise RuntimeError(f"open files or connections {f.path}")
 
         with daemon.DaemonContext(
                 detach_process=not args.foreground,

lobster/core/data/autosense.sh

@@ -6,21 +6,90 @@ shift
 shift
 
 if [[ -z "$release" || -z "$pset" ]]; then
-	echo "usage: autosense.sh release pset args..."
-	exit 1
+  echo "usage: autosense.sh release pset args..."
+  exit 1
 fi
 
 # Make sure we don't have any leftover files to confuse things.
 if [[ -e out.json ]]; then
-	rm -f out.json
+  rm -f out.json
 fi
 
-source /cvmfs/cms.cern.ch/cmsset_default.sh 
+source /cvmfs/cms.cern.ch/cmsset_default.sh
 cd "$release"
 eval $(scramv1 runtime -sh)
 cd - > /dev/null
 
-python <<EOF > /dev/null 2>&1
+# Decide whether to use python or python3 based on the resolved executable path:
+# - if 3rd-to-last element starts with "slc" -> python
+# - if it starts with "el"  -> python3
+# Preference order:
+#   1) Try python (if present); if tag says "slc*" -> use python
+#   2) Else try python3 (if present); if tag says "el*" -> use python3
+#   3) Else fall back to python if present, else python3, else error
+
+resolve_path() {
+  local p="$1"
+  if command -v realpath >/dev/null 2>&1; then
+    realpath "$p"
+  elif command -v readlink >/dev/null 2>&1; then
+    readlink -f "$p" 2>/dev/null || echo "$p"
+  else
+    echo "$p"
+  fi
+}
+
+third_to_last_component() {
+  local p="$1"
+  p="${p%/}"
+  p="${p#/}"              # drop leading "/" so array indexing is clean
+  local IFS='/'
+  read -r -a parts <<< "$p"
+  local n="${#parts[@]}"
+  if (( n >= 3 )); then
+    echo "${parts[n-3]}"
+  else
+    echo ""
+  fi
+}
+
+_py=""
+_py_path=""
+tag=""
+
+# 1) Try python first
+_py_path="$(type -P python 2>/dev/null || true)"
+if [[ -n "$_py_path" ]]; then
+  tag="$(third_to_last_component "$(resolve_path "$_py_path")")"
+  if [[ "$tag" == slc* ]]; then
+    _py="python"
+  fi
+fi
+
+# 2) Then try python3
+if [[ -z "$_py" ]]; then
+  _py_path="$(type -P python3 2>/dev/null || true)"
+  if [[ -n "$_py_path" ]]; then
+    tag="$(third_to_last_component "$(resolve_path "$_py_path")")"
+    if [[ "$tag" == el* ]]; then
+      _py="python3"
+    fi
+  fi
+fi
+
+# 3) Final fallback (python first, then python3), else error
+if [[ -z "$_py" ]]; then
+  if type -P python >/dev/null 2>&1; then
+    _py="python"
+  elif type -P python3 >/dev/null 2>&1; then
+    _py="python3"
+  else
+    echo "error: neither python nor python3 found in PATH" >&2
+    exit 2
+  fi
+fi
+
+"$_py" <<EOF > /dev/null 2>&1
 import imp
 import json
 import shlex
@@ -49,6 +118,4 @@ cat out.json
 # Put an EOL at the end of all this
 echo ""
 
-rm -f out.json
-
-
+rm -f out.json