[pull] master from NVIDIA:master

.ansible-lint

@@ -0,0 +1,26 @@
+---
+# Project-level ansible-lint configuration for ansible-lint 26.x
+# Profile levels: min, basic, moderate, safety, shared, production
+profile: min
+
+# Exclude external/vendored roles and hardware-specific roles
+exclude_paths:
+  - roles/galaxy/
+  - roles/nvidia-dgx/
+  - roles/nvidia-dgx-firmware/
+  - roles/nvidia-gpu-tests/
+  - submodules/
+
+# Skip rules for pre-existing issues across the codebase.
+# These should be fixed incrementally in future PRs.
+skip_list:
+  - fqcn[action-core]    # 634 violations: modules not using FQCN
+  - fqcn[action]         # 24 violations: same for non-core
+  - name[casing]         # 526 violations: task names not capitalized
+  - yaml[truthy]         # 152 violations: yes/no instead of true/false
+  - yaml[octal-values]   # 52 violations: octal file modes
+  - jinja[spacing]       # 20 violations: jinja2 spacing
+  - name[play]           # 12 violations: play names
+  - schema[meta]         # 3 violations: meta/main.yml schema
+  - key-order[task]      # 3 violations: task key ordering
+  - ignore-errors        # pre-existing ignore_errors usage

.claude/skills/test-playbooks.md

@@ -0,0 +1,40 @@
+---
+name: test-playbooks
+description: Test Ansible playbooks across Ubuntu versions on target machines
+---
+
+## Prerequisites
+- Virtualenv activated (`source .venv/bin/activate` or `source /opt/deepops/env/bin/activate`)
+- Target machines provisioned and accessible via SSH
+- Inventory configured: either `config/inventory` (static) or `config/maas-inventory.yml` (MAAS dynamic)
+
+## Steps
+1. Verify connectivity: `ansible -m ping all`
+2. Run playbook: `ansible-playbook playbooks/<playbook>.yml`
+3. Verify results (check playbook output, run smoke tests on targets)
+4. To test another OS version: reprovision targets with the new OS, re-run playbook, verify again
+
+## Test Matrix
+| Playbook | Inventory groups needed | Test on 24.04 | Test on 22.04 |
+|----------|------------------------|---------------|---------------|
+| k8s-cluster.yml | kube_control_plane, kube_node, etcd | yes | yes |
+| slurm-cluster.yml | slurm-master, slurm-node | yes | yes |
+| ngc-ready-server.yml | (any host group) | yes | yes |
+
+## MAAS Users
+If using MAAS dynamic inventory (`scripts/maas_inventory.py`), the deploy script automates provisioning:
+```bash
+./scripts/maas_deploy.sh --status                      # check VM state
+./scripts/maas_deploy.sh --os noble --profile k8s      # deploy + tag for K8s
+./scripts/maas_deploy.sh --os jammy --profile slurm    # deploy + tag for Slurm
+./scripts/maas_deploy.sh --profile k8s --tags-only     # re-tag without redeploying
+./scripts/maas_deploy.sh --release                     # release VMs
+```
+Profiles assign MAAS tags that the dynamic inventory maps to Ansible groups:
+- **k8s**: first machine = `kube_control_plane` + `etcd`, remaining = `kube_node`
+- **slurm**: first machine = `slurm-master`, remaining = `slurm-node`
+
+## Group Naming
+- K8s groups use underscores: `kube_control_plane`, `kube_node`, `k8s_cluster`
+- Slurm groups use hyphens: `slurm-master`, `slurm-node`, `slurm-cluster`
+- Old hyphenated K8s names (`kube-master`, `kube-node`) are accepted via TAG_ALIASES

.github/workflows/ansible-lint-roles.yml

@@ -0,0 +1,31 @@
+---
+name: run ansible-lint on deepops roles
+on:
+  - push
+  - pull_request
+jobs:
+  lint:
+    runs-on: ubuntu-22.04
+    steps:
+
+      - name: check out repo
+        uses: actions/checkout@v4
+        with:
+          path: "${{ github.repository }}"
+
+      - name: set up python
+        uses: actions/setup-python@v4
+        with:
+          python-version: "3.10"
+
+      - name: install dependencies
+        run: |
+          python3 -m pip install --upgrade pip
+          python3 -m pip install ansible-lint==26.1.1 ansible==10.7.0
+
+      - name: run lint script
+        env:
+          ANSIBLE_LINT_EXCLUDE: "nvidia-dgx|nvidia-gpu-tests"
+        run: |
+          cd "${{ github.repository }}"
+          bash ./scripts/deepops/ansible-lint-roles.sh

.github/workflows/codeql.yml

@@ -0,0 +1,41 @@
+name: "CodeQL"
+
+on:
+  push:
+    branches: [ "master", "release-20.02", "release-20.06", "release-20.08", "release-20.10", "release-20.11", "release-20.12", "release-21.03", "release-21.05", "release-21.06", "release-21.09", "release-21.12", "release-22.01", "release-22.04" ]
+  pull_request:
+    branches: [ "master" ]
+  schedule:
+    - cron: "38 16 * * 2"
+
+jobs:
+  analyze:
+    name: Analyze
+    runs-on: ubuntu-latest
+    permissions:
+      actions: read
+      contents: read
+      security-events: write
+
+    strategy:
+      fail-fast: false
+      matrix:
+        language: [ python ]
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Initialize CodeQL
+        uses: github/codeql-action/init@v3
+        with:
+          languages: ${{ matrix.language }}
+          queries: +security-and-quality
+
+      - name: Autobuild
+        uses: github/codeql-action/autobuild@v3
+
+      - name: Perform CodeQL Analysis
+        uses: github/codeql-action/analyze@v3
+        with:
+          category: "/language:${{ matrix.language }}"

.github/workflows/molecule.yml

@@ -0,0 +1,50 @@
+---
+name: test ansible roles with molecule
+on:
+  - push
+  - pull_request
+jobs:
+  build:
+    runs-on: ubuntu-24.04
+    strategy:
+      fail-fast: false
+      matrix:
+        deepops-role:
+          - cachefilesd
+          - facts
+          - kerberos_client
+          - lmod
+          - nfs
+          - nhc
+          - nvidia_dcgm
+          - openmpi
+          - openshift
+          - mofed
+          - spack
+          # Excluded from Docker CI (require systemd services that can't
+          # run in containers): nis_client, rsyslog_client, rsyslog_server,
+          # slurm. Tested end-to-end on real MAAS VMs instead.
+          # Also excluded: singularity_wrapper (broken upstream Galaxy dep)
+    steps:
+      - name: check out repo
+        uses: actions/checkout@v4
+        with:
+          path: "${{ github.repository }}"
+      - name: set up python
+        uses: actions/setup-python@v4
+        with:
+          python-version: "3.12"
+      - name: install dependencies
+        run: |
+          python3 -m pip install --upgrade pip
+          python3 -m pip install ansible==10.7.0 passlib
+          python3 -m pip install molecule molecule-plugins[docker] docker
+      - name: run molecule test
+        env:
+          ANSIBLE_ROLES_PATH: "${{ github.workspace }}/${{ github.repository }}/roles/galaxy:${{ github.workspace }}/${{ github.repository }}/roles"
+        run: |
+          cd "${{ github.repository }}/roles"
+          ansible-galaxy role install --force -r ./requirements.yml
+          ansible-galaxy collection install --force -r ./requirements.yml
+          cd "${{ matrix.deepops-role }}"
+          molecule test

.github/workflows/setup.yml

@@ -0,0 +1,25 @@
+---
+name: test setup.sh
+on:
+  - push
+  - pull_request
+jobs:
+  setup:
+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        os:
+          - ubuntu-22.04
+          - ubuntu-24.04
+    steps:
+      - name: check out repo
+        uses: actions/checkout@v4
+
+      - name: run setup.sh
+        run: bash scripts/setup.sh
+
+      - name: verify ansible in venv
+        run: |
+          source /opt/deepops/env/bin/activate
+          ansible --version
+          python3 -c "from packaging.version import Version; print('packaging OK')"

.github/workflows/stale.yml

@@ -0,0 +1,31 @@
+# This workflow warns and then closes issues and PRs that have had no activity for a specified amount of time.
+#
+# You can adjust the behavior by modifying this file.
+# For more information, see:
+# https://github.com/actions/stale
+name: Mark stale issues and pull requests
+
+on:
+  schedule:
+  - cron: '40 0 * * *'
+
+jobs:
+  stale:
+
+    runs-on: ubuntu-latest
+    permissions:
+      issues: write
+      pull-requests: write
+
+    steps:
+    - uses: actions/stale@v9
+      with:
+        repo-token: ${{ secrets.GITHUB_TOKEN }}
+        stale-issue-message: 'This issue is stale because it has been open for 60 days with no activity. Please update the issue or it will be closed in 7 days.'
+        stale-pr-message: 'This PR is stale because it has been open for 180 days with no activity. Please update the PR or it will be closed in 7 days.'
+        stale-issue-label: 'no-issue-activity'
+        stale-pr-label: 'no-pr-activity'
+        days-before-issue-stale: 60
+        days-before-pr-stale: 180
+        exempt-pr-labels: 'no-stale'
+        exempt-issue-labels: 'no-stale'

.gitignore

@@ -4,6 +4,13 @@
 # misc.
 .*.swp
 
+# virtualenv
+/.venv/
+
+# claude code
+/CLAUDE.md
+/tasks/
+
 # project-specific
 /admin.conf
 /config*/

.gitmodules

@@ -1,6 +1,6 @@
 [submodule "kubespray"]
 	path = submodules/kubespray
-	url = https://github.com/kubernetes-incubator/kubespray.git
+	url = https://github.com/kubernetes-sigs/kubespray.git
 [submodule "packer-maas"]
 	path = submodules/packer-maas
 	url = https://github.com/DeepOps/packer-maas.git

README.md

@@ -1,69 +1,92 @@
-DeepOps
-===
-
-GPU infrastructure and automation tools
+# DeepOps
+
+Infrastructure automation tools for Kubernetes and Slurm clusters with NVIDIA GPUs.
+
+## Table of Contents
+
+- [DeepOps](#deepops)
+  - [Table of Contents](#table-of-contents)
+  - [Overview](#overview)
+  - [Deployment Requirements](#deployment-requirements)
+    - [Provisioning System](#provisioning-system)
+    - [Cluster System](#cluster-system)
+    - [Kubernetes](#kubernetes)
+    - [Slurm](#slurm)
+    - [Hybrid clusters](#hybrid-clusters)
+    - [Virtual](#virtual)
+  - [Updating DeepOps](#updating-deepops)
+  - [Copyright and License](#copyright-and-license)
+  - [Issues](#issues)
+  - [Contributing](#contributing)
 
 ## Overview
 
-The DeepOps project encapsulates best practices in the deployment of GPU server clusters and sharing single powerful nodes (such as [NVIDIA DGX Systems](https://www.nvidia.com/en-us/data-center/dgx-systems/)). DeepOps can also be adapted or used in a modular fashion to match site-specific cluster needs. For example:
+The DeepOps project encapsulates best practices in the deployment of GPU server clusters and sharing single powerful nodes (such as [NVIDIA DGX Systems](https://www.nvidia.com/en-us/data-center/dgx-systems/)). DeepOps may also be adapted or used in a modular fashion to match site-specific cluster needs. For example:
 
-* An on-prem data center of NVIDIA DGX servers where DeepOps provides end-to-end capabilities to set up the entire cluster management stack
-* An existing cluster running Kubernetes where DeepOps scripts are used to deploy Kubeflow and connect NFS storage
-* An existing cluster that needs a resource manager / batch scheduler, where DeepOps is used to install Slurm, Kubernetes, or a hybrid of both
-* A single machine where no scheduler is desired, only NVIDIA drivers, Docker, and the NVIDIA Container Runtime
+- An on-prem data center of NVIDIA DGX servers where DeepOps provides end-to-end capabilities to set up the entire cluster management stack
+- An existing cluster running Kubernetes where DeepOps scripts are used to deploy KubeFlow and connect NFS storage
+- An existing cluster that needs a resource manager / batch scheduler, where DeepOps is used to install Slurm or Kubernetes
+- A single machine where no scheduler is desired, only NVIDIA drivers, Docker, and the NVIDIA Container Runtime
 
-Check out the [video tutorial](https://drive.google.com/file/d/1RNLQYlgJqE8JMv0np8SdEDqeCN2piavF/view) for how to use DeepOps to deploy Kubernetes and Kubeflow on a single DGX Station. This provides a good base test ground for larger deployments.
+Latest release: [DeepOps 26.05 Release](https://github.com/NVIDIA/deepops/releases/tag/26.05)
 
-## Releases
+It is recommended to use the latest release branch for stable code (linked above). All development takes place on the master branch, which is generally [functional](docs/deepops/testing.md) but may change significantly between releases.
 
-Latest release: [DeepOps 21.09 Release](https://github.com/NVIDIA/deepops/releases/tag/21.09)
+## Deployment Requirements
 
-It is recommended to use the latest release branch for stable code (linked above). All development takes place on the master branch, which is generally [functional](docs/deepops/testing.md) but may change significantly between releases.
+### Provisioning System
 
-## Getting Started
+The provisioning system is used to orchestrate the running of all playbooks and one will be needed when instantiating Kubernetes or Slurm clusters. Current release validation focuses on:
 
-For detailed help or guidance, read through our [Getting Started Guide](docs/) or pick one of the deployment options documented below.
+- Ubuntu 22.04 LTS and 24.04 LTS
+- NVIDIA DGX OS 6 and 7
 
-## Deployment Options
+DeepOps still retains legacy/community-maintained paths for older environments such as DGX OS 4/5, Ubuntu 18.04/20.04, and CentOS 7/8. Treat those paths as compatibility references unless your site validates them for the release you deploy.
 
-### Supported Ansible versions
+### Cluster System
 
-DeepOps supports using Ansible 2.9.x.
-Ansible 2.10.x and newer are not currently supported.
+The cluster nodes will follow the requirements described by Slurm or Kubernetes. You may also use a cluster node as a provisioning system but it is not required. Current release validation focuses on:
 
-### Supported distributions
+- Ubuntu 22.04 LTS and 24.04 LTS for generic Kubernetes and Slurm deployments
+- NVIDIA DGX OS 6 and 7 for DGX systems
+- Red Hat Enterprise Linux / Rocky Linux 8 and 9 for DGX platform software installation through the `nvidia-dgx` role
 
-DeepOps currently supports the following Linux distributions:
+DeepOps still retains legacy/community-maintained paths for older environments such as DGX OS 4/5, Ubuntu 18.04/20.04, CentOS 7/8, and the historical DGX EL7 stack. Treat those paths as compatibility references unless your site validates them for the release you deploy.
 
-* NVIDIA DGX OS 4, 5
-* Ubuntu 18.04 LTS, 20.04 LTS
-* CentOS 7, 8
+You may also install a supported operating system on all servers via a 3rd-party solution such as [MAAS](https://maas.io/) or [Foreman](https://www.theforeman.org/), or via an existing site-standard automated installer.
+For new Ubuntu 24.04 or DGX OS 7 deployments, prefer Ubuntu autoinstall/cloud-init or MAAS and then apply DeepOps roles after the OS is present.
+For DGX platform software installation on top of vanilla Ubuntu or Red Hat family operating systems, see the [DGX software stack role guide](docs/deepops/dgx-software-stack.md).
 
 ### Kubernetes
 
-Kubernetes (K8s) is an open-source system for automating deployment, scaling, and management of containerized applications.
+Kubernetes (K8s) is an open-source system for automating deployment, scaling, and management of containerized applications. The instantiation of a Kubernetes cluster is done by [Kubespray](submodules/kubespray). Kubespray runs on bare metal and most clouds, using Ansible as its substrate for provisioning and orchestration. For people with familiarity with Ansible, existing Ansible deployments or the desire to run a Kubernetes cluster across multiple platforms, Kubespray is a good choice. Kubespray does generic configuration management tasks from the "OS operators" ansible world, plus some initial K8s clustering (with networking plugins included) and control plane bootstrapping. DeepOps provides additional playbooks for orchestration and optimization of GPU environments.
 
 Consult the [DeepOps Kubernetes Deployment Guide](docs/k8s-cluster/) for instructions on building a GPU-enabled Kubernetes cluster using DeepOps.
 
 For more information on Kubernetes in general, refer to the [official Kubernetes docs](https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/).
 
 ### Slurm
 
-Slurm is an open source, fault-tolerant, and highly scalable cluster management and job scheduling system for large and small Linux clusters.
+Slurm is an open-source cluster resource management and job scheduling system that strives to be simple, scalable, portable, fault-tolerant, and interconnect agnostic. Slurm currently has been tested only under Linux.
+
+As a cluster resource manager, Slurm provides three key functions. First, it allocates exclusive and/or non-exclusive access to resources (compute nodes) to users for some duration of time so they can perform work. Second, it provides a framework for starting, executing, and monitoring work (normally a parallel job) on the set of allocated nodes. Finally, it arbitrates conflicting requests for resources by managing a queue of pending work. Slurm cluster instantiation is achieved through [SchedMD](https://slurm.schedmd.com/download.html)
 
 Consult the [DeepOps Slurm Deployment Guide](docs/slurm-cluster/) for instructions on building a GPU-enabled Slurm cluster using DeepOps.
 
 For more information on Slurm in general, refer to the [official Slurm docs](https://slurm.schedmd.com/overview.html).
 
-### DGX POD Hybrid Cluster
+### Hybrid clusters
+
+**DeepOps does not test or support a configuration where both Kubernetes and Slurm are deployed on the same physical cluster.**
 
-A hybrid cluster with both Kubernetes and Slurm can also be deployed. This is recommended for [DGX POD](https://www.nvidia.com/en-us/data-center/dgx-pod-reference-architecture/) and other setups that wish to make maximal use of the cluster.
+[NVIDIA Bright Cluster Manager](https://www.brightcomputing.com/brightclustermanager) is recommended as an enterprise solution which enables managing multiple workload managers within a single cluster, including Kubernetes, Slurm, Univa Grid Engine, and PBS Pro.
 
-Consult the [DeepOps DGX POD Deployment Guide](docs/deepops/dgx-pod.md) for step-by-step instructions on building a GPU-enabled hybrid cluster using DeepOps.
+**DeepOps does not test or support a configuration where nodes have a heterogenous OS running.**
+The `nvidia-dgx` role can install NVIDIA DGX platform software on supported DGX systems running Red Hat Enterprise Linux / Rocky Linux 8 or 9; broader Kubernetes or Slurm cluster support on RHEL still requires site-specific validation.
 
 ### Virtual
 
-To try DeepOps before deploying it on an actual cluster, a virtualized version of DeepOps may be deployed on a single node using Vagrant. This can be used for testing, adding new features, or configuring DeepOps to meet deployment-specific needs.
+To try DeepOps before deploying it on an actual cluster, a virtualized version of DeepOps may be deployed on a single node using Vagrant. This path is useful for learning and local experimentation, but it is a legacy/community-supported lab path and should not be treated as release-grade validation for current GPU clusters.
 
 Consult the [Virtual DeepOps Deployment Guide](virtual/README.md) to build a GPU-enabled virtual cluster with DeepOps.