Merge pull request #68 from PyAutoLabs/claude/howtolens-tutorial-restructure-f19y9d

Restructure the lecture series: point sources + formalism lectures, chaining into chapter 2, pixelizations as chapter 3, new Scaling Up Lensing chapter 4

Author: Jammy2211

AGENTS.md

@@ -8,10 +8,14 @@ canonical, agent-agnostic instructions for this repo.
 ## Repository Structure
 
 - `scripts/` — Runnable Python tutorial scripts:
-  - `chapter_1_introduction/` — Grids, profiles, galaxies, ray-tracing, data, fitting
-  - `chapter_2_lens_modeling/` — Non-linear searches, Bayesian inference, lens modeling
-  - `chapter_3_search_chaining/` — Search chaining, prior passing, automated pipelines
-  - `chapter_4_pixelizations/` — Pixelized source reconstruction, inversions, regularization
+  - `chapter_1_introduction/` — Grids, profiles, galaxies, ray-tracing, point sources, the lensing
+    formalism, data, fitting
+  - `chapter_2_lens_modeling/` — Non-linear searches, Bayesian inference, lens modeling, search
+    chaining, prior passing, automated pipelines
+  - `chapter_3_pixelizations/` — Pixelized source reconstruction, inversions, regularization, the
+    Bayesian formalism
+  - `chapter_4_scaling_up_lensing/` — Extra galaxies, multi-galaxy lenses, scaling relations, group
+    and cluster scales, weak lensing
   - `chapter_optional/` — Alternative searches and advanced topics
   - `simulator/` — Simulator scripts that generate the tutorial datasets at runtime
 - `notebooks/` — Jupyter versions, generated from `scripts/` (do not edit directly)

README.md

@@ -21,13 +21,14 @@ they are concise and assume the concepts taught in **HowToLens** as background.
 ## Chapters
 
 - `chapter_1_introduction` — An introduction to strong gravitational lensing and **PyAutoLens**: grids, light
-  and mass profiles, galaxies, ray-tracing, simulated imaging data, and fitting.
+  and mass profiles, galaxies, ray-tracing, point sources, the lensing formalism, simulated imaging data,
+  and fitting.
 - `chapter_2_lens_modeling` — Bayesian inference, non-linear searches, and how to fit a lens model to CCD
-  imaging data with **PyAutoLens**.
-- `chapter_3_search_chaining` — Chaining multiple non-linear searches together to build automated lens
-  modeling pipelines for complex systems.
-- `chapter_4_pixelizations` — Pixelized source reconstructions (inversions) for sources with irregular
-  morphologies.
+  imaging data with **PyAutoLens**, ending with search chaining and automated pipelines.
+- `chapter_3_pixelizations` — Pixelized source reconstructions (inversions) for sources with irregular
+  morphologies, including the Bayesian formalism underpinning them.
+- `chapter_4_scaling_up_lensing` — Scaling lens modeling up beyond a single lens galaxy: extra galaxies,
+  multi-galaxy lenses, scaling relations, group and cluster scales, and weak lensing.
 - `chapter_optional` — Optional tutorials on alternative non-linear searches and other advanced topics.
 
 **HowToLens** currently sits at four chapters. Each chapter will take around a day to work through.

config/build/no_run.yaml

@@ -25,10 +25,3 @@
 #   permanent skips — every mega-run surfaces them with a loud warning
 #   banner. Investigate the failure, fix the underlying bug, and remove
 #   the NEEDS_FIX marker.
-
-- tutorial_5_borders # NEEDS_FIX 2026-08-04 - cap-induced, not a mask problem:
-  # fails only under PYAUTO_SMALL_DATASETS (IndexError: index 371 out of bounds
-  # for axis 0 with size 272) and passes on the same dataset files without the
-  # cap. Hardcoded pix/border indices outrun the capped mesh. Fix by decoupling
-  # the indices (or the mesh shape) from dataset resolution, as HowToGalaxy
-  # chapter_4 tutorial_3 did, then remove this entry.

llms-full.txt

@@ -26,8 +26,8 @@ analysis?" below).
 
 - **Chapter 1 — Introduction** → [scripts/chapter_1_introduction/](scripts/chapter_1_introduction/): Lensing from first principles: grids & galaxies, ray tracing, simulating data, and fitting data. The conceptual foundation everything else builds on.
 - **Chapter 2 — Lens Modeling** → [scripts/chapter_2_lens_modeling/](scripts/chapter_2_lens_modeling/): How to fit a lens model to data with a non-linear search — parameter spaces, priors, dealing with failure, linear light profiles, masking, and reading results.
-- **Chapter 3 — Search Chaining** → [scripts/chapter_3_search_chaining/](scripts/chapter_3_search_chaining/): Breaking a hard fit into a sequence of searches (prior passing), multi-galaxy lenses, complex sources, and an introduction to the SLaM pipelines.
-- **Chapter 4 — Pixelizations** → [scripts/chapter_4_pixelizations/](scripts/chapter_4_pixelizations/): Reconstructing the source on a pixel-grid instead of with light profiles — mappers, inversions, Bayesian regularization, borders, and adaptive meshes.
+- **Chapter 3 — Pixelizations** → [scripts/chapter_3_pixelizations/](scripts/chapter_3_pixelizations/): Reconstructing the source on a pixel-grid instead of with light profiles — mappers, inversions, Bayesian regularization, the Bayesian formalism, borders, and adaptive meshes.
+- **Chapter 4 — Scaling Up Lensing** → [scripts/chapter_4_scaling_up_lensing/](scripts/chapter_4_scaling_up_lensing/): Beyond a single lens galaxy — extra galaxies, multi-galaxy lenses, scaling relations, group and cluster scales, and weak lensing.
 
 ## I want to understand…
 
@@ -37,8 +37,9 @@ analysis?" below).
 - **How does a non-linear search find a model?** → [scripts/chapter_2_lens_modeling/tutorial_1_non_linear_search.py](scripts/chapter_2_lens_modeling/tutorial_1_non_linear_search.py): Parameter space, priors, nested sampling with Nautilus.
 - **My fit found a wrong (local-maxima) solution — what do I do?** → [scripts/chapter_2_lens_modeling/tutorial_4_dealing_with_failure.py](scripts/chapter_2_lens_modeling/tutorial_4_dealing_with_failure.py): Prior tuning, simplifying the model, and other escapes from local maxima.
 - **What are linear light profiles / MGE?** → [scripts/chapter_2_lens_modeling/tutorial_5_linear_profiles.py](scripts/chapter_2_lens_modeling/tutorial_5_linear_profiles.py): Solving intensities linearly; the Multi-Gaussian Expansion basis.
-- **How do I chain searches / what is SLaM?** → [scripts/chapter_3_search_chaining/tutorial_1_search_chaining.py](scripts/chapter_3_search_chaining/tutorial_1_search_chaining.py) then [scripts/chapter_3_search_chaining/tutorial_6_slam.py](scripts/chapter_3_search_chaining/tutorial_6_slam.py).
-- **How do I reconstruct a complex source on a pixel grid?** → [scripts/chapter_4_pixelizations/tutorial_1_pixelizations.py](scripts/chapter_4_pixelizations/tutorial_1_pixelizations.py) through [tutorial_4_bayesian_regularization.py](scripts/chapter_4_pixelizations/tutorial_4_bayesian_regularization.py).
+- **How do I chain searches / what is SLaM?** → [scripts/chapter_2_lens_modeling/tutorial_9_search_chaining.py](scripts/chapter_2_lens_modeling/tutorial_9_search_chaining.py) then [scripts/chapter_2_lens_modeling/tutorial_11_slam.py](scripts/chapter_2_lens_modeling/tutorial_11_slam.py).
+- **How do I reconstruct a complex source on a pixel grid?** → [scripts/chapter_3_pixelizations/tutorial_1_pixelizations.py](scripts/chapter_3_pixelizations/tutorial_1_pixelizations.py) through [tutorial_4_bayesian_regularization.py](scripts/chapter_3_pixelizations/tutorial_4_bayesian_regularization.py).
+- **How do I model lenses with many galaxies, groups, clusters or weak lensing?** → [scripts/chapter_4_scaling_up_lensing/tutorial_1_extra_galaxies.py](scripts/chapter_4_scaling_up_lensing/tutorial_1_extra_galaxies.py) through [tutorial_6_weak_lensing.py](scripts/chapter_4_scaling_up_lensing/tutorial_6_weak_lensing.py).
 
 ## Ready for real analysis?
 

llms.txt

@@ -5,11 +5,11 @@ Every page below is the corresponding example script **fully executed**, rendere
 - [Tutorial 0: Visualization](chapter_1_introduction/tutorial_0_visualization.md) — from `scripts/chapter_1_introduction/tutorial_0_visualization.py`
 - [HowToLens: Introduction](chapter_1_introduction/tutorial_1_grids_and_galaxies.md) — from `scripts/chapter_1_introduction/tutorial_1_grids_and_galaxies.py`
 - [Tutorial 2: Ray Tracing](chapter_1_introduction/tutorial_2_ray_tracing.md) — from `scripts/chapter_1_introduction/tutorial_2_ray_tracing.py`
-- [Tutorial 5: More Ray Tracing](chapter_1_introduction/tutorial_3_more_ray_tracing.md) — from `scripts/chapter_1_introduction/tutorial_3_more_ray_tracing.py`
+- [Tutorial 3: More Ray Tracing](chapter_1_introduction/tutorial_3_more_ray_tracing.md) — from `scripts/chapter_1_introduction/tutorial_3_more_ray_tracing.py`
 - [Tutorial 4: Point Sources](chapter_1_introduction/tutorial_4_point_sources.md) — from `scripts/chapter_1_introduction/tutorial_4_point_sources.py`
 - [Tutorial 5: Lensing Formalism](chapter_1_introduction/tutorial_5_lensing_formalism.md) — from `scripts/chapter_1_introduction/tutorial_5_lensing_formalism.py`
 - [Tutorial 6: Data](chapter_1_introduction/tutorial_6_data.md) — from `scripts/chapter_1_introduction/tutorial_6_data.py`
 - [Tutorial 7: Fitting](chapter_1_introduction/tutorial_7_fitting.md) — from `scripts/chapter_1_introduction/tutorial_7_fitting.py`
-- [Tutorial 9: Summary](chapter_1_introduction/tutorial_8_summary.md) — from `scripts/chapter_1_introduction/tutorial_8_summary.py`
+- [Tutorial 8: Summary](chapter_1_introduction/tutorial_8_summary.md) — from `scripts/chapter_1_introduction/tutorial_8_summary.py`
 
 These pages are regenerated manually by PyAutoHands's `generate_markdown.py` when a curated script changes.

markdown/README.md

@@ -10,8 +10,9 @@ clearly in your Jupyter notebook and on your computer screen.
 
 __Contents__
 
-- **Directories:** **PyAutoLens assumes** the working directory is `autolens_workspace` on your hard-disk.
+- **Directories:** **HowToLens** assumes the working directory is the `HowToLens` repository root on your hard-disk.
 - **Dataset:** Load and plot the strong lens dataset.
+- **Dataset Auto-Simulation:** Create the dataset via its simulator script if it is not on your hard-disk.
 - **Subplots:** In addition to plotting individual figures, **PyAutoLens** can plot `subplots` which show multiple.
 - **Plot Customization:** Does the figure display correctly on your computer screen?
 - **Overlays:** Overlays such as critical curves and image positions are added using the `lines=` and `positions=`.
@@ -25,27 +26,50 @@ from autolens import jax_wrapper  # Sets JAX environment before other imports
 from autolens import setup_notebook; setup_notebook()
 ```
 
+    .../PyAutoNerves/autonerves/workspace.py:206: UserWarning: Cannot verify the workspace at HowToLens/scripts/chapter_1_introduction is compatible with the installed library version (2026.7.23.1): no `version.minimum_library_version` or `version.workspace_version` key in config/general.yaml and no version.txt at the workspace root.
+    
+    If you cloned the workspace from `main` rather than a release tag, set `version.workspace_version_check: False` in config/general.yaml to silence this warning. The `main` branch updates more frequently than library releases, so version mismatches are expected and not actionable for `main`-branch users.
+    
+    You can also set the environment variable PYAUTO_SKIP_WORKSPACE_VERSION_CHECK=1 to disable temporarily.
+      warnings.warn(_missing_version_warning(root, library_version))
+    .../PyAutoNerves/autonerves/workspace.py:206: UserWarning: Cannot verify the workspace at HowToLens/scripts/chapter_1_introduction is compatible with the installed library version (2026.7.23.1): no `version.minimum_library_version` or `version.workspace_version` key in config/general.yaml and no version.txt at the workspace root.
+    
+    If you cloned the workspace from `main` rather than a release tag, set `version.workspace_version_check: False` in config/general.yaml to silence this warning. The `main` branch updates more frequently than library releases, so version mismatches are expected and not actionable for `main`-branch users.
+    
+    You can also set the environment variable PYAUTO_SKIP_WORKSPACE_VERSION_CHECK=1 to disable temporarily.
+      warnings.warn(_missing_version_warning(root, library_version))
     Working Directory has been set to `HowToLens`
+    .../PyAutoNerves/autonerves/workspace.py:206: UserWarning: Cannot verify the workspace at HowToLens/scripts/chapter_1_introduction is compatible with the installed library version (2026.7.23.1): no `version.minimum_library_version` or `version.workspace_version` key in config/general.yaml and no version.txt at the workspace root.
+    
+    If you cloned the workspace from `main` rather than a release tag, set `version.workspace_version_check: False` in config/general.yaml to silence this warning. The `main` branch updates more frequently than library releases, so version mismatches are expected and not actionable for `main`-branch users.
+    
+    You can also set the environment variable PYAUTO_SKIP_WORKSPACE_VERSION_CHECK=1 to disable temporarily.
+      warnings.warn(_missing_version_warning(root, library_version))
+
 
+__Directories__
 
-If the printed working directory does not match the workspace path on your computer, you can manually set it
-as follows (the example below shows the path I would use on my laptop. The code is commented out so you do not
-use this path in this tutorial!
+**HowToLens** assumes the working directory is the `HowToLens` repository root on your hard-disk, so that relative
+paths to `dataset/` and `output/` resolve correctly.
+
+If your working directory does not match this path on your computer, you can manually set it as follows (the
+example below shows the path I would use on my laptop. The code is commented out so you do not use this path in
+this tutorial!
 
 
 ```python
-# workspace_path = "/Users/Jammy/Code/PyAuto/autolens_workspace"
+# workspace_path = "/Users/Jammy/Code/PyAuto/HowToLens"
 # #%cd $workspace_path
 # print(f"Working Directory has been set to `{workspace_path}`")
 ```
 
 __Dataset__
 
 The `dataset_path` specifies where the dataset is located, which is the
-directory `autolens_workspace/dataset/imaging/simple__no_lens_light`.
+directory `dataset/imaging/simple__no_lens_light` of the HowToLens repository.
 
-There are many example simulated images of strong lenses in this directory that will be used throughout the
-**HowToLens** lectures.
+The simulated images of strong lenses used throughout the **HowToLens** lectures are written to the `dataset`
+directory at runtime by the simulator scripts in `scripts/simulator/`.
 
 
 ```python
@@ -57,6 +81,31 @@ import autolens.plot as aplt
 dataset_path = Path("dataset") / "imaging" / "simple__no_lens_light"
 ```
 
+__Dataset Auto-Simulation__
+
+If the dataset does not already exist on your system, it is created by running the corresponding
+simulator script. This ensures every example script can be run without manually simulating data first.
+
+
+```python
+if al.util.dataset.should_simulate(str(dataset_path)):
+    import subprocess
+    import sys
+
+    subprocess.run(
+        [sys.executable, "scripts/simulator/no_lens_light.py"],
+        check=True,
+    )
+```
+
+    Figure(700x700)
+    .../PyAutoArray/autoarray/operators/convolver.py:1424: UserWarning: No blurring_image provided. Only the direct image will be convolved. This may change the correctness of the PSF convolution.
+      warnings.warn(
+    Figure(1800x1800)
+    Figure(1800x1800)
+    Figure(700x700)
+
+
 We now load this dataset from .fits files and create an instance of an `Imaging` object.
 
 
@@ -78,7 +127,7 @@ aplt.plot_array(array=dataset.data, title="Dataset Image")
 
 
 
-![png](tutorial_0_visualization_files/tutorial_0_visualization_9_0.png)
+![png](tutorial_0_visualization_files/tutorial_0_visualization_11_0.png)
 
 
 
@@ -96,7 +145,7 @@ aplt.subplot_imaging_dataset(dataset=dataset)
 
 
 
-![png](tutorial_0_visualization_files/tutorial_0_visualization_11_0.png)
+![png](tutorial_0_visualization_files/tutorial_0_visualization_13_0.png)
 
 
 
@@ -106,7 +155,7 @@ Does the figure display correctly on your computer screen?
 
 If not, the default matplotlib settings can be customized via the config files in:
 
-  autolens_workspace/config/visualize/
+  config/visualize/
 
 Key config entries:
 
@@ -126,7 +175,7 @@ aplt.plot_array(array=dataset.data, title="Dataset Image (Log10)", use_log10=Tru
 
 
 
-![png](tutorial_0_visualization_files/tutorial_0_visualization_13_0.png)
+![png](tutorial_0_visualization_files/tutorial_0_visualization_15_0.png)
 
 
 
@@ -168,7 +217,7 @@ aplt.plot_array(
 
 
 
-![png](tutorial_0_visualization_files/tutorial_0_visualization_15_0.png)
+![png](tutorial_0_visualization_files/tutorial_0_visualization_17_0.png)
 
 
 
@@ -186,8 +235,3 @@ The key plotting functions you'll use are:
 
 Great! Hopefully, visualization in **PyAutoLens** is displaying nicely for us to get on with the
 **HowToLens** lecture series.
-
-
-```python
-
-```