Skip to content

Commit 3e5d4ab

Browse files
scottyhqscottyhq
authored
Simplify python environment setup (#310)
* add pyproject.toml * simplify conda environment management * flake8 fix, remove conda-lock references * rebuild all pages without cache * fix codespell * fix for zarr v3 * use matrix for nocache workflow * fix some broken links
1 parent 869be4a commit 3e5d4ab

34 files changed

+13095
-27593
lines changed

‎.binder/environment.yml

-1
This file was deleted.

‎.binder/environment.yml

+42
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
name: default
2+
channels:
3+
- conda-forge
4+
- nodefaults
5+
dependencies:
6+
- jupyter-book >=1.0.3,<2
7+
- pre-commit >=4.1.0,<5
8+
- dask-labextension >=7.0.0,<8
9+
- jupyterlab >=4.3.5,<5
10+
- jupyter_bokeh >=4.0.5,<5
11+
- jupyterlab-myst >=2.4.2,<3
12+
- jupyter-resource-usage >=1.1.1,<2
13+
- cartopy >=0.24.0,<0.25
14+
- cf_xarray >=0.10.0,<0.11
15+
- dask >=2025.2.0,<2026
16+
- datashader >=0.17.0,<0.18
17+
- distributed >=2025.2.0,<2026
18+
- gcsfs >=2025.2.0,<2026
19+
- geoviews-core >=1.14.0,<2
20+
- gsw >=3.6.19,<4
21+
- hvplot >=0.11.2,<0.12
22+
- h5netcdf >=1.5.0,<2
23+
- ipykernel >=6.29.5,<7
24+
- matplotlib-base >=3.10.0,<4
25+
- netcdf4 >=1.7.2,<2
26+
- numpy >=2.1.3,<3
27+
- pint-xarray >=0.4,<0.5
28+
- pydap >=3.5.3,<4
29+
- python-graphviz >=0.20.3,<0.21
30+
- pooch >=1.8.2,<2
31+
- rioxarray >=0.18.2,<0.19
32+
- scipy >=1.15.2,<2
33+
- sphinx-codeautolink >=0.17.0,<0.18
34+
- sphinxcontrib-mermaid >=1.0.0,<2
35+
- sphinx-notfound-page >=1.0.4,<2
36+
- sphinxext-rediraffe >=0.2.7,<0.3
37+
- s3fs >=2025.2.0,<2026
38+
- xarray >=2025.1.2,<2026
39+
- zarr >=3.0.3,<4
40+
- flox >=0.10.0,<0.11
41+
- numbagg >=0.9.0,<0.10
42+
- python >=3.10

‎.devcontainer/Dockerfile

-1
This file was deleted.

‎.devcontainer/devcontainer.json

+1-4
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,5 @@
11
{
2-
"build": {
3-
"dockerfile": "./Dockerfile",
4-
"context": "../conda"
5-
},
2+
"image": "quay.io/pangeo/pangeo-notebook:latest",
63
"postCreateCommand": {
74
"vscode": "mkdir ${containerWorkspaceFolder}/.vscode && cp ${containerWorkspaceFolder}/.devcontainer/tasks.json ${containerWorkspaceFolder}/.vscode/tasks.json"
85
},

‎.devcontainer/scipy2023/devcontainer.json

+1-4
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,5 @@
11
{
2-
"build": {
3-
"dockerfile": "../Dockerfile",
4-
"context": "../../conda"
5-
},
2+
"image": "quay.io/pangeo/pangeo-notebook:2023.07.05",
63
"postCreateCommand": {
74
"jupyterlab": "mkdir /home/jovyan/.jupyter && cp ${containerWorkspaceFolder}/.devcontainer/scipy2023/jupyter_lab_config.py /home/jovyan/.jupyter/jupyter_lab_config.py",
85
"vscode": "mkdir ${containerWorkspaceFolder}/.vscode && cp ${containerWorkspaceFolder}/.devcontainer/scipy2023/tasks.json ${containerWorkspaceFolder}/.vscode/tasks.json"

‎.devcontainer/scipy2023/jupyter_lab_config.py

-12
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,3 @@
1-
# Configuration file for lab.
2-
31
c = get_config() # noqa
4-
5-
## The default URL to redirect to from `/`
6-
# Default: '/lab'
72
c.LabApp.default_url = '/lab/tree/workshops/scipy2023/index.ipynb'
8-
9-
## Set the Access-Control-Allow-Origin header
10-
#
11-
# Use '*' to allow any origin to access your server.
12-
#
13-
# Takes precedence over allow_origin_pat.
14-
# Default: ''
153
c.ServerApp.allow_origin = '*'

‎.devcontainer/scipy2024/devcontainer.json

+1-4
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,5 @@
11
{
2-
"build": {
3-
"dockerfile": "../Dockerfile",
4-
"context": "../../conda"
5-
},
2+
"image": "quay.io/pangeo/pangeo-notebook:2024.07.08",
63
"postCreateCommand": {
74
"jupyterlab": "mkdir /home/jovyan/.jupyter && cp ${containerWorkspaceFolder}/.devcontainer/scipy2024/jupyter_lab_config.py /home/jovyan/.jupyter/jupyter_lab_config.py",
85
"vscode": "mkdir ${containerWorkspaceFolder}/.vscode && cp ${containerWorkspaceFolder}/.devcontainer/scipy2024/tasks.json ${containerWorkspaceFolder}/.vscode/tasks.json"

‎.devcontainer/scipy2024/jupyter_lab_config.py

-12
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,3 @@
1-
# Configuration file for lab.
2-
31
c = get_config() # noqa
4-
5-
## The default URL to redirect to from `/`
6-
# Default: '/lab'
72
c.LabApp.default_url = '/lab/tree/workshops/scipy2024/index.ipynb'
8-
9-
## Set the Access-Control-Allow-Origin header
10-
#
11-
# Use '*' to allow any origin to access your server.
12-
#
13-
# Takes precedence over allow_origin_pat.
14-
# Default: ''
153
c.ServerApp.allow_origin = '*'

‎.gitattributes

+2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# SCM syntax highlighting & preventing 3-way merges
2+
pixi.lock merge=binary linguist-language=YAML linguist-generated=true

‎.github/dependabot.yml

-4
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,6 @@
11
# Regularly update Docker tags and Actions steps
22
version: 2
33
updates:
4-
- package-ecosystem: "docker"
5-
directory: "/.devcontainer"
6-
schedule:
7-
interval: "daily"
84
- package-ecosystem: "github-actions"
95
directory: "/.github"
106
schedule:

‎.github/workflows/main.yaml

+5-8
Original file line numberDiff line numberDiff line change
@@ -30,18 +30,15 @@ jobs:
3030
with:
3131
path: _build
3232
# NOTE: change key to "jupyterbook-DATE" to force rebuilding cache
33-
key: jupyterbook-20240517
33+
key: jupyterbook-20250221
3434

35-
- name: Install Conda environment with Micromamba
36-
uses: mamba-org/setup-micromamba@v1
35+
- uses: prefix-dev/[email protected]
3736
with:
38-
environment-file: conda/conda-lock.yml
39-
environment-name: xarray-tutorial
40-
cache-environment: true
37+
manifest-path: pyproject.toml
38+
cache: true
39+
activate-environment: true
4140

4241
- name: Build JupyterBook
43-
# NOTE: login shell activates conda environment
44-
shell: bash -el {0}
4542
run: |
4643
jupyter-book build ./ --warningiserror --keep-going
4744

‎.github/workflows/nocache.yaml

+37
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
name: Rebuild Entire Jupyter Book on all Platforms
2+
3+
on:
4+
workflow_dispatch:
5+
6+
# Allow one concurrent deployment
7+
concurrency:
8+
group: ${{ github.workflow }}-${{ github.ref }}
9+
cancel-in-progress: true
10+
11+
jobs:
12+
build:
13+
name: Build on ${{ matrix.runs-on }}
14+
runs-on: ${{ matrix.runs-on }}
15+
strategy:
16+
fail-fast: false
17+
matrix:
18+
runs-on: [ubuntu-latest, macos-latest, windows-latest]
19+
20+
steps:
21+
- name: Checkout repository
22+
uses: actions/checkout@v4
23+
24+
- uses: prefix-dev/[email protected]
25+
with:
26+
manifest-path: pyproject.toml
27+
cache: true
28+
activate-environment: true
29+
30+
- name: Build JupyterBook
31+
run: |
32+
jupyter-book build ./ --warningiserror --keep-going
33+
34+
- name: Dump Build Logs
35+
if: always()
36+
run: |
37+
if (test -a _build/html/reports/*log); then cat _build/html/reports/*log ; fi

‎.github/workflows/pull_request.yaml

+5-10
Original file line numberDiff line numberDiff line change
@@ -13,9 +13,6 @@ concurrency:
1313
jobs:
1414
preview:
1515
runs-on: ubuntu-latest
16-
defaults:
17-
run:
18-
shell: bash -el {0}
1916
steps:
2017
- name: Checkout repository
2118
if: github.event.action != 'closed'
@@ -27,15 +24,13 @@ jobs:
2724
with:
2825
path: _build
2926
# NOTE: change key to "jupyterbook-DATE" to force rebuilding cache
30-
key: jupyterbook-20240517
27+
key: jupyterbook-20250221
3128

32-
- name: Install Conda environment with Micromamba
33-
if: github.event.action != 'closed'
34-
uses: mamba-org/setup-micromamba@v1
29+
- uses: prefix-dev/[email protected]
3530
with:
36-
environment-file: conda/conda-lock.yml
37-
environment-name: xarray-tutorial
38-
cache-environment: true
31+
manifest-path: pyproject.toml
32+
cache: true
33+
activate-environment: true
3934

4035
- name: Build JupyterBook
4136
if: github.event.action != 'closed'

‎.github/workflows/qaqc.yaml

+4-8
Original file line numberDiff line numberDiff line change
@@ -14,19 +14,15 @@ concurrency:
1414
jobs:
1515
quality-control:
1616
runs-on: ubuntu-latest
17-
defaults:
18-
run:
19-
shell: bash -el {0}
2017

2118
steps:
2219
- uses: actions/checkout@v4
2320

24-
- name: Install Conda environment with Micromamba
25-
uses: mamba-org/setup-micromamba@v1
21+
- uses: prefix-dev/[email protected]
2622
with:
27-
environment-file: conda/conda-lock.yml
28-
environment-name: xarray-tutorial
29-
cache-environment: true
23+
manifest-path: pyproject.toml
24+
cache: true
25+
activate-environment: true
3026

3127
# NOTE: this isn't a comprehensive spellcheck, just common typos
3228
- name: Spellcheck

‎.gitignore

+4
Original file line numberDiff line numberDiff line change
@@ -118,3 +118,7 @@ venv.bak/
118118
# misc
119119
.DS_Store
120120
.vscode/
121+
122+
# pixi environments
123+
.pixi
124+
*.egg-info

‎.pre-commit-config.yaml

+5
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,11 @@ repos:
1313
- id: check-yaml
1414
- id: double-quote-string-fixer
1515

16+
- repo: https://github.com/codespell-project/codespell
17+
rev: "v2.3.0"
18+
hooks:
19+
- id: codespell
20+
1621
- repo: https://github.com/psf/black
1722
rev: 25.1.0
1823
hooks:

‎.prettierignore

-1
This file was deleted.

‎CONTRIBUTING.md

+45-12
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,9 @@ guidelines:
2222
1. Be concise and limited to one or two topics, such that a reader can get through the example within a few minutes of reading
2323
1. Be of general relevance to Xarray users, and so not too specific on a particular problem or use case.
2424

25-
## Fork this repository
25+
## Contribution process
26+
27+
### Fork this repository
2628

2729
We recommend first forking this repository and creating a local copy:
2830

@@ -31,24 +33,28 @@ git clone https://github.com/YOURACCOUNT/xarray-tutorial.git
3133
cd xarray-tutorial
3234
```
3335

34-
## Create a Python environment
35-
36-
You'll need `conda` or `mamba`, which can be installed from https://github.com/conda-forge/miniforge
36+
### Create a Python environment
3737

38-
We also use [conda-lock](https://conda.github.io/conda-lock/) to ensure we have reproducible environments across different operating systems
38+
You'll need `pixi` or `conda` or `mamba`, which can be installed from https://github.com/conda-forge/miniforge
3939

4040
We also use [pre-commit hooks](https://pre-commit.com) to run styling and other checks before committing code.
4141

42+
#### Using pixi (recommended)
43+
44+
```
45+
pixi install
46+
pixi shell # exit
4247
```
43-
conda-lock install -f conda/conda-lock.yml
44-
# Or latest package versions: `mamba env create -f conda/environment-unpinned.yml`
4548

46-
conda activate xarray-tutorial
49+
#### Using conda
4750

51+
```
52+
mamba env create -f .binder/environment.yml -n xarray-tutorial
53+
conda activate xarray-tutorial # conda deactivate
4854
pre-commit install
4955
```
5056

51-
## Add content
57+
### Add content
5258

5359
Develop your new content on a branch. See [JupyterBook Docs](https://jupyterbook.org/en/stable/intro.html) for guides on adding `.md`, `.ipynb` and other content.
5460

@@ -58,18 +64,45 @@ git add .
5864
git commit -m "added pages x,y and improved z"
5965
```
6066

61-
## Preview your changes
67+
### Preview your changes
6268

6369
Running jupyterbook will execute notebooks and render HTML pages for the website. Be sure to fix any execution errors and preview the website in your web browser to make sure everything looks good!
6470

6571
```
66-
jb build .
72+
jupyter-book build ./ --warningiserror --keep-going
73+
# Or "pixi run build"
6774
```
6875

69-
## Open a pull request
76+
### Open a pull request
7077

7178
```
7279
git push
7380
```
7481

7582
Follow the link reported in a terminal to open a pull request!
83+
84+
## Instructions for environment management
85+
86+
[`pixi`](https://pixi.sh) can be used to create and update a multi-platform lockfile, so a reproducible set of package versions is installed across different operating systems.
87+
88+
Dependencies (with optional pins) are specified in the `pyproject.toml` file, and specific locked versions for all platforms are kept in `pixi.lock`.
89+
90+
Install environment from the lockfile
91+
92+
```
93+
pixi install
94+
pixi shell # activate environment, "exit" to deactivate
95+
```
96+
97+
Upgrade all packages to latest versions:
98+
99+
```
100+
pixi upgrade
101+
```
102+
103+
## Render conda/mamba environment files
104+
105+
```
106+
pixi project export conda-environment -p linux-64 .binder/environment.yml
107+
pixi project export conda-explicit-spec -p linux-64 /tmp
108+
```

‎README.md

+3-10
Original file line numberDiff line numberDiff line change
@@ -16,13 +16,12 @@ All notebooks can be run via the Mybinder.org 'Launch Binder' badge at the top o
1616

1717
#### Github Codespaces
1818

19-
This tutorial is available to run within [Github Codespaces](https://github.com/features/codespaces) - "a development environment that's hosted in the cloud" - with the conda environment specification in the [`conda-lock.yml`](conda/conda-lock.yml) file.
19+
This tutorial is available to run within [Github Codespaces](https://github.com/features/codespaces) - "a development environment that's hosted in the cloud"
2020

2121
[![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/xarray-contrib/xarray-tutorial/tree/main)
2222

2323
☝️ Click the button above to go to options window to launch a Github codespace.
2424

25-
A codespace is a development environment that's hosted in the cloud.
2625
GitHub currently gives every user [120 vCPU hours per month for free](https://docs.github.com/en/billing/managing-billing-for-github-codespaces/about-billing-for-github-codespaces#monthly-included-storage-and-core-hours-for-personal-accounts), beyond that you must pay. **So be sure to explicitly stop or shut down your codespace when you are done by going to this page (https://github.com/codespaces).**
2726

2827
Once your codespace is launched, the following happens:
@@ -34,18 +33,12 @@ Once your codespace is launched, the following happens:
3433

3534
#### Locally
3635

37-
You can also run these notebooks on your own computer! We recommend using [`micromamba`](https://mamba.readthedocs.io/en/latest/installation.html#micromamba) or [`conda-lock`](https://conda-incubator.github.io/conda-lock/) to ensure a fully reproducible Python environment:
36+
You can also run these notebooks on your own computer! We recommend using [`pixi`](https://pixi.sh/latest/#installation) to ensure a fully reproducible Python environment:
3837

3938
```bash
4039
git clone https://github.com/xarray-contrib/xarray-tutorial.git
4140
cd xarray-tutorial
42-
43-
conda-lock install conda/conda-lock.yml --name xarray-tutorial
44-
# Or `micromamba create -n xarray-tutorial -f conda-lock.yml`
45-
# Or latest package versions: `mamba env create -f conda/environment-unpinned.yml`
46-
47-
conda activate xarray-tutorial
48-
jupyter lab
41+
pixi run tutorial
4942
```
5043

5144
## Contributing

0 commit comments

Comments
 (0)