Milestones
List view
- make runnable on external compute - release - share model weights (huggingface) - prepare datasets to share - scripts/notebooks for running and finetuning
Due by November 8, 2026•0/5 issues closed- inode issues: JSC Meteo Cloud (zarr3), in scratch folders (squashfs? monitoring?) - HPC outages: ensure people can continue developing - More generally: overview of common failure modes and explore backup options.
No due date•0/11 issues closed- How can the code be restructured such that core packages remain stable while enabling a large degree of flexibility for research? - How do we maintain a clean code base while keeping the time to merge fast? - What criteria do we set up to provide clear expectations with regard to code development and merges? - (How) do we organize config directories? Keep a central repo for it or just give guidelines on how people set it up? Which configs to keep in the main repo? What guarantees for those configs? Potential issue: people's own configs with non-recommended args (e.g.: writing uncompressed zarr output consumes many inodes). Ideas: - Part of linting: checking that a function/variable/class is not unused, - Code structure monitoring: How many lines in what packages/subdirectories? Short term goal: how do we merge the work that is currently spread across many branches and forks? Long term goal: how do we keep projects using the same core functionality while building additional features according to their needs?
No due date•4/12 issues closed- Share people's knowledge how to use tools, share setups and skill files. - Plugins for shared project skills, e.g. documentation. Plugins allow opt-in and development. - Strategy for dealing with the increased code volume.
No due dateSuccess criteria: - Set up new MLflow instance or find alternative that implements shared logging. - Automatic near-real-time logging. - Iterate on interface to access the most relevant metrics/plots/evaluation artifacts. - It is widely used.
No due date•0/3 issues closed- No due date•1/1 issues closed
In contrast to model parameters, we focus on data reading/processing/writing.
No due date•1/9 issues closedMemory and throughput optimization of forward and backward passes.
No due date•0/9 issues closedCovers issues related to the impact of scaling on efficiency, both in terms of computational performance and model accuracy.
No due date•0/5 issues closed- No due date•0/8 issues closed
Investigate the bottlenecks for training high-resolution datasets (h512 georing + n320 ERA inputs, n320 output, 2 forecasting steps, see [hedgedoc](https://gitlab.jsc.fz-juelich.de/hedgedoc/9GkwO6AeSROsVP3btJqnGQ) for details) with higher parameter counts, e.g. increasing the depth and width in the global aggregation engine depth compared to the preprint model (ae_global_dim_embed, ae_global_num_blocks) or the local aggregation (ae_local_dim_embed, ae_local_num_blocks). Success criteria: - Identify bottlenecks, - Implement solutions to reach the target of doubling depth and width in the global aggregation engine compared to the preprint model. - Merge solutions.
No due date•0/1 issues closedSuccess criteria: - We can train on h512 georing + o1280 operational analysis inputs, n320 output, 2 forecasting steps (for detailed configuration see https://gitlab.jsc.fz-juelich.de/hedgedoc/1z7jFyTHRGeSUheS0lFynw). - Efficiency: with h512+n320, 2-step forecasting configuration, spatial parallelism = 4, 32 nodes: aim for 0.3 optimizer steps per second. - All changes required to do this have been merged to the develop branch.
Overdue by 18 day(s)•Due by October 15, 2026•2/5 issues closed2026Q3 milestone for the engineering team
No due date•6/11 issues closed- Overdue by 5 month(s)•Due by May 31, 2026
Downscaling ranks among the most critical downstream tasks for a foundation model. This milestone seeks to develop rapid diagnostic metrics and tools for downscaling use cases, incorporating them into a fast‑evaluation package.
Overdue by 8 month(s)•Due by February 28, 2026•10/13 issues closedBeta release of WG-atmo: - all the critical workflows for experimenting have been implemented in the alpha release - focus on stabilization and scaling
Overdue by 6 month(s)•Due by May 1, 2026•26/27 issues closedSteps to implement and test the diffusion model are discussed in the [DesignDoc](https://ecmwf.sharepoint.com/:w:/r/sites/WeatherGenerator-Project/_layouts/15/Doc.aspx?sourcedoc=%7Bb6f5b024-d4b5-4fda-b0cd-ca849e09132b%7D&action=edit&wdPid=3e7a9f1c&wdPreviousSession=50864e37-4085-161d-a5ab-0ac94c456274&wdPreviousSessionSrc=Reload)
Overdue by 10 month(s)•Due by December 17, 2025•14/21 issues closed- Overdue by 10 month(s)•Due by December 17, 2025•12/15 issues closed