Fair Comparison of Pangu-Weather Model with Operational IFS Model

[EloyAnguiano]

Description:

I've been reviewing the Pangu-Weather paper and have a query regarding the fairness of the comparison methodology used. Here's my concern:

When comparing against the Operational IFS model, I believe the temporal gaps between each model might not be equitable. Let me elaborate:

Let's consider that the Operational IFS model intends to predict weather conditions for 2024-01-01 01:00, and at the current time, it's 2024-01-01 00:00, resulting in a 1-hour gap. The Operational IFS model can indeed make this prediction with the available data at 2024-01-01 00:00, hence, the temporal gap for prediction is 1 hour.

However, in the case of Pangu, as it relies on ERA5 data as input, it cannot access the ERA5 data at the initial hour since ERA5 data is published with a 5-day latency, so it would have to use the data from 2023-12-26 00:00 as its most recent input (source: CDS - Reanalysis ERA5):

ERA5 is updated daily with a latency of about 5 days. In case that serious flaws are detected in this early release (called ERA5T), this data could be different from the final release 2 to 3 months later. In case that this occurs users are notified.

Therefore, for a fair comparison in operational terms, and as I understand from the paper, the performance of Pangu should be evaluated at 5 days +1 hour forecasts, as opposed to the +1 hour data available for the Operational IFS model.

Is this assumption correct, or am I misunderstanding the comparison regarding operational perspectives?

[EloyAnguiano]

It seems to me that the way of performing those comparatives for being fare at the usability of this output in real time would be to establish the IFS h0 (for example, the prediction for the 2024-01-01 00:00 made at time 2024-01-01 00:00) as the initial conditions for Pangu. Thus, the Pangu h3 would be comparable to the IFS h3 and so on to see if that is really usefull at real time.

[0rhisia0]

Pangu relies on the ERA5 reanlysis data, which is generated via HRES initial fields. Thus, the comparison is made assuming that access to these initial field access is available at generation time. In technicality you could pay for the initial fields/or generate them yourself and run them off them. That's what ECMWF is doing here (https://www.ecmwf.int/en/forecasts/dataset/machine-learning-model-data), so it makes sense that the models are compared as is without dealing with the 5 day CCDS ERA5 lag