IMERG Rainfall Data Processing for 2022
This repository contains three Python scripts for processing IMERG rainfall data for the year 2022. The data is processed from daily accumulated rainfall .nc files into monthly accumulated rainfall .nc files. Below is a description of each script, their functionalities, and how they work together. Overview
The three scripts handle different stages of processing:
Code 1: Extracts and processes IMERG daily rainfall data for each day (from March 31 to October 31, 2022), then accumulates the precipitation data for each day and saves it into a .nc file.
Code 2: Concatenates multiple daily accumulated rainfall .nc files into a single monthly accumulated rainfall .nc file.
Code 3: Processes the daily .nc files created in Code 1 and combines them into a monthly accumulated rainfall dataset, saving the final output as new .nc files.
Script Breakdown
- Code 1: Daily Accumulated Rainfall Data Creation (daily_accumulation.py)
This script processes daily rainfall data and accumulates precipitation values. It does so by reading multiple IMERG HDF5 files for each day, summing the precipitation data, and then saving the result into a new .nc file.
Input: IMERG HDF5 .nc4 files located in /home/pc/Desktop/IMERG_2022.
Output: Daily accumulated rainfall .nc files saved in /home/pc/Desktop/IMERG_OP_2022.
Process:
Loops through the defined months (March to October 2022).
For each day, it reads the corresponding HDF5 files.
Accumulates precipitation data for each day.
Saves the accumulated precipitation data into a .nc file.
Log File: log_2022_Daily_Acc_Rain.txt records the processing steps, any errors, and the details of each accumulated file.
- Code 2: Concatenate Daily Files into Monthly Data (monthly_accumulation.py)
This script concatenates the daily accumulated .nc files into a single monthly dataset, aggregating the rainfall values for the entire month.
Input: Daily .nc files generated by Code 1.
Output: Monthly accumulated rainfall .nc files.
Process:
For each month (April to October 2022), the script loads daily .nc files from the directory.
Concatenates the data along the time dimension.
Ensures the time dimension is updated and consistent.
Saves the combined monthly dataset to a new .nc file.
Log File: log_2022_Monthly_accumulated.txt records the steps taken, including errors and successful processing details for each month.
- Code 3: Final Monthly Accumulation (final_monthly_accumulation.py)
This script takes the daily .nc files created in Code 1 and combines them into monthly datasets by concatenating daily rainfall files for each month.
Input: Daily .nc files generated by Code 1 (stored in /home/pc/Desktop/IMERG_OP_2022).
Output: Monthly accumulated .nc files for each month (April to October 2022).
Process:
For each month, it loads all daily files (from the first day to the last).
Concatenates the data along the time dimension.
Renames the time dimension to 'day' and processes the data for each month.
Saves the combined dataset into a new .nc file.
Log File: log_2022_Monthly_accumulated.txt (same as Code 2) logs the process for each month.
Setup and Usage Requirements
Python 3.x
xarray (for working with .nc files)
pandas (for handling date operations)
glob (for finding files in directories)
os (for file and directory management)
cftime (for handling time values in IMERG data)
File Structure
The input files should be organized in the following way:
/home/pc/Desktop/IMERG_2022/ <-- Contains the daily IMERG HDF5 files (.nc4) ├── 3B-HHR.MS.MRG.3IMERG.20220331*.HDF5.nc4 └── ... (other files for each day from March 31 to October 31)
/home/pc/Desktop/IMERG_OP_2022/ <-- Output directory for daily and monthly .nc files
How to Run the Scripts
Run Code 1: Execute the first script to process and accumulate rainfall data for each day:
python daily_accumulation.py
This will generate daily .nc files for each day in 2022 (March to October) and log the output.
Run Code 2: Once the daily .nc files are created, run the second script to concatenate them into monthly accumulated rainfall datasets:
python monthly_accumulation.py
This will process the daily files and combine them into monthly datasets, saving the result as .nc files.
Run Code 3: Finally, run the third script to ensure the final monthly datasets are accumulated properly and saved:
python final_monthly_accumulation.py
This script will create the final monthly .nc files for each month and log the results.
Log Files
Daily Accumulation Log: log_2022_Daily_Acc_Rain.txt
Monthly Accumulation Log: log_2022_Monthly_accumulated.txt
These log files will contain details about the files being processed, any errors encountered, and the resulting output files. Summary
By running these three scripts sequentially, you can process IMERG rainfall data for the year 2022. The data will be:
Daily Accumulated: Each day's precipitation is accumulated and saved as a .nc file.
Monthly Accumulated: The daily files for each month are concatenated into a monthly dataset.
Final Output: The monthly datasets are saved and logged.
These scripts facilitate efficient processing and accumulation of large IMERG datasets, and the log files ensure traceability of all steps taken during processing. Log_2022.txt:
Finalized on July 29.
Contains daily files in CSV format with accumulated rainfall data for the year 2022.
Includes daily files in NC format with 49 timestamps for 2022.
Log_2022_Daily_Acc_Rain.txt:
Details IMERG daily accumulated rainfall files (IMERG_Daily_acc_rainfall_{year}-{month}-{day}.nc) for the entire year 2022.
Provides accumulated rainfall data from 3 GMT to 3 GMT at all grid points.
Includes information on accessing and viewing accumulated precipitation values after processing all time stamps in the log file
Log_2022_Monthly_Accumulated.txt:
Details the conversion of IMERG daily accumulated rainfall files (IMERG_Daily_acc_rainfall_{year}-{month}-{day}.nc) into monthly accumulated rainfall data.
Contains files in the format Daily_3gmt_to_3gmt_accumulated_rainfall_{year}-{month}_{1-dayend}.nc for each month of 2022.
Includes seven monthly NC files, each with daily accumulated rainfall data for all days in the month , accumulated precipitation is viewed in log file.
Daily_3gmt_to_3gmt_accumulated_rainfall_2022-10_1-31.nc