ERA5 processing

ERA5 processing

Required files are:

• stat_{s}.txt
  • checked for percentage missing
  • checked no more than 2 variables all missing
• stat30_{s}_nocorr.txt
  • must exist but not used
• stat30_{s}.txt
  • must exist but not used

I think the following is what happens:

• met data is extracted from the ERAI netCDF file and written as yearly CSV files
  • these are the {s}nocorr.csv files.
• the ERAI _nocorr files are regressed against the tower data
  • only 1 stats file so must be all years at once
  • the slope and intercept values in the stat_{s}.txt and stat30_{s}_nocorr.txt files are very similar
  • the slope and intercept values in the stat30_{s}.txt file are close to 1 and 0 respectively
  • this suggests the stat_{s}.txt and stat30_{s}_nocorr.txt files are ERAI vs tower regression values
  • the stat30_{s}.txt file is a check of fitted ERAI vs tower data
• the fitted ERAI data is written to CSV file
  • these are the {s}_.csv files

This is partially confirmed by:

• plotting the corrected vs uncorrected Ta_era gives a straight line, no scatter
• regressing corrected vs uncorrected Ta_era gives slope=0.985 and intercept=-2.018 compared to slope=0.99 and intercept=-2.02 in the stat_US-ARc.txt file

So, I think the way to do the ERA5 processing for ONEFlux is as follows:

1. Read the tower CSV files and concatenate into a single data frame.
2. Read the ERA5 data.
3. Trim the tower and ERA5 data so they cover the same time period.
4. Regress the ERA5 data against the tower data.
   1. Ta, Pa, VPD, WS, Precip, Rg, LWin
5. Write the stat_{s}.txt file.
   1. Var,Unit,Percentage of Gaps,Slope,Intercept,RMSEbc,RMSEac,Corr,Mean,Std,MAEbc,MAEac
   2. Ta, Pa, VPD, WS, Precip, Rg, LWin, LWin_calc
6. Apply the fit to the ERA5 data and write out yearly, corrected ERA5 data as CSV files.