Reading Paired Data
Limitations for Reading Paired Data
Saving paired datasets and reading them back in to create plots and statistics is useful and efficient in MELODIES MONET because pairing the data is the most time-consuming step and typically users like to make several modifications to optimize their plotting and statistical analysis. Currently, these saved paired files have some limitations, so please read this section closely. In MELODIES MONET, prior to pairing, the model data are read in and adjusted based on the YAML file options for each model and then the observation data are read in and adjusted based on the YAML file options for each observation. Then based on the mapping provided in the model portion of the YAML file, the data is paired and saved out to a file. When you read this file back into MELODIES MONET, the code can read in the model and observation data again, so that users can continue to make plots that do not only use the paired data like the spatial overlay plots, but the pairing step is skipped to save time. Therefore, if you adjust anything in the model or obs parts of the YAML file like unit conversions, different number of mapped species, NaN values, etc., these updates will not be adjusted in your paired file. Thus, please make sure to use the exact same model and obs options in your YAML file for plotting as you used for pairing the data except for those specific to plotting as explained below. If you plan to make plots separately for different mapped species, just create several paired files for each mapped species, so that you have extra flexibility.
You can change the following:
The analysis start and end time to update the time window over which to create the plots and calculate statistics.
You can combine multiple paired files across several times periods or flights.
You can also combine multiple models paired on the same observations to evaluate across different models or sensitivity tests using the same model.
You may update anything in the plots and stats section of the YAML input file.
There are two exceptions unique to plotting in the model and obs section of the YAML file that you are able to adjust. In the model section, you can adjust the plot_kwargs section to control line/marker colors and styles and in the obs section, you can adjust the plotting descriptions for each observation variable like the y-axis labels, min/max values, and other plotting specific variables such as ty_scale.
We are working to make the pairing more generalizable in version 2, so more flexibility will be available in later versions. We are also planning to add a feature where you can drop species listed in the mapping table, so that you have more options when reading in this paired file to optimize your analysis plots and statistics.
First let’s just import the driver.
from melodies_monet import driver
Read model and observations
an = driver.analysis()
an.control = "control_wrfchem_saveandread.yaml"
an.read_control()
an.control_dict
an.open_models()
an.open_obs()
wrfchem
example:wrfchem:racm_esrl
**** Reading WRF-Chem model output...
wrfchem
example:wrfchem:racm_esrl_vcp
**** Reading WRF-Chem model output...
Read saved data using control file
The driver will read the data based on the information included in the control file by calling an.read_analysis().
In the control file analysis section, setting method to 'netcdf' for a given attribute of the analysis class (e.g., paired, models, obs) will read NetCDF-4 files and set the appropriate attribute with the data. Filenames must be specified as a dict, with the keys being the pair name and the values being either a string with the filename to be read, or an iterable with multiple filenames to be read. If multiple files (such as several different days) are specified they will be joined by coordinates with xarray’s merge function.
In the control file analysis section, setting method to 'pkl' for a given attribute of the analysis class (e.g., paired, models, obs) will read .pkl files and set the appropriate attribute with the data. Filenames must be specified as either a string or an iterable. If multiple files (such as several different days) are specified, they will be joined by coordinates with xarray’s merge function.
an.read_analysis()
Reading: ./output/save_and_read/0905_airnow_RACM_ESRL.nc4
Reading: ./output/save_and_read/0905_airnow_RACM_ESRL_VCP.nc4
an.paired['airnow_RACM_ESRL'].obj
an.paired['airnow_RACM_ESRL_VCP'].obj
Read data without using control file
Alternatively, the same can be achieved by calling melodies_monet.util.read_util.read_saved_data() directly. The object to set must be an attribute of the instance of the analysis class (e.g., an.paired, an.models, an.obs).
# For netCDF files
from melodies_monet.util.read_util import read_saved_data
read_saved_data(
analysis=an,
filenames={'airnow_wrfchem_v4.2': ['0905_airnow_wrfchem_v4.2.nc4']},
method='netcdf',
attr='paired')
# For pickle files
from melodies_monet.util.read_util import read_saved_data
read_saved_data(analysis=an, filenames=['0905.pkl'], method='pkl', attr='paired')