AirNow and CAM-chem

First we need to import the driver.

from melodies_monet import driver
Please install h5py to open files from the Amazon S3 servers.
Please install h5netcdf to open files from the Amazon S3 servers.
# Needed if you want to make changes to `melodies_monet` and don't want to restart kernel:
%load_ext autoreload

%autoreload 2

Initiate the analysis class

Now lets create an instance of the melodies_monet.driver analysis class. It consists of 4 main parts: model instances, observation instances, a paired instance of both. This will allow us to move things around the plotting function for spatial and overlays and more complex plots.

an = driver.analysis()
an
analysis(
    control='control.yaml',
    control_dict=None,
    models={},
    obs={},
    paired={},
    start_time=None,
    end_time=None,
    download_maps=True,
    output_dir=None,
    debug=False,
)

Control File

Read in the required yaml control file that sets up all the definitions of what we want to pair and plot.

an.control = 'control_camchem.yaml'
an.read_control()
an.control_dict
Hide code cell output
{'analysis': {'start_time': '2019-09-01-00:00:00',
  'end_time': '2019-09-09-00:00:00',
  'output_dir': './output/camchem',
  'download_maps': False,
  'debug': False},
 'model': {'cam-chem': {'files': 'example:camchem:fv',
   'mod_type': 'cesm_fv',
   'radius_of_influence': 150000.0,
   'mapping': {'airnow': {'O3': 'OZONE'}},
   'projection': 'None',
   'plot_kwargs': {'color': 'dodgerblue', 'marker': '+', 'linestyle': '-.'}}},
 'obs': {'airnow': {'use_airnow': True,
   'filename': 'example:airnow:2019-09',
   'obs_type': 'pt_sfc',
   'variables': {'PM2.5': {'unit_scale': 1,
     'unit_scale_method': '*',
     'nan_value': -1.0,
     'ylabel_plot': 'PM2.5 (ug/m3)',
     'ty_scale': 2.0,
     'vmin_plot': 0.0,
     'vmax_plot': 22.0,
     'vdiff_plot': 15.0,
     'nlevels_plot': 23},
    'OZONE': {'unit_scale': 1,
     'unit_scale_method': '*',
     'nan_value': -1.0,
     'ylabel_plot': 'Ozone (ppbv)',
     'vmin_plot': 15.0,
     'vmax_plot': 55.0,
     'vdiff_plot': 20.0,
     'nlevels_plot': 21}}}},
 'plots': {'plot_grp1': {'type': 'timeseries',
   'fig_kwargs': {'figsize': [12, 6]},
   'default_plot_kwargs': {'linewidth': 2.0, 'markersize': 10.0},
   'text_kwargs': {'fontsize': 18.0},
   'domain_type': ['all', 'epa_region'],
   'domain_name': ['CONUS', 'R1'],
   'data': ['airnow_cam-chem'],
   'data_proc': {'rem_obs_nan': True,
    'ts_select_time': 'time',
    'ts_avg_window': 'H',
    'set_axis': False}},
  'plot_grp2': {'type': 'taylor',
   'fig_kwargs': {'figsize': [8, 8]},
   'default_plot_kwargs': {'linewidth': 2.0, 'markersize': 10.0},
   'text_kwargs': {'fontsize': 16.0},
   'domain_type': ['all', 'epa_region'],
   'domain_name': ['CONUS', 'R1'],
   'data': ['airnow_cam-chem'],
   'data_proc': {'rem_obs_nan': True, 'set_axis': True}},
  'plot_grp3': {'type': 'spatial_bias',
   'fig_kwargs': {'states': True, 'figsize': [10, 5]},
   'text_kwargs': {'fontsize': 16.0},
   'domain_type': ['all', 'epa_region'],
   'domain_name': ['CONUS', 'R1'],
   'data': ['airnow_cam-chem'],
   'data_proc': {'rem_obs_nan': True, 'set_axis': True}},
  'plot_grp4': {'type': 'spatial_overlay',
   'fig_kwargs': {'states': True, 'figsize': [10, 5]},
   'text_kwargs': {'fontsize': 16.0},
   'domain_type': ['all', 'epa_region'],
   'domain_name': ['CONUS', 'R1'],
   'data': ['airnow_cam-chem'],
   'data_proc': {'rem_obs_nan': True, 'set_axis': True}},
  'plot_grp5': {'type': 'boxplot',
   'fig_kwargs': {'figsize': [8, 6]},
   'text_kwargs': {'fontsize': 20.0},
   'domain_type': ['all', 'epa_region'],
   'domain_name': ['CONUS', 'R1'],
   'data': ['airnow_cam-chem'],
   'data_proc': {'rem_obs_nan': True, 'set_axis': False}}}}

Load the model data

The driver will automatically loop through the “models” found in the model section of the control file and create model classes for each. Classes include the label, mapping information, and xarray object as well as the filenames. Note it can open multiple files easily by including wildcards. Here we are only opening one CAM-chem file.

an.open_models()
Downloading data from 'https://csl.noaa.gov/groups/csl4/modeldata/melodies-monet/data/example_model_data/cesmfv_example/CAM_chem_merra2_FCSD_1deg_QFED_world_201909-01-09_small_sfc.nc' to file 'C:\Users\zmoon\AppData\Local\pooch\pooch\Cache\f57b114fb465b23ee998209b50df7d9d-CAM_chem_merra2_FCSD_1deg_QFED_world_201909-01-09_small_sfc.nc'.
cam-chem
{'files': 'example:camchem:fv', 'mod_type': 'cesm_fv', 'radius_of_influence': 150000.0, 'mapping': {'airnow': {'O3': 'OZONE'}}, 'projection': 'None', 'plot_kwargs': {'color': 'dodgerblue', 'marker': '+', 'linestyle': '-.'}}
example:camchem:fv
**** Reading CESM model output...
an.models
{'cam-chem': model(
     model='cesm_fv',
     radius_of_influence=150000.0,
     mod_kwargs={},
     file_str='example:camchem:fv',
     label='cam-chem',
     obj=...,
     mapping={'airnow': {'O3': 'OZONE'}},
     label='cam-chem',
     ...
 )}
an.models['cam-chem'].obj
<xarray.Dataset>
Dimensions:  (time: 36, z: 1, lat: 192, lon: 288)
Coordinates:
  * lat      (lat) float64 -90.0 -89.06 -88.12 -87.17 ... 87.17 88.12 89.06 90.0
  * z        (z) float64 992.5
  * lon      (lon) float64 0.0 1.25 2.5 3.75 5.0 ... 355.0 356.2 357.5 358.8
  * time     (time) datetime64[ns] 2019-09-01T06:00:00 ... 2019-09-10
Data variables:
    O3       (time, z, lat, lon) float32 dask.array<chunksize=(36, 1, 192, 288), meta=np.ndarray>
    PM25     (time, z, lat, lon) float32 dask.array<chunksize=(36, 1, 192, 288), meta=np.ndarray>
Attributes:
    Conventions:       CF-1.0
    source:            CAM
    case:              fmerra.2.1003.FCSD.f09.qfedcmip.56L.001.branch02
    logname:           buchholz
    host:              cheyenne3
    initial_file:      /glade/p/cesmdata/cseg/inputdata/atm/cam/inic/fv/f.e20...
    topography_file:   /glade/p/cesmdata/cseg/inputdata/atm/cam/met/MERRA2/0....
    model_doi_url:     https://doi.org/10.5065/D67H1H0V
    time_period_freq:  hour_6
    history:           Mon Feb 28 16:25:23 2022: ncks -7 -L 1 --baa=4 --ppc d...
    NCO:               netCDF Operators version 5.0.6 (Homepage = http://nco....
# All the info in the model class can be called here.
print(an.models['cam-chem'].label)
print(an.models['cam-chem'].mapping)
cam-chem
{'airnow': {'O3': 'OZONE'}}
# All the info in the analysis class can also be called.
print(an.start_time)
print(an.end_time)
print(an.download_maps)
2019-09-01 00:00:00
2019-09-09 00:00:00
True

Open Obs

Now for monet-analysis we will open preprocessed data in either netcdf icartt or some other format. We will not be retrieving data like monetio does for some observations (ie aeronet, airnow, etc….). Instead we will provide utitilies to do this so that users can add more data easily.

Like models we list all obs objects in the yaml file and it will loop through and create driver.observation instances that include the model type, file, objects (i.e. data object) and label

an.control_dict['obs']
{'airnow': {'use_airnow': True,
  'filename': 'example:airnow:2019-09',
  'obs_type': 'pt_sfc',
  'variables': {'PM2.5': {'unit_scale': 1,
    'unit_scale_method': '*',
    'nan_value': -1.0,
    'ylabel_plot': 'PM2.5 (ug/m3)',
    'ty_scale': 2.0,
    'vmin_plot': 0.0,
    'vmax_plot': 22.0,
    'vdiff_plot': 15.0,
    'nlevels_plot': 23},
   'OZONE': {'unit_scale': 1,
    'unit_scale_method': '*',
    'nan_value': -1.0,
    'ylabel_plot': 'Ozone (ppbv)',
    'vmin_plot': 15.0,
    'vmax_plot': 55.0,
    'vdiff_plot': 20.0,
    'nlevels_plot': 21}}}}
an.open_obs()
# All the info in the observation class can also be called.
an.obs['airnow'].obj
<xarray.Dataset>
Dimensions:     (x: 3786, time: 2091, y: 1)
Coordinates:
  * x           (x) int64 0 1 2 3 4 5 6 7 ... 3779 3780 3781 3782 3783 3784 3785
  * time        (time) datetime64[ns] 2019-09-01 ... 2019-09-30T00:30:00
    latitude    (y, x) float64 47.65 47.51 49.02 53.3 ... 36.92 47.93 41.37
    longitude   (y, x) float64 -52.82 -52.79 -55.8 -60.36 ... -94.84 106.9 69.27
    siteid      (y, x) object '000010102' '000010401' ... 'UZB010001'
Dimensions without coordinates: y
Data variables: (12/30)
    BARPR       (time, y, x) float64 ...
    BC          (time, y, x) float64 ...
    CO          (time, y, x) float64 ...
    NO          (time, y, x) float64 ...
    NO2         (time, y, x) float64 ...
    NO2Y        (time, y, x) float64 ...
    ...          ...
    cmsa_name   (y, x) float64 -1.0 -1.0 -1.0 -1.0 -1.0 ... -1.0 -1.0 -1.0 -1.0
    msa_code    (y, x) float64 -1.0 -1.0 -1.0 -1.0 ... -1.0 3.306e+04 -1.0 -1.0
    msa_name    (y, x) object '' '' '' '' '' '' ... '' '' '' ' Miami, OK ' '' ''
    state_name  (y, x) object 'CC' 'CC' 'CC' 'CC' 'CC' 'CC' ... '' '' '' '' ''
    epa_region  (y, x) object 'CA' 'CA' 'CA' 'CA' 'CA' ... '' 'R6' 'DSMG' 'DSUZ'
    time_local  (time, y, x) datetime64[ns] ...
Attributes:
    title:         
    format:        NetCDF-4
    date_created:  2021-06-07

Pair model and obs data

%%time

an.pair_data()
Hide code cell output
[########################################] | 100% Completed |  0.1s
[########################################] | 100% Completed |  0.2s
[########################################] | 100% Completed |  0.2s
[########################################] | 100% Completed |  0.3s
Wall time: 29.2 s
an.paired
{'airnow_cam-chem': pair(
     type='pt_sfc',
     radius_of_influence=1000000.0,
     obs='airnow',
     model='cam-chem',
     model_vars=['O3'],
     obs_vars=['OZONE'],
     filename='airnow_cam-chem.nc',
 )}
an.paired['airnow_cam-chem'].obj
<xarray.Dataset>
Dimensions:     (time: 2091, x: 3786)
Coordinates:
  * time        (time) datetime64[ns] 2019-09-01 ... 2019-09-30T00:30:00
  * x           (x) int32 0 1 2 3 4 5 6 7 ... 3779 3780 3781 3782 3783 3784 3785
Data variables: (12/34)
    BARPR       (time, x) float64 -1.0 -1.0 -1.0 -1.0 ... -1.0 -1.0 -1.0 -1.0
    BC          (time, x) float64 -1.0 -1.0 -1.0 -1.0 ... -1.0 -1.0 -1.0 -1.0
    CO          (time, x) float64 -1.0 -1.0 -1.0 -1.0 ... -1.0 -1.0 -1.0 -1.0
    NO          (time, x) float64 -1.0 -1.0 -1.0 -1.0 ... -1.0 -1.0 -1.0 -1.0
    NO2         (time, x) float64 -1.0 -1.0 -1.0 -1.0 ... -1.0 -1.0 -1.0 -1.0
    NO2Y        (time, x) float64 -1.0 -1.0 -1.0 -1.0 ... -1.0 -1.0 -1.0 -1.0
    ...          ...
    cmsa_name   (x) float64 -1.0 -1.0 -1.0 -1.0 -1.0 ... -1.0 -1.0 -1.0 -1.0
    msa_code    (x) float64 -1.0 -1.0 -1.0 -1.0 ... -1.0 3.306e+04 -1.0 -1.0
    msa_name    (x) object '' '' '' '' '' '' '' ... '' '' '' ' Miami, OK ' '' ''
    state_name  (x) object 'CC' 'CC' 'CC' 'CC' 'CC' 'CC' ... '' '' '' '' '' ''
    epa_region  (x) object 'CA' 'CA' 'CA' 'CA' 'CA' ... '' 'R6' 'DSMG' 'DSUZ'
    siteid      (x) object '000010102' '000010401' ... 'UB1010001' 'UZB010001'

Generate plots

%%time

an.plotting()
Hide code cell output
Warning: ty_scale not specified for OZONE, so default used.
Reference std: 16.085098346255723
Warning: ty_scale not specified for OZONE, so default used.
Reference std: 10.063694443664746
C:\tools\miniconda3\envs\melodies-monet-docs\lib\site-packages\cartopy\io\__init__.py:241: DownloadWarning: Downloading: https://naturalearth.s3.amazonaws.com/10m_cultural/ne_10m_admin_0_boundary_lines_land.zip
  warnings.warn(f'Downloading: {url}', DownloadWarning)
[########################################] | 100% Completed |  0.1s
[########################################] | 100% Completed |  0.1s
[########################################] | 100% Completed |  0.2s
[########################################] | 100% Completed |  0.3s
[########################################] | 100% Completed |  0.1s
[########################################] | 100% Completed |  0.1s
[########################################] | 100% Completed |  0.1s
[########################################] | 100% Completed |  0.1s
Wall time: 33.4 s

10 Figures