Reading Paired Data

First let’s just import the driver.

from melodies_monet import driver

Please install h5py to open files from the Amazon S3 servers.
Please install h5netcdf to open files from the Amazon S3 servers.

Read model and observations

an = driver.analysis()
an.control = "control_wrfchem_saveandread.yaml"
an.read_control()
an.control_dict

an.open_models()
an.open_obs()

example:wrfchem:racm_esrl
**** Reading WRF-Chem model output...
example:wrfchem:racm_esrl_vcp
**** Reading WRF-Chem model output...

Read saved data using control file

The driver will read the data based on the information included in the control file by calling an.read_analysis().

In the control file analysis section, setting method to 'netcdf' for a given attribute of the analysis class (e.g., paired, models, obs) will read NetCDF-4 files and set the appropriate attribute with the data. Filenames must be specified as a dict, with the keys being the pair name and the values being either a string with the filename to be read, or an iterable with multiple filenames to be read. If multiple files (such as several different days) are specified they will be joined by coordinates with xarray’s merge function.

In the control file analysis section, setting method to 'pkl' for a given attribute of the analysis class (e.g., paired, models, obs) will read .pkl files and set the appropriate attribute with the data. Filenames must be specified as either a string or an iterable. If multiple files (such as several different days) are specified, they will be joined by coordinates with xarray’s merge function.

an.read_analysis()

Reading:  ./output/save_and_read/0905_airnow_RACM_ESRL.nc4
Reading:  ./output/save_and_read/0905_airnow_RACM_ESRL_VCP.nc4

an.paired['airnow_RACM_ESRL'].obj

Show code cell output Hide code cell output

<xarray.Dataset>
Dimensions:     (time: 2091, x: 3786)
Coordinates:
  * time        (time) datetime64[ns] 2019-09-01 ... 2019-09-30T00:30:00
  * x           (x) int64 0 1 2 3 4 5 6 7 ... 3779 3780 3781 3782 3783 3784 3785
Data variables: (12/35)
    BARPR       (time, x) float64 ...
    BC          (time, x) float64 ...
    CO          (time, x) float64 ...
    NO          (time, x) float64 ...
    NO2         (time, x) float64 ...
    NO2Y        (time, x) float64 ...
    ...          ...
    cmsa_name   (x) float64 -1.0 -1.0 -1.0 -1.0 -1.0 ... -1.0 -1.0 -1.0 -1.0
    msa_code    (x) float64 -1.0 -1.0 -1.0 -1.0 ... -1.0 3.306e+04 -1.0 -1.0
    msa_name    (x) object '' '' '' '' '' '' '' ... '' '' '' ' Miami, OK ' '' ''
    state_name  (x) object 'CC' 'CC' 'CC' 'CC' 'CC' 'CC' ... '' '' '' '' '' ''
    epa_region  (x) object 'CA' 'CA' 'CA' 'CA' 'CA' ... '' 'R6' 'DSMG' 'DSUZ'
    siteid      (x) object '000010102' '000010401' ... 'UB1010001' 'UZB010001'
Attributes:
    title:         
    format:        NetCDF-4
    date_created:  2022-12-05
    dict_json:     {\n    "type": "pt_sfc",\n    "radius_of_influence": 10000...
    group_name:    airnow_RACM_ESRL

xarray.Dataset

• Dimensions:
  • time: 2091
  • x: 3786
• Coordinates: (2)
  • time
    (time)
    datetime64[ns]
    2019-09-01 ... 2019-09-30T00:30:00

    array(['2019-09-01T00:00:00.000000000', '2019-09-01T00:15:00.000000000',
           '2019-09-01T00:30:00.000000000', ..., '2019-09-30T00:00:00.000000000',
           '2019-09-30T00:15:00.000000000', '2019-09-30T00:30:00.000000000'],
          dtype='datetime64[ns]')

  • x
    (x)
    int64
    0 1 2 3 4 ... 3782 3783 3784 3785

    array([   0,    1,    2, ..., 3783, 3784, 3785])

• Data variables: (35)
  • BARPR
    (time, x)
    float64
    ...

    [7916526 values with dtype=float64]

  • BC
    (time, x)
    float64
    ...

    [7916526 values with dtype=float64]

  • CO
    (time, x)
    float64
    ...

    [7916526 values with dtype=float64]

  • NO
    (time, x)
    float64
    ...

    [7916526 values with dtype=float64]

  • NO2
    (time, x)
    float64
    ...

    [7916526 values with dtype=float64]

  • NO2Y
    (time, x)
    float64
    ...

    [7916526 values with dtype=float64]

  • NOX
    (time, x)
    float64
    ...

    [7916526 values with dtype=float64]

  • NOY
    (time, x)
    float64
    ...

    [7916526 values with dtype=float64]

  • OZONE
    (time, x)
    float64
    ...

    [7916526 values with dtype=float64]

  • PM10
    (time, x)
    float64
    ...

    [7916526 values with dtype=float64]

  • PM2.5
    (time, x)
    float64
    ...

    [7916526 values with dtype=float64]

  • PRECIP
    (time, x)
    float64
    ...

    [7916526 values with dtype=float64]

  • RHUM
    (time, x)
    float64
    ...

    [7916526 values with dtype=float64]

  • RWD
    (time, x)
    float64
    ...

    [7916526 values with dtype=float64]

  • RWS
    (time, x)
    float64
    ...

    [7916526 values with dtype=float64]

  • SO2
    (time, x)
    float64
    ...

    [7916526 values with dtype=float64]

  • SRAD
    (time, x)
    float64
    ...

    [7916526 values with dtype=float64]

  • TEMP
    (time, x)
    float64
    ...

    [7916526 values with dtype=float64]

  • UV-AETH
    (time, x)
    float64
    ...

    [7916526 values with dtype=float64]

  • WD
    (time, x)
    float64
    ...

    [7916526 values with dtype=float64]

  • WS
    (time, x)
    float64
    ...

    [7916526 values with dtype=float64]

  • units
    (time, x)
    object
    ...

    [7916526 values with dtype=object]

  • time_local
    (time, x)
    datetime64[ns]
    ...

    [7916526 values with dtype=datetime64[ns]]

  • PM2_5_DRY
    (time, x)
    float32
    ...

    [7916526 values with dtype=float32]

  • o3
    (time, x)
    float32
    ...

    [7916526 values with dtype=float32]

  • site
    (x)
    object
    ...

    array(["St. John's", 'Mount Pearl', 'Grand Falls Windsor', ..., 'Miami',
           'Ulaanbaatar', 'Tashkent'], dtype=object)

  • utcoffset
    (x)
    int64
    ...

    array([-4, -4, -4, ..., -6,  8,  5])

  • latitude
    (x)
    float64
    ...

    array([47.6528  , 47.505   , 49.0194  , ..., 36.922222, 47.928385, 41.3672  ])

  • longitude
    (x)
    float64
    ...

    array([-52.8167 , -52.7947 , -55.8028 , ..., -94.83889, 106.92947,  69.2725 ])

  • cmsa_name
    (x)
    float64
    ...

    array([-1., -1., -1., ..., -1., -1., -1.])

  • msa_code
    (x)
    float64
    ...

    array([-1.000e+00, -1.000e+00, -1.000e+00, ...,  3.306e+04, -1.000e+00,
           -1.000e+00])

  • msa_name
    (x)
    object
    ...

    array(['', '', '', ..., ' Miami, OK ', '', ''], dtype=object)

  • state_name
    (x)
    object
    ...

    array(['CC', 'CC', 'CC', ..., '', '', ''], dtype=object)

  • epa_region
    (x)
    object
    ...

    array(['CA', 'CA', 'CA', ..., 'R6', 'DSMG', 'DSUZ'], dtype=object)

  • siteid
    (x)
    object
    ...

    array(['000010102', '000010401', '000010501', ..., 'TT9209004', 'UB1010001',
           'UZB010001'], dtype=object)

• Attributes: (5)

  title :

  format :

  NetCDF-4

  date_created :

  2022-12-05

  dict_json :

  { "type": "pt_sfc", "radius_of_influence": 1000000.0, "obs": "airnow", "model": "RACM_ESRL", "model_vars": [ "PM2_5_DRY", "o3" ], "obs_vars": [ "PM2.5", "OZONE" ], "filename": "airnow_RACM_ESRL.nc" }

  group_name :

  airnow_RACM_ESRL

an.paired['airnow_RACM_ESRL_VCP'].obj

Show code cell output Hide code cell output

<xarray.Dataset>
Dimensions:     (time: 2091, x: 3786)
Coordinates:
  * time        (time) datetime64[ns] 2019-09-01 ... 2019-09-30T00:30:00
  * x           (x) int64 0 1 2 3 4 5 6 7 ... 3779 3780 3781 3782 3783 3784 3785
Data variables: (12/35)
    BARPR       (time, x) float64 ...
    BC          (time, x) float64 ...
    CO          (time, x) float64 ...
    NO          (time, x) float64 ...
    NO2         (time, x) float64 ...
    NO2Y        (time, x) float64 ...
    ...          ...
    cmsa_name   (x) float64 -1.0 -1.0 -1.0 -1.0 -1.0 ... -1.0 -1.0 -1.0 -1.0
    msa_code    (x) float64 -1.0 -1.0 -1.0 -1.0 ... -1.0 3.306e+04 -1.0 -1.0
    msa_name    (x) object '' '' '' '' '' '' '' ... '' '' '' ' Miami, OK ' '' ''
    state_name  (x) object 'CC' 'CC' 'CC' 'CC' 'CC' 'CC' ... '' '' '' '' '' ''
    epa_region  (x) object 'CA' 'CA' 'CA' 'CA' 'CA' ... '' 'R6' 'DSMG' 'DSUZ'
    siteid      (x) object '000010102' '000010401' ... 'UB1010001' 'UZB010001'
Attributes:
    title:         
    format:        NetCDF-4
    date_created:  2022-12-05
    dict_json:     {\n    "type": "pt_sfc",\n    "radius_of_influence": 10000...
    group_name:    airnow_RACM_ESRL_VCP

xarray.Dataset

• Dimensions:
  • time: 2091
  • x: 3786
• Coordinates: (2)
  • time
    (time)
    datetime64[ns]
    2019-09-01 ... 2019-09-30T00:30:00

    array(['2019-09-01T00:00:00.000000000', '2019-09-01T00:15:00.000000000',
           '2019-09-01T00:30:00.000000000', ..., '2019-09-30T00:00:00.000000000',
           '2019-09-30T00:15:00.000000000', '2019-09-30T00:30:00.000000000'],
          dtype='datetime64[ns]')

  • x
    (x)
    int64
    0 1 2 3 4 ... 3782 3783 3784 3785

    array([   0,    1,    2, ..., 3783, 3784, 3785])

• Data variables: (35)
  • BARPR
    (time, x)
    float64
    ...

    [7916526 values with dtype=float64]

  • BC
    (time, x)
    float64
    ...

    [7916526 values with dtype=float64]

  • CO
    (time, x)
    float64
    ...

    [7916526 values with dtype=float64]

  • NO
    (time, x)
    float64
    ...

    [7916526 values with dtype=float64]

  • NO2
    (time, x)
    float64
    ...

    [7916526 values with dtype=float64]

  • NO2Y
    (time, x)
    float64
    ...

    [7916526 values with dtype=float64]

  • NOX
    (time, x)
    float64
    ...

    [7916526 values with dtype=float64]

  • NOY
    (time, x)
    float64
    ...

    [7916526 values with dtype=float64]

  • OZONE
    (time, x)
    float64
    ...

    [7916526 values with dtype=float64]

  • PM10
    (time, x)
    float64
    ...

    [7916526 values with dtype=float64]

  • PM2.5
    (time, x)
    float64
    ...

    [7916526 values with dtype=float64]

  • PRECIP
    (time, x)
    float64
    ...

    [7916526 values with dtype=float64]

  • RHUM
    (time, x)
    float64
    ...

    [7916526 values with dtype=float64]

  • RWD
    (time, x)
    float64
    ...

    [7916526 values with dtype=float64]

  • RWS
    (time, x)
    float64
    ...

    [7916526 values with dtype=float64]

  • SO2
    (time, x)
    float64
    ...

    [7916526 values with dtype=float64]

  • SRAD
    (time, x)
    float64
    ...

    [7916526 values with dtype=float64]

  • TEMP
    (time, x)
    float64
    ...

    [7916526 values with dtype=float64]

  • UV-AETH
    (time, x)
    float64
    ...

    [7916526 values with dtype=float64]

  • WD
    (time, x)
    float64
    ...

    [7916526 values with dtype=float64]

  • WS
    (time, x)
    float64
    ...

    [7916526 values with dtype=float64]

  • units
    (time, x)
    object
    ...

    [7916526 values with dtype=object]

  • time_local
    (time, x)
    datetime64[ns]
    ...

    [7916526 values with dtype=datetime64[ns]]

  • PM2_5_DRY
    (time, x)
    float32
    ...

    [7916526 values with dtype=float32]

  • o3
    (time, x)
    float32
    ...

    [7916526 values with dtype=float32]

  • site
    (x)
    object
    ...

    array(["St. John's", 'Mount Pearl', 'Grand Falls Windsor', ..., 'Miami',
           'Ulaanbaatar', 'Tashkent'], dtype=object)

  • utcoffset
    (x)
    int64
    ...

    array([-4, -4, -4, ..., -6,  8,  5])

  • latitude
    (x)
    float64
    ...

    array([47.6528  , 47.505   , 49.0194  , ..., 36.922222, 47.928385, 41.3672  ])

  • longitude
    (x)
    float64
    ...

    array([-52.8167 , -52.7947 , -55.8028 , ..., -94.83889, 106.92947,  69.2725 ])

  • cmsa_name
    (x)
    float64
    ...

    array([-1., -1., -1., ..., -1., -1., -1.])

  • msa_code
    (x)
    float64
    ...

    array([-1.000e+00, -1.000e+00, -1.000e+00, ...,  3.306e+04, -1.000e+00,
           -1.000e+00])

  • msa_name
    (x)
    object
    ...

    array(['', '', '', ..., ' Miami, OK ', '', ''], dtype=object)

  • state_name
    (x)
    object
    ...

    array(['CC', 'CC', 'CC', ..., '', '', ''], dtype=object)

  • epa_region
    (x)
    object
    ...

    array(['CA', 'CA', 'CA', ..., 'R6', 'DSMG', 'DSUZ'], dtype=object)

  • siteid
    (x)
    object
    ...

    array(['000010102', '000010401', '000010501', ..., 'TT9209004', 'UB1010001',
           'UZB010001'], dtype=object)

• Attributes: (5)

  title :

  format :

  NetCDF-4

  date_created :

  2022-12-05

  dict_json :

  { "type": "pt_sfc", "radius_of_influence": 1000000.0, "obs": "airnow", "model": "RACM_ESRL_VCP", "model_vars": [ "PM2_5_DRY", "o3" ], "obs_vars": [ "PM2.5", "OZONE" ], "filename": "airnow_RACM_ESRL_VCP.nc" }

  group_name :

  airnow_RACM_ESRL_VCP

Read data without using control file

Alternatively, the same can be achieved by calling melodies_monet.util.read_util.read_saved_data() directly. The object to set must be an attribute of the instance of the analysis class (e.g., an.paired, an.models, an.obs).

# For netCDF files 
from melodies_monet.util.read_util import read_saved_data

read_saved_data(
    analysis=an,
    filenames={'airnow_wrfchem_v4.2': ['0905_airnow_wrfchem_v4.2.nc4']},
    method='netcdf',
    attr='paired')

# For pickle files 
from melodies_monet.util.read_util import read_saved_data

read_saved_data(analysis=an, filenames=['0905.pkl'], method='pkl', attr='paired')