diff --git a/README.md b/README.md index ae766d5d0..7b19e5b7f 100755 --- a/README.md +++ b/README.md @@ -8,6 +8,7 @@ PCMDI metrics package (PMP) [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.1414560.svg)](https://doi.org/10.5281/zenodo.1414560) [![Anaconda-Server Badge](https://anaconda.org/pcmdi/pcmdi_metrics/badges/installer/conda.svg)](https://conda.anaconda.org/pcmdi) [![Anaconda-Server Badge](https://anaconda.org/pcmdi/pcmdi_metrics/badges/downloads.svg)](https://anaconda.org/pcmdi/pcmdi_metrics) +[![CircleCI](https://circleci.com/gh/PCMDI/pcmdi_metrics.svg?style=svg)](https://circleci.com/gh/PCMDI/pcmdi_metrics) [![Coverage Status](https://coveralls.io/repos/github/PCMDI/pcmdi_metrics/badge.svg)](https://coveralls.io/github/PCMDI/pcmdi_metrics) The PCMDI metrics package is used to provide "quick-look" objective comparisons of Earth System Models (ESMs) with one another and available observations. Results are produced in the context of all model simulations contributed to CMIP6 and earlier CMIP phases. Among other purposes, this enables modeling groups to evaluate changes during the development cycle in the context of the structural error distribution of the multi-model ensemble. Currently, the comparisons emphasize metrics of large- to global-scale annual cycle and both tropcial and extra-tropical modes of variability. Ongoing work in v1.x development branches include established statistics for ENSO, MJO, regional monsoons, and high frequency characteristics of simulated precipitation. diff --git a/doc/jupyter/Demo/Demo_6_ENSO.ipynb b/doc/jupyter/Demo/Demo_6_ENSO.ipynb new file mode 100644 index 000000000..74c4c4671 --- /dev/null +++ b/doc/jupyter/Demo/Demo_6_ENSO.ipynb @@ -0,0 +1,1629 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "f02518f6", + "metadata": {}, + "source": [ + "# ENSO" + ] + }, + { + "cell_type": "markdown", + "id": "1bb7a1ac", + "metadata": {}, + "source": [ + "This notebook provides an overview of running the ENSO metrics. More information can be found in the [README]( ). Example parameter files are located in the [PMP sample setups]( ). \n", + "\n", + "**Reference**: Planton, Y., E. Guilyardi, A. T. Wittenberg, J. Lee, P. J. Gleckler, T. Bayr, S. McGregor, M. J. McPhaden, S. Power, R. Roehrig, A. Voldoire, 2020: Evaluating El Niño in climate models with the CLIVAR 2020 ENSO metrics package. Bulletin of the American Meteorological Society. [doi: 10.1175/BAMS-D-19-0337.1](https://doi.org/10.1175/BAMS-D-19-0337.1)\n", + "\n", + "Description for individual metrics can be found at https://github.com/CLIVAR-PRP/ENSO_metrics/wiki.\n", + "\n", + "### Requirements\n", + "\n", + "The first two sections of this notebook help to set up the demo data and environment.\n", + "\n", + "**Download demo data**: The ENSO metrics demo requires downloading a large sample data set (size 10.8 GB). Some files overlap with the PMP sample data from Demo 0 and will not be downloaded twice if that data is already present.\n", + "\n", + "**Environment**: This demo also requires installing the ENSO Metrics and scipy packages to your current environment. *If you do not want to alter your PCMDI Metrics conda environment*, it is suggested that you create and activate a clone for use with this demo." + ] + }, + { + "cell_type": "markdown", + "id": "7a6383bf", + "metadata": {}, + "source": [ + "## Download demo data" + ] + }, + { + "cell_type": "markdown", + "id": "cdc4b9e2", + "metadata": {}, + "source": [ + "The ENSO metric requires a different set of sample data than the rest of the PMP metrics. This section of the notebook will download that data to your chosen location and generate a basic parameter file." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "da5d715f", + "metadata": {}, + "outputs": [], + "source": [ + "# Lets get the file containing the data needed for this demo\n", + "import requests\n", + "r = requests.get(\"https://pcmdiweb.llnl.gov/pss/pmpdata/pmp_enso_tutorial_files.txt\")\n", + "with open(\"enso_data_files.txt\",\"wb\") as f:\n", + " f.write(r.content)" + ] + }, + { + "cell_type": "markdown", + "id": "ec2d5a6a", + "metadata": {}, + "source": [ + "If you want to change the location where the demo data and output are stored, you can do so here:" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "8e7cc829", + "metadata": {}, + "outputs": [], + "source": [ + "# This is where you will be downloading the sample_data\n", + "demo_data_directory = \"demo_data\"\n", + "# this line is where your output will be stored\n", + "demo_output_directory = \"demo_output\"" + ] + }, + { + "cell_type": "markdown", + "id": "f2386574", + "metadata": {}, + "source": [ + "Then download the data. The total sample data size is 10.8 GB. This will take several minutes." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "520aa1c5", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "MD5: enso_data_files.txt\n", + "All files downloaded\n" + ] + } + ], + "source": [ + "# Let's download the files\n", + "import cdat_info\n", + "try:\n", + " cdat_info.download_sample_data_files(\"enso_data_files.txt\", demo_data_directory)\n", + " print(\"All files downloaded\")\n", + "except:\n", + " print(\"Download failed\")" + ] + }, + { + "cell_type": "markdown", + "id": "11aa0d9f", + "metadata": {}, + "source": [ + "After downloading the data, we generate the parameter file for this demo." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "bf31cbd1", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Preparing parameter file: basic_enso_param.py\n", + "Saving User Choices\n" + ] + } + ], + "source": [ + "from download_sample_data import generate_parameter_files\n", + "filenames=[\"basic_enso_param.py.in\"]\n", + "generate_parameter_files(demo_data_directory, demo_output_directory, filenames=filenames)" + ] + }, + { + "cell_type": "markdown", + "id": "81ab1e49", + "metadata": {}, + "source": [ + "## Environment" + ] + }, + { + "cell_type": "markdown", + "id": "647d09fa", + "metadata": {}, + "source": [ + "[ENSO Metrics package](https://github.com/CLIVAR-PRP/ENSO_metrics) and [scipy](https://www.scipy.org/) installations are needed. This section will clone the ENSO Metrics repository and *install ENSO Metrics and scipy in your current conda environment*. Set the `enso_install_location` below to chose where to clone the ENSO Metrics repository." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "d632f1ab", + "metadata": {}, + "outputs": [], + "source": [ + "enso_install_location = \".\"" + ] + }, + { + "cell_type": "markdown", + "id": "ed720426", + "metadata": {}, + "source": [ + "To clone and install the ENSO Metrics package, un-comment the next cell and run it (delete quotes in lines 1 & 6)." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "aecd5bcc", + "metadata": { + "scrolled": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "'\\n%%bash -s \"$enso_install_location\"\\ncd $1\\ngit clone https://github.com/CLIVAR-PRP/ENSO_metrics.git\\ncd ENSO_metrics\\npython setup.py install\\n'" + ] + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "\"\"\"\n", + "%%bash -s \"$enso_install_location\"\n", + "cd $1\n", + "git clone https://github.com/CLIVAR-PRP/ENSO_metrics.git\n", + "cd ENSO_metrics\n", + "python setup.py install\n", + "\"\"\"" + ] + }, + { + "cell_type": "markdown", + "id": "137de9c7", + "metadata": {}, + "source": [ + "To install scipy, un-comment the next cell and run it (delete quotes in lines 1 & 3)." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "046c7468", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "'\\nconda install -c anaconda scipy \\n'" + ] + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "\"\"\"\n", + "conda install -c anaconda scipy \n", + "\"\"\"" + ] + }, + { + "cell_type": "markdown", + "id": "ab7ac9e8", + "metadata": {}, + "source": [ + "## Usage" + ] + }, + { + "cell_type": "markdown", + "id": "44577389", + "metadata": {}, + "source": [ + "The ENSO driver can be run from the command line as `enso_driver.py`. In this notebook, we will use bash cell magic (cells beginning with `%%bash`) to run the ENSO driver as a subprocess." + ] + }, + { + "cell_type": "markdown", + "id": "b9059757", + "metadata": {}, + "source": [ + "For help, type: \n", + "```\n", + "enso_driver.py --help\n", + "```" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "c113d824", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "usage: enso_driver.py [-h] [--parameters PARAMETERS]\n", + " [--diags OTHER_PARAMETERS [OTHER_PARAMETERS ...]]\n", + " [--mip MIP] [--exp EXP] [--modpath MODPATH]\n", + " [--modpath_lf MODPATH_LF]\n", + " [--modnames MODNAMES [MODNAMES ...]] [-r REALIZATION]\n", + " [--reference_data_path REFERENCE_DATA_PATH]\n", + " [--reference_data_lf_path REFERENCE_DATA_LF_PATH]\n", + " [--metricsCollection METRICSCOLLECTION]\n", + " [--json_name JSON_NAME] [--netcdf_name NETCDF_NAME]\n", + " [--results_dir RESULTS_DIR] [--case_id CASE_ID]\n", + " [--obs_catalogue OBS_CATALOGUE]\n", + " [--obs_cmor_path OBS_CMOR_PATH] [-d [DEBUG]]\n", + " [--obs_cmor [OBS_CMOR]] [--nc_out [NC_OUT]]\n", + "\n", + "optional arguments:\n", + " -h, --help show this help message and exit\n", + " --parameters PARAMETERS, -p PARAMETERS\n", + " --diags OTHER_PARAMETERS [OTHER_PARAMETERS ...]\n", + " Path to other user-defined parameter file. (default:\n", + " None)\n", + " --mip MIP A WCRP MIP project such as CMIP3 and CMIP5 (default:\n", + " cmip5)\n", + " --exp EXP An experiment such as AMIP, historical or pi-contorl\n", + " (default: historical)\n", + " --modpath MODPATH, --mp MODPATH\n", + " Explicit path to model data (default: None)\n", + " --modpath_lf MODPATH_LF\n", + " Directory path to model land fraction field (default:\n", + " None)\n", + " --modnames MODNAMES [MODNAMES ...]\n", + " List of models (default: None)\n", + " -r REALIZATION, --realization REALIZATION\n", + " Consider all accessible realizations as idividual -\n", + " r1i1p1: default, consider only 'r1i1p1' member Or,\n", + " specify realization, e.g, r3i1p1' - *: consider all\n", + " available realizations (default: r1i1p1)\n", + " --reference_data_path REFERENCE_DATA_PATH, --rdp REFERENCE_DATA_PATH\n", + " The path/filename of reference (obs) data. (default:\n", + " None)\n", + " --reference_data_lf_path REFERENCE_DATA_LF_PATH\n", + " Data path to land fraction of reference dataset\n", + " (default: None)\n", + " --metricsCollection METRICSCOLLECTION\n", + " Metrics Collection e.g. ENSO_perf, ENSO_tel, or\n", + " ENSO_proc (default: ENSO_perf)\n", + " --json_name JSON_NAME\n", + " File name for output JSON (default: None)\n", + " --netcdf_name NETCDF_NAME\n", + " File name for output NetCDF (default: None)\n", + " --results_dir RESULTS_DIR, --rd RESULTS_DIR\n", + " The name of the folder where all runs will be stored.\n", + " (default: None)\n", + " --case_id CASE_ID version as date, e.g., v20191116 (yyyy-mm-dd)\n", + " (default: v20210604)\n", + " --obs_catalogue OBS_CATALOGUE\n", + " obs_catalogue JSON file for CMORized observation,\n", + " default is None (default: None)\n", + " --obs_cmor_path OBS_CMOR_PATH\n", + " Directory path for CMORized observation dataset,\n", + " default is None (default: None)\n", + " -d [DEBUG], --debug [DEBUG]\n", + " Option for debug: True / False (defualt) (default:\n", + " False)\n", + " --obs_cmor [OBS_CMOR]\n", + " Use CMORized reference database?: True / False\n", + " (defualt) (default: False)\n", + " --nc_out [NC_OUT] Option for generate netCDF file output: True (default)\n", + " / False (default: True)\n" + ] + } + ], + "source": [ + "%%bash\n", + "enso_driver.py --help" + ] + }, + { + "cell_type": "markdown", + "id": "0701dc96", + "metadata": {}, + "source": [ + "### Basic example" + ] + }, + { + "cell_type": "markdown", + "id": "46928faf", + "metadata": {}, + "source": [ + "Parameters for the ENSO Metrics can be set on the command line or using a parameter file. This first example will use a parameter file, which is shown below." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "70a6b070", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "import os\n", + "\n", + "#\n", + "# OPTIONS ARE SET BY USER IN THIS FILE AS INDICATED BELOW BY:\n", + "#\n", + "#\n", + "\n", + "# MODELS\n", + "modnames = ['ACCESS1-0']\n", + "mip = 'cmip5' # cmip5, cmip6\n", + "exp = 'historical' # historical, piControl\n", + "realization = 'r1i1p1'\n", + "modpath = 'demo_data/CMIP5_demo_data/%(variable)_Amon_%(model)_historical_%(realization)_185001-200512.nc'\n", + "modpath_lf = 'demo_data/CMIP5_demo_data/sftlf_fx_%(model)_amip_r0i0p0.nc'\n", + "\n", + "# OBSERVATIONS\n", + "obs_cmor = True\n", + "obs_cmor_path = \"demo_data/PCMDIobs2\"\n", + "obs_catalogue = \"demo_data/pcmdiobs-CEM2020_monthly_bySource_catalogue_v20210422_demo.json\"\n", + "\n", + "# METRICS COLLECTION\n", + "metricsCollection = 'ENSO_perf' # ENSO_perf, ENSO_tel, ENSO_proc\n", + "\n", + "# OUTPUT\n", + "case_id = 'basicTestEnso'\n", + "results_dir = os.path.join('demo_output',case_id, metricsCollection)\n", + "\n", + "json_name = '%(mip)_%(exp)_%(metricsCollection)_%(case_id)_%(model)_%(realization)'\n", + "netcdf_name = json_name\n", + "nc_out = False\n", + "\n" + ] + } + ], + "source": [ + "with open(\"basic_enso_param.py\") as f:\n", + " print(f.read())" + ] + }, + { + "cell_type": "markdown", + "id": "92d0e015", + "metadata": {}, + "source": [ + "The next cell runs the ENSO driver using the basic parameter file. This may take several minutes." + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "5f8ef3f9", + "metadata": { + "scrolled": false + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "mip: cmip5\n", + "exp: historical\n", + "models: ['ACCESS1-0']\n", + "realization: r1i1p1\n", + "mc_name: ENSO_perf\n", + "outdir: demo_output/basicTestEnso/ENSO_perf\n", + "netcdf_path: demo_output/basicTestEnso/ENSO_perf\n", + "debug: False\n", + "obs_cmor: True\n", + "obs_cmor_path: demo_data/PCMDIobs2\n", + "egg_pth: /Users/ordonez4/Library/Caches/Python-Eggs/pcmdi_metrics-v1.2.1_690_g281220b-py3.8.egg-tmp/share/pmp\n", + "output directory for graphics:demo_output/basicTestEnso/ENSO_perf\n", + "output directory for diagnostic_results:demo_output/basicTestEnso/ENSO_perf\n", + "output directory for metrics_results:demo_output/basicTestEnso/ENSO_perf\n", + "list_variables: ['pr', 'sst', 'taux']\n", + "list_obs: ['AVISO-1-0', 'ERA-INT', 'GPCP-2-3', 'HadISST-1-1']\n", + "PMPdriver: dict_obs readin end\n", + "models: ['ACCESS1-0']\n", + " ----- model: ACCESS1-0 ---------------------\n", + "PMPdriver: var loop start for model ACCESS1-0\n", + "realization: r1i1p1\n", + " --- run: r1i1p1 ---\n", + " --- var: pr ---\n", + "var_in_file: pr\n", + "var, areacell_in_file, realm: pr areacella atmos\n", + "path: demo_data/CMIP5_demo_data/pr_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc\n", + "file_list: ['demo_data/CMIP5_demo_data/pr_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc']\n", + "path: demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\n", + "file_list: ['demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc']\n", + "path: demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\n", + "file_list: ['demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc']\n", + "PMPdriver: var loop end\n", + " --- var: sst ---\n", + "var_in_file: ts\n", + "var, areacell_in_file, realm: sst areacella atmos\n", + "path: demo_data/CMIP5_demo_data/ts_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc\n", + "file_list: ['demo_data/CMIP5_demo_data/ts_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc']\n", + "path: demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\n", + "file_list: ['demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc']\n", + "path: demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\n", + "file_list: ['demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc']\n", + "PMPdriver: var loop end\n", + " --- var: taux ---\n", + "var_in_file: tauu\n", + "var, areacell_in_file, realm: taux areacella atmos\n", + "path: demo_data/CMIP5_demo_data/tauu_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc\n", + "file_list: ['demo_data/CMIP5_demo_data/tauu_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc']\n", + "path: demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\n", + "file_list: ['demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc']\n", + "path: demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\n", + "file_list: ['demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc']\n", + "PMPdriver: var loop end\n", + "dictDatasets:\n", + "{\n", + " \"model\": {\n", + " \"ACCESS1-0_r1i1p1\": {\n", + " \"pr\": {\n", + " \"areaname\": \"areacella\",\n", + " \"landmaskname\": \"sftlf\",\n", + " \"path + filename\": \"demo_data/CMIP5_demo_data/pr_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc\",\n", + " \"path + filename_area\": \"demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\",\n", + " \"path + filename_landmask\": \"demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\",\n", + " \"varname\": \"pr\"\n", + " },\n", + " \"sst\": {\n", + " \"areaname\": \"areacella\",\n", + " \"landmaskname\": \"sftlf\",\n", + " \"path + filename\": \"demo_data/CMIP5_demo_data/ts_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc\",\n", + " \"path + filename_area\": \"demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\",\n", + " \"path + filename_landmask\": \"demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\",\n", + " \"varname\": \"ts\"\n", + " },\n", + " \"taux\": {\n", + " \"areaname\": \"areacella\",\n", + " \"landmaskname\": \"sftlf\",\n", + " \"path + filename\": \"demo_data/CMIP5_demo_data/tauu_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc\",\n", + " \"path + filename_area\": \"demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\",\n", + " \"path + filename_landmask\": \"demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\",\n", + " \"varname\": \"tauu\"\n", + " }\n", + " }\n", + " },\n", + " \"observations\": {\n", + " \"ERA-INT\": {\n", + " \"pr\": {\n", + " \"areaname\": \"areacella\",\n", + " \"landmaskname\": \"sftlf\",\n", + " \"path + filename\": \"demo_data/PCMDIobs2/atmos/mon/pr/ERA-INT/gn/v20200707/pr_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc\",\n", + " \"path + filename_area\": null,\n", + " \"path + filename_landmask\": null,\n", + " \"varname\": \"pr\"\n", + " },\n", + " \"sst\": {\n", + " \"areaname\": \"areacella\",\n", + " \"landmaskname\": \"sftlf\",\n", + " \"path + filename\": \"demo_data/PCMDIobs2/atmos/mon/ts/ERA-INT/gn/v20200707/ts_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc\",\n", + " \"path + filename_area\": null,\n", + " \"path + filename_landmask\": null,\n", + " \"varname\": \"ts\"\n", + " },\n", + " \"taux\": {\n", + " \"areaname\": \"areacella\",\n", + " \"landmaskname\": \"sftlf\",\n", + " \"path + filename\": \"demo_data/PCMDIobs2/atmos/mon/tauu/ERA-INT/gn/v20200707/tauu_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc\",\n", + " \"path + filename_area\": null,\n", + " \"path + filename_landmask\": null,\n", + " \"varname\": \"tauu\"\n", + " }\n", + " },\n", + " \"GPCP-2-3\": {\n", + " \"pr\": {\n", + " \"areaname\": \"areacella\",\n", + " \"landmaskname\": \"sftlf\",\n", + " \"path + filename\": \"demo_data/PCMDIobs2/atmos/mon/pr/GPCP-2-3/gn/v20200707/pr_mon_GPCP-2-3_BE_gn_v20200707_197901-201907.nc\",\n", + " \"path + filename_area\": null,\n", + " \"path + filename_landmask\": null,\n", + " \"varname\": \"pr\"\n", + " }\n", + " }\n", + " }\n", + "}\n", + "\n", + "### Compute the metric collection ###\n", + "\n", + "\u001b[94m ComputeCollection: metric = BiasPrLatRmse\u001b[0m\n", + "\u001b[94m ComputeMetric: oneVarRMSmetric, BiasPrLatRmse = ACCESS1-0_r1i1p1 and ERA-INT\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageZonal\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageZonal\u001b[0m\n", + "\u001b[94m ComputeMetric: oneVarRMSmetric, BiasPrLatRmse = ACCESS1-0_r1i1p1 and GPCP-2-3\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageZonal\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageZonal\u001b[0m\n", + "\u001b[94m ComputeCollection: metric = BiasPrLonRmse\u001b[0m\n", + "\u001b[94m ComputeMetric: oneVarRMSmetric, BiasPrLonRmse = ACCESS1-0_r1i1p1 and ERA-INT\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageMeridional\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageMeridional\u001b[0m\n", + "\u001b[94m ComputeMetric: oneVarRMSmetric, BiasPrLonRmse = ACCESS1-0_r1i1p1 and GPCP-2-3\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageMeridional\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageMeridional\u001b[0m\n", + "\u001b[94m ComputeCollection: metric = BiasSstLonRmse\u001b[0m\n", + "\u001b[94m ComputeMetric: oneVarRMSmetric, BiasSstLonRmse = ACCESS1-0_r1i1p1 and ERA-INT\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageMeridional\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageMeridional\u001b[0m\n", + "\u001b[94m ComputeCollection: metric = BiasTauxLonRmse\u001b[0m\n", + "\u001b[94m ComputeMetric: oneVarRMSmetric, BiasTauxLonRmse = ACCESS1-0_r1i1p1 and ERA-INT\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageHorizontal\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageHorizontal\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageMeridional\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageMeridional\u001b[0m\n", + "\u001b[94m ComputeCollection: metric = EnsoAmpl\u001b[0m\n", + "\u001b[94m ComputeMetric: oneVarmetric = ACCESS1-0_r1i1p1\u001b[0m\n", + "\u001b[94m ComputeMetric: oneVarmetric = ERA-INT\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[94m ComputeCollection: metric = EnsoDuration\u001b[0m\n", + "\u001b[94m ComputeMetric: oneVarmetric = ACCESS1-0_r1i1p1\u001b[0m\n", + "\u001b[94m ComputeMetric: oneVarmetric = ERA-INT\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[94m ComputeCollection: metric = EnsoSeasonality\u001b[0m\n", + "\u001b[94m ComputeMetric: oneVarmetric = ACCESS1-0_r1i1p1\u001b[0m\n", + "\u001b[94m ComputeMetric: oneVarmetric = ERA-INT\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[94m ComputeCollection: metric = EnsoSstDiversity_2\u001b[0m\n", + "\u001b[94m ComputeMetric: oneVarmetric = ACCESS1-0_r1i1p1\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageMeridional\u001b[0m\n", + "\u001b[94m ComputeMetric: oneVarmetric = ERA-INT\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageMeridional\u001b[0m\n", + "\u001b[94m ComputeCollection: metric = EnsoSstLonRmse\u001b[0m\n", + "\u001b[94m ComputeMetric: oneVarRMSmetric, EnsoSstLonRmse = ACCESS1-0_r1i1p1 and ERA-INT\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageMeridional\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageMeridional\u001b[0m\n", + "\u001b[94m ComputeCollection: metric = EnsoSstSkew\u001b[0m\n", + "\u001b[94m ComputeMetric: oneVarmetric = ACCESS1-0_r1i1p1\u001b[0m\n", + "\u001b[94m ComputeMetric: oneVarmetric = ERA-INT\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[94m ComputeCollection: metric = EnsoSstTsRmse\u001b[0m\n", + "\u001b[94m ComputeMetric: oneVarRMSmetric, EnsoSstTsRmse = ACCESS1-0_r1i1p1 and ERA-INT\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[94m ComputeCollection: metric = SeasonalPrLatRmse\u001b[0m\n", + "\u001b[94m ComputeMetric: oneVarRMSmetric, SeasonalPrLatRmse = ACCESS1-0_r1i1p1 and ERA-INT\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageZonal\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageZonal\u001b[0m\n", + "\u001b[94m ComputeMetric: oneVarRMSmetric, SeasonalPrLatRmse = ACCESS1-0_r1i1p1 and GPCP-2-3\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageZonal\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageZonal\u001b[0m\n", + "\u001b[94m ComputeCollection: metric = SeasonalPrLonRmse\u001b[0m\n", + "\u001b[94m ComputeMetric: oneVarRMSmetric, SeasonalPrLonRmse = ACCESS1-0_r1i1p1 and ERA-INT\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageMeridional\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageMeridional\u001b[0m\n", + "\u001b[94m ComputeMetric: oneVarRMSmetric, SeasonalPrLonRmse = ACCESS1-0_r1i1p1 and GPCP-2-3\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageMeridional\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageMeridional\u001b[0m\n", + "\u001b[94m ComputeCollection: metric = SeasonalSstLonRmse\u001b[0m\n", + "\u001b[94m ComputeMetric: oneVarRMSmetric, SeasonalSstLonRmse = ACCESS1-0_r1i1p1 and ERA-INT\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageMeridional\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageMeridional\u001b[0m\n", + "\u001b[94m ComputeCollection: metric = SeasonalTauxLonRmse\u001b[0m\n", + "\u001b[94m ComputeMetric: oneVarRMSmetric, SeasonalTauxLonRmse = ACCESS1-0_r1i1p1 and ERA-INT\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageHorizontal\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageHorizontal\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageMeridional\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageMeridional\u001b[0m\n", + "PMPdriver: model loop end\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/Users/ordonez4/miniconda3/envs/test_pcmdi/lib/python3.8/site-packages/cdms2/MV2.py:318: Warning: arguments order for compress function has changed\n", + "it is now: MV2.copmress(array,condition), if your code seems to not react or act wrong to a call to compress, please check this\n", + " warnings.warn(\n", + "INFO::2021-06-04 14:22::pcmdi_metrics:: Results saved to a json file: /Users/ordonez4/Documents/git/pcmdi_metrics/doc/jupyter/Demo/demo_output/basicTestEnso/ENSO_perf/cmip5_historical_ENSO_perf_basicTestEnso_ACCESS1-0_r1i1p1.json\n", + "INFO::2021-06-04 14:22::pcmdi_metrics:: Results saved to a json file: /Users/ordonez4/Documents/git/pcmdi_metrics/doc/jupyter/Demo/demo_output/basicTestEnso/ENSO_perf/cmip5_historical_ENSO_perf_basicTestEnso_ACCESS1-0_r1i1p1_diveDown.json\n" + ] + } + ], + "source": [ + "%%bash\n", + "enso_driver.py -p basic_enso_param.py" + ] + }, + { + "cell_type": "markdown", + "id": "1da57cf2", + "metadata": {}, + "source": [ + "This run saved metrics to two files: \n", + "basicTestEnso/cmip5_historical_ENSO_perf_basicTestEnso_ACCESS1-0_r1i1p1.json basicTestEnso/cmip5_historical_ENSO_perf_basicTestEnso_ACCESS1-0_r1i1p1_diveDown.json\n", + "\n", + "diveDown metrics are not available in all cases. \n", + "\n", + "The results section of cmip5_historical_ENSO_perf_basicTestEnso_ACCESS1-0_r1i1p1.json is shown below." + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "32ffb4a4", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{\n", + " \"BiasPrLatRmse\": {\n", + " \"diagnostic\": {\n", + " \"ACCESS1-0_r1i1p1\": {\n", + " \"value\": null,\n", + " \"value_error\": null\n", + " },\n", + " \"ERA-INT\": {\n", + " \"value\": null,\n", + " \"value_error\": null\n", + " },\n", + " \"GPCP-2-3\": {\n", + " \"value\": null,\n", + " \"value_error\": null\n", + " }\n", + " },\n", + " \"metric\": {\n", + " \"ERA-INT\": {\n", + " \"value\": 1.107530462384606,\n", + " \"value_error\": null\n", + " },\n", + " \"GPCP-2-3\": {\n", + " \"value\": 1.9301242972172135,\n", + " \"value_error\": null\n", + " }\n", + " }\n", + " },\n", + " \"BiasPrLonRmse\": {\n", + " \"diagnostic\": {\n", + " \"ACCESS1-0_r1i1p1\": {\n", + " \"value\": null,\n", + " \"value_error\": null\n", + " },\n", + " \"ERA-INT\": {\n", + " \"value\": null,\n", + " \"value_error\": null\n", + " },\n", + " \"GPCP-2-3\": {\n", + " \"value\": null,\n", + " \"value_error\": null\n", + " }\n", + " },\n", + " \"metric\": {\n", + " \"ERA-INT\": {\n", + " \"value\": 0.6464917771721342,\n", + " \"value_error\": null\n", + " },\n", + " \"GPCP-2-3\": {\n", + " \"value\": 1.4165839641155153,\n", + " \"value_error\": null\n", + " }\n", + " }\n", + " },\n", + " \"BiasSstLonRmse\": {\n", + " \"diagnostic\": {\n", + " \"ACCESS1-0_r1i1p1\": {\n", + " \"value\": null,\n", + " \"value_error\": null\n", + " },\n", + " \"ERA-INT\": {\n", + " \"value\": null,\n", + " \"value_error\": null\n", + " }\n", + " },\n", + " \"metric\": {\n", + " \"ERA-INT\": {\n", + " \"value\": 0.6404285425003315,\n", + " \"value_error\": null\n", + " }\n", + " }\n", + " },\n", + " \"BiasTauxLonRmse\": {\n", + " \"diagnostic\": {\n", + " \"ACCESS1-0_r1i1p1\": {\n", + " \"value\": null,\n", + " \"value_error\": null\n", + " },\n", + " \"ERA-INT\": {\n", + " \"value\": null,\n", + " \"value_error\": null\n", + " }\n", + " },\n", + " \"metric\": {\n", + " \"ERA-INT\": {\n", + " \"value\": 5.837312275359632,\n", + " \"value_error\": null\n", + " }\n", + " }\n", + " },\n", + " \"EnsoAmpl\": {\n", + " \"diagnostic\": {\n", + " \"ACCESS1-0_r1i1p1\": {\n", + " \"value\": 0.6630260736952586,\n", + " \"value_error\": 0.0530845705527287\n", + " },\n", + " \"ERA-INT\": {\n", + " \"value\": 0.9001341048707652,\n", + " \"value_error\": 0.1423236985494241\n", + " }\n", + " },\n", + " \"metric\": {\n", + " \"ERA-INT\": {\n", + " \"value\": 26.341411784363938,\n", + " \"value_error\": 17.543852271121864\n", + " }\n", + " }\n", + " },\n", + " \"EnsoDuration\": {\n", + " \"diagnostic\": {\n", + " \"ACCESS1-0_r1i1p1\": {\n", + " \"value\": 12.0,\n", + " \"value_error\": null\n", + " },\n", + " \"ERA-INT\": {\n", + " \"value\": 13.0,\n", + " \"value_error\": null\n", + " }\n", + " },\n", + " \"metric\": {\n", + " \"ERA-INT\": {\n", + " \"value\": 7.6923076923076925,\n", + " \"value_error\": null\n", + " }\n", + " }\n", + " },\n", + " \"EnsoSeasonality\": {\n", + " \"diagnostic\": {\n", + " \"ACCESS1-0_r1i1p1\": {\n", + " \"value\": 1.65806080684549,\n", + " \"value_error\": 0.26592975680376785\n", + " },\n", + " \"ERA-INT\": {\n", + " \"value\": 2.0529600453533243,\n", + " \"value_error\": 0.6533381863299897\n", + " }\n", + " },\n", + " \"metric\": {\n", + " \"ERA-INT\": {\n", + " \"value\": 19.235602728930374,\n", + " \"value_error\": 38.65610570118721\n", + " }\n", + " }\n", + " },\n", + " \"EnsoSstDiversity_2\": {\n", + " \"diagnostic\": {\n", + " \"ACCESS1-0_r1i1p1\": {\n", + " \"value\": 10.0,\n", + " \"value_error\": null\n", + " },\n", + " \"ERA-INT\": {\n", + " \"value\": 32.0,\n", + " \"value_error\": null\n", + " }\n", + " },\n", + " \"metric\": {\n", + " \"ERA-INT\": {\n", + " \"value\": 68.75,\n", + " \"value_error\": null\n", + " }\n", + " }\n", + " },\n", + " \"EnsoSstLonRmse\": {\n", + " \"diagnostic\": {\n", + " \"ACCESS1-0_r1i1p1\": {\n", + " \"value\": null,\n", + " \"value_error\": null\n", + " },\n", + " \"ERA-INT\": {\n", + " \"value\": null,\n", + " \"value_error\": null\n", + " }\n", + " },\n", + " \"metric\": {\n", + " \"ERA-INT\": {\n", + " \"value\": 0.1640103265768218,\n", + " \"value_error\": null\n", + " }\n", + " }\n", + " },\n", + " \"EnsoSstSkew\": {\n", + " \"diagnostic\": {\n", + " \"ACCESS1-0_r1i1p1\": {\n", + " \"value\": -0.3339196537261687,\n", + " \"value_error\": -0.02673496883520269\n", + " },\n", + " \"ERA-INT\": {\n", + " \"value\": 0.40501535626049495,\n", + " \"value_error\": 0.06403855065638503\n", + " }\n", + " },\n", + " \"metric\": {\n", + " \"ERA-INT\": {\n", + " \"value\": 182.4461711302128,\n", + " \"value_error\": -19.63686084226418\n", + " }\n", + " }\n", + " },\n", + " \"EnsoSstTsRmse\": {\n", + " \"diagnostic\": {\n", + " \"ACCESS1-0_r1i1p1\": {\n", + " \"value\": null,\n", + " \"value_error\": null\n", + " },\n", + " \"ERA-INT\": {\n", + " \"value\": null,\n", + " \"value_error\": null\n", + " }\n", + " },\n", + " \"metric\": {\n", + " \"ERA-INT\": {\n", + " \"value\": 0.10604060484676657,\n", + " \"value_error\": null\n", + " }\n", + " }\n", + " },\n", + " \"SeasonalPrLatRmse\": {\n", + " \"diagnostic\": {\n", + " \"ACCESS1-0_r1i1p1\": {\n", + " \"value\": null,\n", + " \"value_error\": null\n", + " },\n", + " \"ERA-INT\": {\n", + " \"value\": null,\n", + " \"value_error\": null\n", + " },\n", + " \"GPCP-2-3\": {\n", + " \"value\": null,\n", + " \"value_error\": null\n", + " }\n", + " },\n", + " \"metric\": {\n", + " \"ERA-INT\": {\n", + " \"value\": 1.1594264845137485,\n", + " \"value_error\": null\n", + " },\n", + " \"GPCP-2-3\": {\n", + " \"value\": 1.5589152451460864,\n", + " \"value_error\": null\n", + " }\n", + " }\n", + " },\n", + " \"SeasonalPrLonRmse\": {\n", + " \"diagnostic\": {\n", + " \"ACCESS1-0_r1i1p1\": {\n", + " \"value\": null,\n", + " \"value_error\": null\n", + " },\n", + " \"ERA-INT\": {\n", + " \"value\": null,\n", + " \"value_error\": null\n", + " },\n", + " \"GPCP-2-3\": {\n", + " \"value\": null,\n", + " \"value_error\": null\n", + " }\n", + " },\n", + " \"metric\": {\n", + " \"ERA-INT\": {\n", + " \"value\": 1.2638515328921656,\n", + " \"value_error\": null\n", + " },\n", + " \"GPCP-2-3\": {\n", + " \"value\": 1.4219913613495796,\n", + " \"value_error\": null\n", + " }\n", + " }\n", + " },\n", + " \"SeasonalSstLonRmse\": {\n", + " \"diagnostic\": {\n", + " \"ACCESS1-0_r1i1p1\": {\n", + " \"value\": null,\n", + " \"value_error\": null\n", + " },\n", + " \"ERA-INT\": {\n", + " \"value\": null,\n", + " \"value_error\": null\n", + " }\n", + " },\n", + " \"metric\": {\n", + " \"ERA-INT\": {\n", + " \"value\": 0.2880146500221596,\n", + " \"value_error\": null\n", + " }\n", + " }\n", + " },\n", + " \"SeasonalTauxLonRmse\": {\n", + " \"diagnostic\": {\n", + " \"ACCESS1-0_r1i1p1\": {\n", + " \"value\": null,\n", + " \"value_error\": null\n", + " },\n", + " \"ERA-INT\": {\n", + " \"value\": null,\n", + " \"value_error\": null\n", + " }\n", + " },\n", + " \"metric\": {\n", + " \"ERA-INT\": {\n", + " \"value\": 4.234563899898219,\n", + " \"value_error\": null\n", + " }\n", + " }\n", + " }\n", + "}\n" + ] + } + ], + "source": [ + "import json\n", + "metrics_file=demo_output_directory+\"/basicTestEnso/ENSO_perf/cmip5_historical_ENSO_perf_basicTestEnso_ACCESS1-0_r1i1p1.json\"\n", + "with open(metrics_file) as f:\n", + " results = json.load(f)[\"RESULTS\"][\"model\"][\"ACCESS1-0\"][\"r1i1p1\"][\"value\"]\n", + "print(json.dumps(results, indent = 2))" + ] + }, + { + "cell_type": "markdown", + "id": "d25a4fb7", + "metadata": {}, + "source": [ + "### ENSO Metrics Collections\n", + "\n", + "There are 3 metrics collections available: \n", + "ENSO_perf \n", + "ENSO_tel \n", + "ENSO_proc \n", + "\n", + "They can be selected using the `--metricsCollection` flag. The first example used the \"ENSO_perf\" collection.\n", + "\n", + "The next example runs the teleconnection collection. To save individual metrics in netCDF format, it uses the `--nc_out` flag." + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "931a02c5", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "mip: cmip5\n", + "exp: historical\n", + "models: ['ACCESS1-0']\n", + "realization: r1i1p1\n", + "mc_name: ENSO_tel\n", + "outdir: demo_output/basicTestEnso/ENSO_tel\n", + "netcdf_path: demo_output/basicTestEnso/ENSO_tel\n", + "debug: False\n", + "obs_cmor: True\n", + "obs_cmor_path: demo_data/PCMDIobs2\n", + "egg_pth: /Users/ordonez4/Library/Caches/Python-Eggs/pcmdi_metrics-v1.2.1_690_g281220b-py3.8.egg-tmp/share/pmp\n", + "output directory for graphics:demo_output/basicTestEnso/ENSO_tel\n", + "output directory for diagnostic_results:demo_output/basicTestEnso/ENSO_tel\n", + "output directory for metrics_results:demo_output/basicTestEnso/ENSO_tel\n", + "list_variables: ['pr', 'sst']\n", + "list_obs: ['AVISO-1-0', 'ERA-INT', 'GPCP-2-3', 'HadISST-1-1']\n", + "PMPdriver: dict_obs readin end\n", + "models: ['ACCESS1-0']\n", + " ----- model: ACCESS1-0 ---------------------\n", + "PMPdriver: var loop start for model ACCESS1-0\n", + "realization: r1i1p1\n", + " --- run: r1i1p1 ---\n", + " --- var: pr ---\n", + "var_in_file: pr\n", + "var, areacell_in_file, realm: pr areacella atmos\n", + "path: demo_data/CMIP5_demo_data/pr_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc\n", + "file_list: ['demo_data/CMIP5_demo_data/pr_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc']\n", + "path: demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\n", + "file_list: ['demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc']\n", + "path: demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\n", + "file_list: ['demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc']\n", + "PMPdriver: var loop end\n", + " --- var: sst ---\n", + "var_in_file: ts\n", + "var, areacell_in_file, realm: sst areacella atmos\n", + "path: demo_data/CMIP5_demo_data/ts_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc\n", + "file_list: ['demo_data/CMIP5_demo_data/ts_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc']\n", + "path: demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\n", + "file_list: ['demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc']\n", + "path: demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\n", + "file_list: ['demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc']\n", + "PMPdriver: var loop end\n", + "dictDatasets:\n", + "{\n", + " \"model\": {\n", + " \"ACCESS1-0_r1i1p1\": {\n", + " \"pr\": {\n", + " \"areaname\": \"areacella\",\n", + " \"landmaskname\": \"sftlf\",\n", + " \"path + filename\": \"demo_data/CMIP5_demo_data/pr_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc\",\n", + " \"path + filename_area\": \"demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\",\n", + " \"path + filename_landmask\": \"demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\",\n", + " \"varname\": \"pr\"\n", + " },\n", + " \"sst\": {\n", + " \"areaname\": \"areacella\",\n", + " \"landmaskname\": \"sftlf\",\n", + " \"path + filename\": \"demo_data/CMIP5_demo_data/ts_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc\",\n", + " \"path + filename_area\": \"demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\",\n", + " \"path + filename_landmask\": \"demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\",\n", + " \"varname\": \"ts\"\n", + " }\n", + " }\n", + " },\n", + " \"observations\": {\n", + " \"ERA-INT\": {\n", + " \"pr\": {\n", + " \"areaname\": \"areacella\",\n", + " \"landmaskname\": \"sftlf\",\n", + " \"path + filename\": \"demo_data/PCMDIobs2/atmos/mon/pr/ERA-INT/gn/v20200707/pr_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc\",\n", + " \"path + filename_area\": null,\n", + " \"path + filename_landmask\": null,\n", + " \"varname\": \"pr\"\n", + " },\n", + " \"sst\": {\n", + " \"areaname\": \"areacella\",\n", + " \"landmaskname\": \"sftlf\",\n", + " \"path + filename\": \"demo_data/PCMDIobs2/atmos/mon/ts/ERA-INT/gn/v20200707/ts_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc\",\n", + " \"path + filename_area\": null,\n", + " \"path + filename_landmask\": null,\n", + " \"varname\": \"ts\"\n", + " }\n", + " },\n", + " \"GPCP-2-3\": {\n", + " \"pr\": {\n", + " \"areaname\": \"areacella\",\n", + " \"landmaskname\": \"sftlf\",\n", + " \"path + filename\": \"demo_data/PCMDIobs2/atmos/mon/pr/GPCP-2-3/gn/v20200707/pr_mon_GPCP-2-3_BE_gn_v20200707_197901-201907.nc\",\n", + " \"path + filename_area\": null,\n", + " \"path + filename_landmask\": null,\n", + " \"varname\": \"pr\"\n", + " }\n", + " }\n", + " }\n", + "}\n", + "\n", + "### Compute the metric collection ###\n", + "\n", + "\u001b[94m ComputeCollection: metric = EnsoAmpl\u001b[0m\n", + "\u001b[94m ComputeMetric: oneVarmetric = ACCESS1-0_r1i1p1\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageMeridional\u001b[0m\n", + "\u001b[94m ComputeMetric: oneVarmetric = ERA-INT\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageMeridional\u001b[0m\n", + "\u001b[94m ComputeCollection: metric = EnsoPrMapDjf\u001b[0m\n", + "\u001b[94m ComputeMetric: twoVarRMSmetric, EnsoPrMapDjf = ACCESS1-0_r1i1p1 and ERA-INT_ERA-INT\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[94m ComputeMetric: twoVarRMSmetric, EnsoPrMapDjf = ACCESS1-0_r1i1p1 and ERA-INT_GPCP-2-3\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[94m ComputeCollection: metric = EnsoPrMapJja\u001b[0m\n", + "\u001b[94m ComputeMetric: twoVarRMSmetric, EnsoPrMapJja = ACCESS1-0_r1i1p1 and ERA-INT_ERA-INT\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[94m ComputeMetric: twoVarRMSmetric, EnsoPrMapJja = ACCESS1-0_r1i1p1 and ERA-INT_GPCP-2-3\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[94m ComputeCollection: metric = EnsoSeasonality\u001b[0m\n", + "\u001b[94m ComputeMetric: oneVarmetric = ACCESS1-0_r1i1p1\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageMeridional\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageMeridional\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageMeridional\u001b[0m\n", + "\u001b[94m ComputeMetric: oneVarmetric = ERA-INT\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageMeridional\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageMeridional\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageMeridional\u001b[0m\n", + "\u001b[94m ComputeCollection: metric = EnsoSstLonRmse\u001b[0m\n", + "\u001b[94m ComputeMetric: oneVarRMSmetric, EnsoSstLonRmse = ACCESS1-0_r1i1p1 and ERA-INT\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageMeridional\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageMeridional\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[94m ComputeCollection: metric = EnsoSstMapDjf\u001b[0m\n", + "\u001b[94m ComputeMetric: oneVarRMSmetric, EnsoSstMapDjf = ACCESS1-0_r1i1p1 and ERA-INT\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[94m ComputeCollection: metric = EnsoSstMapJja\u001b[0m\n", + "\u001b[94m ComputeMetric: oneVarRMSmetric, EnsoSstMapJja = ACCESS1-0_r1i1p1 and ERA-INT\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "PMPdriver: model loop end\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/Users/ordonez4/miniconda3/envs/test_pcmdi/lib/python3.8/site-packages/cdms2/MV2.py:318: Warning: arguments order for compress function has changed\n", + "it is now: MV2.copmress(array,condition), if your code seems to not react or act wrong to a call to compress, please check this\n", + " warnings.warn(\n", + "INFO::2021-06-04 14:29::pcmdi_metrics:: Results saved to a json file: /Users/ordonez4/Documents/git/pcmdi_metrics/doc/jupyter/Demo/demo_output/basicTestEnso/ENSO_tel/cmip5_historical_ENSO_tel_basicTestEnso_ACCESS1-0_r1i1p1.json\n", + "INFO::2021-06-04 14:29::pcmdi_metrics:: Results saved to a json file: /Users/ordonez4/Documents/git/pcmdi_metrics/doc/jupyter/Demo/demo_output/basicTestEnso/ENSO_tel/cmip5_historical_ENSO_tel_basicTestEnso_ACCESS1-0_r1i1p1_diveDown.json\n" + ] + } + ], + "source": [ + "%%bash\n", + "enso_driver.py -p basic_enso_param.py \\\n", + "--metricsCollection ENSO_tel \\\n", + "--results_dir 'demo_output/basicTestEnso/ENSO_tel' \\\n", + "--nc_out True" + ] + }, + { + "cell_type": "markdown", + "id": "da0956e3", + "metadata": {}, + "source": [ + "All of the results (netCDF and JSON) are located in the output directory, which uses the metrics collection name." + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "d297d2e5", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "demo_output/basicTestEnso/ENSO_tel/cmip5_historical_ENSO_tel_basicTestEnso_ACCESS1-0_r1i1p1_EnsoAmpl.nc\r\n", + "demo_output/basicTestEnso/ENSO_tel/cmip5_historical_ENSO_tel_basicTestEnso_ACCESS1-0_r1i1p1_EnsoPrMapDjf.nc\r\n", + "demo_output/basicTestEnso/ENSO_tel/cmip5_historical_ENSO_tel_basicTestEnso_ACCESS1-0_r1i1p1_EnsoPrMapJja.nc\r\n", + "demo_output/basicTestEnso/ENSO_tel/cmip5_historical_ENSO_tel_basicTestEnso_ACCESS1-0_r1i1p1_EnsoSeasonality.nc\r\n", + "demo_output/basicTestEnso/ENSO_tel/cmip5_historical_ENSO_tel_basicTestEnso_ACCESS1-0_r1i1p1_EnsoSstLonRmse.nc\r\n", + "demo_output/basicTestEnso/ENSO_tel/cmip5_historical_ENSO_tel_basicTestEnso_ACCESS1-0_r1i1p1_EnsoSstMapDjf.nc\r\n", + "demo_output/basicTestEnso/ENSO_tel/cmip5_historical_ENSO_tel_basicTestEnso_ACCESS1-0_r1i1p1_EnsoSstMapJja.nc\r\n" + ] + } + ], + "source": [ + "!ls {demo_output_directory + \"/basicTestEnso/ENSO_tel/*.nc\"}" + ] + }, + { + "cell_type": "markdown", + "id": "2cd7ff92", + "metadata": {}, + "source": [ + "Finally, this example runs the remaining metrics collection ENSO_proc:" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "4ec6cdde", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "mip: cmip5\n", + "exp: historical\n", + "models: ['ACCESS1-0']\n", + "realization: r1i1p1\n", + "mc_name: ENSO_proc\n", + "outdir: demo_output/basicTestEnso/ENSO_proc\n", + "netcdf_path: demo_output/basicTestEnso/ENSO_proc\n", + "debug: False\n", + "obs_cmor: True\n", + "obs_cmor_path: demo_data/PCMDIobs2\n", + "egg_pth: /Users/ordonez4/Library/Caches/Python-Eggs/pcmdi_metrics-v1.2.1_690_g281220b-py3.8.egg-tmp/share/pmp\n", + "output directory for graphics:demo_output/basicTestEnso/ENSO_proc\n", + "output directory for diagnostic_results:demo_output/basicTestEnso/ENSO_proc\n", + "output directory for metrics_results:demo_output/basicTestEnso/ENSO_proc\n", + "list_variables: ['ssh', 'sst', 'taux', 'thf']\n", + "list_obs: ['AVISO-1-0', 'ERA-INT', 'GPCP-2-3', 'HadISST-1-1']\n", + "\u001b[95mObservation datasetAVISO-1-0 is not given for variable thf\u001b[0m\n", + "\u001b[95mObservation datasetGPCP-2-3 is not given for variable thf\u001b[0m\n", + "\u001b[95mObservation datasetHadISST-1-1 is not given for variable thf\u001b[0m\n", + "PMPdriver: dict_obs readin end\n", + "models: ['ACCESS1-0']\n", + " ----- model: ACCESS1-0 ---------------------\n", + "PMPdriver: var loop start for model ACCESS1-0\n", + "realization: r1i1p1\n", + " --- run: r1i1p1 ---\n", + " --- var: ssh ---\n", + "var_in_file: zos\n", + "var, areacell_in_file, realm: ssh areacello ocean\n", + "path: demo_data/CMIP5_demo_data/zos_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc\n", + "file_list: []\n", + "path: demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\n", + "file_list: ['demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc']\n", + "path: demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\n", + "file_list: ['demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc']\n", + "PMPdriver: var loop end\n", + " --- var: sst ---\n", + "var_in_file: ts\n", + "var, areacell_in_file, realm: sst areacella atmos\n", + "path: demo_data/CMIP5_demo_data/ts_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc\n", + "file_list: ['demo_data/CMIP5_demo_data/ts_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc']\n", + "path: demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\n", + "file_list: ['demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc']\n", + "path: demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\n", + "file_list: ['demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc']\n", + "PMPdriver: var loop end\n", + " --- var: taux ---\n", + "var_in_file: tauu\n", + "var, areacell_in_file, realm: taux areacella atmos\n", + "path: demo_data/CMIP5_demo_data/tauu_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc\n", + "file_list: ['demo_data/CMIP5_demo_data/tauu_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc']\n", + "path: demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\n", + "file_list: ['demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc']\n", + "path: demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\n", + "file_list: ['demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc']\n", + "PMPdriver: var loop end\n", + " --- var: thf ---\n", + "var_in_file: ['hfls', 'hfss', 'rlds', 'rlus', 'rsds', 'rsus']\n", + "var, areacell_in_file, realm: thf areacella atmos\n", + "path: demo_data/CMIP5_demo_data/hfls_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc\n", + "file_list: ['demo_data/CMIP5_demo_data/hfls_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc']\n", + "path: demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\n", + "file_list: ['demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc']\n", + "path: demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\n", + "file_list: ['demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc']\n", + "path: demo_data/CMIP5_demo_data/hfls_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc\n", + "file_list: ['demo_data/CMIP5_demo_data/hfls_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc']\n", + "path: demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\n", + "file_list: ['demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc']\n", + "file_areacell_tmp: demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\n", + "path: demo_data/CMIP5_demo_data/hfss_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc\n", + "file_list: ['demo_data/CMIP5_demo_data/hfss_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc']\n", + "path: demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\n", + "file_list: ['demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc']\n", + "file_areacell_tmp: demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\n", + "path: demo_data/CMIP5_demo_data/rlds_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc\n", + "file_list: ['demo_data/CMIP5_demo_data/rlds_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc']\n", + "path: demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\n", + "file_list: ['demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc']\n", + "file_areacell_tmp: demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\n", + "path: demo_data/CMIP5_demo_data/rlus_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc\n", + "file_list: ['demo_data/CMIP5_demo_data/rlus_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc']\n", + "path: demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\n", + "file_list: ['demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc']\n", + "file_areacell_tmp: demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\n", + "path: demo_data/CMIP5_demo_data/rsds_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc\n", + "file_list: ['demo_data/CMIP5_demo_data/rsds_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc']\n", + "path: demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\n", + "file_list: ['demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc']\n", + "file_areacell_tmp: demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\n", + "path: demo_data/CMIP5_demo_data/rsus_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc\n", + "file_list: ['demo_data/CMIP5_demo_data/rsus_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc']\n", + "path: demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\n", + "file_list: ['demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc']\n", + "file_areacell_tmp: demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\n", + "PMPdriver: var loop end\n", + "dictDatasets:\n", + "{\n", + " \"model\": {\n", + " \"ACCESS1-0_r1i1p1\": {\n", + " \"ssh\": {\n", + " \"areaname\": \"areacello\",\n", + " \"landmaskname\": \"sftlf\",\n", + " \"path + filename\": null,\n", + " \"path + filename_area\": \"demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\",\n", + " \"path + filename_landmask\": null,\n", + " \"varname\": \"zos\"\n", + " },\n", + " \"sst\": {\n", + " \"areaname\": \"areacella\",\n", + " \"landmaskname\": \"sftlf\",\n", + " \"path + filename\": \"demo_data/CMIP5_demo_data/ts_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc\",\n", + " \"path + filename_area\": \"demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\",\n", + " \"path + filename_landmask\": \"demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\",\n", + " \"varname\": \"ts\"\n", + " },\n", + " \"taux\": {\n", + " \"areaname\": \"areacella\",\n", + " \"landmaskname\": \"sftlf\",\n", + " \"path + filename\": \"demo_data/CMIP5_demo_data/tauu_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc\",\n", + " \"path + filename_area\": \"demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\",\n", + " \"path + filename_landmask\": \"demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\",\n", + " \"varname\": \"tauu\"\n", + " },\n", + " \"thf\": {\n", + " \"areaname\": [\n", + " \"areacella\",\n", + " \"areacella\",\n", + " \"areacella\",\n", + " \"areacella\",\n", + " \"areacella\",\n", + " \"areacella\"\n", + " ],\n", + " \"landmaskname\": [\n", + " \"sftlf\",\n", + " \"sftlf\",\n", + " \"sftlf\",\n", + " \"sftlf\",\n", + " \"sftlf\",\n", + " \"sftlf\"\n", + " ],\n", + " \"path + filename\": [\n", + " \"demo_data/CMIP5_demo_data/hfls_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc\",\n", + " \"demo_data/CMIP5_demo_data/hfss_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc\",\n", + " \"demo_data/CMIP5_demo_data/rlds_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc\",\n", + " \"demo_data/CMIP5_demo_data/rlus_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc\",\n", + " \"demo_data/CMIP5_demo_data/rsds_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc\",\n", + " \"demo_data/CMIP5_demo_data/rsus_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc\"\n", + " ],\n", + " \"path + filename_area\": [\n", + " \"demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\",\n", + " \"demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\",\n", + " \"demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\",\n", + " \"demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\",\n", + " \"demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\",\n", + " \"demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\"\n", + " ],\n", + " \"path + filename_landmask\": [\n", + " \"demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\",\n", + " \"demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\",\n", + " \"demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\",\n", + " \"demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\",\n", + " \"demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\",\n", + " \"demo_data/CMIP5_demo_data/sftlf_fx_ACCESS1-0_amip_r0i0p0.nc\"\n", + " ],\n", + " \"varname\": [\n", + " \"hfls\",\n", + " \"hfss\",\n", + " \"rlds\",\n", + " \"rlus\",\n", + " \"rsds\",\n", + " \"rsus\"\n", + " ]\n", + " }\n", + " }\n", + " },\n", + " \"observations\": {\n", + " \"AVISO-1-0\": {\n", + " \"ssh\": {\n", + " \"areaname\": \"areacella\",\n", + " \"landmaskname\": \"sftlf\",\n", + " \"path + filename\": \"demo_data/PCMDIobs2/ocean/mon/zos/AVISO-1-0/gn/v20210422/zos_mon_AVISO-1-0_BE_gn_v20210422_199301-201912.nc\",\n", + " \"path + filename_area\": null,\n", + " \"path + filename_landmask\": null,\n", + " \"varname\": \"zos\"\n", + " }\n", + " },\n", + " \"ERA-INT\": {\n", + " \"sst\": {\n", + " \"areaname\": \"areacella\",\n", + " \"landmaskname\": \"sftlf\",\n", + " \"path + filename\": \"demo_data/PCMDIobs2/atmos/mon/ts/ERA-INT/gn/v20200707/ts_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc\",\n", + " \"path + filename_area\": null,\n", + " \"path + filename_landmask\": null,\n", + " \"varname\": \"ts\"\n", + " },\n", + " \"taux\": {\n", + " \"areaname\": \"areacella\",\n", + " \"landmaskname\": \"sftlf\",\n", + " \"path + filename\": \"demo_data/PCMDIobs2/atmos/mon/tauu/ERA-INT/gn/v20200707/tauu_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc\",\n", + " \"path + filename_area\": null,\n", + " \"path + filename_landmask\": null,\n", + " \"varname\": \"tauu\"\n", + " },\n", + " \"thf\": {\n", + " \"areaname\": [\n", + " \"areacella\",\n", + " \"areacella\",\n", + " \"areacella\",\n", + " \"areacella\",\n", + " \"areacella\",\n", + " \"areacella\"\n", + " ],\n", + " \"landmaskname\": [\n", + " \"sftlf\",\n", + " \"sftlf\",\n", + " \"sftlf\",\n", + " \"sftlf\",\n", + " \"sftlf\",\n", + " \"sftlf\"\n", + " ],\n", + " \"path + filename\": [\n", + " \"demo_data/PCMDIobs2/atmos/mon/rsus/ERA-INT/gn/v20200707/rsus_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc\"\n", + " ],\n", + " \"path + filename_area\": [\n", + " null,\n", + " null,\n", + " null,\n", + " null,\n", + " null,\n", + " null\n", + " ],\n", + " \"path + filename_landmask\": null,\n", + " \"varname\": [\n", + " \"hfls\",\n", + " \"hfss\",\n", + " \"rlds\",\n", + " \"rlus\",\n", + " \"rsds\",\n", + " \"rsus\"\n", + " ]\n", + " }\n", + " }\n", + " }\n", + "}\n", + "\n", + "### Compute the metric collection ###\n", + "\n", + "\u001b[94m ComputeCollection: metric = BiasSstLonRmse\u001b[0m\n", + "\u001b[94m ComputeMetric: oneVarRMSmetric, BiasSstLonRmse = ACCESS1-0_r1i1p1 and ERA-INT\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageMeridional\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageMeridional\u001b[0m\n", + "\u001b[94m ComputeCollection: metric = BiasTauxLonRmse\u001b[0m\n", + "\u001b[94m ComputeMetric: oneVarRMSmetric, BiasTauxLonRmse = ACCESS1-0_r1i1p1 and ERA-INT\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageHorizontal\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageHorizontal\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageMeridional\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageMeridional\u001b[0m\n", + "\u001b[94m ComputeCollection: metric = EnsoAmpl\u001b[0m\n", + "\u001b[94m ComputeMetric: oneVarmetric = ACCESS1-0_r1i1p1\u001b[0m\n", + "\u001b[94m ComputeMetric: oneVarmetric = ERA-INT\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[94m ComputeCollection: metric = EnsodSstOce_2\u001b[0m\n", + "\u001b[94m ComputeMetric: twoVarmetric = ACCESS1-0_r1i1p1\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib ReadAndSelectRegion\u001b[0m\n", + "\u001b[93m hfls sign reversed\u001b[0m\n", + "\u001b[93m range old = +1.99 to +243.32\u001b[0m\n", + "\u001b[93m range new = -243.32 to -1.99\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib ReadAndSelectRegion\u001b[0m\n", + "\u001b[93m hfss sign reversed\u001b[0m\n", + "\u001b[93m range old = -3.28 to +31.07\u001b[0m\n", + "\u001b[93m range new = -31.07 to +3.28\u001b[0m\n", + "\u001b[94m ComputeMetric: twoVarmetric = ERA-INT_ERA-INT\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "No such variable, hfls\n", + "\u001b[94m ComputeCollection: metric = EnsoFbSshSst\u001b[0m\n", + "\u001b[94m ComputeCollection: ENSO_proc, metric EnsoFbSshSst not computed\u001b[0m\n", + "\u001b[94m reason(s):\u001b[0m\n", + "\u001b[94m no modeled ssh given\u001b[0m\n", + "\u001b[94m ComputeCollection: metric = EnsoFbSstTaux\u001b[0m\n", + "\u001b[94m ComputeMetric: twoVarmetric = ACCESS1-0_r1i1p1\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageHorizontal\u001b[0m\n", + "\u001b[94m ComputeMetric: twoVarmetric = ERA-INT_ERA-INT\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageHorizontal\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[94m ComputeCollection: metric = EnsoFbSstThf\u001b[0m\n", + "\u001b[94m ComputeMetric: twoVarmetric = ACCESS1-0_r1i1p1\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib ReadAndSelectRegion\u001b[0m\n", + "\u001b[93m hfls sign reversed\u001b[0m\n", + "\u001b[93m range old = +1.99 to +243.32\u001b[0m\n", + "\u001b[93m range new = -243.32 to -1.99\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib ReadAndSelectRegion\u001b[0m\n", + "\u001b[93m hfss sign reversed\u001b[0m\n", + "\u001b[93m range old = -3.28 to +31.07\u001b[0m\n", + "\u001b[93m range new = -31.07 to +3.28\u001b[0m\n", + "\u001b[94m ComputeMetric: twoVarmetric = ERA-INT_ERA-INT\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "No such variable, hfls\n", + "\u001b[94m ComputeCollection: metric = EnsoFbTauxSsh\u001b[0m\n", + "\u001b[94m ComputeCollection: ENSO_proc, metric EnsoFbTauxSsh not computed\u001b[0m\n", + "\u001b[94m reason(s):\u001b[0m\n", + "\u001b[94m no modeled ssh given\u001b[0m\n", + "\u001b[94m ComputeCollection: metric = EnsoSeasonality\u001b[0m\n", + "\u001b[94m ComputeMetric: oneVarmetric = ACCESS1-0_r1i1p1\u001b[0m\n", + "\u001b[94m ComputeMetric: oneVarmetric = ERA-INT\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[94m ComputeCollection: metric = EnsoSstLonRmse\u001b[0m\n", + "\u001b[94m ComputeMetric: oneVarRMSmetric, EnsoSstLonRmse = ACCESS1-0_r1i1p1 and ERA-INT\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageMeridional\u001b[0m\n", + "\u001b[93m EnsoUvcdatToolsLib AverageMeridional\u001b[0m\n", + "\u001b[94m ComputeCollection: metric = EnsoSstSkew\u001b[0m\n", + "\u001b[94m ComputeMetric: oneVarmetric = ACCESS1-0_r1i1p1\u001b[0m\n", + "\u001b[94m ComputeMetric: oneVarmetric = ERA-INT\u001b[0m\n", + "\u001b[93m NOTE: Estimated landmask applied\u001b[0m\n", + "PMPdriver: model loop end\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/Users/ordonez4/miniconda3/envs/test_pcmdi/lib/python3.8/site-packages/cdms2/MV2.py:318: Warning: arguments order for compress function has changed\n", + "it is now: MV2.copmress(array,condition), if your code seems to not react or act wrong to a call to compress, please check this\n", + " warnings.warn(\n", + "INFO::2021-06-04 14:32::pcmdi_metrics:: Results saved to a json file: /Users/ordonez4/Documents/git/pcmdi_metrics/doc/jupyter/Demo/demo_output/basicTestEnso/ENSO_proc/cmip5_historical_ENSO_proc_basicTestEnso_ACCESS1-0_r1i1p1.json\n", + "INFO::2021-06-04 14:32::pcmdi_metrics:: Results saved to a json file: /Users/ordonez4/Documents/git/pcmdi_metrics/doc/jupyter/Demo/demo_output/basicTestEnso/ENSO_proc/cmip5_historical_ENSO_proc_basicTestEnso_ACCESS1-0_r1i1p1_diveDown.json\n" + ] + } + ], + "source": [ + "%%bash\n", + "enso_driver.py -p basic_enso_param.py \\\n", + "--metricsCollection ENSO_proc \\\n", + "--results_dir 'demo_output/basicTestEnso/ENSO_proc'" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.8.10" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/doc/jupyter/Demo/basic_enso_param.py.in b/doc/jupyter/Demo/basic_enso_param.py.in new file mode 100644 index 000000000..8138e07fe --- /dev/null +++ b/doc/jupyter/Demo/basic_enso_param.py.in @@ -0,0 +1,30 @@ +import os + +# +# OPTIONS ARE SET BY USER IN THIS FILE AS INDICATED BELOW BY: +# +# + +# MODELS +modnames = ['ACCESS1-0'] +mip = 'cmip5' # cmip5, cmip6 +exp = 'historical' # historical, piControl +realization = 'r1i1p1' +modpath = '$INPUT_DIR$/CMIP5_demo_data/%(variable)_Amon_%(model)_historical_%(realization)_185001-200512.nc' +modpath_lf = '$INPUT_DIR$/CMIP5_demo_data/sftlf_fx_%(model)_amip_r0i0p0.nc' + +# OBSERVATIONS +obs_cmor = True +obs_cmor_path = "$INPUT_DIR$/PCMDIobs2" +obs_catalogue = "demo_data/pcmdiobs-CEM2020_monthly_bySource_catalogue_v20210422_demo.json" + +# METRICS COLLECTION +metricsCollection = 'ENSO_perf' # ENSO_perf, ENSO_tel, ENSO_proc + +# OUTPUT +case_id = 'basicTestEnso' +results_dir = os.path.join('$OUTPUT_DIR$',case_id, metricsCollection) + +json_name = '%(mip)_%(exp)_%(metricsCollection)_%(case_id)_%(model)_%(realization)' +netcdf_name = json_name +nc_out = False diff --git a/doc/jupyter/Demo/demo_data/pcmdiobs-CEM2020_monthly_bySource_catalogue_v20210422.json b/doc/jupyter/Demo/demo_data/pcmdiobs-CEM2020_monthly_bySource_catalogue_v20210422.json new file mode 100644 index 000000000..9248bfbb1 --- /dev/null +++ b/doc/jupyter/Demo/demo_data/pcmdiobs-CEM2020_monthly_bySource_catalogue_v20210422.json @@ -0,0 +1,504 @@ +{ + "20CR": { + "psl": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "fc21b9030f19abb3f752bd0bf8e42c00", + "RefName": "20CR", + "RefTrackingDate": "Tue Jul 7 11:22:07 2020", + "filename": "psl_mon_20CR_BE_gn_v20200707_187101-201212.nc", + "period": "187101-201212", + "shape": "(1704, 91, 180)", + "template": "atmos/mon/psl/20CR/gn/v20200707/psl_mon_20CR_BE_gn_v20200707_187101-201212.nc" + }, + "ts": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "b596b372edd1c5b861b026ad2630b301", + "RefName": "20CR", + "RefTrackingDate": "Tue Jul 7 11:22:11 2020", + "filename": "ts_mon_20CR_BE_gn_v20200707_187101-201212.nc", + "period": "187101-201212", + "shape": "(1704, 94, 192)", + "template": "atmos/mon/ts/20CR/gn/v20200707/ts_mon_20CR_BE_gn_v20200707_187101-201212.nc" + } + }, + "AVISO-1-0": { + "zos": { + "CMIP_CMOR_TABLE": "Omon", + "MD5sum": "50dfba202220048192f8b3092ac4f60b", + "RefName": "AVISO-1-0", + "RefTrackingDate": "Thu Apr 22 10:34:12 2021", + "filename": "zos_mon_AVISO-1-0_BE_gn_v20210422_199301-201912.nc", + "period": "199301-201912", + "shape": "(324, 720, 1440)", + "template": "ocean/mon/zos/AVISO-1-0/gn/v20210422/zos_mon_AVISO-1-0_BE_gn_v20210422_199301-201912.nc" + } + }, + "CERES-EBAF-4-1": { + "rlds": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "4dcf399726fd5071983dec93c9aa806d", + "RefName": "CERES-EBAF-4-1", + "RefTrackingDate": "Tue Jul 7 11:14:50 2020", + "filename": "rlds_mon_CERES-EBAF-4-1_BE_gn_v20200707_200301-201812.nc", + "period": "200301-201812", + "shape": "(192, 180, 360)", + "template": "atmos/mon/rlds/CERES-EBAF-4-1/gn/v20200707/rlds_mon_CERES-EBAF-4-1_BE_gn_v20200707_200301-201812.nc" + }, + "rldscs": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "51d912b2e4cdc94de13e640321f0d7c6", + "RefName": "CERES-EBAF-4-1", + "RefTrackingDate": "Tue Jul 7 11:14:52 2020", + "filename": "rldscs_mon_CERES-EBAF-4-1_BE_gn_v20200707_200301-201812.nc", + "period": "200301-201812", + "shape": "(192, 180, 360)", + "template": "atmos/mon/rldscs/CERES-EBAF-4-1/gn/v20200707/rldscs_mon_CERES-EBAF-4-1_BE_gn_v20200707_200301-201812.nc" + }, + "rltcre": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "0cd2d283b897e22462bed7a2bf92a31c", + "RefName": "CERES-EBAF-4-1", + "RefTrackingDate": "Tue Jul 7 11:14:40 2020", + "filename": "rltcre_mon_CERES-EBAF-4-1_BE_gn_v20200707_200301-201812.nc", + "period": "200301-201812", + "shape": "(192, 180, 360)", + "template": "atmos/mon/rltcre/CERES-EBAF-4-1/gn/v20200707/rltcre_mon_CERES-EBAF-4-1_BE_gn_v20200707_200301-201812.nc" + }, + "rlus": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "3ce5153e94312058f829d0da1c5a45f3", + "RefName": "CERES-EBAF-4-1", + "RefTrackingDate": "Tue Jul 7 11:14:45 2020", + "filename": "rlus_mon_CERES-EBAF-4-1_BE_gn_v20200707_200301-201812.nc", + "period": "200301-201812", + "shape": "(192, 180, 360)", + "template": "atmos/mon/rlus/CERES-EBAF-4-1/gn/v20200707/rlus_mon_CERES-EBAF-4-1_BE_gn_v20200707_200301-201812.nc" + }, + "rlut": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "8534afbd8ab49e7a48b64b1f92c813f9", + "RefName": "CERES-EBAF-4-1", + "RefTrackingDate": "Tue Jul 7 11:14:31 2020", + "filename": "rlut_mon_CERES-EBAF-4-1_BE_gn_v20200707_200301-201812.nc", + "period": "200301-201812", + "shape": "(192, 180, 360)", + "template": "atmos/mon/rlut/CERES-EBAF-4-1/gn/v20200707/rlut_mon_CERES-EBAF-4-1_BE_gn_v20200707_200301-201812.nc" + }, + "rlutcs": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "624a294db3d290a4b90c196d921da1a2", + "RefName": "CERES-EBAF-4-1", + "RefTrackingDate": "Tue Jul 7 11:14:35 2020", + "filename": "rlutcs_mon_CERES-EBAF-4-1_BE_gn_v20200707_200301-201812.nc", + "period": "200301-201812", + "shape": "(192, 180, 360)", + "template": "atmos/mon/rlutcs/CERES-EBAF-4-1/gn/v20200707/rlutcs_mon_CERES-EBAF-4-1_BE_gn_v20200707_200301-201812.nc" + }, + "rsds": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "35c1391356cccb661f8e00bf822ca1d0", + "RefName": "CERES-EBAF-4-1", + "RefTrackingDate": "Tue Jul 7 11:14:54 2020", + "filename": "rsds_mon_CERES-EBAF-4-1_BE_gn_v20200707_200301-201812.nc", + "period": "200301-201812", + "shape": "(192, 180, 360)", + "template": "atmos/mon/rsds/CERES-EBAF-4-1/gn/v20200707/rsds_mon_CERES-EBAF-4-1_BE_gn_v20200707_200301-201812.nc" + }, + "rsdscs": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "35e852cabfd67a362461520083bfb37b", + "RefName": "CERES-EBAF-4-1", + "RefTrackingDate": "Tue Jul 7 11:14:55 2020", + "filename": "rsdscs_mon_CERES-EBAF-4-1_BE_gn_v20200707_200301-201812.nc", + "period": "200301-201812", + "shape": "(192, 180, 360)", + "template": "atmos/mon/rsdscs/CERES-EBAF-4-1/gn/v20200707/rsdscs_mon_CERES-EBAF-4-1_BE_gn_v20200707_200301-201812.nc" + }, + "rsdt": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "05171a406a531ebafa7c359e147d5338", + "RefName": "CERES-EBAF-4-1", + "RefTrackingDate": "Tue Jul 7 11:14:38 2020", + "filename": "rsdt_mon_CERES-EBAF-4-1_BE_gn_v20200707_200301-201812.nc", + "period": "200301-201812", + "shape": "(192, 180, 360)", + "template": "atmos/mon/rsdt/CERES-EBAF-4-1/gn/v20200707/rsdt_mon_CERES-EBAF-4-1_BE_gn_v20200707_200301-201812.nc" + }, + "rstcre": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "799c04d94aeb1661a51c577bed59de0d", + "RefName": "CERES-EBAF-4-1", + "RefTrackingDate": "Tue Jul 7 11:14:42 2020", + "filename": "rstcre_mon_CERES-EBAF-4-1_BE_gn_v20200707_200301-201812.nc", + "period": "200301-201812", + "shape": "(192, 180, 360)", + "template": "atmos/mon/rstcre/CERES-EBAF-4-1/gn/v20200707/rstcre_mon_CERES-EBAF-4-1_BE_gn_v20200707_200301-201812.nc" + }, + "rsus": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "b24d205b00cfd7de7f4060ac2d36db61", + "RefName": "CERES-EBAF-4-1", + "RefTrackingDate": "Tue Jul 7 11:14:47 2020", + "filename": "rsus_mon_CERES-EBAF-4-1_BE_gn_v20200707_200301-201812.nc", + "period": "200301-201812", + "shape": "(192, 180, 360)", + "template": "atmos/mon/rsus/CERES-EBAF-4-1/gn/v20200707/rsus_mon_CERES-EBAF-4-1_BE_gn_v20200707_200301-201812.nc" + }, + "rsuscs": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "4b4e154d921d522b8090c83467382b5b", + "RefName": "CERES-EBAF-4-1", + "RefTrackingDate": "Tue Jul 7 11:14:48 2020", + "filename": "rsuscs_mon_CERES-EBAF-4-1_BE_gn_v20200707_200301-201812.nc", + "period": "200301-201812", + "shape": "(192, 180, 360)", + "template": "atmos/mon/rsuscs/CERES-EBAF-4-1/gn/v20200707/rsuscs_mon_CERES-EBAF-4-1_BE_gn_v20200707_200301-201812.nc" + }, + "rsut": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "0b7201cb1ddbf22fac61bb41c2394f0c", + "RefName": "CERES-EBAF-4-1", + "RefTrackingDate": "Tue Jul 7 11:14:32 2020", + "filename": "rsut_mon_CERES-EBAF-4-1_BE_gn_v20200707_200301-201812.nc", + "period": "200301-201812", + "shape": "(192, 180, 360)", + "template": "atmos/mon/rsut/CERES-EBAF-4-1/gn/v20200707/rsut_mon_CERES-EBAF-4-1_BE_gn_v20200707_200301-201812.nc" + }, + "rsutcs": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "fc6ad09a8867e028ff7795f1c5f7f01c", + "RefName": "CERES-EBAF-4-1", + "RefTrackingDate": "Tue Jul 7 11:14:34 2020", + "filename": "rsutcs_mon_CERES-EBAF-4-1_BE_gn_v20200707_200301-201812.nc", + "period": "200301-201812", + "shape": "(192, 180, 360)", + "template": "atmos/mon/rsutcs/CERES-EBAF-4-1/gn/v20200707/rsutcs_mon_CERES-EBAF-4-1_BE_gn_v20200707_200301-201812.nc" + }, + "rt": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "279d926f651627bf657d1bd018ca21fa", + "RefName": "CERES-EBAF-4-1", + "RefTrackingDate": "Tue Jul 7 11:14:37 2020", + "filename": "rt_mon_CERES-EBAF-4-1_BE_gn_v20200707_200301-201812.nc", + "period": "200301-201812", + "shape": "(192, 180, 360)", + "template": "atmos/mon/rt/CERES-EBAF-4-1/gn/v20200707/rt_mon_CERES-EBAF-4-1_BE_gn_v20200707_200301-201812.nc" + } + }, + "CMAP-V1902": { + "pr": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "6e723645101fd3cb0dd2bca7c101a1b1", + "RefName": "CMAP-V1902", + "RefTrackingDate": "Tue Jul 7 11:15:23 2020", + "filename": "pr_mon_CMAP-V1902_BE_gn_v20200707_197901-201901.nc", + "period": "197901-201901", + "shape": "(481, 72, 144)", + "template": "atmos/mon/pr/CMAP-V1902/gn/v20200707/pr_mon_CMAP-V1902_BE_gn_v20200707_197901-201901.nc" + } + }, + "ERA-20C": { + "psl": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "a71e5170399b06ce43578078779d808a", + "RefName": "ERA-20C", + "RefTrackingDate": "Tue Jul 7 11:22:24 2020", + "filename": "psl_mon_ERA-20C_BE_gn_v20200707_190001-201012.nc", + "period": "190001-201012", + "shape": "(1332, 181, 360)", + "template": "atmos/mon/psl/ERA-20C/gn/v20200707/psl_mon_ERA-20C_BE_gn_v20200707_190001-201012.nc" + }, + "ts": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "616aea2f5d5701a13602d9fc2595ce22", + "RefName": "ERA-20C", + "RefTrackingDate": "Tue Jul 7 11:22:36 2020", + "filename": "ts_mon_ERA-20C_BE_gn_v20200707_190001-201012.nc", + "period": "190001-201012", + "shape": "(1332, 181, 360)", + "template": "atmos/mon/ts/ERA-20C/gn/v20200707/ts_mon_ERA-20C_BE_gn_v20200707_190001-201012.nc" + } + }, + "ERA-5": { + "psl": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "2435c5923028d7b4ba0b1b6bbd6f70cc", + "RefName": "ERA-5", + "RefTrackingDate": "Tue Jul 7 12:16:11 2020", + "filename": "psl_mon_ERA-5_BE_gn_v20200707_197901-201912.nc", + "period": "197901-201912", + "shape": "(492, 721, 1440)", + "template": "atmos/mon/psl/ERA-5/gn/v20200707/psl_mon_ERA-5_BE_gn_v20200707_197901-201912.nc" + }, + "tas": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "05bdbf064dd7b410ba024abf8b18b5ac", + "RefName": "ERA-5", + "RefTrackingDate": "Tue Jul 7 15:16:15 2020", + "filename": "tas_mon_ERA-5_BE_1x1_v20200707_197901-201912.nc", + "period": "197901-201912", + "shape": "(492, 180, 360)", + "template": "atmos/mon/tas/ERA-5/1x1/v20200707/tas_mon_ERA-5_BE_1x1_v20200707_197901-201912.nc" + }, + "ts": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "473b2e16a273db44bb7c459fb3d48c90", + "RefName": "ERA-5", + "RefTrackingDate": "Tue Jul 7 12:18:28 2020", + "filename": "ts_mon_ERA-5_BE_gn_v20200707_197901-201912.nc", + "period": "197901-201912", + "shape": "(492, 721, 1440)", + "template": "atmos/mon/ts/ERA-5/gn/v20200707/ts_mon_ERA-5_BE_gn_v20200707_197901-201912.nc" + } + }, + "ERA-INT": { + "hfls": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "931115171bd83691053ada406daf60a3", + "RefName": "ERA-INT", + "RefTrackingDate": "Tue Jul 7 11:17:48 2020", + "filename": "hfls_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc", + "period": "197901-201903", + "shape": "(483, 241, 480)", + "template": "atmos/mon/hfls/ERA-INT/gn/v20200707/hfls_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc" + }, + "hfss": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "6aa1e0d9f239b68ab218b0155e23acdd", + "RefName": "ERA-INT", + "RefTrackingDate": "Tue Jul 7 11:17:57 2020", + "filename": "hfss_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc", + "period": "197901-201903", + "shape": "(483, 241, 480)", + "template": "atmos/mon/hfss/ERA-INT/gn/v20200707/hfss_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc" + }, + "pr": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "596bccd7f74acf87130da23360dbb433", + "RefName": "ERA-INT", + "RefTrackingDate": "Tue Jul 7 11:18:14 2020", + "filename": "pr_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc", + "period": "197901-201903", + "shape": "(483, 241, 480)", + "template": "atmos/mon/pr/ERA-INT/gn/v20200707/pr_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc" + }, + "psl": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "c25c6e044f72bf0bf04ce9f373b0b765", + "RefName": "ERA-INT", + "RefTrackingDate": "Tue Jul 7 11:18:05 2020", + "filename": "psl_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc", + "period": "197901-201903", + "shape": "(483, 241, 480)", + "template": "atmos/mon/psl/ERA-INT/gn/v20200707/psl_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc" + }, + "rlds": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "76dcae175a8ff42777845d99263e93ea", + "RefName": "ERA-INT", + "RefTrackingDate": "Tue Jul 7 11:18:30 2020", + "filename": "rlds_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc", + "period": "197901-201903", + "shape": "(483, 241, 480)", + "template": "atmos/mon/rlds/ERA-INT/gn/v20200707/rlds_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc" + }, + "rlus": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "b39190423e9ec4874da0bb0672670a79", + "RefName": "ERA-INT", + "RefTrackingDate": "Tue Jul 7 11:18:22 2020", + "filename": "rlus_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc", + "period": "197901-201903", + "shape": "(483, 241, 480)", + "template": "atmos/mon/rlus/ERA-INT/gn/v20200707/rlus_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc" + }, + "rsds": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "4b6d51361cfb64fa3c2c8832eac701af", + "RefName": "ERA-INT", + "RefTrackingDate": "Tue Jul 7 11:18:38 2020", + "filename": "rsds_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc", + "period": "197901-201903", + "shape": "(483, 241, 480)", + "template": "atmos/mon/rsds/ERA-INT/gn/v20200707/rsds_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc" + }, + "rsus": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "5148ad12cad32475f0c6102de3b44b80", + "RefName": "ERA-INT", + "RefTrackingDate": "Tue Jul 7 11:18:46 2020", + "filename": "rsus_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc", + "period": "197901-201903", + "shape": "(483, 241, 480)", + "template": "atmos/mon/rsus/ERA-INT/gn/v20200707/rsus_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc" + }, + "sfcWind": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "68c612decadbc820eb478297523b0c07", + "RefName": "ERA-INT", + "RefTrackingDate": "Tue Jul 7 11:18:55 2020", + "filename": "sfcWind_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc", + "period": "197901-201903", + "shape": "(483, 241, 480)", + "template": "atmos/mon/sfcWind/ERA-INT/gn/v20200707/sfcWind_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc" + }, + "tauu": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "b3f3638e211b5b0090565c0ce9859324", + "RefName": "ERA-INT", + "RefTrackingDate": "Tue Jul 7 11:19:22 2020", + "filename": "tauu_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc", + "period": "197901-201903", + "shape": "(483, 241, 480)", + "template": "atmos/mon/tauu/ERA-INT/gn/v20200707/tauu_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc" + }, + "tauv": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "b4979b789e23c1e4f9af92817c8b3889", + "RefName": "ERA-INT", + "RefTrackingDate": "Tue Jul 7 11:19:32 2020", + "filename": "tauv_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc", + "period": "197901-201903", + "shape": "(483, 241, 480)", + "template": "atmos/mon/tauv/ERA-INT/gn/v20200707/tauv_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc" + }, + "ts": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "fc8d87f5f3f791125b63570fdf23abf4", + "RefName": "ERA-INT", + "RefTrackingDate": "Tue Jul 7 11:19:39 2020", + "filename": "ts_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc", + "period": "197901-201903", + "shape": "(483, 241, 480)", + "template": "atmos/mon/ts/ERA-INT/gn/v20200707/ts_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc" + }, + "uas": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "9dd9c9d291fc7589277cea1ec848b387", + "RefName": "ERA-INT", + "RefTrackingDate": "Tue Jul 7 11:19:04 2020", + "filename": "uas_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc", + "period": "197901-201903", + "shape": "(483, 241, 480)", + "template": "atmos/mon/uas/ERA-INT/gn/v20200707/uas_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc" + }, + "vas": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "2e30ab7a3b3840a5e8649e9cce4aa1e9", + "RefName": "ERA-INT", + "RefTrackingDate": "Tue Jul 7 11:19:13 2020", + "filename": "vas_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc", + "period": "197901-201903", + "shape": "(483, 241, 480)", + "template": "atmos/mon/vas/ERA-INT/gn/v20200707/vas_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc" + } + }, + "GPCP-2-3": { + "pr": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "b6741c3f979b77a23509778f3a28403d", + "RefName": "GPCP-2-3", + "RefTrackingDate": "Tue Jul 7 11:15:25 2020", + "filename": "pr_mon_GPCP-2-3_BE_gn_v20200707_197901-201907.nc", + "period": "197901-201907", + "shape": "(487, 72, 144)", + "template": "atmos/mon/pr/GPCP-2-3/gn/v20200707/pr_mon_GPCP-2-3_BE_gn_v20200707_197901-201907.nc" + } + }, + "HadISST-1-1": { + "ts": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "4c983e9fc232f1348dc950abd1900ec5", + "RefName": "HadISST-1-1", + "RefTrackingDate": "Thu Apr 22 10:31:26 2021", + "filename": "ts_mon_HadISST-1-1_BE_gn_v20210422_187001-201907.nc", + "period": "187001-201907", + "shape": "(1795, 180, 360)", + "template": "atmos/mon/ts/HadISST-1-1/gn/v20210422/ts_mon_HadISST-1-1_BE_gn_v20210422_187001-201907.nc" + } + }, + "REMSS-PRW-v07r01": { + "sfcWind": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "3fb2c98eace4bc4998af5458ad354091", + "RefName": "REMSS-PRW-v07r01", + "RefTrackingDate": "Tue Jul 7 11:15:32 2020", + "filename": "sfcWind_mon_REMSS-PRW-v07r01_BE_gn_v20200707_198801-201812.nc", + "period": "198801-201812", + "shape": "(372, 180, 360)", + "template": "atmos/mon/sfcWind/REMSS-PRW-v07r01/gn/v20200707/sfcWind_mon_REMSS-PRW-v07r01_BE_gn_v20200707_198801-201812.nc" + } + }, + "TropFlux-1-0": { + "hfls": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "babdc4f8bb7f152ab4696ee1fb7ce1cd", + "RefName": "TropFlux-1-0", + "RefTrackingDate": "Thu Apr 22 10:06:09 2021", + "filename": "hfls_mon_TropFlux-1-0_BE_gn_v20210422_197901-201707.nc", + "period": "197901-201707", + "shape": "(463, 60, 350)", + "template": "atmos/mon/hfls/TropFlux-1-0/gn/v20210422/hfls_mon_TropFlux-1-0_BE_gn_v20210422_197901-201707.nc" + }, + "hfns": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "1672fa7b9846522b4b09a65e3517016e", + "RefName": "TropFlux-1-0", + "RefTrackingDate": "Thu Apr 22 10:06:05 2021", + "filename": "hfns_mon_TropFlux-1-0_BE_gn_v20210422_197901-201707.nc", + "period": "197901-201707", + "shape": "(463, 60, 350)", + "template": "atmos/mon/hfns/TropFlux-1-0/gn/v20210422/hfns_mon_TropFlux-1-0_BE_gn_v20210422_197901-201707.nc" + }, + "hfss": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "972c255c09545f70d12c82cb87e6dd28", + "RefName": "TropFlux-1-0", + "RefTrackingDate": "Thu Apr 22 10:06:13 2021", + "filename": "hfss_mon_TropFlux-1-0_BE_gn_v20210422_197901-201707.nc", + "period": "197901-201707", + "shape": "(463, 60, 350)", + "template": "atmos/mon/hfss/TropFlux-1-0/gn/v20210422/hfss_mon_TropFlux-1-0_BE_gn_v20210422_197901-201707.nc" + }, + "tas": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "1af826deba5eee25f968de23bdad6371", + "RefName": "TropFlux-1-0", + "RefTrackingDate": "Thu Apr 22 10:05:57 2021", + "filename": "tas_mon_TropFlux-1-0_BE_gn_v20210422_197901-201707.nc", + "period": "197901-201707", + "shape": "(463, 60, 350)", + "template": "atmos/mon/tas/TropFlux-1-0/gn/v20210422/tas_mon_TropFlux-1-0_BE_gn_v20210422_197901-201707.nc" + }, + "tauu": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "6347cbc998a572a5db3152c1ac74a5a8", + "RefName": "TropFlux-1-0", + "RefTrackingDate": "Thu Apr 22 10:06:17 2021", + "filename": "tauu_mon_TropFlux-1-0_BE_gn_v20210422_197901-201707.nc", + "period": "197901-201707", + "shape": "(463, 60, 350)", + "template": "atmos/mon/tauu/TropFlux-1-0/gn/v20210422/tauu_mon_TropFlux-1-0_BE_gn_v20210422_197901-201707.nc" + }, + "tauv": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "ef07080d7c131e8d10a1d380e5e4113f", + "RefName": "TropFlux-1-0", + "RefTrackingDate": "Thu Apr 22 10:06:21 2021", + "filename": "tauv_mon_TropFlux-1-0_BE_gn_v20210422_197901-201707.nc", + "period": "197901-201707", + "shape": "(463, 60, 350)", + "template": "atmos/mon/tauv/TropFlux-1-0/gn/v20210422/tauv_mon_TropFlux-1-0_BE_gn_v20210422_197901-201707.nc" + }, + "ts": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "0e4870e0a7c2b5091a100afa00740961", + "RefName": "TropFlux-1-0", + "RefTrackingDate": "Thu Apr 22 10:06:01 2021", + "filename": "ts_mon_TropFlux-1-0_BE_gn_v20210422_197901-201707.nc", + "period": "197901-201707", + "shape": "(463, 60, 350)", + "template": "atmos/mon/ts/TropFlux-1-0/gn/v20210422/ts_mon_TropFlux-1-0_BE_gn_v20210422_197901-201707.nc" + } + } +} diff --git a/doc/jupyter/Demo/demo_data/pcmdiobs-CEM2020_monthly_bySource_catalogue_v20210422_demo.json b/doc/jupyter/Demo/demo_data/pcmdiobs-CEM2020_monthly_bySource_catalogue_v20210422_demo.json new file mode 100644 index 000000000..ffc4eb28a --- /dev/null +++ b/doc/jupyter/Demo/demo_data/pcmdiobs-CEM2020_monthly_bySource_catalogue_v20210422_demo.json @@ -0,0 +1,180 @@ +{ + "AVISO-1-0": { + "zos": { + "CMIP_CMOR_TABLE": "Omon", + "MD5sum": "50dfba202220048192f8b3092ac4f60b", + "RefName": "AVISO-1-0", + "RefTrackingDate": "Thu Apr 22 10:34:12 2021", + "filename": "zos_mon_AVISO-1-0_BE_gn_v20210422_199301-201912.nc", + "period": "199301-201912", + "shape": "(324, 720, 1440)", + "template": "ocean/mon/zos/AVISO-1-0/gn/v20210422/zos_mon_AVISO-1-0_BE_gn_v20210422_199301-201912.nc" + } + }, + "ERA-INT": { + "hfls": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "931115171bd83691053ada406daf60a3", + "RefName": "ERA-INT", + "RefTrackingDate": "Tue Jul 7 11:17:48 2020", + "filename": "hfls_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc", + "period": "197901-201903", + "shape": "(483, 241, 480)", + "template": "atmos/mon/hfls/ERA-INT/gn/v20200707/hfls_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc" + }, + "hfss": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "6aa1e0d9f239b68ab218b0155e23acdd", + "RefName": "ERA-INT", + "RefTrackingDate": "Tue Jul 7 11:17:57 2020", + "filename": "hfss_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc", + "period": "197901-201903", + "shape": "(483, 241, 480)", + "template": "atmos/mon/hfss/ERA-INT/gn/v20200707/hfss_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc" + }, + "pr": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "596bccd7f74acf87130da23360dbb433", + "RefName": "ERA-INT", + "RefTrackingDate": "Tue Jul 7 11:18:14 2020", + "filename": "pr_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc", + "period": "197901-201903", + "shape": "(483, 241, 480)", + "template": "atmos/mon/pr/ERA-INT/gn/v20200707/pr_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc" + }, + "psl": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "c25c6e044f72bf0bf04ce9f373b0b765", + "RefName": "ERA-INT", + "RefTrackingDate": "Tue Jul 7 11:18:05 2020", + "filename": "psl_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc", + "period": "197901-201903", + "shape": "(483, 241, 480)", + "template": "atmos/mon/psl/ERA-INT/gn/v20200707/psl_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc" + }, + "rlds": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "76dcae175a8ff42777845d99263e93ea", + "RefName": "ERA-INT", + "RefTrackingDate": "Tue Jul 7 11:18:30 2020", + "filename": "rlds_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc", + "period": "197901-201903", + "shape": "(483, 241, 480)", + "template": "atmos/mon/rlds/ERA-INT/gn/v20200707/rlds_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc" + }, + "rlus": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "b39190423e9ec4874da0bb0672670a79", + "RefName": "ERA-INT", + "RefTrackingDate": "Tue Jul 7 11:18:22 2020", + "filename": "rlus_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc", + "period": "197901-201903", + "shape": "(483, 241, 480)", + "template": "atmos/mon/rlus/ERA-INT/gn/v20200707/rlus_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc" + }, + "rsds": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "4b6d51361cfb64fa3c2c8832eac701af", + "RefName": "ERA-INT", + "RefTrackingDate": "Tue Jul 7 11:18:38 2020", + "filename": "rsds_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc", + "period": "197901-201903", + "shape": "(483, 241, 480)", + "template": "atmos/mon/rsds/ERA-INT/gn/v20200707/rsds_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc" + }, + "rsus": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "5148ad12cad32475f0c6102de3b44b80", + "RefName": "ERA-INT", + "RefTrackingDate": "Tue Jul 7 11:18:46 2020", + "filename": "rsus_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc", + "period": "197901-201903", + "shape": "(483, 241, 480)", + "template": "atmos/mon/rsus/ERA-INT/gn/v20200707/rsus_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc" + }, + "sfcWind": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "68c612decadbc820eb478297523b0c07", + "RefName": "ERA-INT", + "RefTrackingDate": "Tue Jul 7 11:18:55 2020", + "filename": "sfcWind_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc", + "period": "197901-201903", + "shape": "(483, 241, 480)", + "template": "atmos/mon/sfcWind/ERA-INT/gn/v20200707/sfcWind_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc" + }, + "tauu": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "b3f3638e211b5b0090565c0ce9859324", + "RefName": "ERA-INT", + "RefTrackingDate": "Tue Jul 7 11:19:22 2020", + "filename": "tauu_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc", + "period": "197901-201903", + "shape": "(483, 241, 480)", + "template": "atmos/mon/tauu/ERA-INT/gn/v20200707/tauu_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc" + }, + "tauv": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "b4979b789e23c1e4f9af92817c8b3889", + "RefName": "ERA-INT", + "RefTrackingDate": "Tue Jul 7 11:19:32 2020", + "filename": "tauv_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc", + "period": "197901-201903", + "shape": "(483, 241, 480)", + "template": "atmos/mon/tauv/ERA-INT/gn/v20200707/tauv_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc" + }, + "ts": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "fc8d87f5f3f791125b63570fdf23abf4", + "RefName": "ERA-INT", + "RefTrackingDate": "Tue Jul 7 11:19:39 2020", + "filename": "ts_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc", + "period": "197901-201903", + "shape": "(483, 241, 480)", + "template": "atmos/mon/ts/ERA-INT/gn/v20200707/ts_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc" + }, + "uas": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "9dd9c9d291fc7589277cea1ec848b387", + "RefName": "ERA-INT", + "RefTrackingDate": "Tue Jul 7 11:19:04 2020", + "filename": "uas_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc", + "period": "197901-201903", + "shape": "(483, 241, 480)", + "template": "atmos/mon/uas/ERA-INT/gn/v20200707/uas_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc" + }, + "vas": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "2e30ab7a3b3840a5e8649e9cce4aa1e9", + "RefName": "ERA-INT", + "RefTrackingDate": "Tue Jul 7 11:19:13 2020", + "filename": "vas_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc", + "period": "197901-201903", + "shape": "(483, 241, 480)", + "template": "atmos/mon/vas/ERA-INT/gn/v20200707/vas_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc" + } + }, + "GPCP-2-3": { + "pr": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "b6741c3f979b77a23509778f3a28403d", + "RefName": "GPCP-2-3", + "RefTrackingDate": "Tue Jul 7 11:15:25 2020", + "filename": "pr_mon_GPCP-2-3_BE_gn_v20200707_197901-201907.nc", + "period": "197901-201907", + "shape": "(487, 72, 144)", + "template": "atmos/mon/pr/GPCP-2-3/gn/v20200707/pr_mon_GPCP-2-3_BE_gn_v20200707_197901-201907.nc" + } + }, + "HadISST-1-1": { + "ts": { + "CMIP_CMOR_TABLE": "Amon", + "MD5sum": "4c983e9fc232f1348dc950abd1900ec5", + "RefName": "HadISST-1-1", + "RefTrackingDate": "Thu Apr 22 10:31:26 2021", + "filename": "ts_mon_HadISST-1-1_BE_gn_v20210422_187001-201907.nc", + "period": "187001-201907", + "shape": "(1795, 180, 360)", + "template": "atmos/mon/ts/HadISST-1-1/gn/v20210422/ts_mon_HadISST-1-1_BE_gn_v20210422_187001-201907.nc" + } + } +} diff --git a/doc/jupyter/Demo/download_sample_data.py b/doc/jupyter/Demo/download_sample_data.py index f0f2fe535..b810e4fa9 100644 --- a/doc/jupyter/Demo/download_sample_data.py +++ b/doc/jupyter/Demo/download_sample_data.py @@ -1,13 +1,15 @@ import glob -def generate_parameter_files(demo_data_directory, demo_output_directory): +def generate_parameter_files(demo_data_directory, demo_output_directory, filenames=[]): # This prepares the various parameter files used in the demo notebooks # to reflect where you downloaded the data sub_dict = { "INPUT_DIR": demo_data_directory, "OUTPUT_DIR": demo_output_directory } - for name in glob.glob("*.in"): + if len(filenames) < 1: + filenames = glob.glob("*.in") + for name in filenames: with open(name) as template_file: print("Preparing parameter file: {}".format(name[:-3])) template = template_file.read() diff --git a/pcmdi_metrics/enso/__init__.py b/pcmdi_metrics/enso/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/pcmdi_metrics/enso/enso_driver.py b/pcmdi_metrics/enso/enso_driver.py new file mode 100755 index 000000000..1c88502ac --- /dev/null +++ b/pcmdi_metrics/enso/enso_driver.py @@ -0,0 +1,445 @@ +#!/usr/bin/env python +# ================================================= +# Dependencies +# ------------------------------------------------- +import cdms2 +import glob +import json +import os +import pkg_resources +import sys + +from genutil import StringConstructor +from pcmdi_metrics.enso.lib import AddParserArgument +from pcmdi_metrics.enso.lib import metrics_to_json +from pcmdi_metrics.enso.lib import find_realm, get_file +from pcmdi_metrics.enso.lib import CLIVAR_LargeEnsemble_Variables +from pcmdi_metrics.enso.lib import sort_human +from EnsoMetrics.EnsoCollectionsLib import CmipVariables, defCollection, ReferenceObservations +from EnsoMetrics.EnsoComputeMetricsLib import ComputeCollection + +# To avoid below error when using multi cores +# OpenBLAS blas_thread_init: pthread_create failed for thread XX of 96: Resource temporarily unavailable +os.environ['OPENBLAS_NUM_THREADS'] = '1' + +# ================================================= +# Collect user defined options +# ------------------------------------------------- +param = AddParserArgument() + +# Pre-defined options +mip = param.mip +exp = param.exp +print('mip:', mip) +print('exp:', exp) + +# Path to model data as string template +modpath = param.process_templated_argument("modpath") +modpath_lf = param.process_templated_argument("modpath_lf") + +# Check given model option +models = param.modnames + +# Include all models if conditioned +if ('all' in [m.lower() for m in models]) or (models == 'all'): + model_index_path = param.modpath.split('/')[-1].split('.').index("%(model)") + models = ([p.split('/')[-1].split('.')[model_index_path] for p in glob.glob(modpath( + mip=mip, exp=exp, model='*', realization='*', variable='ts'))]) + # remove duplicates + models = sorted(list(dict.fromkeys(models)), key=lambda s: s.lower()) + +print('models:', models) + +# Realizations +realization = param.realization +print('realization: ', realization) + +# Metrics Collection +mc_name = param.metricsCollection +dict_mc = defCollection(mc_name) +list_metric = sorted(dict_mc['metrics_list'].keys()) +print('mc_name:', mc_name) + +# case id +case_id = param.case_id + +# Output +outdir_template = param.process_templated_argument("results_dir") +outdir = StringConstructor(str(outdir_template( + output_type='%(output_type)', + mip=mip, exp=exp, metricsCollection=mc_name, case_id=case_id))) +netcdf_path = outdir(output_type='diagnostic_results') +json_name_template = param.process_templated_argument("json_name") +netcdf_name_template = param.process_templated_argument("netcdf_name") + +print('outdir:', str(outdir_template( + output_type='%(output_type)', + mip=mip, exp=exp, metricsCollection=mc_name))) +print('netcdf_path:', netcdf_path) + +# Switches +debug = param.debug +print('debug:', debug) + +obs_cmor = param.obs_cmor +print('obs_cmor:', obs_cmor) + +obs_cmor_path = param.obs_cmor_path +print('obs_cmor_path:', obs_cmor_path) + +obs_catalogue_json = param.obs_catalogue + +# ================================================= +# Prepare loop iteration +# ------------------------------------------------- +# Environmental setup +try: + egg_pth = pkg_resources.resource_filename( + pkg_resources.Requirement.parse("pcmdi_metrics"), "share/pmp") +except Exception: + egg_pth = os.path.join(sys.prefix, "share", "pmp") +print('egg_pth:', egg_pth) + +# Create output directory +for output_type in ['graphics', 'diagnostic_results', 'metrics_results']: + if not os.path.exists(outdir(output_type=output_type)): + os.makedirs(outdir(output_type=output_type)) + print('output directory for ' + output_type + ':' + + outdir(output_type=output_type)) + +# list of variables +list_variables = list() +for metric in list_metric: + listvar = dict_mc['metrics_list'][metric]['variables'] + for var in listvar: + if var not in list_variables: + list_variables.append(var) +list_variables = sorted(list_variables) +print('list_variables:', list_variables) + +# list of observations +list_obs = list() +if obs_cmor and obs_catalogue_json is not None: + with open(obs_catalogue_json) as jobs: + obs_catalogue_dict = json.load(jobs) + list_obs = list(obs_catalogue_dict.keys()) +else: + for metric in list_metric: + dict_var_obs = dict_mc['metrics_list'][metric]['obs_name'] + for var in dict_var_obs.keys(): + for obs in dict_var_obs[var]: + if obs not in list_obs: + list_obs.append(obs) +list_obs = sorted(list_obs) +print('list_obs:', list_obs) + +# +# finding file and variable name in file for each observations dataset +# +dict_obs = dict() + +for obs in list_obs: + if obs_cmor: + dict_var = CmipVariables()["variable_name_in_file"] + else: + # be sure to add your datasets to EnsoCollectionsLib.ReferenceObservations if needed + dict_var = ReferenceObservations(obs)['variable_name_in_file'] + + dict_obs[obs] = dict() + for var in list_variables: + # + # finding variable name in file + # + try: + var_in_file = dict_var[var]['var_name'] + except Exception: + print('\033[95m' + str(var) + " is not available for " + str(obs) + " or unscripted" + '\033[0m') + else: + if isinstance(var_in_file, list): + var0 = var_in_file[0] + else: + var0 = var_in_file + + try: + # finding file for 'obs', 'var' + if obs_cmor and obs_catalogue_json is not None: + if var0 in list(obs_catalogue_dict[obs].keys()): + file_name = os.path.join(obs_cmor_path, obs_catalogue_dict[obs][var0]["template"]) + if not os.path.isfile(file_name): + file_name = None + else: + file_name = None + + if debug: + print('file_name:', file_name) + else: + file_name = param.reference_data_path[obs].replace('VAR', var0) + file_areacell = None # temporary for now + try: + file_landmask = param.reference_data_lf_path[obs] + except Exception: + file_landmask = None + try: + areacell_in_file = dict_var['areacell']['var_name'] + except Exception: + areacell_in_file = None + try: + landmask_in_file = dict_var['landmask']['var_name'] + except Exception: + landmask_in_file = None + # if var_in_file is a list (like for thf) all variables should be read from the same realm + if isinstance(var_in_file, list): + list_files = list() + if obs_cmor and obs_catalogue_json is not None: + for var1 in var_in_file: + file_name1 = os.path.join(obs_cmor_path, obs_catalogue_dict[obs][var1]["template"]) + if not os.path.isfile(file_name1): + file_name1 = None + else: + for var1 in var_in_file: + file_name1 = param.reference_data_path[obs].replace('VAR', var1) + list_files.append(file_name1) + list_areacell = [file_areacell for var1 in var_in_file] + list_name_area = [areacell_in_file for var1 in var_in_file] + try: + list_landmask = [param.reference_data_lf_path[obs] for var1 in var_in_file] + except Exception: + list_landmask = None + list_name_land = [landmask_in_file for var1 in var_in_file] + else: + list_files = file_name + list_areacell = file_areacell + list_name_area = areacell_in_file + list_landmask = file_landmask + list_name_land = landmask_in_file + + if list_files is not None: + dict_obs[obs][var] = {'path + filename': list_files, 'varname': var_in_file, + 'path + filename_area': list_areacell, 'areaname': list_name_area, + 'path + filename_landmask': list_landmask, 'landmaskname': list_name_land} + except Exception: + print('\033[95m' + 'Observation dataset' + str(obs) + + " is not given for variable " + str(var) + '\033[0m') + + if len(list(dict_obs[obs].keys())) == 0: + del dict_obs[obs] + +print('PMPdriver: dict_obs readin end') + +# ================================================= +# Loop for Models +# ------------------------------------------------- +dict_metric, dict_dive = dict(), dict() + +print('models:', models) + +for mod in models: + print(' ----- model: ', mod, ' ---------------------') + print('PMPdriver: var loop start for model ', mod) + + # finding file and variable name in file for each observations dataset + if "CLIVAR_LE" == mip and mod in ['CESM1-CAM5']: + dict_var = CLIVAR_LargeEnsemble_Variables()['variable_name_in_file'] + else: + dict_var = CmipVariables()['variable_name_in_file'] + + dict_mod = {mod: {}} + dict_metric[mod], dict_dive[mod] = dict(), dict() + + realm, areacell_in_file = find_realm('ts', mip) + + model_path_list = glob.glob( + modpath(mip=mip, exp=exp, realm=realm, model=mod, realization=realization, variable='ts')) + + model_path_list = sort_human(model_path_list) + if debug: + print('modpath:', modpath(mip=mip, exp=exp, realm=realm, model=mod, realization=realization, variable='ts')) + print('model_path_list:', model_path_list) + + # Find where run can be gripped from given filename template for modpath + print('realization:', realization) + try: + if mip == "CLIVAR_LE": + inline_separator = '_' + else: + inline_separator = '.' + run_in_modpath = modpath(mip=mip, exp=exp, realm=realm, model=mod, realization=realization, + variable='ts').split('/')[-1].split(inline_separator).index(realization) + print('run_in_modpath:', run_in_modpath) + # Collect available runs + runs_list = [model_path.split('/')[-1].split(inline_separator)[run_in_modpath] + for model_path in model_path_list] + except Exception: + if realization not in ["all", "*"]: + runs_list = [realization] + + if debug: + print('runs_list:', runs_list) + + # ================================================= + # Loop for Realizations + # ------------------------------------------------- + for run in runs_list: + + print(' --- run: ', run, ' ---') + mod_run = '_'.join([mod, run]) + dict_mod = {mod_run: {}} + + if debug: + print('list_variables:', list_variables) + + try: + for var in list_variables: + print(' --- var: ', var, ' ---') + # finding variable name in file + var_in_file = dict_var[var]['var_name'] + print('var_in_file:', var_in_file) + if isinstance(var_in_file, list): + var0 = var_in_file[0] + else: + var0 = var_in_file + # finding variable type (atmos or ocean) + realm, areacell_in_file = find_realm(var0, mip) + if realm == 'Amon': + realm2 = 'atmos' + elif realm == 'Omon': + realm2 = 'ocean' + else: + realm2 = realm + print('var, areacell_in_file, realm:', var, areacell_in_file, realm) + # + # finding file for 'mod', 'var' + # + file_name = get_file(modpath(mip=mip, realm=realm, exp=exp, model=mod, realization=run, variable=var0)) + file_areacell = get_file(modpath_lf(mip=mip, realm=realm2, model=mod, variable=areacell_in_file)) + file_landmask = get_file(modpath_lf(mip=mip, realm=realm2, + model=mod, variable=dict_var['landmask']['var_name'])) + # -- TEMPORARY -- + if mip == 'cmip6': + if mod in ['IPSL-CM6A-LR', 'CNRM-CM6-1']: + file_landmask = ('/work/lee1043/ESGF/CMIP6/CMIP/' + mod + + '/sftlf_fx_' + mod + '_historical_r1i1p1f1_gr.nc') + elif mod in ['GFDL-ESM4']: + file_landmask = modpath_lf(mip=mip, realm="atmos", model='GFDL-CM4', + variable=dict_var['landmask']['var_name']) + if mip == 'cmip5': + if mod == "BNU-ESM": + # Incorrect latitude in original sftlf fixed + file_landmask = "/work/lee1043/ESGF/CMIP5/BNU-ESM/sftlf_fx_BNU-ESM_historical_r0i0p0.nc" + elif mod == "HadCM3": + # Inconsistent lat/lon between sftlf and other variables + file_landmask = None + # Inconsistent grid between areacella and tauu (probably staggering grid system) + file_areacell = None + # -- TEMPORARY END -- + """ + try: + areacell_in_file = dict_var['areacell']['var_name'] + except Exception: + areacell_in_file = None + """ + try: + landmask_in_file = dict_var['landmask']['var_name'] + except Exception: + landmask_in_file = None + + if isinstance(var_in_file, list): + list_areacell, list_files, list_landmask, list_name_area, list_name_land = \ + list(), list(), list(), list(), list() + for var1 in var_in_file: + realm, areacell_in_file = find_realm(var1, mip) + modpath_tmp = get_file(modpath(mip=mip, exp=exp, realm=realm, model=mod, + realization=realization, variable=var1)) + file_areacell_tmp = get_file(modpath_lf(mip=mip, realm=realm2, model=mod, + variable=areacell_in_file)) + print("file_areacell_tmp:", file_areacell_tmp) + list_files.append(modpath_tmp) + list_areacell.append(file_areacell_tmp) + list_name_area.append(areacell_in_file) + list_landmask.append(file_landmask) + list_name_land.append(landmask_in_file) + else: + list_files = file_name + list_areacell = file_areacell + list_name_area = areacell_in_file + list_landmask = file_landmask + list_name_land = landmask_in_file + + # Variable from ocean grid + if var in ['ssh']: + list_landmask = None + # Temporay control of areacello for models with zos on gr instead on gn + if mod in ['BCC-ESM1', 'CESM2', 'CESM2-FV2', 'CESM2-WACCM', 'CESM2-WACCM-FV2', + 'GFDL-CM4', 'GFDL-ESM4', 'MRI-ESM2-0', # cmip6 + 'BCC-CSM1-1', 'BCC-CSM1-1-M', 'GFDL-CM3', 'GISS-E2-R', + 'MRI-CGCM3']: # cmip5 + list_areacell = None + + dict_mod[mod_run][var] = { + 'path + filename': list_files, 'varname': var_in_file, + 'path + filename_area': list_areacell, 'areaname': list_name_area, + 'path + filename_landmask': list_landmask, 'landmaskname': list_name_land} + + print('PMPdriver: var loop end') + + # dictionary needed by EnsoMetrics.ComputeMetricsLib.ComputeCollection + dictDatasets = {'model': dict_mod, 'observations': dict_obs} + print('dictDatasets:') + print(json.dumps(dictDatasets, indent=4, sort_keys=True)) + + # regridding dictionary (only if you want to specify the regridding) + dict_regrid = {} + """ + # Usage of dict_regrid (select option as below): + dict_regrid = { + 'regridding': { + 'model_orand_obs': 2, 'regridder': 'cdms', 'regridTool': 'esmf', 'regridMethod': 'linear', + 'newgrid_name': 'generic 1x1deg'}, + } + """ + + # Prepare netcdf file setup + json_name = json_name_template(mip=mip, exp=exp, metricsCollection=mc_name, + case_id=case_id, model=mod, realization=run) + netcdf_name = netcdf_name_template(mip=mip, exp=exp, metricsCollection=mc_name, + case_id=case_id, model=mod, realization=run) + netcdf = os.path.join(netcdf_path, netcdf_name) + + if obs_cmor: + obs_interpreter = "CMIP" + else: + obs_interpreter = None + + # Computes the metric collection + print("\n### Compute the metric collection ###\n") + cdms2.setAutoBounds('on') + dict_metric[mod][run], dict_dive[mod][run] = ComputeCollection( + mc_name, dictDatasets, mod_run, netcdf=param.nc_out, + netcdf_name=netcdf, debug=debug, obs_interpreter=obs_interpreter) + + if debug: + print('file_name:', file_name) + print('list_files:', list_files) + print('netcdf_name:', netcdf_name) + print('json_name:', json_name) + print('dict_metric:') + print(json.dumps(dict_metric, indent=4, sort_keys=True)) + + # OUTPUT METRICS TO JSON FILE (per simulation) + metrics_to_json(mc_name, dict_obs, dict_metric, dict_dive, egg_pth, outdir, json_name, mod=mod, run=run) + + except Exception as e: + print('failed for ', mod, run) + print(e) + if not debug: + pass + +print('PMPdriver: model loop end') + +# ================================================= +# OUTPUT METRICS TO JSON FILE (for all simulations) +# ------------------------------------------------- +# json_name = json_name_template(mip=mip, exp=exp, metricsCollection=mc_name, model='all', realization='all') +# metrics_to_json(mc_name, dict_obs, dict_metric, dict_dive, egg_pth, outdir, json_name) + +sys.exit(0) diff --git a/pcmdi_metrics/enso/lib/__init__.py b/pcmdi_metrics/enso/lib/__init__.py new file mode 100644 index 000000000..ed0008278 --- /dev/null +++ b/pcmdi_metrics/enso/lib/__init__.py @@ -0,0 +1 @@ +from .enso_lib import AddParserArgument, tree, metrics_to_json, find_realm, get_file, CLIVAR_LargeEnsemble_Variables, sort_human # noqa diff --git a/pcmdi_metrics/enso/lib/enso_lib.py b/pcmdi_metrics/enso/lib/enso_lib.py new file mode 100755 index 000000000..ad586365e --- /dev/null +++ b/pcmdi_metrics/enso/lib/enso_lib.py @@ -0,0 +1,261 @@ +from __future__ import print_function +from collections import defaultdict +from pcmdi_metrics.driver.pmp_parser import PMPParser + +import collections +import copy +import datetime +import glob +import os +import pcmdi_metrics +import re + + +def AddParserArgument(): + + P = PMPParser() # Includes all default options + + P.add_argument("--mip", + type=str, + default="cmip5", + help="A WCRP MIP project such as CMIP3 and CMIP5") + P.add_argument("--exp", + type=str, + default="historical", + help="An experiment such as AMIP, historical or pi-contorl") + P.use("--modpath") + P.add_argument("--modpath_lf", + type=str, + dest='modpath_lf', + help="Directory path to model land fraction field") + P.add_argument("--modnames", + type=str, + nargs='+', + default=None, + help="List of models") + P.add_argument("-r", "--realization", + type=str, + default="r1i1p1", + help="Consider all accessible realizations as idividual\n" + "- r1i1p1: default, consider only 'r1i1p1' member\n" + " Or, specify realization, e.g, r3i1p1'\n" + "- *: consider all available realizations") + P.use("--reference_data_path") + P.add_argument("--reference_data_lf_path", + type=str, + dest='reference_data_lf_path', + help="Data path to land fraction of reference dataset") + P.add_argument("--metricsCollection", + type=str, + dest='metricsCollection', + default="ENSO_perf", + help="Metrics Collection e.g. ENSO_perf, ENSO_tel, or ENSO_proc") + P.add_argument("--json_name", + type=str, + dest='json_name', + help="File name for output JSON") + P.add_argument("--netcdf_name", + type=str, + dest='netcdf_name', + help="File name for output NetCDF") + P.use("--results_dir") + P.add_argument("--case_id", + type=str, + dest="case_id", + default="{:v%Y%m%d}".format(datetime.datetime.now()), + help="version as date, e.g., v20191116 (yyyy-mm-dd)") + P.add_argument("--obs_catalogue", + type=str, + default=None, + dest='obs_catalogue', + help="obs_catalogue JSON file for CMORized observation, default is None") + P.add_argument("--obs_cmor_path", + type=str, + default=None, + dest='obs_cmor_path', + help="Directory path for CMORized observation dataset, default is None") + # Switches + P.add_argument("-d", "--debug", nargs='?', + const=True, default=False, + type=bool, + help="Option for debug: True / False (defualt)") + P.add_argument("--obs_cmor", nargs='?', + const=True, default=False, + type=bool, + help="Use CMORized reference database?: True / False (defualt)") + P.add_argument("--nc_out", nargs='?', + const=True, default=True, + type=bool, + help="Option for generate netCDF file output: True (default) / False") + + param = P.get_parameter() + + return param + + +# Dictionary to save result +def tree(): return defaultdict(tree) + + +# Prepare outputing metrics to JSON file +def metrics_to_json(mc_name, dict_obs, dict_metric, dict_dive, egg_pth, outdir, json_name, mod=None, run=None): + # disclaimer and reference for JSON header + disclaimer = open( + os.path.join( + egg_pth, + "disclaimer.txt")).read() + + if mc_name == 'MC1': + reference = ("The statistics in this file are based on Bellenger et al. " + + "Clim Dyn (2014) 42:1999-2018. doi:10.1007/s00382-013-1783-z") + elif mc_name == 'ENSO_perf': + reference = "MC for ENSO Performance..." + elif mc_name == 'ENSO_tel': + reference = "MC for ENSO Teleconnection..." + elif mc_name == 'ENSO_proc': + reference = "MC for ENSO Process..." + else: + reference = mc_name + + enso_stat_dic = tree() # Use tree dictionary to avoid declearing everytime + + # First JSON for metrics results + enso_stat_dic['obs'] = dict_obs + if mod is not None and run is not None: + enso_stat_dic['model'][mod][run] = dict_metric[mod][run] + else: + enso_stat_dic['model'] = dict_metric + metrics_dictionary = collections.OrderedDict() + metrics_dictionary["DISCLAIMER"] = disclaimer + metrics_dictionary["REFERENCE"] = reference + metrics_dictionary["RESULTS"] = enso_stat_dic + + OUT = pcmdi_metrics.io.base.Base(outdir(output_type='metrics_results'), json_name+'.json') + OUT.write( + metrics_dictionary, + json_structure=["type", "data", "metric", "item", "value or description"], + indent=4, + separators=( + ',', + ': '), + sort_keys=True) + + # Second JSON for dive down information + diveDown_dictionary = collections.OrderedDict() + diveDown_dictionary["DISCLAIMER"] = disclaimer + diveDown_dictionary["REFERENCE"] = reference + diveDown_dictionary["RESULTS"] = {} + if mod is not None and run is not None: + diveDown_dictionary["RESULTS"]["model"] = {} + diveDown_dictionary["RESULTS"]["model"][mod] = {} + diveDown_dictionary["RESULTS"]["model"][mod][run] = {} + diveDown_dictionary["RESULTS"]["model"][mod][run] = dict_dive[mod][run] + else: + diveDown_dictionary["RESULTS"]["model"] = dict_dive + + OUT2 = pcmdi_metrics.io.base.Base(outdir(output_type='metrics_results'), json_name+'_diveDown.json') + OUT2.write( + dict_dive, + json_structure=["type", "data", "metric", "item", "value or description"], + indent=4, + separators=( + ',', + ': '), + sort_keys=True) + + +def find_realm(varname, mip): + if varname in ["tos", "tauuo", "zos", "areacello", "SSH", "ssh"]: + if mip == "CLIVAR_LE": + realm = "Omon" + else: + realm = "ocean" + areacell_in_file = "areacello" + else: + if mip == "CLIVAR_LE": + realm = "Amon" + else: + realm = "atmos" + areacell_in_file = "areacella" + return realm, areacell_in_file + + +def get_file(path): + file_list = glob.glob(path) + print("path: ", path) + print("file_list: ", file_list) + + if len(file_list) > 1: + print("Multiple files detected in get_file function. file_list: ", file_list) + path_to_return = sorted(file_list)[0] + elif len(file_list) == 1: + path_to_return = file_list[0] + elif len(file_list) == 0: + path_to_return = path + + if not os.path.isfile(path_to_return): + path_to_return = None + + return path_to_return + + +def CLIVAR_LargeEnsemble_Variables(): + dict_cmip_variables = { + 'reference': 'http://cfconventions.org/Data/cf-standard-names/46/build/cf-standard-name-table.html', + 'variable_name_in_file': { + # line keys: + # '':{'var_name':'','cf_name':, + # 'cf_unit':''} + # areacell + 'areacell': {'var_name': 'areacella', 'cf_name': 'cell_area', 'cf_units': 'm2'}, + # landmask + 'landmask': {'var_name': 'sftlf', 'cf_name': 'cell_area', 'cf_units': '1'}, + # latent heat flux (on ocean grid or ocean points only) + 'lhf': {'var_name': 'hfls', 'cf_name': 'surface_upward_latent_heat_flux', 'cf_units': 'W m-2'}, + # longwave radiation computed from these variables IN THAT ORDER (on ocean grid or ocean points only) + # lwr = rlds - rlus + # sometimes lwr is included in the datasets in a variable called 'rls' + 'lwr': {'var_name': 'flns', 'cf_name': 'net_longwave_flux_at_surface', 'cf_units': 'W m-2'}, + # Rainfall Flux + 'pr': {'var_name': 'pr', 'cf_name': 'precipitation_flux', 'cf_units': 'kg m-2 s-1'}, + # Sea Level Pressure + 'slp': {'var_name': 'psl', 'cf_name': 'air_pressure_at_sea_level', 'cf_units': 'Pa'}, + # sensible heat flux (on ocean grid or ocean points only) + 'shf': {'var_name': 'hfss', 'cf_name': 'surface_upward_sensible_heat_flux', 'cf_units': 'W m-2'}, + # sea surface height + 'ssh': {'var_name': 'ssh', 'cf_name': 'Sea Surface Height', 'cf_units': 'm'}, + # sea surface temperature + 'sst': {'var_name': 'ts', 'cf_name': 'surface_temperature', 'cf_units': 'K'}, + # shortwave radiation computed from these variables IN THAT ORDER + # swr = rsds - rsus + # sometimes swr is included in the datasets in a variable called 'rss' + 'swr': {'var_name': 'fsns', 'cf_name': 'net_shortwave_flux_at_surface', 'cf_units': 'W m-2'}, + # zonal surface wind stress + 'taux': {'var_name': 'tauu', 'cf_name': 'surface_downward_eastward_stress', 'cf_units': 'Pa'}, + # total heat flux computed from these variables IN THAT ORDER + # tfh = hfls + hfss + rlds - rlus + rsds - rsus + # sometimes rls = rlds - rlus and rss = rsds - rsus + # sometimes thf is included in the datasets in a variable called 'hfds', 'netflux', 'thflx',... + 'thf': { + 'var_name': ['hfls', 'hfss', 'flns', 'fsns'], + 'cf_name': ['surface_upward_latent_heat_flux', 'surface_upward_sensible_heat_flux', + 'net_longwave_flux_at_surface', + 'net_shortwave_flux_at_surface'], + 'cf_units': 'W m-2', 'algebric_calculation': ['plus', 'plus', 'plus', 'plus'] + }, + }, + } + return dict_cmip_variables + + +def sort_human(input_list): + lst = copy.copy(input_list) + + def convert(text): + return int(text) if text.isdigit() else text + + def alphanum(key): + return [convert(c) for c in re.split('([0-9]+)', key)] + + lst.sort(key=alphanum) + return lst diff --git a/pcmdi_metrics/enso/param/my_Param_ENSO.py b/pcmdi_metrics/enso/param/my_Param_ENSO.py new file mode 100755 index 000000000..5fda7d8bd --- /dev/null +++ b/pcmdi_metrics/enso/param/my_Param_ENSO.py @@ -0,0 +1,84 @@ +import datetime +import glob +import os + + +def find_latest(path): + dir_list = [p for p in glob.glob(path+"/v????????")] + return sorted(dir_list)[-1] + + +# ================================================= +# Background Information +# ------------------------------------------------- +mip = 'cmip6' # cmip5, cmip6 +exp = 'historical' # historical, piControl + +# ================================================= +# Miscellaneous +# ------------------------------------------------- +# debug = False +debug = True +nc_out = True + +# ================================================= +# Observation +# ------------------------------------------------- +reference_data_path = { + 'ERA-Interim': '/p/user_pub/PCMDIobs/PCMDIobs2/atmos/mon/VAR/ERA-INT/gn/v20200707/VAR_mon_ERA-INT_BE_gn_v20200707_197901-201903.nc', # noqa + 'HadISST': '/work/lee1043/DATA/HadISSTv1.1/HadISSTv1.1.xml', + 'OISST': '/work/lee1043/DATA/OISST/xmls/OISST_tos_mo.xml', + 'Tropflux': '/work/lee1043/DATA/TropFlux/monthly/xmls/Tropflux_VAR_mo.xml', + # 'Tropflux': '/p/user_pub/PCMDIobs/PCMDIobs2.0/atmos/mon/VAR/TropFlux-1-0/gn/v20190912/VAR_mon_TropFlux-1-0_BE_gn_197901-201707.nc', # noqa + # 'OAFlux': '/work/lee1043/DATA/OAFlux/xmls/OAFlux_VAR_mo.xml', + 'GPCPv2.3': '/p/user_pub/pmp/pmp_obs_preparation/orig/data/GPCP_v2.3_mon_jwl/precip.mon.mean.nc', + # 'GPCPv2.3': '/p/user_pub/PCMDIobs/PCMDIobs2.0/atmos/mon/pr/GPCP-2-3/gn/v20200117/pr_mon_GPCP-2-3_BE_gn_197901-201907.nc', # noqa + # 'AVISO': '/p/user_pub/PCMDIobs/PCMDIobs2.1/ocean/mon/zos/AVISO-1-0/gn/v20190912/zos_mon_AVISO-1-0_BE_gn_199210-201012.nc', # noqa + 'AVISO': '/work/lee1043/DATA/AVISO/sla_aviso_199301-201812.xml', +} + +reference_data_lf_path = { + 'GPCPv2.3': '/work/lee1043/DATA/GPCP/gpcp_25_lsmask.nc' +} +# ================================================= +# Models +# ------------------------------------------------- +modpath = os.path.join( + find_latest('/p/user_pub/pmp/pmp_results/pmp_v1.1.2/additional_xmls/latest'), + '%(mip)/%(exp)/%(realm)/mon/%(variable)', + '%(mip).%(exp).%(model).%(realization).mon.%(variable).xml') + +modpath_lf = os.path.join( + find_latest('/p/user_pub/pmp/pmp_results/pmp_v1.1.2/additional_xmls/latest'), + '%(mip)/historical/%(realm)/fx/%(variable)', + '%(mip).historical.%(model).r0i0p0.fx.%(variable).xml') + +modnames = ['all'] + +if debug: + modnames = ['IPSL-CM6A-LR'] + +realization = 'r1i1p1f1' # r1i1p1 (cmip5), r1i1p1f1 (cmip6), * (all) +# realization = '*' + +# ================================================= +# Metrics Collection +# ------------------------------------------------- +metricsCollection = 'ENSO_perf' # ENSO_perf, ENSO_tel, ENSO_proc + +# ================================================= +# Output +# ------------------------------------------------- +case_id = "{:v%Y%m%d}".format(datetime.datetime.now()) +pmprdir = '/p/user_pub/pmp/pmp_results/pmp_v1.1.2' + +if debug: + pmprdir = '/work/lee1043/imsi/result_test' + +results_dir = os.path.join( + pmprdir, + '%(output_type)', 'enso_metric', + '%(mip)', '%(exp)', '%(case_id)', '%(metricsCollection)') + +json_name = '%(mip)_%(exp)_%(metricsCollection)_%(case_id)_%(model)_%(realization)' +netcdf_name = json_name diff --git a/pcmdi_metrics/enso/param/my_Param_ENSO_PCMDIobs.py b/pcmdi_metrics/enso/param/my_Param_ENSO_PCMDIobs.py new file mode 100755 index 000000000..bc2628193 --- /dev/null +++ b/pcmdi_metrics/enso/param/my_Param_ENSO_PCMDIobs.py @@ -0,0 +1,79 @@ +import datetime +import glob +import os + + +def find_latest(path): + dir_list = [p for p in glob.glob(path+"/v????????")] + return sorted(dir_list)[-1] + + +# ================================================= +# Background Information +# ------------------------------------------------- +mip = 'cmip6' # cmip5, cmip6 +exp = 'historical' # historical, piControl + +# ================================================= +# Miscellaneous +# ------------------------------------------------- +# debug = False +debug = True +nc_out = True + +# ================================================= +# Observation +# ------------------------------------------------- +reference_data_lf_path = { + 'GPCPv2.3': '/work/lee1043/DATA/GPCP/gpcp_25_lsmask.nc', + 'GPCP-2-3': '/work/lee1043/DATA/GPCP/gpcp_25_lsmask.nc' +} + +obs_cmor = True +obs_cmor_path = "/p/user_pub/PCMDIobs/PCMDIobs2" +obs_catalogue = "/p/user_pub/PCMDIobs/catalogue/pcmdiobs_monthly_bySource_catalogue_v20210422.json" + +# ================================================= +# Models +# ------------------------------------------------- +modpath = os.path.join( + find_latest('/p/user_pub/pmp/pmp_results/pmp_v1.1.2/additional_xmls/latest'), + '%(mip)/%(exp)/%(realm)/mon/%(variable)', + '%(mip).%(exp).%(model).%(realization).mon.%(variable).xml') + +modpath_lf = os.path.join( + find_latest('/p/user_pub/pmp/pmp_results/pmp_v1.1.2/additional_xmls/latest'), + '%(mip)/historical/%(realm)/fx/%(variable)', + '%(mip).historical.%(model).r0i0p0.fx.%(variable).xml') + +modnames = ['IPSL-CM6A-LR'] +realization = 'r1i1p1f1' + +# modnames = ['all'] +# realization = '*' + +if debug: + modnames = ['IPSL-CM6A-LR'] + realization = 'r1i1p1f1' # r1i1p1 (cmip5), r1i1p1f1 (cmip6), * (all) + +# ================================================= +# Metrics Collection +# ------------------------------------------------- +metricsCollection = 'ENSO_perf' # ENSO_perf, ENSO_tel, ENSO_proc + +# ================================================= +# Output +# ------------------------------------------------- +case_id = "{:v%Y%m%d}".format(datetime.datetime.now()) +pmprdir = '/p/user_pub/pmp/pmp_results/pmp_v1.1.2' + +if debug: + pmprdir = '/work/lee1043/imsi/result_test' + +results_dir = os.path.join( + pmprdir, + '%(output_type)', 'enso_metric', + '%(mip)', '%(exp)', '%(case_id)', '%(metricsCollection)') + +json_name = '%(mip)_%(exp)_%(metricsCollection)_%(case_id)_%(model)_%(realization)' +netcdf_name = json_name diff --git a/pcmdi_metrics/enso/param/my_Param_ENSO_PCMDIobs_CLIVAR_LE_CEMS1-CAM5.py b/pcmdi_metrics/enso/param/my_Param_ENSO_PCMDIobs_CLIVAR_LE_CEMS1-CAM5.py new file mode 100755 index 000000000..137e3b0b3 --- /dev/null +++ b/pcmdi_metrics/enso/param/my_Param_ENSO_PCMDIobs_CLIVAR_LE_CEMS1-CAM5.py @@ -0,0 +1,73 @@ +import datetime +import glob +import os + + +def find_latest(path): + dir_list = [p for p in glob.glob(path+"/v????????")] + return sorted(dir_list)[-1] + + +# ================================================= +# Background Information +# ------------------------------------------------- +mip = 'CLIVAR_LE' # cmip5, cmip6 +exp = 'historical' # historical, piControl + +# ================================================= +# Miscellaneous +# ------------------------------------------------- +debug = False +# debug = True +nc_out = True + +# ================================================= +# Observation +# ------------------------------------------------- +reference_data_lf_path = { + 'GPCPv2.3': '/work/lee1043/DATA/GPCP/gpcp_25_lsmask.nc', + 'GPCP-2-3': '/work/lee1043/DATA/GPCP/gpcp_25_lsmask.nc' +} + +obs_cmor = True +obs_cmor_path = "/p/user_pub/PCMDIobs/PCMDIobs2" +obs_catalogue = "/p/user_pub/PCMDIobs/catalogue/pcmdiobs_monthly_bySource_catalogue_v20210422.json" + +# ================================================= +# Models +# ------------------------------------------------- +modpath = os.path.join('/work/lee1043/ESGF/ESG_NCAR', + '%(mip)/%(model)/mon/%(variable)/rewrite', + '%(variable)_%(realm)_%(model)_%(exp)_%(realization)_????01-200512_rewrite.nc') +modpath_lf = os.path.join('/work/lee1043/ESGF/ESG_NCAR', + 'CESM_LE/mon/%(realm)/%(variable)', + 'b.e11.B20TRC5CNBDRD.f09_g16.002.cam.h0.%(variable).????01-200512.nc') + +modnames = ['CESM1-CAM5'] +realization = '*' + +if debug: + modnames = ['CESM1-CAM5'] + realization = 'r1i1p1' # r1i1p1 (cmip5), r1i1p1f1 (cmip6), * (all) + +# ================================================= +# Metrics Collection +# ------------------------------------------------- +metricsCollection = 'ENSO_perf' # ENSO_perf, ENSO_tel, ENSO_proc + +# ================================================= +# Output +# ------------------------------------------------- +case_id = "{:v%Y%m%d}".format(datetime.datetime.now()) +pmprdir = '/p/user_pub/pmp/pmp_results/pmp_v1.1.2' + +if debug: + pmprdir = '/work/lee1043/imsi/result_test' + +results_dir = os.path.join( + pmprdir, + '%(output_type)', 'enso_metric', + '%(mip)', '%(exp)', '%(case_id)', '%(metricsCollection)') + +json_name = '%(mip)_%(exp)_%(metricsCollection)_%(case_id)_%(model)_%(realization)' +netcdf_name = json_name diff --git a/pcmdi_metrics/enso/param/my_Param_ENSO_PCMDIobs_CLIVAR_LE_CanESM2.py b/pcmdi_metrics/enso/param/my_Param_ENSO_PCMDIobs_CLIVAR_LE_CanESM2.py new file mode 100755 index 000000000..117dfe7e7 --- /dev/null +++ b/pcmdi_metrics/enso/param/my_Param_ENSO_PCMDIobs_CLIVAR_LE_CanESM2.py @@ -0,0 +1,74 @@ +import datetime +import glob +import os + + +def find_latest(path): + dir_list = [p for p in glob.glob(path+"/v????????")] + return sorted(dir_list)[-1] + + +# ================================================= +# Background Information +# ------------------------------------------------- +mip = 'CLIVAR_LE' # cmip5, cmip6 +exp = 'historical' # historical, piControl + +# ================================================= +# Miscellaneous +# ------------------------------------------------- +debug = False +# debug = True +nc_out = True + +# ================================================= +# Observation +# ------------------------------------------------- +reference_data_lf_path = { + 'GPCPv2.3': '/work/lee1043/DATA/GPCP/gpcp_25_lsmask.nc', + 'GPCP-2-3': '/work/lee1043/DATA/GPCP/gpcp_25_lsmask.nc' +} + +obs_cmor = True +obs_cmor_path = "/p/user_pub/PCMDIobs/PCMDIobs2" +obs_catalogue = "/p/user_pub/PCMDIobs/catalogue/pcmdiobs_monthly_bySource_catalogue_v20210422.json" + +# ================================================= +# Models +# ------------------------------------------------- +modpath = os.path.join('/work/lee1043/ESGF/ESG_NCAR', + '%(mip)/%(model)/mon/%(variable)/rewrite', + '%(variable)_%(realm)_%(model)_%(exp)_%(realization)_????01-200512_rewrite.nc') +modpath_lf = os.path.join( + find_latest('/p/user_pub/pmp/pmp_results/pmp_v1.1.2/additional_xmls/latest'), + 'cmip5/historical/%(realm)/fx/%(variable)', + 'cmip5.historical.%(model).r0i0p0.fx.%(variable).xml') + +modnames = ['CanESM2'] +realization = '*' + +if debug: + # modnames = ['CESM1-CAM5'] + realization = 'r1i1p1' # r1i1p1 (cmip5), r1i1p1f1 (cmip6), * (all) + +# ================================================= +# Metrics Collection +# ------------------------------------------------- +metricsCollection = 'ENSO_perf' # ENSO_perf, ENSO_tel, ENSO_proc + +# ================================================= +# Output +# ------------------------------------------------- +case_id = "{:v%Y%m%d}".format(datetime.datetime.now()) +pmprdir = '/p/user_pub/pmp/pmp_results/pmp_v1.1.2' + +if debug: + pmprdir = '/work/lee1043/imsi/result_test' + +results_dir = os.path.join( + pmprdir, + '%(output_type)', 'enso_metric', + '%(mip)', '%(exp)', '%(case_id)', '%(metricsCollection)') + +json_name = '%(mip)_%(exp)_%(metricsCollection)_%(case_id)_%(model)_%(realization)' +netcdf_name = json_name diff --git a/pcmdi_metrics/enso/param/my_Param_ENSO_obs2obs.py b/pcmdi_metrics/enso/param/my_Param_ENSO_obs2obs.py new file mode 100755 index 000000000..6a418f48b --- /dev/null +++ b/pcmdi_metrics/enso/param/my_Param_ENSO_obs2obs.py @@ -0,0 +1,85 @@ +import datetime +import glob +import os + + +def find_latest(path): + dir_list = [p for p in glob.glob(path+"/v????????")] + return sorted(dir_list)[-1] + + +# ================================================= +# Background Information +# ------------------------------------------------- +mip = 'obs2obs' # cmip5, cmip6 +exp = 'historical' # historical, piControl + +# ================================================= +# Miscellaneous +# ------------------------------------------------- +debug = False +nc_out = True + +# ================================================= +# Observation +# ------------------------------------------------- +reference_data_path = { + 'ERA-Interim': '/p/user_pub/PCMDIobs/PCMDIobs2/atmos/mon/VAR/ERA-INT/gn/v20200402/VAR_mon_ERA-INT_BE_gn_v20200402_197901-201903.nc', # noqa + 'HadISST': '/work/lee1043/DATA/HadISSTv1.1/HadISSTv1.1.xml', + 'OISST': '/work/lee1043/DATA/OISST/xmls/OISST_tos_mo.xml', + 'Tropflux': '/work/lee1043/DATA/TropFlux/monthly/xmls/Tropflux_VAR_mo.xml', + # 'Tropflux': '/p/user_pub/PCMDIobs/PCMDIobs2.0/atmos/mon/VAR/TropFlux-1-0/gn/v20190912/VAR_mon_TropFlux-1-0_BE_gn_197901-201707.nc', # noqa + # 'OAFlux': '/work/lee1043/DATA/OAFlux/xmls/OAFlux_VAR_mo.xml', + 'GPCPv2.3': '/p/user_pub/pmp/pmp_obs_preparation/orig/data/GPCP_v2.3_mon_jwl/precip.mon.mean.nc', + # 'GPCPv2.3': '/p/user_pub/PCMDIobs/PCMDIobs2.0/atmos/mon/pr/GPCP-2-3/gn/v20200117/pr_mon_GPCP-2-3_BE_gn_197901-201907.nc', # noqa + # 'AVISO': '/p/user_pub/PCMDIobs/PCMDIobs2.1/ocean/mon/zos/AVISO-1-0/gn/v20190912/zos_mon_AVISO-1-0_BE_gn_199210-201012.nc', # noqa + 'AVISO': '/work/lee1043/DATA/AVISO/sla_aviso_199301-201812.xml', +} + +reference_data_lf_path = { + 'GPCPv2.3': '/work/lee1043/DATA/GPCP/gpcp_25_lsmask.nc' +} +# ================================================= +# Models +# ------------------------------------------------- +modpath = os.path.join( + '/p/user_pub/PCMDIobs/PCMDIobs2/%(realm)/mon/%(variable)', + '%(model)/gn/', + 'v????????', + '%(variable)_mon_%(model)_BE_gn_v????????_??????-??????.nc') + +modpath_lf = os.path.join( + find_latest('/p/user_pub/pmp/pmp_results/pmp_v1.1.2/additional_xmls/latest'), + '%(mip)/historical/%(realm)/fx/%(variable)', + '%(mip).historical.%(model).r0i0p0.fx.%(variable).xml') + +modnames = ['20CR', 'ERA-20C', 'ERA-INT', 'TropFlux-1-0', 'CMAP-V1902', 'GPCP-2-3', 'TRMM-3B43v-7', 'ERA-5', + 'CERES-EBAF-4-0', 'CERES-EBAF-4-1', 'AVISO-1-0'] + +if debug: + modnames = ['ERA-INT'] + +realization = 'r1i1p1f1' # r1i1p1 (cmip5), r1i1p1f1 (cmip6), * (all) +# realization = '*' + +# ================================================= +# Metrics Collection +# ------------------------------------------------- +metricsCollection = 'ENSO_perf' # ENSO_perf, ENSO_tel, ENSO_proc + +# ================================================= +# Output +# ------------------------------------------------- +case_id = "{:v%Y%m%d}".format(datetime.datetime.now()) +pmprdir = '/p/user_pub/pmp/pmp_results/pmp_v1.1.2' + +if debug: + pmprdir = '/work/lee1043/imsi/result_test' + +results_dir = os.path.join( + pmprdir, + '%(output_type)', 'enso_metric', + '%(mip)', '%(exp)', '%(case_id)', '%(metricsCollection)') + +json_name = '%(mip)_%(exp)_%(metricsCollection)_%(case_id)_%(model)_%(realization)' +netcdf_name = json_name diff --git a/pcmdi_metrics/enso/scripts_pcmdi/README.md b/pcmdi_metrics/enso/scripts_pcmdi/README.md new file mode 100644 index 000000000..db8b1853b --- /dev/null +++ b/pcmdi_metrics/enso/scripts_pcmdi/README.md @@ -0,0 +1,16 @@ +# Scripts for PCMDI Metrics Package + +- `run_pmp.sh`: Compute metrics using single CPU. + - `PMPdriver_EnsoMetrics.py` +- `run_pmp_parallel.sh`: Compute metrics using multiple CPUs. + - `parallel_driver.py` + - `PMPdriver_EnsoMetrics.py` + - Input parameter file: `my_Param_ENSO.py` or `my_Param_ENSO_obs2obs.py` +- `run_pmp_palallel_obs2obs.sh`: Compute metrics using multiple CPUs but for observation to observation comparison. + - `parallel_driver.py` + - `PMPdriver_EnsoMetrics.py` +- `run_pmp_plot_parallel.sh`: Generate dive down plots using multiple CPUs. + - `parallel_driver_plot.py` + - `PMPdriver_plot.py` + +**NOTE**: *More clean up needed for obs2obs task* diff --git a/pcmdi_metrics/enso/scripts_pcmdi/enso_driver_obsOnly.py b/pcmdi_metrics/enso/scripts_pcmdi/enso_driver_obsOnly.py new file mode 100755 index 000000000..713b1e4ad --- /dev/null +++ b/pcmdi_metrics/enso/scripts_pcmdi/enso_driver_obsOnly.py @@ -0,0 +1,208 @@ +#!/usr/bin/env python +# ================================================= +# Dependencies +# ------------------------------------------------- +from __future__ import print_function + +import glob +import json +import os +import pkg_resources +import sys + +from genutil import StringConstructor +from pcmdi_metrics.enso.lib import AddParserArgument +from pcmdi_metrics.enso.lib import metrics_to_json +from EnsoMetrics.EnsoCollectionsLib import defCollection, ReferenceObservations +# from EnsoMetrics.EnsoComputeMetricsLib import ComputeCollection +from EnsoMetrics.EnsoComputeMetricsLib import ComputeCollection_ObsOnly + +# To avoid below error when using multi cores +# OpenBLAS blas_thread_init: pthread_create failed for thread XX of 96: Resource temporarily unavailable +os.environ['OPENBLAS_NUM_THREADS'] = '1' + +# ================================================= +# Collect user defined options +# ------------------------------------------------- +param = AddParserArgument() + +# Pre-defined options +mip = param.mip +exp = param.exp +print('mip:', mip) +print('exp:', exp) + +# Path to model data as string template +modpath = param.process_templated_argument("modpath") +modpath_lf = param.process_templated_argument("modpath_lf") + +# Check given model option +models = param.modnames + +# Include all models if conditioned +if ('all' in [m.lower() for m in models]) or (models == 'all'): + model_index_path = param.modpath.split('/')[-1].split('.').index("%(model)") + models = ([p.split('/')[-1].split('.')[model_index_path] for p in glob.glob(modpath( + mip=mip, exp=exp, model='*', realization='*', variable='ts'))]) + # remove duplicates + models = sorted(list(dict.fromkeys(models)), key=lambda s: s.lower()) + +print('models:', models) + +# Realizations +realization = param.realization +print('realization: ', realization) + +# Metrics Collection +mc_name = param.metricsCollection +dict_mc = defCollection(mc_name) +list_metric = sorted(dict_mc['metrics_list'].keys()) +print('mc_name:', mc_name) + +# case id +case_id = param.case_id + +# Output +outdir_template = param.process_templated_argument("results_dir") +outdir = StringConstructor(str(outdir_template( + output_type='%(output_type)', + mip=mip, exp=exp, metricsCollection=mc_name, case_id=case_id))) +netcdf_path = outdir(output_type='diagnostic_results') +json_name_template = param.process_templated_argument("json_name") +netcdf_name_template = param.process_templated_argument("netcdf_name") + +print('outdir:', str(outdir_template( + output_type='%(output_type)', + mip=mip, exp=exp, metricsCollection=mc_name))) +print('netcdf_path:', netcdf_path) + +# Switches +debug = param.debug +print('debug:', debug) + +# ================================================= +# Prepare loop iteration +# ------------------------------------------------- +# Environmental setup +try: + egg_pth = pkg_resources.resource_filename( + pkg_resources.Requirement.parse("pcmdi_metrics"), "share/pmp") +except Exception: + egg_pth = os.path.join(sys.prefix, "share", "pmp") +print('egg_pth:', egg_pth) + +# Create output directory +for output_type in ['graphics', 'diagnostic_results', 'metrics_results']: + if not os.path.exists(outdir(output_type=output_type)): + os.makedirs(outdir(output_type=output_type)) + print(outdir(output_type=output_type)) + +# list of variables +list_variables = list() +for metric in list_metric: + listvar = dict_mc['metrics_list'][metric]['variables'] + for var in listvar: + if var not in list_variables: + list_variables.append(var) +list_variables = sorted(list_variables) +print(list_variables) + +# list of observations +list_obs = list() +for metric in list_metric: + dict_var_obs = dict_mc['metrics_list'][metric]['obs_name'] + for var in dict_var_obs.keys(): + for obs in dict_var_obs[var]: + if obs not in list_obs: + list_obs.append(obs) +list_obs = sorted(list_obs) + +# +# finding file and variable name in file for each observations dataset +# +dict_obs = dict() + +for obs in list_obs: + # be sure to add your datasets to EnsoCollectionsLib.ReferenceObservations if needed + dict_var = ReferenceObservations(obs)['variable_name_in_file'] + dict_obs[obs] = dict() + for var in list_variables: + # + # finding variable name in file + # + try: + var_in_file = dict_var[var]['var_name'] + except Exception: + print('\033[95m' + str(var) + " is not available for " + str(obs) + " or unscripted" + '\033[0m') + else: + if isinstance(var_in_file, list): + var0 = var_in_file[0] + else: + var0 = var_in_file + + try: + # finding file for 'obs', 'var' + file_name = param.reference_data_path[obs].replace('VAR', var0) + file_areacell = None # temporary for now + try: + file_landmask = param.reference_data_lf_path[obs] + except Exception: + file_landmask = None + try: + areacell_in_file = dict_var['areacell']['var_name'] + except Exception: + areacell_in_file = None + try: + landmask_in_file = dict_var['landmask']['var_name'] + except Exception: + landmask_in_file = None + # if var_in_file is a list (like for thf) all variables should be read from the same realm + if isinstance(var_in_file, list): + list_files = list() + list_files = [param.reference_data_path[obs].replace('VAR', var1) for var1 in var_in_file] + list_areacell = [file_areacell for var1 in var_in_file] + list_name_area = [areacell_in_file for var1 in var_in_file] + try: + list_landmask = [param.reference_data_lf_path[obs] for var1 in var_in_file] + except Exception: + list_landmask = None + list_name_land = [landmask_in_file for var1 in var_in_file] + else: + list_files = file_name + list_areacell = file_areacell + list_name_area = areacell_in_file + list_landmask = file_landmask + list_name_land = landmask_in_file + dict_obs[obs][var] = {'path + filename': list_files, 'varname': var_in_file, + 'path + filename_area': list_areacell, 'areaname': list_name_area, + 'path + filename_landmask': list_landmask, 'landmaskname': list_name_land} + except Exception: + print('\033[95m' + 'Observation dataset' + str(obs) + + " is not given for variable " + str(var) + '\033[0m') + +print('PMPdriver: dict_obs readin end') + +# Prepare computing the metric collection (OBS to OBS) +dictDatasets = {'observations': dict_obs} +netcdf_path = "/work/lee1043/imsi/result_test/enso_metric/test_obs2obs_yann" +netcdf_name = 'YANN_PLANTON_' + mc_name + "_OBSNAME" +netcdf = os.path.join(netcdf_path, netcdf_name) +if debug: + print('file_name:', file_name) + print('list_files:', list_files) + print('netcdf_name:', netcdf_name) + print('dict_obs:') + print(json.dumps(dict_obs, indent=4, sort_keys=True)) + with open("dict_obs_" + mc_name + ".json", "w") as f_dict_obs: + json.dump(dict_obs, f_dict_obs, indent=4, sort_keys=True) + +# Compute the metric collection (OBS to OBS) +dict_metric, dict_dive = ComputeCollection_ObsOnly(mc_name, dictDatasets, debug=True, netcdf=True, netcdf_name=netcdf) +if debug: + print('dict_metric:') + print(json.dumps(dict_metric, indent=4, sort_keys=True)) + +# OUTPUT METRICS TO JSON FILE (per simulation) +outdir = netcdf_path +json_name = netcdf_name +metrics_to_json(mc_name, dict_obs, dict_metric, dict_dive, egg_pth, outdir, json_name, mod='obs', run='test') diff --git a/pcmdi_metrics/enso/scripts_pcmdi/parallel_driver.py b/pcmdi_metrics/enso/scripts_pcmdi/parallel_driver.py new file mode 100755 index 000000000..8b4850039 --- /dev/null +++ b/pcmdi_metrics/enso/scripts_pcmdi/parallel_driver.py @@ -0,0 +1,193 @@ +#!/usr/bin/env python + +""" +Usage example: +1. First realization per model +./parallel_driver.py -p my_Param_ENSO.py --mip cmip6 --modnames all --realization r1i1p1f1 --metricsCollection ENSO_perf +2. All realizations of individual models +./parallel_driver.py -p my_Param_ENSO.py --mip cmip6 --modnames all --realization all --metricsCollection ENSO_perf +""" + +from __future__ import print_function +from genutil import StringConstructor +from subprocess import Popen + +from pcmdi_metrics.enso.lib import AddParserArgument, find_realm +from pcmdi_metrics.variability_mode.lib import sort_human + +import glob +import os +import sys +import time + +# To avoid below error +# OpenBLAS blas_thread_init: pthread_create failed for thread XX of 96: Resource temporarily unavailable +os.environ['OPENBLAS_NUM_THREADS'] = '1' + +# Must be done before any CDAT library is called. +# https://github.com/CDAT/cdat/issues/2213 +if 'UVCDAT_ANONYMOUS_LOG' not in os.environ: + os.environ['UVCDAT_ANONYMOUS_LOG'] = 'no' + +# ================================================= +# Collect user defined options +# ------------------------------------------------- +param = AddParserArgument() + +# Pre-defined options +mip = param.mip +exp = param.exp +print('mip:', mip) +print('exp:', exp) + +# Path to model data as string template +modpath = param.process_templated_argument("modpath") + +# Check given model option +models = param.modnames +print('models:', models) + +# Include all models if conditioned +if mip == "CLIVAR_LE": + inline_separator = '_' +else: + inline_separator = '.' + +if ('all' in [m.lower() for m in models]) or (models == 'all'): + model_index_path = param.modpath.split('/')[-1].split(inline_separator).index("%(model)") + models = ([p.split('/')[-1].split(inline_separator)[model_index_path] for p in glob.glob(modpath( + mip=mip, exp=exp, model='*', realization='*', variable='ts'))]) + # remove duplicates + models = sorted(list(dict.fromkeys(models)), key=lambda s: s.lower()) + +print('models:', models) +print('number of models:', len(models)) + +# Realizations +realization = param.realization +if ('all' in [r.lower() for r in realization]) or (realization == 'all'): + realization = '*' +print('realization: ', realization) + +# Metrics Collection +mc_name = param.metricsCollection + +# case id +case_id = param.case_id +print('case_id:', case_id) + +# Output +outdir_template = param.process_templated_argument("results_dir") +outdir = StringConstructor(str(outdir_template( + output_type='%(output_type)', + mip=mip, exp=exp, metricsCollection=mc_name, case_id=case_id))) + +# Debug +debug = param.debug +print('debug:', debug) + +# ================================================= +# Create output directories +# ------------------------------------------------- +for output_type in ['graphics', 'diagnostic_results', 'metrics_results']: + if not os.path.exists(outdir(output_type=output_type)): + os.makedirs(outdir(output_type=output_type)) + print(outdir(output_type=output_type)) + +# ================================================= +# Generates list of command +# ------------------------------------------------- +if mip == "obs2obs": + param_file = '../param/my_Param_ENSO_obs2obs.py' +if mip == "CLIVAR_LE": + # param_file = '../param/my_Param_ENSO_PCMDIobs_CLIVAR_LE-CESM1-CAM5.py' + param_file = '../param/my_Param_ENSO_PCMDIobs_CLIVAR_LE_CanESM2.py' +else: + param_file = '../param/my_Param_ENSO_PCMDIobs.py' + +cmds_list = [] +for model in models: + print(' ----- model: ', model, ' ---------------------') + # Find all xmls for the given model + realm, areacell_in_file = find_realm('ts', mip) + model_path_list = glob.glob( + modpath(mip=mip, exp=exp, realm=realm, model=model, realization="*", variable='ts')) + # sort in nice way + model_path_list = sort_human(model_path_list) + if debug: + print('model_path_list:', model_path_list) + try: + # Find where run can be gripped from given filename template for modpath + run_in_modpath = modpath(mip=mip, exp=exp, realm=realm, model=model, realization=realization, + variable='ts').split('/')[-1].split(inline_separator).index(realization) + if debug: + print('run_in_modpath:', run_in_modpath) + # Collect available runs + runs_list = [model_path.split('/')[-1].split(inline_separator)[run_in_modpath] + for model_path in model_path_list] + except Exception: + if realization not in ["*", "all"]: + runs_list = [realization] + if debug: + print('runs_list (all):', runs_list) + # Check if given run member is included. If not for all runs and given run member is not included, + # take alternative run + if realization != "*": + if realization in runs_list: + runs_list = [realization] + else: + runs_list = runs_list[0:1] + if debug: + print('runs_list (revised):', runs_list) + for run in runs_list: + cmd = ['enso_driver.py', + '-p', param_file, + '--mip', mip, '--metricsCollection', mc_name, + '--case_id', case_id, + '--modnames', model, + '--realization', run] + cmds_list.append(cmd) + +if debug: + for cmd in cmds_list: + print(' '.join(cmd)) + +# ================================================= +# Run subprocesses in parallel +# ------------------------------------------------- +# log dir +log_dir = os.path.join("log", case_id, mc_name) + +if not os.path.exists(log_dir): + os.makedirs(log_dir) + +# number of tasks to submit at the same time +num_workers = 7 +# num_workers = 10 +# num_workers = 30 +# num_workers = 25 + +print("Start : %s" % time.ctime()) + +# submit tasks and wait for subset of tasks to complete +procs_list = [] +for p, cmd in enumerate(cmds_list): + timenow = time.ctime() + print(timenow, p, ' '.join(cmd)) + model = cmd[-3] + run = cmd[-1] + log_filename = '_'.join(['log_enso', mc_name, mip, exp, model, run, case_id]) + log_file = os.path.join(log_dir, log_filename) + with open(log_file+"_stdout.txt", "wb") as out, open(log_file+"_stderr.txt", "wb") as err: + procs_list.append(Popen(cmd, stdout=out, stderr=err)) + time.sleep(1) + if ((p > 0 and p % num_workers == 0) or (p == len(cmds_list)-1)): + print('wait...') + for proc in procs_list: + proc.wait() + print("Tasks end : %s" % time.ctime()) + procs_list = [] + +# tasks done +print("End : %s" % time.ctime()) +sys.exit('DONE') diff --git a/pcmdi_metrics/enso/scripts_pcmdi/post_process_merge_jsons.py b/pcmdi_metrics/enso/scripts_pcmdi/post_process_merge_jsons.py new file mode 100755 index 000000000..4cf2a3e42 --- /dev/null +++ b/pcmdi_metrics/enso/scripts_pcmdi/post_process_merge_jsons.py @@ -0,0 +1,102 @@ +#!/usr/bin/env python + +from __future__ import print_function +from genutil import StringConstructor +from pcmdi_metrics.variability_mode.lib import dict_merge + +import copy +import glob +import json +import os + + +def main(): + mips = ["cmip5", "cmip6"] + # mips = ["cmip5"] + # mips = ["cmip6"] + # mips = ["obs2obs"] + mips = ["CLIVAR_LE"] + + exps = ["historical"] + + # MCs = ["ENSO_perf", "ENSO_tel", "ENSO_proc", "test_tel"] + MCs = ["ENSO_perf", "ENSO_tel", "ENSO_proc"] + # MCs = ["ENSO_tel"] + # MCs = ["test_tel"] + + pmprdir = '/p/user_pub/pmp/pmp_results/pmp_v1.1.2' + # pmprdir = "/work/lee1043/imsi/result_test" + + for mip in mips: + for exp in exps: + for MC in MCs: + case_id = find_latest(pmprdir, mip, exp, MC) + print("mip, exp, MC, case_id:", mip, exp, MC, case_id) + merge_jsons(mip, exp, case_id, MC, pmprdir) + + +def merge_jsons(mip, exp, case_id, metricsCollection, pmprdir): + json_file_dir_template = os.path.join( + pmprdir, + '%(output_type)', 'enso_metric', + '%(mip)', '%(exp)', '%(case_id)', '%(metricsCollection)') + json_file_dir_template = StringConstructor(json_file_dir_template) + json_file_dir = json_file_dir_template( + output_type='metrics_results', mip=mip, exp=exp, case_id=case_id, metricsCollection=metricsCollection) + + json_file_template = '_'.join(['%(mip)_%(exp)_%(metricsCollection)', '%(case_id)', '%(model)', '%(realization)']) + json_file_template = '%(mip)_%(exp)_%(metricsCollection)_%(case_id)_%(model)_%(realization)' + json_file_template = StringConstructor(json_file_template) + + # Search for individual JSONs + json_files = sorted(glob.glob( + os.path.join( + json_file_dir, + json_file_template( + mip=mip, exp=exp, metricsCollection=metricsCollection, + case_id=case_id, model='*', realization='*') + '.json'))) + + # Remove diveDown JSONs and previously generated merged JSONs if included + json_files_revised = copy.copy(json_files) + for j, json_file in enumerate(json_files): + filename_component = json_file.split('/')[-1].split('.')[0].split('_') + if 'diveDown' in filename_component: + json_files_revised.remove(json_file) + elif 'allModels' in filename_component: + json_files_revised.remove(json_file) + elif 'allRuns' in filename_component: + json_files_revised.remove(json_file) + + # Load individual JSON and merge to one big dictionary + for j, json_file in enumerate(json_files_revised): + print(j, json_file) + f = open(json_file) + dict_tmp = json.loads(f.read()) + if j == 0: + dict_final = dict_tmp.copy() + else: + dict_merge(dict_final, dict_tmp) + f.close() + + # Dump final dictionary to JSON + final_json_filename = json_file_template( + mip=mip, exp=exp, metricsCollection=metricsCollection, case_id=case_id, + model='allModels', realization='allRuns')+'.json' + final_json_file = os.path.join(json_file_dir, final_json_filename) + + with open(final_json_file, 'w') as fp: + json.dump(dict_final, fp, sort_keys=True, indent=4) + + print("Done: check ", final_json_file) + + +def find_latest(pmprdir, mip, exp, MC): + versions = sorted([r.split('/')[-2] for r in glob.glob(os.path.join( + pmprdir, "metrics_results", "enso_metric", + mip, exp, "v????????", MC))]) + latest_version = versions[-1] + return latest_version + + +if __name__ == "__main__": + main() diff --git a/pcmdi_metrics/enso/scripts_pcmdi/run_pmp.sh b/pcmdi_metrics/enso/scripts_pcmdi/run_pmp.sh new file mode 100755 index 000000000..6d918436c --- /dev/null +++ b/pcmdi_metrics/enso/scripts_pcmdi/run_pmp.sh @@ -0,0 +1,36 @@ +#!/bin/sh +set -a + +# Working conda env in Crunchy: pmp_nightly_20180830 + +ver=`date +"%Y%m%d-%H%M"` + +#mips='cmip5 cmip6' +#mips='cmip5' +mips='cmip6' +#mips='CLIVAR_LE' +#mips='obs2obs' + +#MCs='ENSO_perf ENSO_tel ENSO_proc' +MCs='ENSO_perf' +#MCs='ENSO_tel ENSO_proc' +#MCs='ENSO_tel' +#MCs='ENSO_proc' + +param_dir='../param' + +param_file='my_Param_ENSO.py' +#param_file='my_Param_ENSO_PCMDIobs.py' +#param_file='my_Param_ENSO_PCMDIobs_CLIVAR_LE.py' +#param_file='my_Param_ENSO_obs2obs.py' +#param_file='my_Param_ENSO_obs2obs_combinedDataSource.py' + +mkdir -p log + +for mip in $mips; do + for MC in $MCs; do + echo $mip $MC + enso_driver.py -p $param_dir/$param_file --mip ${mip} --metricsCollection ${MC} >& log/log.${mip}.${MC}.all.v${ver}.txt & + disown + done +done diff --git a/pcmdi_metrics/enso/scripts_pcmdi/run_pmp_parallel.sh b/pcmdi_metrics/enso/scripts_pcmdi/run_pmp_parallel.sh new file mode 100755 index 000000000..b390666a7 --- /dev/null +++ b/pcmdi_metrics/enso/scripts_pcmdi/run_pmp_parallel.sh @@ -0,0 +1,55 @@ +#!/bin/sh +set -a + +# To avoid below error +# OpenBLAS blas_thread_init: pthread_create failed for thread XX of 96: Resource temporarily unavailable +export OMP_NUM_THREADS=1 + +# Working conda env in gates: cdat82_20191107_py27 + +case_id="v"`date +"%Y%m%d"` +#case_id="v20200224" + +#mips='cmip5 cmip6' +#mips='cmip5' +#mips='cmip6' +mips='CLIVAR_LE' +#mips='obs2obs' + +#MCs='ENSO_perf ENSO_tel ENSO_proc' +#MCs='ENSO_perf' +MCs='ENSO_tel ENSO_proc' +#MCs='ENSO_tel' +#MCs='ENSO_proc' + +param_dir='../param' + +modnames='all' +#modnames='IPSL-CM5A-LR' + +realization='all' + +mkdir -p log/$case_id + +for mip in $mips; do + if [ $mip == 'cmip5' ]; then + #realization='r1i1p1' + #modnames="BNU-ESM HadCM3" + param_file='my_Param_ENSO_PCMDIobs.py' + elif [ $mip == 'cmip6' ]; then + #realization='r1i1p1f1' + #modnames="BCC-ESM1 CESM2 CESM2-FV2 CESM2-WACCM CESM2-WACCM-FV2 GFDL-CM4 GFDL-ESM4 MRI-ESM2-0" + param_file='my_Param_ENSO_PCMDIobs.py' + elif [ $mip == 'CLIVAR_LE' ]; then + param_file='my_Param_ENSO_PCMDIobs_CLIVAR_LE.py' + elif [ $mip == 'obs2obs' ]; then + param_file='my_Param_ENSO_obs2obs.py' + fi + + for MC in $MCs; do + echo $mip $MC $realization $case_id + python -u ./parallel_driver.py -p $param_dir/$param_file --mip $mip --case_id=$case_id --modnames $modnames --metricsCollection $MC --realization $realization >& log/$case_id/log_parallel.${mip}.${MC}.all.${case_id}.txt & + disown + sleep 1 + done +done diff --git a/pcmdi_metrics/enso/scripts_pcmdi/run_pmp_parallel_obs2obs.sh b/pcmdi_metrics/enso/scripts_pcmdi/run_pmp_parallel_obs2obs.sh new file mode 100755 index 000000000..80efe5d44 --- /dev/null +++ b/pcmdi_metrics/enso/scripts_pcmdi/run_pmp_parallel_obs2obs.sh @@ -0,0 +1,26 @@ +#!/bin/sh +set -a + +# To avoid below error +# OpenBLAS blas_thread_init: pthread_create failed for thread XX of 96: Resource temporarily unavailable +export OMP_NUM_THREADS=1 + +# Working conda env in gates: cdat82_20191107_py27, cdat82_20200128_py27 + +case_id="v"`date +"%Y%m%d"` + +mips='obs2obs' + +MCs='ENSO_perf ENSO_tel ENSO_proc' +modnames='20CR ERA-20C ERA-INT TropFlux-1-0 CMAP-V1902 GPCP-2-3 TRMM-3B43v-7 ERA-5 CERES-EBAF-4-0 CERES-EBAF-4-1 AVISO-1-0' + +mkdir -p log/$case_id + +for mip in $mips; do + for MC in $MCs; do + echo $mip $MC $realization $case_id + python -u ./parallel_driver.py -p my_Param_ENSO_obs2obs.py --mip $mip --case_id=$case_id --modnames $modnames --metricsCollection $MC >& log/$case_id/log_parallel.${mip}.${MC}.all.${case_id}.txt & + disown + sleep 1 + done +done diff --git a/setup.py b/setup.py index 226de5da6..0f6823eab 100755 --- a/setup.py +++ b/setup.py @@ -64,7 +64,8 @@ 'pcmdi_metrics/monsoon_wang/scripts/mpindex_compute.py', 'pcmdi_metrics/monsoon_sperber/scripts/driver_monsoon_sperber.py', 'pcmdi_metrics/mjo/scripts/mjo_metrics_driver.py', - 'pcmdi_metrics/variability_mode/variability_modes_driver.py' + 'pcmdi_metrics/variability_mode/variability_modes_driver.py', + 'pcmdi_metrics/enso/enso_driver.py' ] # scripts += glob.glob("pcmdi_metrics/diurnal/scripts/*.py")