notebooks/CSSP_20CRDS_Tutorials/Introduction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CSSP 20CR dataset - Tutorials"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Contents:\n",
    "1. [Introduction](#introduction)\n",
    "2. [Description of datasets](#dataset)\n",
    "3. [Learning objectives](#objectives)\n",
    "4. [Jupyter notebook](#notebook)\n",
    "5. [Data format and python libraries](#libs)\n",
    "6. [Instructions to create an environment](#env)\n",
    "7. [Resources](#resources)\n",
    "\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Introduction<a id='introduction'></a>\n",
    "\n",
    "This short course is an introductory set of tutorials on accessing a large (~3Tb) dataset hosted on a cloud server. By putting the data and the computer resources in the same place, users no longer have to spend time downloading data, finding local storage for and manging the software needed to analyse the data. These notebooks explain how to use this cloud based platform to analyse the 20CR-DS dataset.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Learning objectives<a id='objectives'></a>\n",
    "\n",
    "The high level learning objectives for these tutorials are:\n",
    "- To access and explore variables of interest\n",
    "- To convert data into different formats (xarrays and iris) \n",
    "- To prepare data for analysis\n",
    "- To carry out basic analyses\n",
    "- To carry out advanced analysis\n",
    "- To visualise the results  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Description of the tutorial dataset<a id='dataset'></a>\n",
    "\n",
    "A climate reanalysis gives a numerical description of the recent climate, produced by combining models with observations. The Twentieth Century Reanalysis Project (20CR-V2c) is a global reanalysis carried out by the National Oceanic and Atmospheric Administration (NOAA). The outputs from this dataset include temperature, pressure, winds, moisture, solar radiation and clouds, from the surface to the top of the atmosphere as far back as the mid-1800s. More information are available from [climate-reanalysis](https://www.ecmwf.int/en/research/climate-reanalysis) and [20CR-V2c](https://www.esrl.noaa.gov/psd/data/gridded/data.20thC_ReanV2c.html).\n",
    "\n",
    "At the UK Met Office we have increased the resolution of the 20CR-V2c reanalysis dataset using a process known as dynamical downscaling and it now covers the whole of China for the period 1851 to 2010 at a horizontal resolution of 25 km [(Amato et al., 2019)](https://doi.org/10.1175/JAMC-D-19-0083.1). (https://zenodo.org/record/2558135#.XJj2uaD7RWE). This work was funded through the Climate Science for Service Partnership China (CSSP-China) project.\n",
    "\n",
    "The Climate Science for Service Partnership China (CSSP China) is a scientific research project that is building the basis for services to support climate and weather resilient economic development and social welfare through strong, strategic partnerships harnessing UK scientific expertise. Through CSSP China (supported by the Newton Fund and the Department for Business, Energy & Industrial Strategy (BEIS) UK-China Research Innovation Partnership Fund) we are developing a strongly bilateral partnership between the Met Office, the China Meteorological Administration (CMA), the Institute of Atmospheric Physics (IAP) at the Chinese Academy of Sciences, and other key institutes within China and the UK. See the [CSSP-China](https://www.metoffice.gov.uk/research/approach/collaboration/newton/cssp-china/index) for more information.\n",
    "\n",
    "<figure>\n",
    "  <img src=\"images/region.PNG\" alt=\"Trulli\" style=\"width:60%\">\n",
    "  <figcaption>Figure: Downscaled domain of 20CR datasets (Amato et al., 2019)</figcaption>\n",
    "</figure>\n",
    "\n",
    "\n",
    "The area of interests are devided into seven subregions, shown in figure, are considered for analysis [Burke and Stott (2017)](https://journals.ametsoc.org/jcli/article/30/14/5205/97096/Impact-of-Anthropogenic-Climate-Change-on-the-East). The coordinates of these seven regions are: \n",
    "\n",
    "\n",
    "\n",
    "North Central (NC): 104°–113°E, 32°–39°N\n",
    "\n",
    "North East Coast (NEC): 113°–122°E, 32°–39°N\n",
    "\n",
    "North East (NE): 113°–131°E, 39°–44°N\n",
    "\n",
    "Tibetan Plateau (TP): 77°–104°E, 26°–36°N\n",
    "\n",
    "South Central (SC): 104°–113°E, 26°–32°N\n",
    "\n",
    "South East Coast (SEC): 113°–122°E, 26°–32°N\n",
    "\n",
    "South East (SE): 107°–120°E, 21°–26°N"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dataset include monthly, daily, 3 hourly and hourly frequencies for the historical period of 1851-2010. Though in the tutorials we have only used monthly, daily and hourly frequencies. The details of variables and frequencies can be found in [supplementary material](variableslist.pdf). The data for this course is held online in an Azure Blob Storage Service. To access this we use a SAS (shared access signature). You should have been given the credentials for this service before the course, but if not please ask your instructor. \n",
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Jupyter notebook<a id='notebook'></a>\n",
    "Jupyter is an open source platform that contains a suite of tools including Jupyter Notebook: A browser-based application that allows you to create and share documents (i.e. Jupyter Notebook files such as this notebook you are reading right now!). These notebooks can contain simple text content and live code, equations, visualizations and narrative text. It is an Integrated Development Environment (IDE) that allows you to write code, navigate files on the system and inspect variables and more. The Jupyter Notebook file format (.ipynb ) allows you to combine descriptive text, code blocks and code output in a single file. You can then share the notebook itself with anyone who might want to run it and also convert the notebook to a PDF or HTML format that can be viewed like a report.\n",
    "\n",
    "##### How to run Jupyter Notebook\n",
    "A Jupyter Notebook file (.ipynb) has three main parts, which are highlighted in the image below:\n",
    "\n",
    "- Menu bar\n",
    "- Toolbar\n",
    "- Cells\n",
    "\n",
    "\n",
    "<figure>\n",
    "  <img src=\"images/jupyter.png\" alt=\"Trulli\" style=\"width:60%\">\n",
    "</figure>\n",
    "\n",
    "\n",
    "Cells can be specified to store documentation text such as Markdown or programming code such as Python. Text written using the Markdown syntax can be rendered in a cell that is of the cell type Markdown. You can run code (e.g. Python) using the Code as cell type write you code and then either click on the run the selected cell button on top or use the Shift+Enter keyboard combination. When you run the code in a Code cell, the code output displayed below.\n",
    "\n",
    "**Example:** click on the cell below and press `Shift+Enter` or `Ctrl+Enter`, It will print the output below the cell. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"CSSP 20CR dataset\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also execute shell commands from the cell. For Example cell below list down the contents of the *~/cloudfiles/code/Users* directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls ~/cloudfiles/code/Users"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that **`Shift+Enter`** runs the cell and moves the cursor to next cell but **`Ctrl+Enter`** runs the cell but the cursor stays in the same cell.\n",
    "\n",
    "An important component of a Jupyter Notebook is its Kernel. A kernel runs your code in a specific programming language. Jupyter Notebook supports over 40 different languages. In this tutorials, we will use the Python kernel within the Jupyter Notebook IDE.\n",
    "\n",
    "To learn more about Jupyter Notebooks use the introductory free online course available from [Here](https://www.earthdatascience.org/courses/intro-to-earth-data-science/open-reproducible-science/jupyter-python/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Data format and python libraries<a id='libs'></a>\n",
    "\n",
    "##### ZARR\n",
    "The data used in our tutorials have been converted from the [Met Office's PP file format](https://help.ceda.ac.uk/article/4424-pp-binary-forma) to Zarr. Zarr is a [specification](https://zarr.readthedocs.io/en/stable/spec.html) for how to store gridded data in a key-value interface (such as Amazon S3 object store), where each chunk of data is a separate value with a corresponding key indicating its position in the full dataset. This has advantage over NetCDF format as it allows for a highly parallel data access where many CPUs can simultaneously read different parts of the same dataset. Zarr is also a [Python library](https://zarr.readthedocs.io/en/stable/api.html) implementation of this specification that allows you to read and write data in a Zarr store.\n",
    "\n",
    "##### Iris\n",
    "In order to explore and analyse our dataset in these tutorials we make use of a Python library called Iris. Iris is a key tool in the [SciTools](https://scitools.org.uk/) project which is a collaborative effort to produce and maintain python-based open-source tools for Earth scientists. Iris is a useful toolkit as it supports read/write access to a range of data formats, including (CF-)netCDF, GRIB, and PP; fundamental data manipulation operations, such as arithmetic, interpolation, and statistics; and a range of integrated plotting options.  See [latest Iris documentation](https://scitools.org.uk/iris/docs/latest/) for more information.\n",
    "\n",
    "##### CATNIP\n",
    "At Met Office we have also developed a python library called CATNIP (Climate Analysis Tools: Now In Python). This library is a collection of routines to make frequently used climate data analysis and visualisation tasks in Iris easier and quicker to perform. We will make use of some of CATNIP's routines in these tutorials. See [CATNIP documentation](https://metoffice.github.io/CATNIP/#) for more information.\n",
    "\n",
    "##### CONDA\n",
    "For these tutorials we have used  the [CONDA](https://docs.conda.io/en/latest/) package management system to install the packages for our development environment. Conda is a package manager that helps you find and install packages. It allows you to set up Python environments (and other software packages) in a reproducible way that can be transferred between computers.\n",
    "\n",
    "You should have already set this environment up following the steps in the joining instructions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Resources<a id='resources'></a>\n",
    "\n",
    "The following are the links you can follow for further information for the packages that we have installed and use in these tutorials.\n",
    "\n",
    "- [python](https://docs.python.org/3/library/)\n",
    "- [zarr](https://zarr.readthedocs.io/en/stable/)\n",
    "- [iris](https://scitools.org.uk/iris/docs/latest/)\n",
    "- [numpy](https://numpy.org/)\n",
    "- [matplotlib](https://matplotlib.org/)\n",
    "- [xarray](http://xarray.pydata.org/en/stable/)\n",
    "- [jupyterlab](https://jupyterlab.readthedocs.io/en/stable/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p><img src=\"../img/MO_MASTER_black_mono_for_light_backg_RBG.png\" alt=\"Met Office Logo\" style=\"float: center; height: 100px;\"/></p>\n",
    "<center>© Crown Copyright, Met Office. All rights reserved.</center>\n",
    "<center>This file is part of PyPrecis and is released under the BSD 3-Clause license.</center>\n",
    "<center>See LICENSE in the root of the repository for full licensing details.</center>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}