updated documentation throughout

README.rst

@@ -33,34 +33,25 @@ Welcome to The REsource eXtraction (rex) tool!
 
 .. inclusion-intro
 
-rex command line tools
-======================
-
-- `rex <https://nrel.github.io/rex/_cli/rex.html#rex>`_
-- `NSRDBX <https://nrel.github.io/rex/_cli/NSRDBX.html#NSRDBX>`_
-- `WINDX <https://nrel.github.io/rex/_cli/WINDX.html#WINDX>`_
-- `US-wave <https://nrel.github.io/rex/_cli/US-wave.html#US-wave>`_
-- `WaveX <https://nrel.github.io/rex/_cli/WaveX.html#Wavex>`_
-- `MultiYearX <https://nrel.github.io/rex/_cli/MultiYearX.html#MultiYearX>`_
-- `rechunk <https://nrel.github.io/rex/_cli/rechunk.html#rechunk>`_
-- `temporal-stats <https://nrel.github.io/rex/_cli/temporal-stats.html#temporal-stats>`_
-- `wind-rose <https://nrel.github.io/rex/_cli/wind-rose.html#wind-rose>`_
-
-Using Eagle Env
-===============
-
-If you would like to run `rex` on Eagle (NREL's HPC) you can use a pre-compiled
-conda env:
-
-.. code-block:: bash
-
-    conda activate /shared-projects/rev/modulefiles/conda/envs/rev/
-
-or
-
-.. code-block:: bash
-
-    source activate /shared-projects/rev/modulefiles/conda/envs/rev/
+What is rex?
+=============
+``rex`` stands for **REsource eXtraciton** tool.
+
+``rex`` enables the efficient and scalable extraction, manipulation, and
+computation with NRELs flagship renewable resource datasets such as: the Wind
+Integration National Dataset (WIND Toolkit), the National Solar Radiation
+Database (NSRDB), the Ocean Surface Wave Hindcast (US Wave) Data, and the
+High-resolution downscaled climate change data (Sup3rCC).
+
+To get started accessing NREL's datasets, see the primer on `NREL Renewable
+Energy Resource Data
+<https://nrel.github.io/rex/misc/examples.nrel_data.html>`_ or the
+`installation instructions <https://nrel.github.io/rex/#installing-rex>`_.
+
+You might also want to check out the basic `Resource Class
+<https://nrel.github.io/rex/_autosummary/rex.resource.Resource.html>`_ that
+can be used to efficiently query NREL data, or our various `example use cases
+<https://nrel.github.io/rex/misc/examples.html>`_.
 
 Installing rex
 ==============
@@ -78,12 +69,13 @@ Option 1: Install from PIP or Conda (recommended for analysts):
 2. Activate directory:
     ``conda activate rex``
 
-3. Install rex:
-    1) ``pip install NREL-rex`` or
-    2) ``conda install nrel-rex --channel=nrel``
+3. Basic ``rex`` install:
+    1) ``pip install NREL-rex``
+    2) or ``conda install nrel-rex --channel=nrel``
 
-       - NOTE: If you install using conda and want to use `HSDS <https://github.com/NREL/hsds-examples>`_
-         you will also need to install h5pyd manually: ``pip install h5pyd``
+4. Install for users outside of NREL that want to access data via HSDS or S3 as per the instructions `here <https://nrel.github.io/rex/misc/examples.nrel_data.html#data-location-external-users>`_:
+    1) ``pip install NREL-rex[s3]`` for easy no-setup direct access of the data on S3 via ``fsspec`` as per `this example <https://nrel.github.io/rex/misc/examples.fsspec.html>`_
+    2) or ``pip install NREL-rex[hsds]`` for more performant access of the data on HSDS with slightly more setup as per `this example <https://nrel.github.io/rex/misc/examples.hsds.html>`_
 
 Option 2: Clone repo (recommended for developers)
 -------------------------------------------------
@@ -109,11 +101,15 @@ Option 2: Clone repo (recommended for developers)
         - ``WINDX``
         - ``US-wave``
 
-Recommended Citation
-====================
-
-Update with current version and DOI:
+rex command line tools
+======================
 
-Michael Rossol, Grant Buster. The REsource Extraction Tool (rex).
-https://github.com/NREL/rex (version v0.2.43), 2021.
-https://doi.org/10.5281/zenodo.4499033.
+- `rex <https://nrel.github.io/rex/_cli/rex.html#rex>`_
+- `NSRDBX <https://nrel.github.io/rex/_cli/NSRDBX.html#NSRDBX>`_
+- `WINDX <https://nrel.github.io/rex/_cli/WINDX.html#WINDX>`_
+- `US-wave <https://nrel.github.io/rex/_cli/US-wave.html#US-wave>`_
+- `WaveX <https://nrel.github.io/rex/_cli/WaveX.html#Wavex>`_
+- `MultiYearX <https://nrel.github.io/rex/_cli/MultiYearX.html#MultiYearX>`_
+- `rechunk <https://nrel.github.io/rex/_cli/rechunk.html#rechunk>`_
+- `temporal-stats <https://nrel.github.io/rex/_cli/temporal-stats.html#temporal-stats>`_
+- `wind-rose <https://nrel.github.io/rex/_cli/wind-rose.html#wind-rose>`_

docs/source/index.rst

@@ -10,14 +10,4 @@
 rex documentation
 *******************
 
-What is rex?
-=============
-rex stands for **REsource eXtraciton** tool.
-
-rex enables the efficient and scalable extraction, manipulation, and
-computation with NRELs flagship renewable resource datasets:
-the Wind Integration National Dataset (WIND Toolkit), and the National Solar
-Radiation Database (NSRDB)
-
 .. include:: ../../README.rst
-   :start-after: inclusion-intro

docs/source/misc/installation.rst

@@ -5,13 +5,6 @@ Installation
    :start-after: Installing rex
    :end-before: Recommended Citation
 
-Usage on Eagle
-==============
-
-.. include:: ../../../README.rst
-   :start-after: Using Eagle Env
-   :end-before: Installing rex
-
 Command Line Tools
 ==================
 

examples/HSDS/README.rst

@@ -1,7 +1,26 @@
 Highly Scalable Data Service (HSDS)
 ===================================
 
-`The Highly Scalable Data Service (HSDS) <https://www.hdfgroup.org/solutions/highly-scalable-data-service-hsds/>`_ is a cloud-optimized solution for storing and accessing HDF5 files, e.g. the NREL wind and solar datasets. You can access NREL data via HSDS in a few ways. Read below to find out more.
+`The Highly Scalable Data Service (HSDS)
+<https://www.hdfgroup.org/solutions/highly-scalable-data-service-hsds/>`_ is a
+cloud-optimized solution for storing and accessing HDF5 files, e.g. the NREL
+wind and solar datasets. You can access NREL data via HSDS in a few ways. Read
+below to find out more.
+
+Note that raw NREL .h5 data files are hosted on AWS S3. In contrast, the files
+on HSDS are not real "files". They are just domains that you can access with
+h5pyd or rex tools to stream small chunks of the files stored on S3. The
+multi-terabyte .h5 files on S3 would be incredibly cumbersome to access
+otherwise.
+
+Extra Requirements
+------------------
+
+You may need some additional software beyond the basic ``rex`` install to run this example:
+
+.. code-block:: bash
+
+    pip install NREL-rex[hsds]
 
 NREL Developer API
 ------------------

examples/NREL_Data/README.rst

@@ -70,29 +70,24 @@ However, this method is slow. The most performant method is via ``HSDS``.
 or server for just the time or space domain you're interested in.
 
 See `this docs page <https://nrel.github.io/rex/misc/examples.fsspec.html>`_
-for easy (but slow) access of the source .h5 files on s3 with ``fsspec``.
+for easy (but slow) access of the source .h5 files on s3 with ``fsspec`` that
+requires basically zero setup. To find relevant S3 files, you can explore the
+S3 directory structure on `OEDI <https://openei.org/wiki/Main_Page>`_ or
+with the `AWS CLI <https://aws.amazon.com/cli/>`_
 
 See `this docs page <https://nrel.github.io/rex/misc/examples.hsds.html>`_ for
-instructions on how to set up HSDS and then continue on to the Data Access
-Examples section below.
+instructions on how to set up HSDS for more performant data access that
+requires a bit of setup. To find relevant HSDS files, you can use HSDS and
+h5pyd to explore the NREL public data directory listings. For example, if you
+are running an HSDS local server, you can use the CLI utility ``hsls``, for
+example, run: ``$ hsls /nrel/`` or ``$ hsls /nrel/nsrdb/v3/``. You can also use
+h5pyd to do the same thing. In a python kernel, ``import h5pyd`` and then run:
+``print(list(h5pyd.Folder('/nrel/')))`` to list the ``/nrel/`` directory.
 
 There is also an experiment with using `zarr
 <https://nrel.github.io/rex/misc/examples.zarr.html>`_, but the examples below
 may not work with these utilities and the zarr example is not regularly tested.
 
-To find relevant HSDS files, you can use HSDS and h5pyd to explore the NREL
-public data directory listings. For example, if you are running an HSDS local
-server, you can use the CLI utility ``hsls``, for example, run: ``$ hsls
-/nrel/`` or ``$ hsls /nrel/nsrdb/v3/``. You can also use h5pyd to do the same
-thing. In a python kernel, ``import h5pyd`` and then run:
-``print(list(h5pyd.Folder('/nrel/')))`` to list the ``/nrel/`` directory.
-
-Note that raw NREL .h5 data files are hosted on AWS S3. In contrast, the files
-on HSDS are not real "files". They are just domains that you can access with
-h5pyd or rex tools to stream small chunks of the files stored on S3. The
-multi-terabyte .h5 files on S3 would be incredibly cumbersome to access
-otherwise.
-
 The `Open Energy Data Initiative (OEDI) <https://openei.org/wiki/Main_Page>`_
 is also invaluable for finding the source s3 filepaths and for finding
 energy-relevant public datasets that are not necessarily spatiotemporal h5
@@ -105,7 +100,6 @@ Data Access Examples
 If you are on the NREL HPC, update the file paths with the relevant NREL HPC
 file paths in ``/datasets/``.
 
-
 If you are not at NREL, see the "Data Location - External Users" section above
 for S3 instructions or for how to setup HSDS and how to find the files that
 you're interested in. Then update the file paths to the files you want either

examples/fsspec/README.rst

@@ -12,14 +12,14 @@ You may need some additional software beyond the basic ``rex`` install to run th
 
 .. code-block:: bash
 
-    pip install fsspec s3fs
+    pip install NREL-rex[s3]
 
 Code Example
 ------------
 
-To open an .h5 file hosted on AWS S3, simply use a path to an S3 resource.:
+To open an .h5 file hosted on AWS S3, simply use a path to an S3 resource with any of the ``rex`` file handlers:
 
-- Change ``fp`` to your desired AWS .h5 resource paths (find the s3 paths on `Open Energy Data Initiative (OEDI) <https://openei.org/wiki/Main_Page>`_).
+- Change ``fp`` to your desired AWS .h5 resource paths (find the s3 paths on `OEDI <https://openei.org/wiki/Main_Page>`_ or with the `AWS CLI <https://aws.amazon.com/cli/>`_).
 - Running this example on a laptop, it takes ~14 seconds to read the meta data, and another ~14 seconds to read the GHI timeseries. This may be faster when running on AWS services in the same region hosting the .h5 file. It is much slower when running on the NREL VPN.
 
 .. code-block:: python

rex/multi_time_resource.py

@@ -232,7 +232,8 @@ def _get_hsds_file_paths(h5_path, hsds_kwargs=None):
             import h5pyd
         except Exception as e:
             msg = (f'Tried to open hsds file path: "{h5_path}" with '
-                   'h5pyd but could not import, try `pip install h5pyd`')
+                   'h5pyd but could not import, try '
+                   '`pip install NREL-rex[hsds]`')
             logger.error(msg)
             raise ImportError(msg) from e
 
@@ -291,7 +292,7 @@ def _get_s3_file_paths(h5_path):
         except Exception as e:
             msg = (f'Tried to open s3 file path: "{h5_path}" with '
                    'fsspec but could not import, try '
-                   '`pip install fsspec s3fs`')
+                   '`pip install NREL-rex[s3]`')
             logger.error(msg)
             raise ImportError(msg) from e
 

rex/resource.py

@@ -1150,7 +1150,8 @@ def open_file(cls, file_path, mode='r', hsds=False, hsds_kwargs=None):
                 import h5pyd
             except Exception as e:
                 msg = (f'Tried to open hsds file path: "{file_path}" with '
-                       'h5pyd but could not import, try `pip install h5pyd`')
+                       'h5pyd but could not import, try '
+                       '`pip install NREL-rex[hsds]`')
                 logger.error(msg)
                 raise ImportError(msg) from e
 
@@ -1171,7 +1172,7 @@ def open_file(cls, file_path, mode='r', hsds=False, hsds_kwargs=None):
             except Exception as e:
                 msg = (f'Tried to open s3 file path: "{file_path}" with '
                        'fsspec but could not import, try '
-                       '`pip install fsspec s3fs`')
+                       '`pip install NREL-rex[s3]`')
                 logger.error(msg)
                 raise ImportError(msg) from e