4dn-dcic · willronchetti · Feb 3, 2020 · Oct 21, 2019 · Oct 21, 2019 · Oct 21, 2019
diff --git a/docs/source/conf.py b/docs/source/conf.py
@@ -49,4 +49,13 @@
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = ['_static']
+html_static_path = ['_static']
+
+# can add a logo on sidebar with:
+# html_logo = docs/source/img/...
+
+# Read the Docs configuration.
+# See: https://sphinx-rtd-theme.readthedocs.io/en/stable/configuring.html
+html_theme_options = {
+    'navigation_depth': 2
+}
diff --git a/docs/source/es_indexing.rst b/docs/source/es_indexing.rst
@@ -0,0 +1,17 @@
+Elasticsearch Indexing
+=====================
+
+**Work in progress!**
+
+Indexing is the process of building a complete document that contains multiple views of an item, then putting that document into Elasticsearch (ES). This is done whenever an item is created or changed, and acts as one of the backbones of Snovault, allowing searching of data and quick reading of complex views for items that are "cached" by using ES as a right storage.
+
+.. image:: img/indexing.png
+
+Figure 1: Diagram of the indexing process.
+
+Code
+-----------------
+* `indexer.py <https://github.com/4dn-dcic/snovault/blob/master/src/snovault/elasticsearch/indexer.py>`_: index endpoint and initialization, Indexer class
+* `mpindexer.py <https://github.com/4dn-dcic/snovault/blob/master/src/snovault/elasticsearch/mpindexer.py>`_: MPIndexer class and helper functions
+* `indexer_queue.py <https://github.com/4dn-dcic/snovault/blob/master/src/snovault/elasticsearch/indexer_queue.py>`_: QueueManager and endpoints for queueing and checking indexing
+* `indexing_views.py <https://github.com/4dn-dcic/snovault/blob/master/src/snovault/indexing_views.py>`_: index-data view and some other related endpoints
diff --git a/docs/source/es-mapping.rst → docs/source/es_mapping.rst b/docs/source/es-mapping.rst → docs/source/es_mapping.rst
diff --git a/docs/source/img/connection_storage.png b/docs/source/img/connection_storage.png
diff --git a/docs/source/img/indexing.png b/docs/source/img/indexing.png
diff --git a/docs/source/img/traversal.png b/docs/source/img/traversal.png
diff --git a/docs/source/index.rst b/docs/source/index.rst
@@ -1,75 +1,25 @@
+Snovault Documentation
 ========================
-Snovault
-========================
-
-Snovault is a JSON-LD Database Framework that serves as the backend for the 4DN Data portal and CGAP.
 
 |Build status|_
 
 .. |Build status| image:: https://travis-ci.org/4dn-dcic/snovault.svg?branch=master
 .. _Build status: https://travis-ci.org/4dn-dcic/snovault
 
-Installation Instructions
-=========================
-
-Currently these are for Mac OSX using homebrew. If using linux, install dependencies with a different package manager.
-
-Step 0: Install Xcode (from App Store) and homebrew: http://brew.sh::
-
-Step 1: Verify that homebrew is working properly::
-
-    $ sudo brew doctor
-
-
-Step 2: Install or update dependencies::
-
-    $ brew install libevent libmagic libxml2 libxslt openssl postgresql graphviz python3
-    $ brew install freetype libjpeg libtiff littlecms webp  # Required by Pillow
-    $ brew tap homebrew/versions
-    $ brew install elasticsearch@5.6
-
-If you need to update dependencies::
-
-    $ brew update
-    $ brew upgrade
-
-Step 3: Run buildout::
-
-    $ python3 bootstrap.py --buildout-version 2.9.5 --setuptools-version 36.6.0
-    $ bin/buildout
-
-    NOTE:
-    If you have issues with postgres or the python interface to it (psycogpg2) you probably need to install postgresql
-    via homebrew (as above)
-    If you have issues with Pillow you may need to install new xcode command line tools:
-    - First update Xcode from AppStore (reboot)
-    $ xcode-select --install
-    If you are running macOS Mojave, you may need to run the below command as well:
-    $ sudo installer -pkg /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg -target /
-
-
-
-If you wish to completely rebuild the application, or have updated dependencies:
-    $ make clean
-
-    Then goto Step 3.
-
-
-Running tests
-=============
-
-To run specific tests locally::
-
-    $ bin/test -k test_name
-
-To run with a debugger::
-
-    $ bin/test --pdb
-
-Specific tests to run locally for schema changes::
+Snovault is a JSON-LD Database Framework that serves as the backend for the `4DN Data portal <https://github.com/4dn-dcic/fourfront>`_ and `CGAP <https://github.com/dbmi-bgm/cgap-portal>`_. It is a very divergent fork of the work of the same name written by the ENCODE team at Stanford University. `See here <https://github.com/ENCODE-DCC/snovault>`_ for the original version.
 
-    $ bin/test -k test_load_workbook
+Since Snovault is used for multiple deployments across a couple projects, we use `GitHub releases <https://github.com/4dn-dcic/snovault/releases>_` to version it. This page also acts as a changelog.
 
-Run the Pyramid tests with::
+To get started, read the following documentation on setting up and developing Snovault:
 
-    $ bin/test
+.. toctree::
+  :titlesonly:
+  local_installation
+  testing
+  resources
+  storage_overview
+  traversal
+  resource_views
+  es_mapping
+  es_indexing
+  snowflakes
diff --git a/docs/source/local_installation.rst b/docs/source/local_installation.rst
@@ -0,0 +1,45 @@
+Local Installation
+==================
+
+Currently these are for macOS using homebrew. If using linux, install dependencies with a different package manager.
+
+Snovault is known to work with Python 3.6.x and will not work with Python 3.7 or greater. If part of the HMS team, it is recommended to use Python 3.4.3, since that's what is running on our servers. A good tool to manage multiple python versions is `pyenv <https://github.com/pyenv/pyenv>_`. It is best practice to create a fresh Python virtualenv using one of these versions before proceeding to the following steps.
+
+Step 0: Obtain AWS keys. These will need to added to your environment variables or through the AWS CLI (installed later in this process).
+
+Step 1: Verify that homebrew is working properly::
+
+    $ sudo brew doctor
+
+
+Step 2: Install or update dependencies::
+
+    $ brew install libevent libmagic libxml2 libxslt openssl postgresql graphviz
+    $ brew install freetype libjpeg libtiff littlecms webp  # Required by Pillow
+    $ brew tap homebrew/versions
+    $ brew install elasticsearch@5.6
+
+If you need to update dependencies::
+
+    $ brew update
+    $ brew upgrade
+
+Step 3: Run buildout::
+
+    $ python3 bootstrap.py --buildout-version 2.9.5 --setuptools-version 36.6.0
+    $ bin/buildout
+
+    NOTE:
+    If you have issues with postgres or the python interface to it (psycogpg2) you probably need to install postgresql
+    via homebrew (as above)
+    If you have issues with Pillow you may need to install new xcode command line tools:
+    - First update Xcode from AppStore (reboot)
+    $ xcode-select --install
+    If you are running macOS Mojave, you may need to run the below command as well:
+    $ sudo installer -pkg /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg -target /
+
+
+If you wish to completely rebuild the application, or have updated dependencies:
+    $ make clean
+
+    Then go to Step 3.
diff --git a/docs/source/resource_views.rst b/docs/source/resource_views.rst
@@ -0,0 +1,10 @@
+Resource Views
+===========================
+
+**Work in progress!**
+
+This document outlines the different base resource views and their sources. May be worth first reading the `traversal <https://snovault.readthedocs.io/en/latest/traversal.html>`_ and `storage <https://snovault.readthedocs.io/en/latest/storage_overview.html>`_ documentation.
+
+**TODO: outline each resource view with context=Item.**
+
+**TODO: Include relationship to storage and traversal (context and embed.py)**
diff --git a/docs/source/resources.rst b/docs/source/resources.rst
@@ -0,0 +1,12 @@
+Resources
+===========================
+
+**Work in progress!**
+
+This document outlines different classes that compose a base Snovault item. Code is located in the following files:
+
+- `resources.py <https://github.com/4dn-dcic/snovault/blob/master/src/snovault/resources.py>`_: Root, AbstractCollection, Collection, Item classes
+- `typeinfo.py <https://github.com/4dn-dcic/snovault/blob/master/src/snovault/typeinfo.py>`_: AbstractTypeInfo, TypeInfo, TypesTool
+- `config.py <https://github.com/4dn-dcic/snovault/blob/master/src/snovault/config.py>`_: CollectionsTool, collection and abstract_collection decorators
+
+**TODO: outline the role of each resource class. Include a complete example**
diff --git a/docs/source/search_info.rst b/docs/source/search_info.rst
diff --git a/docs/source/snowflakes.rst b/docs/source/snowflakes.rst
@@ -1,21 +1,20 @@
-================
 Snowflakes
 ================
 
 General
-^^^^^^^^
+-----------------
 
 Snowflakes used to be the front-end component of Snovault meant to serve as a demo. Since we at 4DN have our own Snovault-backed application (Fourfront, CGAP), snowflakes has been entirely removed from our version of Snovault. It is still present in ENCODE's version which you can find `here <https://github.com/ENCODE-DCC/snovault>`_ .
 
 Removing Snowflakes from Snovault proved more challenging than one may expect. Some parts of snowflakes were actually required for snovault to run, such as ``root.py``. These files have all been migrated into Snovault.
 
 Testing
-^^^^^^^^
+-----------------
 
 In addition, several relevant tests that lived in Snowflakes have been migrated into Snovault. These tests include only those that are specific to Snovault and are not covered in existing Fourfront/CGAP testing. Properly configuring the tests proved challenging as the test framework as previously configured intertwined Snowflakes and Snovault in such a way that Snovault tests could not function without the presence of Snovault.
 
 To fix this, several aspects of the tests have been refactored. We now load test schemas from files and have migrated many of the relevant fixtures from Snowflakes. ``config.py`` also required changes to account for behavior Snovault expected that it inherited from Snowflakes due to how includes work in PyTest.
 
-Test coverage for Snovault should still be fairly strong, especially when combined with that of Fourfront/CGAP. Some indexing tests are marked as flaky as we've found they experience intermittent failures. Updating how we clear the SQS queue has also helped to remidy this issue.
+Test coverage for Snovault should still be fairly strong, especially when combined with that of Fourfront/CGAP. Some indexing tests are marked as flaky as we've found they experience intermittent failures. Updating how we clear the SQS queue has also helped to remedy this issue.
 
-One issue of note that was not solved involved a particular logging related test that appears to pass on local and fail on Travis. The associated test is ``test_indexing_logging``. This tests makes a index post on the application and checks to see that a correct log message was emitted. The log message itself is emitted but for some reason on Travis it is truncated. Even spinning up Travis on an identical container could not reproduce the issue. The relevant line is marked in the test file. 
+One issue of note that was not solved involved a particular logging related test that appears to pass on local and fail on Travis. The associated test is ``test_indexing_logging``. This tests makes a index post on the application and checks to see that a correct log message was emitted. The log message itself is emitted but for some reason on Travis it is truncated. Even spinning up Travis on an identical container could not reproduce the issue. The relevant line is marked in the test file.