Timeseries Analysis and Plotting (TAP) Tool

Abstract

The Timeseries Analysis and Plotting (TAP) Tool is being developed to provide exploratory data analysis for the Land Change Monitoring, Assessment, and Projection (LCMAP) project at USGS EROS. TAP is an open-source project written in python that uses PyQt5 and matplotlib for generating interactive GUIs and plots. Currently, it provides the following functionality:

• Plot a timeseries of Landsat Analysis Ready Data (ARD)
  • Landsat 4-7 Bands 1, 2, 3, 4, 5, 7 Surface Reflectance
  • Landsat 8 Bands 2-7 Surface Reflectance
  • Landsat 4-7 Band 6 Brightness Temperature
  • Landsat 8 Band 10 Brightness Temperature
• Plot model parameters and curve-fits from the Continuous Change Detection and Classification (CCDC) algorithm
• Calculate indices on the fly for plotting and visualization
  • NDVI
  • MSAVI
  • SAVI
  • EVI
  • NDMI
  • NBR
  • NBR-2
• Display RGB visualizations of Landsat ARD
• Export plotted Landsat ARD surface reflectance, brightness temperature, and index values to a .csv file
• Save a plot figure as a .png file
• Save a Landsat ARD RGB display as a .png file
• Save ESRI point shapefile for plotted coordinates

Installation

These instructions work for both Windows and Linux systems.

lcmap_tap has been installed and tested successfully on Windows 7 & 10, and CentOS Linux (version 7).

Note: some functionality could be unstable or intermittent as development of LCMAP operations is ongoing.

System Requirements

python >= 3.6

PyQt5 == 5.10.1*

matplotlib

numpy

GDAL

pandas

pyyaml

cytoolz

cython

requests

lcmap-merlin

* Note on PyQt5: For development any version of PyQt5>=5.9 is currently suitable. For running TAP, it is recommended to use PyQt5==5.10.1 because of an issue in later versions in which Qtwebengineprocess does not close on TAP exit. If building a stand-alone executable with PyInstaller, PyQt5==5.10.1 seems to work best at the moment.

---

The tool currently requires the following data inputs:

• Serialized change and cover results generated by PyCCD.
• Landsat ARD (obtained via HTTP request)

Landsat ARD and the PyCCD algorithm are publicly available. However, TAP currently requires access to ARD via a web-service that is currently available to on-site personnel at USGS EROS only. In addition, the pre-generated PyCCD results must be stored locally and are expected to be in a specific format. For these reasons, TAP is not intended for general public use until these datasets become widely available at a future date.

---

Note

It is recommended to use an Anaconda or Miniconda virtual environment since it provides an easier method of installation for GDAL, cython, and cytoolz. Otherwise, information for installing GDAL manually can be found here.

---

Create a Virtual Environment

• Install Anaconda or Miniconda

  • Download here

• Create a virtual environment with python 3.6.7, include the following dependencies in the environment creation:

  • GDAL 2.2.2
  • cython
  • cytoolz
  • numpy 1.15.4
  • pandas

  conda create -n <env-name> python=3.6.7 gdal=2.2.2 cython cytoolz numpy=1.15.4 pandas
  
  The following NEW packages will be INSTALLED:
  ...
  Proceed ([y]/n)? y

• If you experience SSL cert errors, ssl cert verification can be temporarily disabled for conda via:

  conda config --set ssl_verify false

• Download a ZIP archive of the TAP source code for the target tag/branch.

  • Master
  • Develop

• Extract the zipped folder.

• From the command line, cd into the extracted folder.

Install TAP Source Code

• From the command line, activate the virtual environment (must use source if using bash)

  conda activate <env-name>

• Use pip to install TAP and the remaining dependencies (matplotlib, PyQt, PyYAML, and requests)

  • The reason for using pip to install PyQt is that the most recent version available on conda is 5.9.2 as of 2/15/2019

    Windows and Linux

    # From inside the extraced lcmap-tap ZIP archive (with setup.py)
    pip install --trusted-host pypi.org --trusted-host python.pypi.org --trusted-host files.pythonhosted.org .
    ...
    

Run the Tool

Once installed, lcmap_tap can be executed directly from the command line if the virtual environment is activated:

activate env-name
lcmap_tap

Packaging

Packaging tap-tool using PyInstaller for distribution of an executable binary.

Pre-Reqs

• The following instructions assume the above Installation steps have already been completed.
• Conda virtual environment is activated
• Current working directory is inside the extracted lcmap-tap ZIP archive

Instructions

• Install the most recent version of PyInstaller with pip directly from github:

  pip install --trusted-host python.pypi.org --trusted-host pypi.org --trusted-host files.pythonhosted.org --trusted-host github.com --trusted-host codeload.github.com https://github.com/pyinstaller/pyinstaller/archive/develop.zip

• Create a new file - a short script that calls lcmap-tap, save as 'run_lcmap_tap.py'

  from lcmap_tap.__main__ import main
  main()

• Freeze the script – this will recursively search all imported modules in lcmap_tap to include in the build environment:

  pyinstaller run_lcmap_tap.py

• Create a new file 'config.yaml' in the lcmap-tap code top level directory (should contain setup.py). Contents should be:

  URL: http://lcmap-test.cr.usgs.gov/ARD_CU_C01_V01
  CCD: Z:\bulk\tiles

• Once complete, open the run_lcmap_tap.spec file to edit.

• We need to tell PyInstaller to include certain non-python data files. Add the following to the 'datas' list.

  datas=[( '.\\config.yaml', 'lcmap_tap'),
         ('.\\lcmap_tap\\Auxiliary\\icon.png', 'lcmap_tap/Auxiliary'),
         ('.\\lcmap_tap\\MapCanvas\\UseWebEngineView', 'lcmap_tap/MapCanvas/UseWebEngineView'),
         ('.\\lcmap_tap\\MapCanvas\\UseWebView', 'lcmap_tap/MapCanvas/UseWebView')],

  • Note: the spec file is read as a python file. Datas is a list of tuples, each tuple contains two strings, the first is the path to the data file or folder, the second is the destination for the data file in the build module.

• Update the hidden imports in the run_lcmap_tap.spec file to include:

  hiddenimports=['PyQt5.sip', 'gdal', 'ogr', 'osr', 'merlin'],

  • Note: Doing this seems to fix an issue with not being able to import gdal later on. I think it also helps avoid errors with importing these other modules as well.

• Re-run PyInstaller, but point to the spec file rather than the script file

pyinstaller run_lcmap_tap.spec

• The resulting 'dist' folder contains the stand-alone TAP tool and this is what should be distributed on to users' systems.

  • Zip up this directory for distribution: dist/run_lcmap_tap

• Run TAP simply by double-clicking the executable file, or a shortcut that points to this executable:

dist/run_lcmap_tap/run_lcmap_tap.exe