Merge branch 'develop'

Dockerfile

@@ -73,22 +73,35 @@ RUN ./configure CPPFLAGS="-I /usr/include/gdal" CXXFLAGS=-fpermissive \
   && make clean
 
 # Build FORCE from source
-RUN mkdir -p $INSTALL_DIR/force
+RUN mkdir -p $INSTALL_DIR/force && \
+  # This is needed in case of develop branch
+  mkdir -p /develop
 WORKDIR $INSTALL_DIR/force
 COPY . . 
-# Conditionally enable SPLITS which is disabled by default
-ARG splits=false 
-RUN if [ "$splits" = "true" ] ; then ./splits.sh enable; else ./splits.sh disable; fi
-# Conditionally enable DEBUG mode
+ARG splits=true 
 ARG debug=false 
-RUN if [ "$debug" = "true" ] ; then ./debug.sh enable; else ./debug.sh disable; fi
-RUN make -j7 \
+# Conditionally disable SPLITS which is enabled by default
+RUN if [ "$splits" = "false" ] ; then ./splits.sh disable; else ./splits.sh enable; fi && \
+  # Conditionally enable DEBUG mode
+  if [ "$debug" = "true" ] ; then ./debug.sh enable; else ./debug.sh disable; fi && \
+  # Compile FORCE
+  sed -i 's+BINDIR=/develop+BINDIR=/usr/local/bin+' Makefile && \
+  make -j7 \
   && make install \
   && make clean
 
 # Cleanup after successfull builds
 RUN rm -rf $INSTALL_DIR
 RUN apt-get purge -y --auto-remove apt-utils cmake git build-essential software-properties-common
 
-# Test FORCE run
-RUN force
+# Create a dedicated 'docker' group and user for running FORCE commands
+RUN groupadd docker && \
+  useradd -m docker -g docker -p docker && \
+  chgrp docker /usr/local/bin/ && \
+  chgrp docker /develop
+# Use this user by default
+USER docker
+
+WORKDIR /home/docker
+
+RUN force

README.md

@@ -2,7 +2,7 @@
 
 **Framework for Operational Radiometric Correction for Environmental monitoring**
 
-**Version 3.5.0**
+**Version 3.5.1**
 
 ![FORCE Logo](/images/force.png)
 

bash/force-level1-csd.sh

@@ -138,6 +138,7 @@ show_progress() {
   SIZEDONE=$(awk -v done=$SIZEDONE -v fsize=$FILESIZE 'BEGIN { print (done + fsize) }' )
   PERCDONE=$(awk -v total=$TOTALSIZE -v done=$SIZEDONE 'BEGIN { printf( "%.2f\n", (100 / total * done) )}')
   local WIDTH=$(($(tput cols) - 9)) PERCINT=$(( $(echo $PERCDONE | cut -d"." -f1) + 1 ))
+  if [ $ITER -eq 1 ]; then local PERCINT=0; fi
   printf -v INCREMENT "%*s" "$(( $PERCINT*$WIDTH/100 ))" ""; INCREMENT=${INCREMENT// /=}
   printf "\r\e[K|%-*s| %3d %% %s" "$WIDTH" "$INCREMENT" "$PERCINT" "$*"
 }
@@ -170,7 +171,7 @@ which_satellite() {
 # Initialize arguments and parse command line input
 SENSIN="LT04,LT05,LE07,LC08,S2A,S2B"
 DATEMIN="19700101"
-DATEMAX=$(date +%F)
+DATEMAX=$(date +%Y%m%d)
 CCMIN=0
 CCMAX=100
 TIER="T1"
@@ -518,6 +519,8 @@ get_data() {
       fi
       FILESIZE=$(( $FILESIZEBYTE / 1048576 ))
 
+      show_progress
+
       TILEPATH=$POOL/$TILE    
       SCENEPATH=$TILEPATH/$SCENEID
       if [ $SATELLITE = "sentinel2" ]; then
@@ -548,8 +551,7 @@ get_data() {
       lockfile-create $QUEUE
       echo "$SCENEPATH QUEUED" >> $QUEUE
       lockfile-remove $QUEUE
-
-      show_progress 
+
       ((ITER++))
     done
   fi

docs/source/components/higher-level/tsa/param.rst

@@ -524,6 +524,9 @@ The following parameter descriptions are a print-out of ``force-parameter``, whi
 
 .. _tsa-param-polar:
 
+
+* **Land surface phenology parameters (Polar-based)**
+
   .. note::
      The Polar-based Land Surface Phenology (LSP) metrics are always available (they do not rely on SPLITS).
 

docs/source/history/v3.rst

@@ -1,7 +1,112 @@
 .. _v3:
 
-Version 3
-=========
+FORCE v. 3.5.0
+--------------
+
+Release: 15.09.2020
+
+
+* **General changes**
+
+  * GDAL >= 3.0 support:
+
+    Since GDAL >= 3.0, the coordinates from coordinate transformation operations are no longer sorted as X/Y or LON/LAT, but in the typical order of each coordinate system.
+    This has caused an incompatibility of FORCE with GDAL >= 3.
+    This is fixed now. Still, do not use FORCE < 3.5 with GDAL >= 3. Update FORCE instead.
+
+  * Fixed a small bug that prevented program execution when the parameterfile was too long.
+
+  * FORCE programs will now transition to a Unix-typical usage, where non-mandatory options are specified with -o or --long-option.
+    This will happen from time to time in the next releases. 
+    The new program force-level1-csd is a prototype for this.
+
+  * Reduced the amount of compiler warnings when compiling with gcc 9.3.0 under Ubuntu 20.40 LTS.
+    This is ongoing work. The goal is to get rid of all warnings (they are not critical, though).
+
+* **FORCE LEVEL 1 ARCHIVING SYSTEM**
+
+  * new program force-level1-csd:
+
+    The new force-csd tool (cloud storage download) was kindly contributed by Stefan Ernst.
+    This tool goes way beyond the capabilities of the former sensor-specific scripts force-level1-landsat and force-level1-sentinel2.
+    It downloads Sentinel-2 AND Landsat data from the Google Cloud Storage.
+    You need the gutils python package, and you might need an account on Google's end. 
+    This is currently free. After communicating with Google, we expect this to remain free in the future.
+    force-csd is very efficient, you can use parallel downloads. 
+    You can filter the datapool w.r.t. sensor, date, cloud cover, tier level etc. 
+    The area of interest can be specified in a variety of ways, e.g. coordinate string as in force-level1-sentinel2, but a list of WRS-2/MGRS scenes, or vector geometries (e.g. shapefile) are also supported.
+    Of course, it also takes care about the file queues needed for Level 2 Processing.
+
+  * deprecated programs: force-level1-landsat and force-level1-sentinel2:
+
+    Due to the superiority of force-level1-csd, force-level1-landsat and force-level1-sentinel2 are now deprecated.
+    They will remain in the repo for a while, but will only receive minimal support in the future. 
+    Users are encouraged to change their workflow accordingly.
+
+* **FORCE HIGHER LEVEL**
+
+  * in force-higher-level, Level 3 sub-module: 
+
+    Included a safety query, which ensures that at least one score should be > 0.
+    Setting all scores to 0 resulted in some crashes.
+    Thanks to Jonas Ardö to report on this.
+
+  * in force-higher-level, TSA sub-module: 
+
+    A new suite of metrics was implemented: Land Surface Phenology descriptors based on a polar transformation, Polarmetrics it is.
+    This is based on the paper by Bjorn-Gustaf J. Brooks: https://www.mdpi.com/1999-4907/11/6/606, but was modified (e.g. to consider interannual shifts in the start of the phenological year) and complemented by many more metrics.
+    A couple of new parameters are now necessary, force-parameter has been updated to include these in the skeleton.
+    See :ref:`tsa-param-polar`.
+
+  * in force-higher-level, TSA sub-module: 
+
+    The domain tag of the bandwise FORCE metadata domain was updated to be interoperable with a 4D data model (see force-stack below).
+
+  * in force-higher-level, TSA sub-module: 
+
+    When folding the time series, the quantile statistics were broken with a memory error.
+    This is fixed now. Thanks for Christoph Raab and Benjamin Jakimow for reporting this.
+
+  * in force-higher-level, TSA sub-module, trend and CAT analyses: 
+
+    Added a new band to for computing relative change, i.e. gain/loss relative to initial value: (slope*timesteps)/offset. 
+    This makes most sense when the index has a physical meaning like fractional cover. 
+    This does not make sense at all when offset is negative! Take care.
+
+  * in force-higher-level, TSA sub-module, trend and CAT analyses: 
+
+    fixed an issue that compromised signifance levels for trends, which were computed on DOY-based phenometrics.
+
+* **FORCE AUX**
+
+  * in force-mosaic:
+
+    force-mosaic now copies all metadata to the generated mosaics.
+    For this, the new aux tool force-mdcp is used
+
+  * new program force-mdcp:
+
+    This new helper tool copies metadata from one file to another.
+    This included the FORCE metadata domains.
+    This program is now used by force-mosaic to carry the metadata to the generated mosaics.
+
+  * new program force-stack:
+
+    This new tool stacks files in VRT format.
+    This works with physical images (e.g. GeoTiffs), as well as VRT files (as e.g. generated by force-mosaic).
+    A variable number of input files can be stacked, wildcards are supported.
+    Most basically, this is e.g. useful to look at RGB combinations in QGIS, where RGB visualizations cannot be used when the channels are not in the same file.
+    If the number of bands in the input files is different, the images are stacked after another, e.g. file 1 band 1, file 1 band 2, file 2 band 1.
+    If the number of bands in the input files is the same, the images are stacked with band interleave, e.g. file 1 band 1, file 2 band 1, file 1 band 2, file 2 band 2.
+    The second option implements a 4D data model (think of time series), which is consistent with the QGIS plugins Raster Time Series Manager and Raster Data Plotting (C) Andreas Rabe.
+    A tutorial is in development to illustrate the interoperable use between FORCE and said QGIS plugins.
+
+  * in force-magic-parameters:
+
+    The user can now change between "all combinations" or "paired combinations".
+    Please refer to the program description:
+    https://force-eo.readthedocs.io/en/latest/components/auxilliary/magic-parameters.htm
+
 
 FORCE v. 3.4.0
 --------------

docs/source/history/vdev.rst

@@ -8,104 +8,4 @@ FORCE-dev
 
 Master release: TBA
 
-* **General changes**
-
-  * GDAL >= 3.0 support:
-
-    Since GDAL >= 3.0, the coordinates from coordinate transformation operations are no longer sorted as X/Y or LON/LAT, but in the typical order of each coordinate system.
-    This has caused an incompatibility of FORCE with GDAL >= 3.
-    This is fixed now. Still, do not use FORCE < 3.5 with GDAL >= 3. Update FORCE instead.
-
-  * Fixed a small bug that prevented program execution when the parameterfile was too long.
-
-  * FORCE programs will now transition to a Unix-typical usage, where non-mandatory options are specified with -o or --long-option.
-    This will happen from time to time in the next releases. 
-    The new program force-level1-csd is a prototype for this.
-
-  * Reduced the amount of compiler warnings when compiling with gcc 9.3.0 under Ubuntu 20.40 LTS.
-    This is ongoing work. The goal is to get rid of all warnings (they are not critical, though).
-
-* **FORCE LEVEL 1 ARCHIVING SYSTEM**
-
-  * new program force-level1-csd:
-
-    The new force-csd tool (cloud storage download) was kindly contributed by Stefan Ernst.
-    This tool goes way beyond the capabilities of the former sensor-specific scripts force-level1-landsat and force-level1-sentinel2.
-    It downloads Sentinel-2 AND Landsat data from the Google Cloud Storage.
-    You need the gutils python package, and you might need an account on Google's end. 
-    This is currently free. After communicating with Google, we expect this to remain free in the future.
-    force-csd is very efficient, you can use parallel downloads. 
-    You can filter the datapool w.r.t. sensor, date, cloud cover, tier level etc. 
-    The area of interest can be specified in a variety of ways, e.g. coordinate string as in force-level1-sentinel2, but a list of WRS-2/MGRS scenes, or vector geometries (e.g. shapefile) are also supported.
-    Of course, it also takes care about the file queues needed for Level 2 Processing.
-
-  * deprecated programs: force-level1-landsat and force-level1-sentinel2:
-
-    Due to the superiority of force-level1-csd, force-level1-landsat and force-level1-sentinel2 are now deprecated.
-    They will remain in the repo for a while, but will only receive minimal support in the future. 
-    Users are encouraged to change their workflow accordingly.
-
-* **FORCE HIGHER LEVEL**
-
-  * in force-higher-level, Level 3 sub-module: 
-
-    Included a safety query, which ensures that at least one score should be > 0.
-    Setting all scores to 0 resulted in some crashes.
-    Thanks to Jonas Ardö to report on this.
-
-  * in force-higher-level, TSA sub-module: 
-
-    A new suite of metrics was implemented: Land Surface Phenology descriptors based on a polar transformation, Polarmetrics it is.
-    This is based on the paper by Bjorn-Gustaf J. Brooks: https://www.mdpi.com/1999-4907/11/6/606, but was modified (e.g. to consider interannual shifts in the start of the phenological year) and complemented by many more metrics.
-    A couple of new parameters are now necessary, force-parameter has been updated to include these in the skeleton.
-    See :ref:`tsa-param-polar`.
-
-  * in force-higher-level, TSA sub-module: 
-
-    The domain tag of the bandwise FORCE metadata domain was updated to be interoperable with a 4D data model (see force-stack below).
-
-  * in force-higher-level, TSA sub-module: 
-
-    When folding the time series, the quantile statistics were broken with a memory error.
-    This is fixed now. Thanks for Christoph Raab and Benjamin Jakimow for reporting this.
-
-  * in force-higher-level, TSA sub-module, trend and CAT analyses: 
-
-    Added a new band to for computing relative change, i.e. gain/loss relative to initial value: (slope*timesteps)/offset. 
-    This makes most sense when the index has a physical meaning like fractional cover. 
-    This does not make sense at all when offset is negative! Take care.
-
-  * in force-higher-level, TSA sub-module, trend and CAT analyses: 
-
-    fixed an issue that compromised signifance levels for trends, which were computed on DOY-based phenometrics.
-
-* **FORCE AUX**
-
-  * in force-mosaic:
-
-    force-mosaic now copies all metadata to the generated mosaics.
-    For this, the new aux tool force-mdcp is used
-
-  * new program force-mdcp:
-
-    This new helper tool copies metadata from one file to another.
-    This included the FORCE metadata domains.
-    This program is now used by force-mosaic to carry the metadata to the generated mosaics.
-
-  * new program force-stack:
-
-    This new tool stacks files in VRT format.
-    This works with physical images (e.g. GeoTiffs), as well as VRT files (as e.g. generated by force-mosaic).
-    A variable number of input files can be stacked, wildcards are supported.
-    Most basically, this is e.g. useful to look at RGB combinations in QGIS, where RGB visualizations cannot be used when the channels are not in the same file.
-    If the number of bands in the input files is different, the images are stacked after another, e.g. file 1 band 1, file 1 band 2, file 2 band 1.
-    If the number of bands in the input files is the same, the images are stacked with band interleave, e.g. file 1 band 1, file 2 band 1, file 1 band 2, file 2 band 2.
-    The second option implements a 4D data model (think of time series), which is consistent with the QGIS plugins Raster Time Series Manager and Raster Data Plotting (C) Andreas Rabe.
-    A tutorial is in development to illustrate the interoperable use between FORCE and said QGIS plugins.
-
-  * in force-magic-parameters:
-
-    The user can now change between "all combinations" or "paired combinations".
-    Please refer to the program description:
-    https://force-eo.readthedocs.io/en/latest/components/auxilliary/magic-parameters.htm
-
+No changes yet, master is in sync with develop.

docs/source/index.rst

@@ -5,7 +5,7 @@ FORCE documentation
 
 **FORCE: Framework for Operational Radiometric Correction for Environmental monitoring**
 
-**Version 3.5.0**
+**Version 3.5.1**
 
 `Download from Github <https://github.com/davidfrantz/force>`_.