Version 1.3.1

Major changes

• Documentation was improved with three new vignettes about GUI usage, command line usage and output structure.

Bug fixes

• Restore checking environmental variable in init_python() (this urgent bug fixing was the reason of the urgent release of the current version).
• Fix building offline footprint.
• Fix the Dockerfile.

Changes in default argument values

• Accept both sen2r(..., rm_safe = "yes") and "all".
• In the GUI, the maximum allowed cloud cover was set to 100% accordingly to the sen2r() defaults.

Minor changes

• Add check on max_mask - mask_type coherence (if max_mask < 100 and mask_type is not specified, a warning is returned).
• Suppress some useless warnings.
• Switch examples on 2019 dates (previously examples were based on 2017 images, which were partially moved on LTA).
• Do not print progress bars in logs.

Version 1.3.0

Major changes

• Support for GDAL 3 / PROJ 6 was added:
  • Edit internal function to deal with CRS instead than PROJ.4 strings
  • Allow st_crs2() accepting also WKT strings / EPSG codes in the form "EPSG:xxxx"
  • Edit GUI to accept WKT / to warn if a PROJ.4 is passed
• After running sen2r() a short report is returned to output summarising the status of the required processing (see new internal function sen2r_process_report()). This should facilitate applying sen2r in a cycle until all "expected dates" are processed.
• Management of Sen2Cor GIPP parameters: now the user can process L1C SAFE images applying a topographic correction (as done to produce L2A images on ESA Hub) or specifying other parameters managed by L2A_GIPP.xml file. To allow doing that, new arguments were added to sen2cor() and sen2r() (see below) and a new option was added in the GUI.

New functions

• read_gipp() and set_gipp() to read / create GIPP XML files with the parameters passed to Sen2Cor.
• Internal function st_as_text_2() which returns WKT or WKT2 depending on PROJ version
• Internal function sen2r_process_report() to manage processing reports returned by sen2r().

Deprecated functions

• Internal function init_python() was deprecated (Python is no longer managed by reticulate).

Changes in default argument values

• New argument bigtiff for BigTIFF management in functions s2_translate(), s2_merge(), s2_mask(), s2_rgb() and s2_calcindices().
• New arguments use_dem / gipp (function sen2cor()) and sen2cor_use_dem / sen2cor_gipp (sen2r()) to manage GIPP parameters in Sen2Cor.
• use_python argument in sen2r() was deprecated (no longer needed).

Minor changes

• All components of the order (available, ordered and notordered) are now saved in JSON files. This allows using them to re-do an order, specifying if re-ordering already ordered datasets or only order the ones identified as "notordered" (based on new argument "reorder").
• Output messages are formatted so not to exceed output line length.
• Add support for BigTIFF format.
• Dependencies reticulate and magrittr were removed.
• Checks on the validity of SAFE products were improved.
• Add NDBI / NDVIre indices (#241).
• Add footprint among metadata retrieved using safe_getMetadata() (from existing offline SAFE archives) and s2_list() (from SciHub online metadata).
• Filter offline SAFE archives by footprint instead than by S2 tile ID, so skipping using images which do not contain any data for a specific Area Of Interest.
• Add method to convert class safelist to sf (using footprint).
• Rewrite st_crs2() using class methods (see the function help for newly accepted inputs).
• safe_getMetadata() now directly read XML SAFE files instead than calling GDAL through reticulate (this avoids errors in particular OS conditions).
• Do not use PROJ4 string with PROJ > 3 (https://rsbivand.github.io/ECS530_h19/ECS530_III.html).
• Retrieve the GDAL installation path using gdal-config instead than whereis gdalinfo.
• Add a test for reprojection without EPSG.
• Return a warning if SciHub is down.
• Improve documentation about the problem occurring using newly created credentials.

Fixes

• Fix errors "DLL load failed" with Python GDAL scripts on Windows (see #231 and #234) due to a missing proj.db file in rgdal.
• Hide warning using PROJ >= 6 (rspatial/raster#78).
• Other various bug fixes (see issues).

Version 1.2.1

Major changes

• New safelist class for lists of SAFE Sentinel-2 archives (see safelist-class);
• sen2r() execution now causes saving the used parameters in a json file located in ~/.sen2r/proc_par (this can be used to take trace of the executed processing chains);
• safe_getMetadata() function was rewritten: now files are scanned only if the user requires metadata which must be retrieved from file content, otherwise only file names are analysed.
  Inputs can be also safelist objects.
  The support for oldname SAFE products is deprecated.

New functions / methods

• link_sen2cor() can be used to link an existing Sen2Cor installation to sen2r;
• as() methods can be used to convert safelist from/to character, data.frame and data.table (as well as as.character()', as.data.frame()andas.data.table()` functions).

Changes in default values

• safe_getMetadata(): new arguments format, simplify and allow_oldnames (see the function reference for details);
• s2_list(): argument output_type was deprecated (use as.data.table to obtain a data.table instead than the new default safelist object);
• safe_shortname(): arguments tiles, force_tiles, set.seed and multiple_names were deprecated, since they are not used with SAFE compact names (old names are no more supported);

Minor changes

• Sys.setenv() effects now do not affect the R environment after exiting from sen2r() execution;
• safe_shortname() does no more support oldname SAFE products;
• the Docker version of the package was modified to support last package changes;
• the delay between SciHub registration of new credentials and the possibility to use them on API hub is now documented in the GUI;
• return a warning in case some LTA orders cannot be processed because user quota exceeded.

Version 1.2.0

Starting from this version sen2r supports ordering products from Long Term Archive (LTA) (see news at https://inthub.copernicus.eu/userguide/LongTermArchive). Now the user can automatically order SAFE products which are not available for direct download, and use them when made available. Some internal functions can be exploited to manually manage orders.

Here above the related changes:

New functions

• safe_is_valid() to check if an order was processed;
• s2_order() to order products from LTA.

New arguments

• sen2r() and s2_download() have a new argument order_lta (default: TRUE) to order SAFE archives not available for direct download;
• s2_gui() has a new checkbox to set the previous argument.

Other changes (not related with LTA)

• Function build_example_param_file() does no more compute TOA and RGB432T (this to avoid downloading 2 SAFE archives).
• Code coverage was expanded.

Version 1.1.0

Changes in default values

• The default value for "list_prods" argument in sen2r() is NULL instead of 'BOA'.
• Remove 'no' value for argument "step_atmcorr" in sen2r() (use 'l2a' without selecting any L2A products instead).
• Default values for "smoothing" and "buffer" arguments in s2_mask() are now 0 (like in sen2r()) instead than 10/20 m.
• Set parallel = FALSE as default in sen2r() (parallelisation must be explicitly required).
• Since old name products were repackaged by ESA, these products are no longer supported.
• In the case the users manually defines the path to be used as temporary directory and this directory already exists, a casual subdirectory of the user-defined folder is used (this to prevent to accidentally delete existing files).

Major changes

• sen2r package directory is no more used to store files, but a subfolder .sen2r of the user Home directory is created and used.
  In this way, reinstalling sen2r will not require anymore to reinstall runtime dependencies / to reconfigure the package.
  The permission to write on this folder is asked to the user when the package is loaded for the first time.
• s2_list() and s2_download() now use internal methods instead of calling python function derived from s2download scripts by Hagolle (this substantially improve the speed of SAFE search), which were removed.
• sen2r is now supported also on MacOS.

New/removed dependencies

• Remove sp and rgdal from explicit dependencies.
• Remove stringr and gdalUtils dependencies.
• Add stars dependency.

New functions

• str_pad2(), equivalent to str_pad() without the needing to load "stringr".
• raster_metadata(), to get raster metadata, without rgdal dependency.
• check_scihub_login() to check if SciHub credentials are right.
• check_scihub_connection() to check for internet connectivity.
• build_example_param_file() (not exported) to build an example parameter file.

Other changes

• Packages stability was improved by adding test functions, working with testthat.
• S2 tiles are not provided as kmz within the package, but a rds file is downloaded after installing the package (in this way, package installation is lighter, and S2 tiles are read faster).
• Function check_gdal() was rewritten without using gdalUtils, in order to be simpler and faster.
• Some new indices were added:
  • NDWI-NDWI2 (#184);
  • CRred-BDred-CRred2 (#168);
  • Indices from Sentinel-hub indices were checked.
• The minimum convex hull of the extent is used instead of the bbox to search SAFE images.
• Several "soft" error/warning messages were introduced.
• ENVI headers now includes some information about band names (TOA/BOA) and SCL classes.
• Coupling SAFE: in offline mode, if some L2A products exists but not the corresponding L1C, and TOA-derived products are required, only the "coupled" SAFE archives are used.
• Small base example files were added to the package for testing purposes.
• Sen2Cor version which the user wants to install can be chosen in the GUI.
• Documentation was updated and improved, in particular adding function examples.
• Unused old code was removed.
• Several bugs were fixed.

Version 1.0.2

Changes in default argument values

• the default running order is now "by groups" instead than "step by step": this ensures best computing speed and improves the management of disk usage, although it is not possible to see all the output messages in standard output (to see them, it is necessary to redirect them in a log file).
• cloud masks: "cloud_low_proba" is not yet available in the GUI (if is still a valid argument value for compatibility), and "cloud_and_shadow" now includes class "unclassified". This in order to include "unclassified" only in "conservative" mask values, and to exclude it in non-conservative ones.

Edits in the GUI

• add commands to manage parallelisation and running order (with documentation)
• add button to export output messages in a log file
• add MathML code for some custom spectral indices

New dependencies

• httr: a change in wget available version for Windows (1.20.3 instead than 1.19.4) caused the old link to break; moreover, downloading wget using the new link was not possible with download.file() due to a redirect. So, httr dependency was introduced, and downlod of wget, aria2 and sen2cor is now performed with GET function.
• dplyr: the internal function tiles_intersects() now perform dissolve operations over polygons of S2 tiles using dplyr commands (group_by() and summarise()).

Minor improvements

• check the list of parameters whenever it is loaded

Various fixes

(see issues)

Version 1.0.1

Changes from version 1.0.0

Major changes

• A new algorithm was implemented in s2_calcindices(): now internal raster routines are used by default instead than gdal_calc.py, allowing reducing RAM usage and computation time.
• Better RAM usage and reduction of execution time: sen2r_getElements() was rewritten in order to:
  1. avoid a RAM leak present in the function, which was causing a high RAM use in the first part of the execution of sen2r();
  2. considerably speed up the execution (now the metadata of 100 files is retrieved in 4 milliseconds instead than 80).

Changes in function defaults

• s2_calcindices() uses internal routines; the legacy mode can be used with the argument proc_mode = "gdal_calc" (default value is "raster").
• list_indices() now returns only checked indices by default (specify all = TRUE to return also non-checked ones).
• the default output format of sen2r_getElements() is now a data.table instead than a list (use format = "list" to reproduce the old default).

Various fixes.

Version 1.0.0

First stable release of package sen2r! See the announcement here.

Changes from version 0.3.4

GUI changes:

• Highlight selected tiles in the map (now red) in respect to unselected ones (grey)
• Hide TCI product (replaced by RGB images)
• Implement orbit selector and change the style of tile selector
• Make boxes collapsible
• Use shinyfiles to load extent (for coherence with other file selectors in the GUI)
• Copying path_out to path_rgb / path_indices is not yet necessary (if empty, path_out is used)
• Add loading spinner
• Other style improvements

New functions

• tiles_intersects(): when an extent is loaded, now only required tiles are automatically used, instead than all the overlapping ones.

Update dependencies

• after V8 major update, Installation page was edited to refer to v8 instead than to v8-3.14

Bug fixing:

• Fix bug in SciHub login
• Fix maps (tiles were hidden)
• Fix:missing dependency lwgeom
• Various fixes (#140, #141, #142, etc.)

Version 0.3.4

Major improvements

• Implement 4 different processing orders in order to fasterize processing. See "processing_order" in sen2r() documentation. The 4 different modes allow to privilege stability, velocity or disk space. In this version, the legacy mode ("by_step") is maintained as the default.
• Add new function s2_rgb() to produce RGB images, integrating it in sen2r() and in the GUI.
• Add a dockerised version of sen2r: https://hub.docker.com/r/ranghetti/sen2r

Changes

• Change base cloud masks:
  • class "Saturated and defective" is masked within all masks;
  • class "Dark area pixels" is masked in masks in which "Cloud shadows" was masked;
  • mask "cloud_shadow_cirrus" was renamed to "clear_sky";
  • a new "land" mask was added.
• Add "nomask" level to s2_mask()
• Names of ratio indices now uses ":" instead of "/" (#123)
• Index TCI_idx now is TCIdx
• Use a default extent name (leaving it empty is now not allowed, in order not to confuse with naming convention of merged files)

Other improvements

• Improve SAFE filtering: removing duplicates; filtering by position (only the ones overlapping the extent); do not perform a double check of existing products in offline mode.
• Improve GUI: improve cloud mask helper (documentation within s2_gui() was improved by adding a modaldialog which explains how to apply buffer and smoothing to cloud masks)
• Allow sen2r working with GDAL 2.1.2
• Allow GDAL path to be specified in check_gdal()
• Implements a faster method for masking
• Add max_cloud_safe argument to sen2r()
• Add a function safe_isvalid() to check for valid SAFE archive names, and use it to filter existing SAFE folders (#104)
• Update installation documentation (Linux dependencies and Docker)

Fixes

• Important fix: fix indices computation (before that, indices with additive values were wrongly generated).
• Avoid errors when a SAFE directory exists but not in the right format
• Fix the fiddly error which caused several blocks when startig s2_merge() or s2_calcindices()
• Fix searching oldname products with "auto" corr_type
• Fix loading a custom mask from json parameter file
• remove duplicated EVI2 index
• #112, #115, #120, #124, and others

Version 0.3.3

This is an improvement of version 0.3.2, with several fixes and improvements.

Major changes

• Add internal logging (argument log of function sen2r())
• New function s2_dop() to know the Dates Of Passage over orbits
• Cloud mask can now be smoothed and buffered, both using sen2r() and the GUI
• Add apihub parameter in sen2r() and in the GUI: It is now possible to specify a custom apihub.txt file for specific executions (i.e. for scheduled processes).

Changes

• Minor edits in GUI
• Parallelise functions and manage errors
• Create load_binpaths() (#87: Create a function to load paths of external executables and adapt other functions to use it.
• Implement search for ingestion date
• Split names_missing into names_cloudcovered and names_missing
• Add some burn indices and update spectral indices from IDB
• Change some function names (s2_getMetadata -> safe_getMetadata; s2_shortname -> safe_shortname; fs2nc_getElements -> sen2r_getElements)

Fixes

• Ignore baseline in computing required SAFE names
• Fix #82 (Now BOA and TOA uses GTiff intermediate files instead of VRT (gdal_calc failed using vrt as input, computing only the first band))
• Uniform some default argument values
• Various fixes