Dev 20241209 merge to v1.3.0 (#109)

* updated CMS path for prod

* Update step1_processor_vhi.py HOTFIX: if no data is available exit

ON prod fixed with e170eeb

* Initial Version of  JUPYTER notebooks

* fix typo

* fixed typos and correted binder to dev

* added qgis cog tutorial

* First version of VHI

* commit all changes

* Update main_utils.py

Setting default values if an image collection is empty

* Update dev_config.py

adjusted product name

* Update step1_processor_vhi_hist.py

- renaming of the bands for Landsat 8 to match the band names of Landsat 5 and 7
- thus removing the L8 specific functions
- some adjustements due to errors regarding <str> and <String> between python and GEE

* previsous version with different functions for Landsa 5 and 7 vs. Landsat 8

* Deactivate the Export to Drive Switch for debugging

* Create vhi-plausability.ipynb

* Update step1_processor_vhi_hist.py

- taking the variables water_binary and DEM into the functions that need them
- improve code for all dates from GEE copies to purely python
- fixed the sorting of Landsat collections

* Added get_follection_info_landsat function

The metadata for landsat collections is differently stored than for sentinel collection, thus a second function retrieving the dates of the first and last image of a collection and the total count of images in the collection for Landsat collections.

* Update vhi-plausability.ipynb

Added Spearman's Rank Correlation

* Remove empty code cell from stac-introduction notebook

* Add utility functions for generating CDI and VHI maps

* Add detailed documentation for CDI and VHI maps generation script

* Fix import statement for geopandas in util_cdi_vhi_maps.py

* Add VHI Map Grid Generator script for generating monthly VHI maps

* Update step1_processor_vhi_hist.py

- fixed the parameter 'left' is required issue: within the function of the topographic correction the linear relation between illumination and reflectance is used. If a band is empty, this correction doesn't work. Thus, there's now a check for the validity of the bands this correction is applied to. If there are empty bands, these are excluded and carried on as they were without the correction.
- fixed the incorrect timestamps for start_date and end_date VHI properties, due to a mismatch while translating from JavaScript to python since timestamp() works in number of seconds, and the GEE expects the imestamps in milliseconds.
- Further, used to method chaining for applying the pre-processing functions for the Landsat data.

* Add VHIExtractor class and CSV processing functions for VHI and mask data extraction

* Add documentation and utility functions for tree network data extraction

* Update step1_processor_vhi_hist.py

Fix the issue of the date for the last day of the month. relativedelta(months=1) instead of .replace([...].month+1) which would end with a 13

* Update dev_config.py for VHI HIST product paths (set to int "step1_collection": 'projects/satromo-int/assets/VHI_HIST_SWISS');   adjust main_utils.py removed breakpoint; update satromo_publish.py to include additional file checks for 30m.tif for warnregions generation; add "collection" metadata  for VHI mosaic processing.

* Update date references and enhance processing scripts for VHI data handling for thumbnail

* Adjusting prod path

* Add get_collection_strings function

Display only the Landsat collection names from which images where used for the specific asset.

* Update date handling in main_extract_warnregions.py  for  fix of #107

* Changes in getting the collection info

* Remove get_collection_strings function

Got error 429 too many requests (concurrent aggregations).
The collection filtering did not work properly.
Decided to post all three Landsat collection in the property 'Collection' because from the 'NDVI_index_list' it is also clear which sensors exactely were used.

* Wrong TCI formula corrected

* Wrong TCI formula corrected

work on the apply_SCSccorr function still ongoing.

* Improve error handling/logging for subprocess calls (#108)

LGTM

* fix cdi colors

* Add support for Sentinel-2C acquisition plans and enhance download reporting

* Add STAC asset and item deletion script

* Remove unused imports in util_stac_delete.py

* Updates to the topographic correction

- check if bands are empty before applying topographic correction
- check if there are unmasked pixels, if everything is masked don't apply topo correction
- check if coefficients are valid

* New filter_by_valid_pixels function

- filter out scenes from the image collection for which the cloud mask does not at least leave 9km2 unmasked

* matching 'global_date' to 'datetime'

* prepare deploy

---------

Co-authored-by: David Oesch <david.oesch@gmail.com>
Co-authored-by: Stefan Oderbolz <oderbolz@gmail.com>

Author: 3 people

codegallery/jupyter/Python/stac/stac-introduction.ipynb

@@ -59,13 +59,15 @@
    ]
   },
   {
+
    "cell_type": "code",
    "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": []
   },
   {
+
    "cell_type": "markdown",
    "metadata": {},
    "source": [

codegallery/jupyter/Python/stac/vhi-plausability.ipynb

@@ -124,14 +124,14 @@
     "geocat_id": "bc4d0e6b-e92e-4f28-a7d2-f41bf61e98bc",
     "temporal_coverage": 1,  # Months
     "spatial_scale_export": 30,  # Meters
-    "product_name": "ch.swisstopo.swisseo_vhi_hist_v100",
+    "product_name": "ch.swisstopo.swisseo_vhi_v100",
     "no_data": 255,
     "missing_data": 110,
     "asset_size": 2,
     'NDVI_reference_data': 'projects/satromo-prod/assets/col/1991-2020_NDVI_SWISS_M',
     'LST_reference_data': 'projects/satromo-prod/assets/col/1991-2020_LST_SWISS_M',
     'LST_current_data': 'projects/satromo-prod/assets/col/LST_SWISS',
-    "step1_collection": 'projects/satromo-int/assets/VHI_HIST_SWISS',
+    "step1_collection": 'projects/satromo-prod/assets/col/VHI_HIST_SWISS',
     "step0_collection": 'projects/satromo-prod/assets/col/LST_SWISS'
 }
 

configuration/dev_config.py

@@ -89,7 +89,7 @@
     "spatial_scale_export_mask": 10,
     "product_name": "ch.swisstopo.swisseo_s2-sr_v100",
     "no_data": 9999,
-    #"step0_collection": "projects/satromo-prod/assets/col/S2_SR_HARMONIZED_SWISS"
+    "step0_collection": "projects/satromo-prod/assets/col/S2_SR_HARMONIZED_SWISS"
 }
 
 # VHI – Trockenstress ch.swisstopo.swisseo_vhi_v100
@@ -107,7 +107,7 @@
     'LST_reference_data': 'projects/satromo-prod/assets/col/1991-2020_LST_SWISS',
     'LST_current_data': 'projects/satromo-prod/assets/col/LST_SWISS',
     "step1_collection": 'projects/satromo-prod/assets/col/VHI_SWISS',
-    "step0_collection": "projects/satromo-prod/assets/col/S2_SR_HARMONIZED_SWISS"
+    #"step0_collection": "projects/satromo-prod/assets/col/S2_SR_HARMONIZED_SWISS"
 }
 
 # MSG – MeteoSchweiz: only used for repreocessing

configuration/oed_prod_config.py

@@ -12,7 +12,7 @@
 GDRIVE_SECRETS = os.path.join("secrets", "geetest-credentials-int.secret")
 RCLONE_SECRETS = os.path.join("secrets", "rclone.conf")
 FSDI_SECRETS = os.path.join("secrets", "stac_fsdi-prod.json")
-CMS_SECRETS = os.path.join("secrets", "INT_CMS.json")
+CMS_SECRETS = os.path.join("secrets", "PROD_CMS.json")
 
 # File and directory paths
 GEE_RUNNING_TASKS = os.path.join("processing", "running_tasks.csv")

configuration/prod_config.py

@@ -20,7 +20,7 @@
 # shapefile with warnregions
 shape_file = r"C:\temp\topo-satromo\assets\warnregionen_vhi_2056.shp"
 
-# raster file from URL / file 
+# raster file from URL / file
 raster_url = r"C:\temp\topo-satromo\ch.swisstopo.swisseo_vhi_v100_mosaic_2023-07-13T235959_vegetation-10m.tif"
 
 # missing data values
@@ -134,7 +134,7 @@ def export(raster_url, shape_file, filename, dateISO8601, missing_values):
 
     # Add the date to each region in UTC
     # gdf[date_column] = datetime.strptime(dateISO8601, "%Y-%m-%dT%H:%M:%SZ")
-    gdf[date_column] = pd.to_datetime(dateISO8601).tz_convert('UTC')
+    gdf[date_column]= pd.to_datetime(dateISO8601).tz_convert('UTC').floor('S')
     # gdf[date_column] = gdf[date_column].astype()
 
     # Export the converted GeoDataFrame to a geoparquet file

main_functions/main_extract_warnregions.py

@@ -230,11 +230,12 @@ def create_thumbnail(inputfile_name, product):
 
         except subprocess.CalledProcessError as e:
             print(f"Error: {e}")
+            print("Output")
+            print(e.output)
             return False
 
     # VHI Use case
-
-    elif product.startswith("ch.swisstopo.swisseo_vhi") and inputfile_name.endswith("forest-10m.tif"):
+    elif product.startswith("ch.swisstopo.swisseo_vhi") and (inputfile_name.endswith("forest-10m.tif") or inputfile_name.endswith("forest-30m.tif")):
         # https://github.com/radiantearth/stac-spec/blob/master/best-practices.md#visual
         # It should be called just   "thumbnail.jpg"
         # thumbnail_name = inputfile_name.replace(

main_functions/main_thumbnails.py

@@ -6,6 +6,7 @@
 import os
 import json
 import pandas as pd
+import dateutil
 
 
 def is_date_in_empty_asset_list(collection, check_date_str):
@@ -86,15 +87,15 @@ def get_github_info():
 
 def get_product_from_techname(techname):
     """
-    This function searches for a dictionary in the 'config' module that contains 
+    This function searches for a dictionary in the 'config' module that contains
     'product_name' with a specified value and returns it.
 
     Parameters:
-    techname (str): The value of 'product_name' to search for. 
+    techname (str): The value of 'product_name' to search for.
                     For example, 'ch.swisstopo.swisseo_s2-sr_v100'.
 
     Returns:
-    dict: The dictionary that contains 'product_name' with the value of 'techname'. 
+    dict: The dictionary that contains 'product_name' with the value of 'techname'.
           If no such dictionary is found, it returns None.
     """
 
@@ -172,28 +173,30 @@ def get_collection_info(collection):
 
     Returns:
         A tuple containing the first date, last date, and total number of images in the collection.
+        Returns (None, None, 0) for empty collections.
     """
     # Sort the collection by date in ascending order
+
     sorted_collection = collection.sort('system:time_start')
 
     # Get the first and last image from the sorted collection
     first_image = sorted_collection.first()
     last_image = sorted_collection.sort('system:time_start', False).first()
 
-    # Get the dates of the first and last image
-    first_date = ee.Date(first_image.get('system:time_start')
-                         ).format('YYYY-MM-dd').getInfo()
-    last_date = ee.Date(last_image.get('system:time_start')
-                        ).format('YYYY-MM-dd').getInfo()
-
-    # Get the count of images in the filtered collection
-    image_count = collection.size()
-
-    # Get the scenes count
-    total_scenes = image_count.getInfo()
+    try:
+        # Get the count of images in the collection
+        image_count = collection.size().getInfo()
+        # Get the dates of the first and last image
+        first_date = ee.Date(first_image.get('system:time_start')).format('YYYY-MM-dd').getInfo()
+        last_date = ee.Date(last_image.get('system:time_start')).format('YYYY-MM-dd').getInfo()
+    except ee.EEException:
+        image_count = 0
+        # Handle cases where date information might be missing
+        first_date = None
+        last_date = None
 
     # Return the first date, last date, and total number of scenes
-    return first_date, last_date, total_scenes
+    return first_date, last_date, image_count
 
 
 def get_quadrants(roi):
@@ -440,7 +443,7 @@ def prepare_export(roi, productitem, productasset, productname, scale, image, se
 
     Args:
         roi (ee.Geometry): Region of interest for the export.
-        productitem (str): Timestamp of assets YYYYMMDThhmmss, "YYYYMMDDT235959" for a day 
+        productitem (str): Timestamp of assets YYYYMMDThhmmss, "YYYYMMDDT235959" for a day
         productasset (str): Base filename for the exported files.
         productname (str): Product name of the exported files.
         scale (str): Scalenumber in [m] of the exported file
@@ -507,3 +510,28 @@ def prepare_export(roi, productitem, productasset, productname, scale, image, se
         json.dump(image_info_gee, json_file)
 
     return None
+
+
+def get_collection_info_landsat(collection):
+    """
+    Retrieves information about an image collection for the line of Landsat satellites
+
+    Args:
+        collection: The landsat image collection to retrieve information from.
+
+    Returns:
+        A tuple containing the first date, last date, and total number of images in the collection.
+        Returns (None, None, 0) for empty collections.
+    """
+    # Sort the collection by date in ascending order
+    index_list = collection.aggregate_array('system:index')
+
+    dates_list = [dateutil.parser.parse(i.split('_')[-1]) for i in index_list.getInfo()]
+
+    # Get the first and last image and size of image collection
+    image_count = len(dates_list) if len(dates_list)>0 else 0
+    first_date = min(dates_list) if image_count>0 else None
+    last_date = max(dates_list) if image_count>0 else None
+
+    # Return the first date, last date, and total number of scenes
+    return first_date, last_date, image_count