Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

[Assess bug] RuntimeError: Attempt to create new tiff file #170

Open
JinIgarashi opened this issue Feb 12, 2025 · 3 comments
Open

[Assess bug] RuntimeError: Attempt to create new tiff file #170

JinIgarashi opened this issue Feb 12, 2025 · 3 comments
Labels
bug Something isn't working

Comments

@JinIgarashi
Copy link
Contributor

JinIgarashi commented Feb 12, 2025 •
edited
Loading

In Jupyter notebook, Assess command fails with error of Attempt to create new tiff file

logs

RuntimeError: Attempt to create new tiff file `/data/jin/20240213_training/sample_project/data/population/female_elderly/RWA_female_elderly.tif' failed: /data/jin/20240213_training/sample_project/data/population/female_elderly/RWA_female_elderly.tif: No such file or directory
May be caused by: Deleting /data/jin/20240213_training/sample_project/data/population/female_elderly/RWA_female_elderly.tif failed:
No such file or directory
Full logs 
/data/jin/20240213_training/sample_project /data/jin/20240213_training
2025-02-12 05:38:05,613 - INFO - Assessing component "population" ━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
2025-02-12 05:38:05,618 - INFO - Assessing variable female_elderly━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Going to process 15 variables ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Going to process 15 variables ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Going to process 15 variables ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Going to process 15 variables ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Going to process 15 variables ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Going to process 15 variables ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Going to process 15 variables ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Going to process 15 variables ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Going to process 15 variables ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Going to process 15 variables ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Going to process 15 variables ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Going to process 15 variables ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Going to process 15 variables ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Going to process 15 variables ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Going to process 15 variables ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Going to process 15 variables ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Going to process 15 variables ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Going to process 15 variables ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Going to process 15 variables ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
2025-02-12 05:38:07,868 - INFO - Going to compute female_elderly from 4 source  -:--:--
files
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Going to process 15 variables ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
2025-02-12 05:38:09,142 - INFO - Evaluating variable female_elderly using zonal  0% -:--:--
stats
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
2025-02-12 05:38:10,810 - INFO - Created 22 records━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
2025-02-12 05:38:11,374 - INFO - Assessing variable elderly_dependency[0m   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
2025-02-12 05:38:11,477 - INFO - Assessing variable elderly_total━━━━━━━━━━━━━   7% -:--:--
Assessing population            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
2025-02-12 05:38:11,480 - INFO - Going to compute ━━━━━━━━━━━━━━━━━━━━━━━━━━━━   7% -:--:--
elderly_total=male_elderly+female_elderly
Assessing population            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
2025-02-12 05:38:11,509 - INFO - Assessing variable female_elderly━━━━━━━━━━━━   7% -:--:--
Assessing population            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing female_elderly in RWA ━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   7% -:--:--
Assessing population            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing female_elderly in RWA ━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   7% -:--:--
Assessing population            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing female_elderly in RWA ━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   7% -:--:--
Assessing population            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing female_elderly in RWA ━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   7% -:--:--
Assessing population            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing female_elderly in RWA ━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   7% -:--:--
Assessing population            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing female_elderly in RWA ━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   7% -:--:--
Assessing population            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing female_elderly in RWA ━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   7% -:--:--
Assessing population            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing female_elderly in RWA ━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   7% -:--:--
Assessing population            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing female_elderly in RWA ━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   7% -:--:--
Assessing population            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing female_elderly in RWA ━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   7% -:--:--
Assessing population            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing female_elderly in RWA ━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   7% -:--:--
Assessing population            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing female_elderly in RWA ━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   7% -:--:--
2025-02-12 05:38:12,630 - INFO - Going to compute female_elderly from 4 source  -:--:--
files
Assessing population            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing female_elderly in RWA ━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   7% -:--:--
Assessing population            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing female_elderly in RWA ━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   7% -:--:--
Traceback (most recent call last):
  File "/app/.venv/bin/rapida", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/app/.venv/lib/python3.12/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/cbsurge/assess.py", line 140, in assess
    component(progress=progress, variables=variables, force_compute=force_compute)
  File "/app/cbsurge/components/population/__init__.py", line 165, in __call__
    v(year=self.year, country=country, **kwargs)
  File "/app/cbsurge/core.py", line 184, in __call__
    sources = self.resolve(evaluate=True, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/cbsurge/components/population/__init__.py", line 228, in resolve
    var_local_path = var(**kwargs) # assess
                     ^^^^^^^^^^^^^
  File "/app/cbsurge/core.py", line 181, in __call__
    self.compute(**kwargs)
  File "/app/cbsurge/components/population/__init__.py", line 212, in compute
    sources = self.resolve(**kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^
  File "/app/cbsurge/components/population/__init__.py", line 228, in resolve
    var_local_path = var(**kwargs) # assess
                     ^^^^^^^^^^^^^
  File "/app/cbsurge/core.py", line 173, in __call__
    self.compute(**kwargs)
  File "/app/cbsurge/components/population/__init__.py", line 202, in compute
    computed_file = sumup(src_rasters=downloaded_files,dst_raster=local_path, overwrite=overwrite)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/cbsurge/stats/zst.py", line 123, in sumup
    ds = Calc(calc='sum(a,axis=0)', a=files_to_sum, outfile=dst_raster, projectionCheck=True, format='GTiff',
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/osgeo_utils/auxiliary/util.py", line 46, in enable_exceptions_wrapper
    return fun(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/osgeo_utils/gdal_calc.py", line 413, in Calc
    myOut = myOutDrv.Create(
            ^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/osgeo/gdal.py", line 3817, in Create
    return _gdal.Driver_Create(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Attempt to create new tiff file `/data/jin/20240213_training/sample_project/data/population/female_elderly/RWA_female_elderly.tif' failed: /data/jin/20240213_training/sample_project/data/population/female_elderly/RWA_female_elderly.tif: No such file or directory
May be caused by: Deleting /data/jin/20240213_training/sample_project/data/population/female_elderly/RWA_female_elderly.tif failed:
No such file or directory

reproduction

In jupyter notebook

from cbsurge.admin.ocha import fetch_admin

bbox = [28.729248, -2.88698, 31.047363, -0.954565]
BBOX = ",".join(map(str, bbox))
ADMIN_FILE="admin_2_osm_rwa.gpkg"

print(f"Download data for {BBOX}")

! rapida admin osm --bbox={BBOX} --admin_level=2 --clip --dst_path {ADMIN_FILE} --layer-name "admin"
from cbsurge.project import create

NAME="sample_project"

! rapida create --name={NAME} --polygons={ADMIN_FILE}
! pushd {NAME} && rapida assess -c population -f && popd
@JinIgarashi JinIgarashi added the bug Something isn't working label Feb 12, 2025
@JinIgarashi
Copy link
Contributor Author

I am not sure how to reproduce this bug exactly, but I assume

  • if execute assess command without -v option to assess all variables
  • if execute assess command under existing project with data folder.

when I ran the command rapida assess -c population -v male_total -v female_total -f in a blank project folder, the command was done successfully without error.

@JinIgarashi
Copy link
Contributor Author

JinIgarashi commented Feb 12, 2025 •
edited
Loading

I tried small area for only Kigali (Bounding Box (BBOX): [29.934998, -2.088163, 30.326385, -1.763564]) with rapida assess -c population -v total -v male_total -v female_total -f command. it failed with the same error.

both CLI and notebooks has same error.

@JinIgarashi
Copy link
Contributor Author

the bug present with -f option. it works without -f

@iferencik iferencik removed their assignment Feb 19, 2025
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@JinIgarashi @iferencik