Add RFC 102 text: Embedding resource files into libgdal

[rouault]

Rendered view

[sgillies]

@rouault I'm using CPLFindFile() to detect whether a GDAL install is inside a Python package: https://github.com/rasterio/rasterio/blob/df409de086efa437fef2b66b4e0d2e1e387fe474/rasterio/_env.pyx#L250. Could that function be updated to search in libgdal? Or should I depend on not using this option?

[rouault]

Could that function be updated to search in libgdal?

no, embedded resources are not easily discoverables as sub-files. That would involve complicated low-level compiler/OS-specific trick

Or should I depend on not using this option?

Well, the intent is that your GDAL builds for Rasterio/Fiona binary wheels do use the new CMake option EMBED_RESOURCE_FILES=ON, and probably also USE_ONLY_EMBEDDED_RESOURCE_FILES=ON, since it is unlikely users would want to tune the default resource files, and that you ultimately get rid of the use of CPLFindFile()

[rouault]

FYI @Maxxen w.r.t proj.db embedding

[eschnett]

This feature could easily be implemented in Perl or Python with some makefile support, generating a C source file from a binary that can then be #included whenever #embed is not available.

We have been using this feature in the Einstein Toolkit for many years (hence Perl) in the Formaline thorn. This code does many other things as well; the actual code is in the makeblob.pl and makemetablob.pl Perl files in the src/util directory.

One important thing at the time (decades ago) was to ensure that the generated source files are not too large since certain compilers would run out of memory with large literal arrays. That may or may not be necessary any more.

[rouault]

This feature could easily be implemented in Perl or Python with some makefile support

yes, that's what I mention in the "alternatives" paragraph. But I don't want to implement both C23 #embed support, and older tricks. Too painful too maintain. Let's go to the future straight ahead.

One important thing at the time (decades ago) was to ensure that the generated source files are not too large since certain compilers would run out of memory with large literal arrays. That may or may not be necessary any more.

GCC 15dev #embed unoptimized implementation shows that's still a concern. CLang 19 #embed implementation uses collaboration between the pre-processor and later stages of the compilation to have time and RAM efficient embedding.

[jjimenezshaw]

Is there going to be a similar option only in PROJ? (Without complling GDAL)

[rouault]

Is there going to be a similar option only in PROJ? (Without complling GDAL)

yes

[Maxxen]

One thing that's on my todo list is investigating using sqlites memvfs in combination with its zipvfs file system, potentially reducing the size of the embedded PROJ database even further.

[rouault]

PROJ specificities are now covered by a dedicated PROJ RFC-8: OSGeo/PROJ#4274