Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

[FEATURE] New output archive type for individual files #536

Closed
msbarry opened this issue Mar 27, 2023 · 0 comments · Fixed by #761
Closed

[FEATURE] New output archive type for individual files #536

msbarry opened this issue Mar 27, 2023 · 0 comments · Fixed by #761

Comments

@msbarry
Copy link
Contributor

msbarry commented Mar 27, 2023

Now that we have the ability to switch between output formats easily with string URIs from #520, we should add a new output type that just writes vector tiles directly to disk.

See #533 (reply in thread)
@bbilger bbilger mentioned this issue Jul 30, 2023
Merged
@cmahnke cmahnke mentioned this issue Sep 21, 2023
Open
bbilger added a commit to bbilger/planetiler that referenced this issue Dec 20, 2023
@bbilger
Add support for "files"-archive
bf1a047 
i.e. write individual pbf-files to disk in the format <base>/z/x/y.pbf

in order to use that format it must be passed as "--ouput=/path/to/tiles?format=files"

Fixes onthegomap#536
@bbilger bbilger mentioned this issue Dec 20, 2023
Merged
bbilger added a commit to bbilger/planetiler that referenced this issue Dec 20, 2023
@bbilger
Add support for "files"-archive
68bfcb0 
i.e. write individual pbf-files to disk in the format <base>/z/x/y.pbf

in order to use that format it must be passed as "--ouput=/path/to/tiles?format=files"

Fixes onthegomap#536
bbilger added a commit to bbilger/planetiler that referenced this issue Dec 20, 2023
@bbilger
Add support for "files"-archive
c40d7ae 
i.e. write individual pbf-files to disk in the format <base>/z/x/y.pbf

in order to use that format it must be passed as "--ouput=/path/to/tiles?format=files"

Fixes onthegomap#536
bbilger added a commit to bbilger/planetiler that referenced this issue Dec 20, 2023
@bbilger
Add support for "files"-archive
6e8effd 
i.e. write individual pbf-files to disk in the format <base>/z/x/y.pbf

in order to use that format it must be passed as "--ouput=/path/to/tiles?format=files"

Fixes onthegomap#536
bbilger added a commit to bbilger/planetiler that referenced this issue Dec 20, 2023
@bbilger
Add support for "files"-archive
f1cf90e 
i.e. write individual pbf-files to disk in the format <base>/z/x/y.pbf

in order to use that format it must be passed as "--ouput=/path/to/tiles?format=files"

Fixes onthegomap#536
bbilger added a commit to bbilger/planetiler that referenced this issue Dec 20, 2023
@bbilger
Add support for "files"-archive
0acaca5 
i.e. write individual pbf-files to disk in the format <base>/z/x/y.pbf

in order to use that format it must be passed as "--ouput=/path/to/tiles?format=files"

Fixes onthegomap#536
bbilger added a commit to bbilger/planetiler that referenced this issue Dec 30, 2023
@bbilger
Add support for "files"-archive
05366db 
i.e. write individual pbf-files to disk in the format <base>/z/x/y.pbf

in order to use that format it must be passed as "--ouput=/path/to/tiles?format=files"

Fixes onthegomap#536
msbarry pushed a commit that referenced this issue Jan 3, 2024
@bbilger
Add support for "files"-archive (#761)
c480b35 
* Add support for "files"-archive

i.e. write individual pbf-files to disk in the format <base>/z/x/y.pbf

in order to use that format it must be passed as "--ouput=/path/to/tiles?format=files"

Fixes #536

* default to files format

...if no explict format query param given, path ends with a slash, or no extension given

* output metadata.json in files-archive

and refactor TileArchiveMetadata
1. put zoom into center (CoordinateXY->Coordinate) - in sync with mbtiles-format
2. add (De-)Serializer for Coordinate+Envelop => avoid duplication and cleaner
3. change the json and proto output for TileArchiveMetadata to be (more) in sync with mbtiles-format

* add support for custom tile scheme in files-archive

{z}/{x}/{y}.pbf is the default and can be configured as needed - e.g.:
- different order: {x}/{y}/{z}.pbf
- with intermediate dirs: {x}/a/{y}/b/{z}.pbf
- with different extension: {z}/{y}/{y}.pbf.gz

instead of {x} and {y}, {xs} and {xy} can be used which breaks up
x and y into 2 directories each and ensures that each directory has <1000 children

* fix issues with multiple writers

1. call finish archive only once after all writers are finished
   ...and not every time a writer finishes
2. log "zoom-progress" for the first tile write only
   (Finished z11 ... now starting z12)
3. remove file/dir-size progress logger bottleneck for files archive
   => each archive now reports the bytes written, which also fixes
   the issues of stream-archives reporting the size incorrectly
4. introduce printStats-hook on archive-level

* add async file write support to files archive

...allow to use virtual threads ExecturService (bound only!) for tile writing

also add some benchmark for writing tiles to disk: fixed, bound virtual, async, unbound virtual

* Revert "add async file write support to files archive"

This reverts commit b8cfa56.

* few improvements

- extract TileSchemeEncoding
- use Counter.MultithreadCounter rather than LongAdder to count bytes written
- add some JavaDoc

* simplify files archive usage

1. allow to pass tile scheme directly via output: --output=tiles/{x}/{y}/{z}.pbf
2. auto-encode { (%7B) and } (%7D) => no need to encode it the URI on CLI

* few more adjustments according to PR feeback

1. use WriteableTileArchive#bytesWritten in summmary as well
2. call WriteableTileArchive#init in a safer manner

..and a few more adjustments

* more PR feedback
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add support for "files"-archive bbilger/planetiler
1 participant
@msbarry