Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

add support for simple file output streams: CSV, JSON, protobuf #639

Merged
merged 11 commits into from
Aug 24, 2023
Merged

add support for simple file output streams: CSV, JSON, protobuf #639

merged 11 commits into from
Aug 24, 2023

Conversation

bbilger
Copy link
Contributor

@bbilger bbilger commented Jul 30, 2023 •
edited
Loading

The primary use case for writing tile data to files using simple formats, is to provide a simple plugin mechanism. By using named pipes data can be transformed and processed directly - e.g. to upload to s3 or to store in a DB, without any intermediate file (mbtiles/pmtiles).

This - depending on the use-case - also allows concurrent writing to separate files. I left a few questions in TileArchiveWriter about what needs to be done to handle that there.

Formats

JSON

{"type":"initialization","metadata":{"name":"0", ...}}
{"type":"tile","x":0,"y":0,"z":0,"encodedData":"..."}
{"type":"tile","x":1,"y":2,"z":3,"encodedData":"..."}
{"type":"finish","metadata":{"name":"1", ...}}

entries are separated by newline (can be changed via config), and the output of non-tile entries can be suppressed

CSV

0,0,0,encodedDataBase64
1,2,3,encodedDataBase64

the value separator can be changed, and no meta data is output in that format

protobuf

protobuf-message are length-delimited

in Java the messages can be read from the file/pipe as follows:

// note: do not use nio (Files.newInputStream) for pipes
try (var in = new FileInputStream(...)) {
  StreamArchiveProto.Entry entry;
  while ((entry = StreamArchiveProto.Entry.parseDelimitedFrom(in)) != null) {
    ...
  }
}

Use cases

s3

A very simple example to push tiles directly to s3 could look likes this:

mkfifo /tmp/data/output.csv
./s3-upload < /tmp/data/output.csv
java -Xmx1g -jar planetiler.jar --download --area=monaco --append --output=/tmp/data/output.csv

with s3-upload being something like this:

#! /bin/bash
while IFS="," read -r x y z encoded
do
  echo "pushing tile z=$z x=$x y=$y"
  echo $encoded | base64 -d | aws s3 cp - s3://BUCKET/map/$z/$x/$y.pbf --content-type=application/x-protobuf --content-encoding=gzip
  # echo $encoded | base64 -d | gzip -d | aws s3 cp - s3://BUCKET/map/$z/$x/$y.pbf --content-type=application/x-protobuf
done < "${1:-/dev/stdin}"

Parallel:

mkfifo /tmp/data/output.csv
mkfifo /tmp/data/output.csv1
./s3-upload < /tmp/data/output.csv
./s3-upload < /tmp/data/output.csv1
java -Xmx1g -jar planetiler.jar --download --area=monaco --append --output=/tmp/data/output.csv --tile_write_threads=2

DB

MySQL
mkfifo /tmp/data/output.csv
java -Xmx1g -jar planetiler.jar --download --area=monaco --append --output=/tmp/data/output.csv

mysql>...create tile(s) table
mysql> LOAD DATA INFILE '/tmp/data/output.csv'
  -> INTO TABLE tiles
  -> FIELDS TERMINATED BY ','
  -> LINES TERMINATED BY '\n'
  -> (tile_column, tile_row, zoom_level, @var1)
  -> SET tile_data = FROM_BASE64(@var1);
PostgreSQL
mkfifo /tmp/data/output_raw.csv
mkfifo /tmp/data/output_transformed.csv
# prefix hex-data with '\x' for the postgres import
cat /tmp/data/output_raw.csv | sed -r 's/^([0-9]+,)([0-9]+,)([0-9]+,)(.*)$/\1\2\3\\x\4/' > /tmp/data/output_transformed.csv

java -Xmx1g -jar planetiler.jar --download --area=monaco --append --output=/tmp/data/output_raw.csv --csv_binary_encoding=hex

...create tile(s) table

postgres=# \copy tiles(tile_column, tile_row, zoom_level, tile_data) from /tmp/data/output_transformed.csv DELIMITER ',' CSV;

This is related to:

@bbilger bbilger force-pushed the add-file-stream-outputs branch from 6226783 to d4cc04e Compare July 30, 2023 22:25
@github-actions
Copy link

github-actions bot commented Jul 30, 2023 •
edited
Loading

Base a8e432d This Branch fbb6ab5 
0:02:11 DEB [archive] - Tile stats:
0:02:11 DEB [archive] - z0 avg:7.9k max:7.9k
0:02:11 DEB [archive] - z1 avg:4k max:4k
0:02:11 DEB [archive] - z2 avg:9.4k max:9.4k
0:02:11 DEB [archive] - z3 avg:4k max:6.4k
0:02:11 DEB [archive] - z4 avg:1.6k max:4.6k
0:02:11 DEB [archive] - z5 avg:1.4k max:7.1k
0:02:11 DEB [archive] - z6 avg:974 max:22k
0:02:11 DEB [archive] - z7 avg:769 max:58k
0:02:11 DEB [archive] - z8 avg:417 max:127k
0:02:11 DEB [archive] - z9 avg:281 max:298k
0:02:11 DEB [archive] - z10 avg:161 max:256k
0:02:11 DEB [archive] - z11 avg:106 max:136k
0:02:11 DEB [archive] - z12 avg:85 max:114k
0:02:11 DEB [archive] - z13 avg:72 max:128k
0:02:11 DEB [archive] - z14 avg:68 max:303k
0:02:11 DEB [archive] - all avg:70 max:303k
0:02:11 DEB [archive] -  # features: 5,437,996
0:02:11 DEB [archive] -     # tiles: 4,115,061
0:02:11 INF [archive] - Finished in 36s cpu:1m10s gc:1s avg:2
0:02:11 INF [archive] -   read    1x(2% 0.7s wait:34s)
0:02:11 INF [archive] -   encode  2x(63% 23s)
0:02:11 INF [archive] -   write   1x(13% 4s wait:29s)
0:02:11 INF - Finished in 2m11s cpu:4m1s gc:5s avg:1.8
0:02:11 INF - FINISHED!
0:02:11 INF - 
0:02:11 INF - ----------------------------------------
0:02:11 INF - data errors:
0:02:11 INF - 	render_snap_fix_input	16,460
0:02:11 INF - 	osm_boundary_missing_way	63
0:02:11 INF - 	osm_multipolygon_missing_way	57
0:02:11 INF - 	merge_snap_fix_input	14
0:02:11 INF - 	feature_centroid_if_convex_osm_invalid_multipolygon_empty_after_fix	2
0:02:11 INF - 	feature_polygon_osm_invalid_multipolygon_empty_after_fix	2
0:02:11 INF - 	omt_park_area_osm_invalid_multipolygon_empty_after_fix	1
0:02:11 INF - ----------------------------------------
0:02:11 INF - 	overall          2m11s cpu:4m1s gc:5s avg:1.8
0:02:11 INF - 	lake_centerlines 4s cpu:6s avg:1.8
0:02:11 INF - 	  read     1x(50% 2s done:2s)
0:02:11 INF - 	  process  2x(7% 0.2s wait:2s done:1s)
0:02:11 INF - 	  write    1x(0% 0s wait:2s done:1s)
0:02:11 INF - 	water_polygons   31s cpu:57s gc:3s avg:1.8
0:02:11 INF - 	  read     1x(55% 17s sys:1s wait:4s)
0:02:11 INF - 	  process  2x(31% 10s wait:13s)
0:02:11 INF - 	  write    1x(2% 0.7s wait:30s)
0:02:11 INF - 	natural_earth    12s cpu:15s avg:1.2
0:02:11 INF - 	  read     1x(73% 9s sys:1s done:2s)
0:02:11 INF - 	  process  2x(12% 1s wait:9s done:2s)
0:02:11 INF - 	  write    1x(0% 0s wait:10s done:2s)
0:02:11 INF - 	osm_pass1        4s cpu:7s avg:1.7
0:02:11 INF - 	  read     1x(1% 0s wait:4s)
0:02:11 INF - 	  parse    1x(67% 3s)
0:02:11 INF - 	  process  1x(41% 2s wait:2s)
0:02:11 INF - 	osm_pass2        41s cpu:1m21s avg:2
0:02:11 INF - 	  read     1x(0% 0s wait:22s done:19s)
0:02:11 INF - 	  process  2x(76% 32s)
0:02:11 INF - 	  write    1x(1% 0.4s wait:41s)
0:02:11 INF - 	boundaries       0s cpu:0s avg:1.7
0:02:11 INF - 	sort             2s cpu:4s avg:1.6
0:02:11 INF - 	  worker  1x(83% 2s)
0:02:11 INF - 	archive          36s cpu:1m10s gc:1s avg:2
0:02:11 INF - 	  read    1x(2% 0.7s wait:34s)
0:02:11 INF - 	  encode  2x(63% 23s)
0:02:11 INF - 	  write   1x(13% 4s wait:29s)
0:02:11 INF - ----------------------------------------
0:02:11 INF - 	archive	109MB
0:02:11 INF - 	features	283MB
-rw-r--r-- 1 runner docker 66M Aug 23 14:55 run.jar

0:02:18 DEB [archive] - Tile stats:
0:02:18 DEB [archive] - z0 avg:7.9k max:7.9k
0:02:18 DEB [archive] - z1 avg:4k max:4k
0:02:18 DEB [archive] - z2 avg:9.4k max:9.4k
0:02:18 DEB [archive] - z3 avg:4k max:6.4k
0:02:18 DEB [archive] - z4 avg:1.6k max:4.6k
0:02:18 DEB [archive] - z5 avg:1.4k max:7.1k
0:02:18 DEB [archive] - z6 avg:974 max:22k
0:02:18 DEB [archive] - z7 avg:769 max:58k
0:02:18 DEB [archive] - z8 avg:417 max:127k
0:02:18 DEB [archive] - z9 avg:281 max:298k
0:02:18 DEB [archive] - z10 avg:161 max:256k
0:02:18 DEB [archive] - z11 avg:106 max:136k
0:02:18 DEB [archive] - z12 avg:85 max:114k
0:02:18 DEB [archive] - z13 avg:72 max:128k
0:02:18 DEB [archive] - z14 avg:68 max:303k
0:02:18 DEB [archive] - all avg:70 max:303k
0:02:18 DEB [archive] -  # features: 5,437,996
0:02:18 DEB [archive] -     # tiles: 4,115,061
0:02:18 INF [archive] - Finished in 35s cpu:1m8s gc:1s avg:1.9
0:02:18 INF [archive] -   read    1x(2% 0.7s wait:33s)
0:02:18 INF [archive] -   encode  2x(64% 23s wait:1s)
0:02:18 INF [archive] -   write   1x(13% 4s wait:29s)
0:02:18 INF - Finished in 2m18s cpu:4m7s gc:5s avg:1.8
0:02:18 INF - FINISHED!
0:02:18 INF - 
0:02:18 INF - ----------------------------------------
0:02:18 INF - data errors:
0:02:18 INF - 	render_snap_fix_input	16,460
0:02:18 INF - 	osm_boundary_missing_way	63
0:02:18 INF - 	osm_multipolygon_missing_way	57
0:02:18 INF - 	merge_snap_fix_input	14
0:02:18 INF - 	feature_centroid_if_convex_osm_invalid_multipolygon_empty_after_fix	2
0:02:18 INF - 	feature_polygon_osm_invalid_multipolygon_empty_after_fix	2
0:02:18 INF - 	omt_park_area_osm_invalid_multipolygon_empty_after_fix	1
0:02:18 INF - ----------------------------------------
0:02:18 INF - 	overall          2m18s cpu:4m7s gc:5s avg:1.8
0:02:18 INF - 	lake_centerlines 4s cpu:7s avg:1.8
0:02:18 INF - 	  read     1x(44% 2s done:2s)
0:02:18 INF - 	  process  2x(7% 0.3s wait:2s done:2s)
0:02:18 INF - 	  write    1x(0% 0s wait:2s done:2s)
0:02:18 INF - 	water_polygons   31s cpu:56s gc:3s avg:1.8
0:02:18 INF - 	  read     1x(54% 17s wait:4s)
0:02:18 INF - 	  process  2x(31% 10s wait:13s)
0:02:18 INF - 	  write    1x(2% 0.6s wait:30s)
0:02:18 INF - 	natural_earth    19s cpu:21s avg:1.2
0:02:18 INF - 	  read     1x(48% 9s sys:1s done:9s)
0:02:18 INF - 	  process  2x(8% 1s wait:9s done:9s)
0:02:18 INF - 	  write    1x(0% 0s wait:10s done:9s)
0:02:18 INF - 	osm_pass1        4s cpu:7s avg:1.7
0:02:18 INF - 	  read     1x(1% 0s wait:4s)
0:02:18 INF - 	  parse    1x(67% 3s)
0:02:18 INF - 	  process  1x(40% 2s wait:2s)
0:02:18 INF - 	osm_pass2        41s cpu:1m21s avg:2
0:02:18 INF - 	  read     1x(0% 0s wait:22s done:19s)
0:02:18 INF - 	  process  2x(77% 32s)
0:02:18 INF - 	  write    1x(1% 0.5s wait:41s)
0:02:18 INF - 	boundaries       0s cpu:0s avg:1.1
0:02:18 INF - 	sort             3s cpu:4s avg:1.3
0:02:18 INF - 	  worker  1x(71% 2s)
0:02:18 INF - 	archive          35s cpu:1m8s gc:1s avg:1.9
0:02:18 INF - 	  read    1x(2% 0.7s wait:33s)
0:02:18 INF - 	  encode  2x(64% 23s wait:1s)
0:02:18 INF - 	  write   1x(13% 4s wait:29s)
0:02:18 INF - ----------------------------------------
0:02:18 INF - 	archive	109MB
0:02:18 INF - 	features	283MB
-rw-r--r-- 1 runner docker 66M Aug 23 14:53 run.jar

https://github.com/onthegomap/planetiler/actions/runs/5952929664

ℹ️ Base Logs a8e432d 
0:00:00 DEB - argument: config=null (path to config file)
0:00:00 DEB - argument: area=rhode island (name of the extract to download if osm_url/osm_path not specified (i.e. 'monaco' 'rhode island' 'australia' or 'planet'))
0:00:00 INF - argument: stats=use in-memory stats
0:00:00 DEB - argument: madvise=true (default value for whether to use linux madvise(random) to improve memory-mapped read performance for temporary storage)
0:00:00 DEB - argument: storage=mmap (default storage type for temporary data, one of [ram, mmap, direct])
0:00:00 DEB - argument: threads=2 (num threads)
0:00:00 DEB - argument: write_threads=1 (number of threads to use when writing temp features)
0:00:00 DEB - argument: process_threads=2 (number of threads to use when processing input features)
0:00:00 DEB - argument: bounds=Env[-74.07 : -17.84, 21.34 : 43.55] (bounds)
0:00:00 DEB - argument: polygon=null (a .poly file that limits output to tiles intersecting the shape)
0:00:00 DEB - argument: minzoom=0 (minimum zoom level)
0:00:00 DEB - argument: maxzoom=14 (maximum zoom level up to 15)
0:00:00 DEB - argument: render_maxzoom=14 (maximum rendering zoom level up to 15)
0:00:00 DEB - argument: feature_read_threads=1 (number of threads to use when reading features at tile write time)
0:00:00 DEB - argument: loginterval=10 seconds (time between logs)
0:00:00 DEB - argument: force=false (overwriting output file and ignore disk/RAM warnings)
0:00:00 DEB - argument: gzip_temp=false (gzip temporary feature storage (uses more CPU, but less disk space))
0:00:00 DEB - argument: mmap_temp=true (use memory-mapped IO for temp feature files)
0:00:00 DEB - argument: sort_max_readers=6 (maximum number of concurrent read threads to use when sorting chunks)
0:00:00 DEB - argument: sort_max_writers=6 (maximum number of concurrent write threads to use when sorting chunks)
0:00:00 DEB - argument: nodemap_type=sparsearray (type of node location map, one of [noop, sortedtable, sparsearray, array])
0:00:00 DEB - argument: nodemap_storage=mmap (storage for node location map, one of [ram, mmap, direct])
0:00:00 DEB - argument: nodemap_madvise=true (use linux madvise(random) for node locations)
0:00:00 DEB - argument: multipolygon_geometry_storage=mmap (storage for multipolygon geometries, one of [ram, mmap, direct])
0:00:00 DEB - argument: multipolygon_geometry_madvise=true (use linux madvise(random) for temporary multipolygon geometry storage)
0:00:00 DEB - argument: http_user_agent=Planetiler downloader (https://github.com/onthegomap/planetiler) (User-Agent header to set when downloading files over HTTP)
0:00:00 DEB - argument: http_timeout=30 seconds (Timeout to use when downloading files over HTTP)
0:00:00 DEB - argument: http_retries=1 (Retries to use when downloading files over HTTP)
0:00:00 DEB - argument: download_chunk_size_mb=100 (Size of file chunks to download in parallel in megabytes)
0:00:00 DEB - argument: download_threads=1 (Number of parallel threads to use when downloading each file)
0:00:00 DEB - argument: download_max_bandwidth= (Maximum bandwidth to consume when downloading files in units mb/s, mbps, kbps, etc.)
0:00:00 DEB - argument: min_feature_size_at_max_zoom=0.0625 (Default value for the minimum size in tile pixels of features to emit at the maximum zoom level to allow for overzooming)
0:00:00 DEB - argument: min_feature_size=1.0 (Default value for the minimum size in tile pixels of features to emit below the maximum zoom level)
0:00:00 DEB - argument: simplify_tolerance_at_max_zoom=0.0625 (Default value for the tile pixel tolerance to use when simplifying features at the maximum zoom level to allow for overzooming)
0:00:00 DEB - argument: simplify_tolerance=0.1 (Default value for the tile pixel tolerance to use when simplifying features below the maximum zoom level)
0:00:00 DEB - argument: osm_lazy_reads=true (Read OSM blocks from disk in worker threads)
0:00:00 DEB - argument: skip_filled_tiles=false (Skip writing tiles containing only polygon fills to the output)
0:00:00 DEB - argument: tile_warning_size_mb=1.0 (Maximum size in megabytes of a tile to emit a warning about)
0:00:00 DEB - argument: color=null (Color the terminal output)
0:00:00 DEB - argument: keep_unzipped=true (keep unzipped sources by default after reading)
0:00:00 DEB - argument: tmpdir=data/tmp (temp directory)
0:00:00 DEB - argument: only_download=false (download source data then exit)
0:00:00 DEB - argument: download=false (download sources)
0:00:00 DEB - argument: temp_nodes=data/tmp/node.db (temp node db location)
0:00:00 DEB - argument: temp_multipolygons=data/tmp/multipolygon.db (temp multipolygon db location)
0:00:00 DEB - argument: temp_features=data/tmp/feature.db (temp feature db location)
0:00:00 DEB - argument: osm_parse_node_bounds=false (parse bounds from OSM nodes instead of header)
0:00:00 DEB - argument: only_fetch_wikidata=false (fetch wikidata translations then quit)
0:00:00 DEB - argument: fetch_wikidata=false (fetch wikidata translations then continue)
0:00:00 DEB - argument: use_wikidata=true (use wikidata translations)
0:00:00 DEB - argument: wikidata_cache=data/sources/wikidata_names.json (wikidata cache file)
0:00:00 DEB - argument: lake_centerlines_path=data/sources/lake_centerline.shp.zip (lake_centerlines shapefile path)
0:00:00 DEB - argument: free_lake_centerlines_after_read=false (delete lake_centerlines input file after reading to make space for output (reduces peak disk usage))
0:00:00 DEB - argument: water_polygons_path=data/sources/water-polygons-split-3857.zip (water_polygons shapefile path)
0:00:00 DEB - argument: free_water_polygons_after_read=false (delete water_polygons input file after reading to make space for output (reduces peak disk usage))
0:00:00 DEB - argument: natural_earth_path=data/sources/natural_earth_vector.sqlite.zip (natural_earth sqlite db path)
0:00:00 DEB - argument: free_natural_earth_after_read=false (delete natural_earth input file after reading to make space for output (reduces peak disk usage))
0:00:00 DEB - argument: natural_earth_keep_unzipped=true (keep unzipped natural_earth after reading)
0:00:00 DEB - argument: osm_path=data/sources/rhode_island.osm.pbf (osm OSM input file path)
0:00:00 DEB - argument: free_osm_after_read=false (delete osm input file after reading to make space for output (reduces peak disk usage))
0:00:00 DEB - argument: output=data/out.mbtiles (output tile archive path)
0:00:00 DEB - argument: version=false (show version then exit)
0:00:00 INF - Planetiler build git hash: a8e432dfd1016efe4b54ecc4db3e7570cfa3287a
0:00:00 INF - Planetiler build version: 0.6-SNAPSHOT
0:00:00 INF - Planetiler build timestamp: 2023-08-23T14:52:09.440Z
0:00:00 DEB - argument: transliterate=true (attempt to transliterate latin names)
0:00:00 DEB - argument: languages=am,ar,az,be,bg,br,bs,ca,co,cs,cy,da,de,el,en,eo,es,et,eu,fi,fr,fy,ga,gd,he,hi,hr,hu,hy,id,is,it,ja,ja_kana,ja_rm,ja-Latn,ja-Hira,ka,kk,kn,ko,ko-Latn,ku,la,lb,lt,lv,mk,mt,ml,nl,no,oc,pl,pt,rm,ro,ru,sk,sl,sq,sr,sr-Latn,sv,ta,te,th,tr,uk,zh (languages to use)
0:00:00 DEB - argument: only_layers= (Include only certain layers)
0:00:00 DEB - argument: exclude_layers= (Exclude certain layers)
0:00:00 DEB - argument: boundary_country_names=true (boundary layer: add left/right codes of neighboring countries)
0:00:00 DEB - argument: boundary_osm_only=false (boundary layer: only use OSM, even at low zoom levels)
0:00:00 DEB - argument: transportation_z13_paths=false (transportation(_name) layer: show all paths on z13)
0:00:00 DEB - argument: building_merge_z13=true (building layer: merge nearby buildings at z13)
0:00:00 DEB - argument: transportation_name_brunnel=false (transportation_name layer: set to false to omit brunnel and help merge long highways)
0:00:00 DEB - argument: transportation_name_size_for_shield=false (transportation_name layer: allow road names on shorter segments (ie. they will have a shield))
0:00:00 DEB - argument: transportation_name_limit_merge=false (transportation_name layer: limit merge so we don't combine different relations to help merge long highways)
0:00:00 DEB - argument: transportation_name_minor_refs=false (transportation_name layer: include name and refs from minor road networks if not present on a way)
0:00:00 DEB - argument: help=false (show arguments then exit)
0:00:00 INF - Building OpenMapTilesProfile profile into file:///home/runner/work/planetiler/planetiler/data/out.mbtiles in these phases:
0:00:00 INF -   lake_centerlines: Process features in data/sources/lake_centerline.shp.zip
0:00:00 INF -   water_polygons: Process features in data/sources/water-polygons-split-3857.zip
0:00:00 INF -   natural_earth: Process features in data/sources/natural_earth_vector.sqlite.zip
0:00:00 INF -   osm_pass1: Pre-process OpenStreetMap input (store node locations then relation members)
0:00:00 INF -   osm_pass2: Process OpenStreetMap nodes, ways, then relations
0:00:00 INF -   sort: Sort rendered features by tile ID
0:00:00 INF -   archive: Encode each tile and write to TileArchiveConfig[format=MBTILES, scheme=FILE, uri=file:///home/runner/work/planetiler/planetiler/data/out.mbtiles, options={}]
0:00:00 INF - no wikidata translations found, run with --fetch-wikidata to download
0:00:00 DEB - ✓ 197M storage on / (/dev/root) requested for read phase disk, 20G available
0:00:00 DEB -  - 44M used for temporary node location cache
0:00:00 DEB -  - 6.7M used for temporary multipolygon geometry cache
0:00:00 DEB -  - 146M used for temporary feature storage
0:00:00 DEB - ✓ 219M storage on / (/dev/root) requested for write phase disk, 20G available
0:00:00 DEB -  - 146M used for temporary feature storage
0:00:00 DEB -  - 73M used for archive output
0:00:00 DEB - ✓ 313M JVM heap requested for read phase, 4.2G available
0:00:00 DEB -  - 300M used for sparsearray node location in-memory index
0:00:00 DEB -  - 13M used for temporary profile storage
0:00:00 DEB - ✓ 51M storage on / (/dev/root) requested for read phase, 20G available
0:00:00 DEB -  - 44M used for sparsearray node location cache
0:00:00 DEB -  - 6.7M used for multipolygon way geometries
0:00:00 DEB - ✓ 51M temporary files and 2.9G of free memory for OS to cache them
0:00:00 DEB - argument: archive_name=OpenMapTiles ('name' attribute for tileset metadata)
0:00:00 DEB - argument: archive_description=A tileset showcasing all layers in OpenMapTiles. https://openmaptiles.org ('description' attribute for tileset metadata)
0:00:00 DEB - argument: archive_attribution=<a href="https://www.openmaptiles.org/" target="_blank">&copy; OpenMapTiles</a> <a href="https://www.openstreetmap.org/copyright" target="_blank">&copy; OpenStreetMap contributors</a> ('attribution' attribute for tileset metadata)
0:00:00 DEB - argument: archive_version=3.14.0 ('version' attribute for tileset metadata)
0:00:00 DEB - argument: archive_type=baselayer ('type' attribute for tileset metadata)
0:00:00 DEB - argument: archive_format=pbf ('format' attribute for tileset metadata)
0:00:00 DEB - argument: compact=true (mbtiles: reduce the DB size by separating and deduping the tile data)
0:00:00 DEB - argument: no_index=false (mbtiles: skip adding index to sqlite DB)
0:00:00 DEB - argument: vacuum_analyze=false (mbtiles: vacuum analyze sqlite DB after writing)
0:00:00 INF - Using merge sort feature map, chunk size=1431mb max workers=2
0:00:01 INF [lake_centerlines] - 
0:00:01 INF [lake_centerlines] - Starting...
0:00:04 INF [lake_centerlines] -  read: [  59k 100%  28k/s ] write: [    0    0/s ] 0    
    cpus: 1.8 gc:  3% heap: 179M/4.2G direct: 237k postGC: 78M
    ->     (0/3) -> read( -%) ->    (0/1k) -> process( -%  -%) ->   (0/53k) -> write( -%)
0:00:04 INF [lake_centerlines] - Finished in 4s cpu:6s avg:1.8
0:00:04 INF [lake_centerlines] -   read     1x(50% 2s done:2s)
0:00:04 INF [lake_centerlines] -   process  2x(7% 0.2s wait:2s done:1s)
0:00:04 INF [lake_centerlines] -   write    1x(0% 0s wait:2s done:1s)
0:00:04 INF [water_polygons] - 
0:00:04 INF [water_polygons] - Starting...
0:00:14 INF [water_polygons] -  read: [ 1.8k  13%  188/s ] write: [  94k 9.3k/s ] 1.4G 
    cpus: 1.9 gc: 11% heap: 871M/4.2G direct: 54M postGC: 926M
    ->     (0/3) -> read(52%) ->    (0/1k) -> process(22% 40%) ->  (1k/53k) -> write( 0%)
0:00:24 INF [water_polygons] -  read: [ 4.5k  32%  270/s ] write: [ 268k  17k/s ] 1.4G 
    cpus: 1.7 gc:  8% heap: 2.2G/4.2G direct: 54M postGC: 1.5G
    ->     (0/3) -> read(71%) ->    (0/1k) -> process(25% 10%) -> (828/53k) -> write( 0%)
0:00:34 INF [water_polygons] -  read: [ 9.7k  68%  519/s ] write: [ 2.5M 229k/s ] 1.4G 
    cpus: 1.8 gc:  8% heap: 2.2G/4.2G direct: 54M postGC: 440M
    ->     (0/3) -> read(47%) ->   (1k/1k) -> process(35% 42%) -> (746/53k) -> write( 3%)
0:00:35 INF [water_polygons] -  read: [  14k 100% 3.9k/s ] write: [ 4.3M 1.4M/s ] 193M 
    cpus: 1.7 gc:  0% heap: 2.1G/4.2G direct: 54M postGC: 433M
    ->     (0/3) -> read( -%) ->    (0/1k) -> process( -%  -%) ->   (0/53k) -> write( -%)
0:00:35 INF [water_polygons] - Finished in 31s cpu:57s gc:3s avg:1.8
0:00:35 INF [water_polygons] -   read     1x(55% 17s sys:1s wait:4s)
0:00:35 INF [water_polygons] -   process  2x(31% 10s wait:13s)
0:00:35 INF [water_polygons] -   write    1x(2% 0.7s wait:30s)
0:00:35 INF [natural_earth] - 
0:00:35 INF [natural_earth] - Starting...
0:00:47 INF [natural_earth] -  read: [ 349k 100%  35k/s ] write: [  181   18/s ] 193M 
    cpus: 1.4 gc:  0% heap: 2.8G/4.2G direct: 54M postGC: 436M
    ->     (0/3) -> read( -%) ->    (0/1k) -> process( -%  -%) ->   (0/53k) -> write( -%)
0:00:47 INF [natural_earth] - Finished in 12s cpu:15s avg:1.2
0:00:47 INF [natural_earth] -   read     1x(73% 9s sys:1s done:2s)
0:00:47 INF [natural_earth] -   process  2x(12% 1s wait:9s done:2s)
0:00:47 INF [natural_earth] -   write    1x(0% 0s wait:10s done:2s)
0:00:47 INF [osm_pass1] - 
0:00:47 INF [osm_pass1] - Starting...
0:00:50 INF [osm_pass1:process] - Finished nodes: 4,633,223 (1.7M/s) in 3s cpu:4s avg:1.6
0:00:51 INF [osm_pass1:process] - Finished ways: 344,953 (380k/s) in 0.9s cpu:2s avg:2
0:00:51 INF [osm_pass1:process] - Finished relations: 8,097 (67k/s) in 0.1s cpu:0.2s avg:2
0:00:51 INF [osm_pass1] -  nodes: [ 4.6M 1.2M/s ] 478M  ways: [ 344k  90k/s ] rels: [   8k 2.1k/s ] blocks: [  626  164/s ]
    cpus: 1.7 gc:  0% heap: 3.6G/4.2G direct: 54M postGC: 457M hppc: 468k
    read( -%) ->     (0/4) -> parse( -%) ->     (0/4) -> process( -%)
0:00:51 DEB [osm_pass1] - Processed 626 blocks:
0:00:51 DEB [osm_pass1] -   nodes: 4,633,223 (1.7M/s) in 3s cpu:4s avg:1.6
0:00:51 DEB [osm_pass1] -   ways: 344,953 (380k/s) in 0.9s cpu:2s avg:2
0:00:51 DEB [osm_pass1] -   relations: 8,097 (67k/s) in 0.1s cpu:0.2s avg:2
0:00:51 INF [osm_pass1] - Finished in 4s cpu:7s avg:1.7
0:00:51 INF [osm_pass1] -   read     1x(1% 0s wait:4s)
0:00:51 INF [osm_pass1] -   parse    1x(67% 3s)
0:00:51 INF [osm_pass1] -   process  1x(41% 2s wait:2s)
0:00:51 INF [osm_pass2] - 
0:00:51 INF [osm_pass2] - Starting...
0:00:54 DEB [osm_pass2:process] - Sorting long long multimap...
0:00:54 INF [osm_pass2:process] - Finished nodes: 4,633,223 (1.5M/s) in 3s cpu:6s avg:2
0:00:54 DEB [osm_pass2:process] - Sorted long long multimap 0s cpu:0s avg:1.8
0:00:54 WAR [osm_pass2:process] - No GB polygon for inferring route network types
0:01:01 INF [osm_pass2] -  nodes: [ 4.6M 100% 462k/s ] 478M  ways: [ 119k  35%  11k/s ] rels: [    0   0%    0/s ] features: [ 4.8M  49k/s ] 1.6G  blocks: [  594  95%   59/s ]
    cpus: 2 gc:  1% heap: 1.8G/4.2G direct: 54M postGC: 1.2G relInfo: 427k mpGeoms: 424k 
    read( 0%) ->   (11/13) -> process(61% 63%) -> (619/53k) -> write( 2%)
0:01:11 INF [osm_pass2] -  nodes: [ 4.6M 100%    0/s ] 478M  ways: [ 264k  77%  14k/s ] rels: [    0   0%    0/s ] features: [ 5.2M  39k/s ] 1.6G  blocks: [  612  98%    1/s ]
    cpus: 2 gc:  1% heap: 1.3G/4.2G direct: 54M postGC: 1.2G relInfo: 427k mpGeoms: 14M  
    read( 0%) ->   (11/13) -> process(82% 84%) -> (361/53k) -> write( 1%)
0:01:15 INF [osm_pass2:process] - Finished ways: 344,953 (16k/s) in 21s cpu:41s avg:2
0:01:21 INF [osm_pass2] -  nodes: [ 4.6M 100%    0/s ] 478M  ways: [ 344k 100%   8k/s ] rels: [ 3.1k  39%  318/s ] features: [ 5.4M  20k/s ] 1.6G  blocks: [  625 100%    1/s ]
    cpus: 2 gc:  1% heap: 2.4G/4.2G direct: 54M postGC: 1.2G relInfo: 427k mpGeoms: 19M  
    read( -%) ->    (0/13) -> process(70% 72%) -> (751/53k) -> write( 1%)
0:01:29 INF [osm_pass2:process] - Finished relations: 8,097 (583/s) in 14s cpu:27s avg:2
0:01:31 INF [osm_pass2] -  nodes: [ 4.6M 100%    0/s ] 478M  ways: [ 344k 100%    0/s ] rels: [ 7.4k  92%  429/s ] features: [ 5.4M 2.2k/s ] 1.6G  blocks: [  626 100%   <1/s ]
    cpus: 2 gc:  1% heap: 2.3G/4.2G direct: 54M postGC: 1.2G relInfo: 427k mpGeoms: 19M  
    read( -%) ->    (0/13) -> process(91% 89%) -> (684/53k) -> write( 0%)
0:01:33 INF [osm_pass2] -  nodes: [ 4.6M 100%    0/s ] 478M  ways: [ 344k 100%    0/s ] rels: [   8k 100%  494/s ] features: [ 5.4M   3k/s ] 283M  blocks: [  626 100%    0/s ]
    cpus: 2 gc:  0% heap: 3.4G/4.2G direct: 54M postGC: 1.2G relInfo: 427k mpGeoms: 19M  
    read( -%) ->    (0/13) -> process( -%  -%) ->   (0/53k) -> write( -%)
0:01:33 DEB [osm_pass2] - Processed 626 blocks:
0:01:33 DEB [osm_pass2] -   nodes: 4,633,223 (1.5M/s) in 3s cpu:6s avg:2
0:01:33 DEB [osm_pass2] -   ways: 344,953 (16k/s) in 21s cpu:41s avg:2
0:01:33 DEB [osm_pass2] -   relations: 8,097 (583/s) in 14s cpu:27s avg:2
0:01:33 INF [osm_pass2] - Finished in 41s cpu:1m21s avg:2
0:01:33 INF [osm_pass2] -   read     1x(0% 0s wait:22s done:19s)
0:01:33 INF [osm_pass2] -   process  2x(76% 32s)
0:01:33 INF [osm_pass2] -   write    1x(1% 0.4s wait:41s)
0:01:33 INF [boundaries] - 
0:01:33 INF [boundaries] - Starting...
0:01:33 INF [boundaries] - Creating polygons for 1 boundaries
0:01:33 WAR [boundaries] - Unable to form closed polygon for OSM relation 148838 (likely missing edges)
0:01:33 INF [boundaries] - Finished creating 0 country polygons
0:01:33 INF [boundaries] - Finished in 0s cpu:0s avg:1.7
0:01:33 INF - Deleting node.db to make room for output file
0:01:33 INF [sort] - 
0:01:33 INF [sort] - Starting...
0:01:33 INF [sort] - Grouped 8 chunks into 1
0:01:35 INF [sort] -  chunks: [   1 /   1 100% ] 283M 
    cpus: 1.6 gc: 10% heap: 1.5G/4.2G direct: 54M postGC: 1.4G
    ->     (0/3) -> worker( -%)
0:01:35 INF [sort] - Finished in 2s cpu:4s avg:1.6
0:01:35 INF [sort] -   worker  1x(83% 2s)
0:01:35 INF - read:0s write:0s sort:1s
0:01:35 INF [archive] - 
0:01:35 INF [archive] - Starting...
0:01:36 DEB [archive:write] - Execute mbtiles: create table metadata (name text, value text);
0:01:36 DEB [archive:write] - Execute mbtiles: create unique index name on metadata (name);
0:01:36 DEB [archive:write] - Execute mbtiles: create table tiles_shallow (
  zoom_level integer,
  tile_column integer,
  tile_row integer,
  tile_data_id integer

  , primary key(zoom_level,tile_column,tile_row)

) without rowid

0:01:36 DEB [archive:write] - Execute mbtiles: create table tiles_data (
  tile_data_id integer primary key,
  tile_data blob
)

0:01:36 DEB [archive:write] - Execute mbtiles: create view tiles AS
select
  tiles_shallow.zoom_level as zoom_level,
  tiles_shallow.tile_column as tile_column,
  tiles_shallow.tile_row as tile_row,
  tiles_data.tile_data as tile_data
from tiles_shallow
join tiles_data on tiles_shallow.tile_data_id = tiles_data.tile_data_id

0:01:36 DEB [archive:write] - Set mbtiles metadata: format=pbf
0:01:36 DEB [archive:write] - Set mbtiles metadata: center=-45.955,32.445,3
0:01:36 DEB [archive:write] - Set mbtiles metadata: bounds=-74.07,21.34,-17.84,43.55
0:01:36 DEB [archive:write] - Set mbtiles metadata: json={"vector_layers":[{"id":"aerodrome_label","fields":{"name_int":"String","iata":"String","ele_ft":"Number","name_de":"String","name":"String","icao":"String","name:en":"String","class":"String","name_en":"String","name:latin":"String","ele":"Number"},"minzoom":10,"maxzoom":14},{"id":"aeroway","fields":{"ref":"String","class":"String"},"minzoom":10,"maxzoom":14},{"id":"boundary","fields":{"disputed":"Number","admin_level":"Number","maritime":"Number"},"minzoom":0,"maxzoom":14},{"id":"building","fields":{"colour":"String","render_height":"Number","render_min_height":"Number","hide_3d":"Boolean"},"minzoom":13,"maxzoom":14},{"id":"housenumber","fields":{"housenumber":"String"},"minzoom":14,"maxzoom":14},{"id":"landcover","fields":{"subclass":"String","class":"String","_numpoints":"Number"},"minzoom":7,"maxzoom":14},{"id":"landuse","fields":{"class":"String"},"minzoom":4,"maxzoom":14},{"id":"mountain_peak","fields":{"name_int":"String","customary_ft":"Number","ele_ft":"Number","name_de":"Str... 2358 more characters
0:01:36 DEB [archive:write] - Set mbtiles metadata: name=OpenMapTiles
0:01:36 DEB [archive:write] - Set mbtiles metadata: description=A tileset showcasing all layers in OpenMapTiles. https://openmaptiles.org
0:01:36 DEB [archive:write] - Set mbtiles metadata: attribution=<a href="https://www.openmaptiles.org/" target="_blank">&copy; OpenMapTiles</a> <a href="https://www.openstreetmap.org/copyright" target="_blank">&copy; OpenStreetMap contributors</a>
0:01:36 DEB [archive:write] - Set mbtiles metadata: version=3.14.0
0:01:36 DEB [archive:write] - Set mbtiles metadata: type=baselayer
0:01:36 DEB [archive:write] - Set mbtiles metadata: minzoom=0
0:01:36 DEB [archive:write] - Set mbtiles metadata: maxzoom=14
0:01:36 DEB [archive:write] - Set mbtiles metadata: planetiler:version=0.6-SNAPSHOT
0:01:36 DEB [archive:write] - Set mbtiles metadata: planetiler:githash=a8e432dfd1016efe4b54ecc4db3e7570cfa3287a
0:01:36 DEB [archive:write] - Set mbtiles metadata: planetiler:buildtime=2023-08-23T14:52:09.440Z
0:01:36 DEB [archive:write] - Set mbtiles metadata: planetiler:osm:osmosisreplicationtime=2023-08-22T20:21:53Z
0:01:36 DEB [archive:write] - Set mbtiles metadata: planetiler:osm:osmosisreplicationseq=3799
0:01:36 DEB [archive:write] - Set mbtiles metadata: planetiler:osm:osmosisreplicationurl=http://download.geofabrik.de/north-america/us/rhode-island-updates
0:01:38 INF [archive:write] - Starting z0
0:01:38 INF [archive:write] - Finished z0 in 0s cpu:0s avg:0, now starting z1
0:01:38 INF [archive:write] - Finished z1 in 0s cpu:0s avg:0, now starting z2
0:01:38 INF [archive:write] - Finished z2 in 0s cpu:0s avg:0, now starting z3
0:01:38 INF [archive:write] - Finished z3 in 0s cpu:0s avg:0, now starting z4
0:01:38 INF [archive:write] - Finished z4 in 0s cpu:0s avg:0, now starting z5
0:01:38 INF [archive:write] - Finished z5 in 0s cpu:0s avg:0, now starting z6
0:01:38 INF [archive:write] - Finished z6 in 0s cpu:0s avg:0, now starting z7
0:01:39 INF [archive:write] - Finished z7 in 1s cpu:2s avg:2, now starting z8
0:01:42 INF [archive:write] - Finished z8 in 3s cpu:5s avg:2, now starting z9
0:01:44 INF [archive:write] - Finished z9 in 3s cpu:5s avg:2, now starting z10
0:01:46 INF [archive] -  features: [ 128k   2%  12k/s ] 283M  tiles: [ 4.7k  477/s ] 1.2M 
    cpus: 2 gc:  2% heap: 2.5G/4.2G direct: 54M postGC: 1.7G
    read( 1%) -> (213/217) -> encode(63% 59%) -> (215/216) -> write( 1%)
    last tile: 10/308/380 (z10 4%) https://www.openstreetmap.org/#map=10/42.03297/-71.71875
0:01:46 INF [archive:write] - Finished z10 in 2s cpu:4s avg:2, now starting z11
0:01:49 INF [archive:write] - Finished z11 in 3s cpu:7s avg:2, now starting z12
0:01:54 INF [archive:write] - Finished z12 in 5s cpu:9s avg:2, now starting z13
0:01:56 INF [archive] -  features: [ 671k  12%  54k/s ] 283M  tiles: [ 288k  28k/s ] 13M  
    cpus: 2 gc:  2% heap: 3.3G/4.2G direct: 54M postGC: 1.8G
    read( 1%) -> (214/217) -> encode(64% 63%) -> (215/216) -> write( 4%)
    last tile: 13/2465/3046 (z13 4%) https://www.openstreetmap.org/#map=13/41.83683/-71.67480
0:02:05 INF [archive:write] - Finished z13 in 11s cpu:21s avg:2, now starting z14
0:02:06 INF [archive] -  features: [ 1.9M  36% 129k/s ] 283M  tiles: [ 1.1M  86k/s ] 38M  
    cpus: 2 gc:  4% heap: 1.8G/4.2G direct: 54M postGC: 1.8G
    read( 2%) -> (214/217) -> encode(75% 69%) -> (215/216) -> write( 9%)
    last tile: 14/4934/6202 (z14 4%) https://www.openstreetmap.org/#map=14/40.01079/-71.58691
0:02:11 DEB [archive:write] - Shallow tiles written: 4,115,061
0:02:11 DEB [archive:write] - Tile data written: 17,863 (100% omitted)
0:02:11 DEB [archive:write] - Unique tile hashes: 8,984
0:02:11 INF [archive:write] - Finished z14 in 6s cpu:11s avg:1.9
0:02:11 INF [archive] -  features: [ 5.4M 100% 671k/s ] 283M  tiles: [ 4.1M 573k/s ] 109M 
    cpus: 1.9 gc:  2% heap: 2.9G/4.2G direct: 54M postGC: 1.8G
    read( -%) ->   (0/217) -> encode( -%  -%) ->   (0/216) -> write( -%)
    last tile: 14/7380/5985 (z14 100%) https://www.openstreetmap.org/#map=14/43.56447/-17.84180
0:02:11 DEB [archive] - Tile stats:
0:02:11 DEB [archive] - z0 avg:7.9k max:7.9k
0:02:11 DEB [archive] - z1 avg:4k max:4k
0:02:11 DEB [archive] - z2 avg:9.4k max:9.4k
0:02:11 DEB [archive] - z3 avg:4k max:6.4k
0:02:11 DEB [archive] - z4 avg:1.6k max:4.6k
0:02:11 DEB [archive] - z5 avg:1.4k max:7.1k
0:02:11 DEB [archive] - z6 avg:974 max:22k
0:02:11 DEB [archive] - z7 avg:769 max:58k
0:02:11 DEB [archive] - z8 avg:417 max:127k
0:02:11 DEB [archive] - z9 avg:281 max:298k
0:02:11 DEB [archive] - z10 avg:161 max:256k
0:02:11 DEB [archive] - z11 avg:106 max:136k
0:02:11 DEB [archive] - z12 avg:85 max:114k
0:02:11 DEB [archive] - z13 avg:72 max:128k
0:02:11 DEB [archive] - z14 avg:68 max:303k
0:02:11 DEB [archive] - all avg:70 max:303k
0:02:11 DEB [archive] -  # features: 5,437,996
0:02:11 DEB [archive] -     # tiles: 4,115,061
0:02:11 INF [archive] - Finished in 36s cpu:1m10s gc:1s avg:2
0:02:11 INF [archive] -   read    1x(2% 0.7s wait:34s)
0:02:11 INF [archive] -   encode  2x(63% 23s)
0:02:11 INF [archive] -   write   1x(13% 4s wait:29s)
0:02:11 INF - Finished in 2m11s cpu:4m1s gc:5s avg:1.8
0:02:11 INF - FINISHED!
0:02:11 INF - 
0:02:11 INF - ----------------------------------------
0:02:11 INF - data errors:
0:02:11 INF - 	render_snap_fix_input	16,460
0:02:11 INF - 	osm_boundary_missing_way	63
0:02:11 INF - 	osm_multipolygon_missing_way	57
0:02:11 INF - 	merge_snap_fix_input	14
0:02:11 INF - 	feature_centroid_if_convex_osm_invalid_multipolygon_empty_after_fix	2
0:02:11 INF - 	feature_polygon_osm_invalid_multipolygon_empty_after_fix	2
0:02:11 INF - 	omt_park_area_osm_invalid_multipolygon_empty_after_fix	1
0:02:11 INF - ----------------------------------------
0:02:11 INF - 	overall          2m11s cpu:4m1s gc:5s avg:1.8
0:02:11 INF - 	lake_centerlines 4s cpu:6s avg:1.8
0:02:11 INF - 	  read     1x(50% 2s done:2s)
0:02:11 INF - 	  process  2x(7% 0.2s wait:2s done:1s)
0:02:11 INF - 	  write    1x(0% 0s wait:2s done:1s)
0:02:11 INF - 	water_polygons   31s cpu:57s gc:3s avg:1.8
0:02:11 INF - 	  read     1x(55% 17s sys:1s wait:4s)
0:02:11 INF - 	  process  2x(31% 10s wait:13s)
0:02:11 INF - 	  write    1x(2% 0.7s wait:30s)
0:02:11 INF - 	natural_earth    12s cpu:15s avg:1.2
0:02:11 INF - 	  read     1x(73% 9s sys:1s done:2s)
0:02:11 INF - 	  process  2x(12% 1s wait:9s done:2s)
0:02:11 INF - 	  write    1x(0% 0s wait:10s done:2s)
0:02:11 INF - 	osm_pass1        4s cpu:7s avg:1.7
0:02:11 INF - 	  read     1x(1% 0s wait:4s)
0:02:11 INF - 	  parse    1x(67% 3s)
0:02:11 INF - 	  process  1x(41% 2s wait:2s)
0:02:11 INF - 	osm_pass2        41s cpu:1m21s avg:2
0:02:11 INF - 	  read     1x(0% 0s wait:22s done:19s)
0:02:11 INF - 	  process  2x(76% 32s)
0:02:11 INF - 	  write    1x(1% 0.4s wait:41s)
0:02:11 INF - 	boundaries       0s cpu:0s avg:1.7
0:02:11 INF - 	sort             2s cpu:4s avg:1.6
0:02:11 INF - 	  worker  1x(83% 2s)
0:02:11 INF - 	archive          36s cpu:1m10s gc:1s avg:2
0:02:11 INF - 	  read    1x(2% 0.7s wait:34s)
0:02:11 INF - 	  encode  2x(63% 23s)
0:02:11 INF - 	  write   1x(13% 4s wait:29s)
0:02:11 INF - ----------------------------------------
0:02:11 INF - 	archive	109MB
0:02:11 INF - 	features	283MB
-rw-r--r-- 1 runner docker 66M Aug 23 14:55 run.jar
ℹ️ This Branch Logs fbb6ab5 
0:00:00 DEB - argument: config=null (path to config file)
0:00:00 DEB - argument: area=rhode island (name of the extract to download if osm_url/osm_path not specified (i.e. 'monaco' 'rhode island' 'australia' or 'planet'))
0:00:00 INF - argument: stats=use in-memory stats
0:00:00 DEB - argument: madvise=true (default value for whether to use linux madvise(random) to improve memory-mapped read performance for temporary storage)
0:00:00 DEB - argument: storage=mmap (default storage type for temporary data, one of [ram, mmap, direct])
0:00:00 DEB - argument: threads=2 (num threads)
0:00:00 DEB - argument: write_threads=1 (number of threads to use when writing temp features)
0:00:00 DEB - argument: process_threads=2 (number of threads to use when processing input features)
0:00:00 DEB - argument: bounds=Env[-74.07 : -17.84, 21.34 : 43.55] (bounds)
0:00:00 DEB - argument: polygon=null (a .poly file that limits output to tiles intersecting the shape)
0:00:00 DEB - argument: minzoom=0 (minimum zoom level)
0:00:00 DEB - argument: maxzoom=14 (maximum zoom level up to 15)
0:00:00 DEB - argument: render_maxzoom=14 (maximum rendering zoom level up to 15)
0:00:00 DEB - argument: feature_read_threads=1 (number of threads to use when reading features at tile write time)
0:00:00 DEB - argument: tile_write_threads=1 (number of threads used to write tiles - only supported by [csv, tsv, proto, pbf, json])
0:00:00 DEB - argument: loginterval=10 seconds (time between logs)
0:00:00 DEB - argument: force=false (overwriting output file and ignore disk/RAM warnings)
0:00:00 DEB - argument: append=false (append to the output file - only supported by [csv, tsv, proto, pbf, json])
0:00:00 DEB - argument: gzip_temp=false (gzip temporary feature storage (uses more CPU, but less disk space))
0:00:00 DEB - argument: mmap_temp=true (use memory-mapped IO for temp feature files)
0:00:00 DEB - argument: sort_max_readers=6 (maximum number of concurrent read threads to use when sorting chunks)
0:00:00 DEB - argument: sort_max_writers=6 (maximum number of concurrent write threads to use when sorting chunks)
0:00:00 DEB - argument: nodemap_type=sparsearray (type of node location map, one of [noop, sortedtable, sparsearray, array])
0:00:00 DEB - argument: nodemap_storage=mmap (storage for node location map, one of [ram, mmap, direct])
0:00:00 DEB - argument: nodemap_madvise=true (use linux madvise(random) for node locations)
0:00:00 DEB - argument: multipolygon_geometry_storage=mmap (storage for multipolygon geometries, one of [ram, mmap, direct])
0:00:00 DEB - argument: multipolygon_geometry_madvise=true (use linux madvise(random) for temporary multipolygon geometry storage)
0:00:00 DEB - argument: http_user_agent=Planetiler downloader (https://github.com/onthegomap/planetiler) (User-Agent header to set when downloading files over HTTP)
0:00:00 DEB - argument: http_timeout=30 seconds (Timeout to use when downloading files over HTTP)
0:00:00 DEB - argument: http_retries=1 (Retries to use when downloading files over HTTP)
0:00:00 DEB - argument: download_chunk_size_mb=100 (Size of file chunks to download in parallel in megabytes)
0:00:00 DEB - argument: download_threads=1 (Number of parallel threads to use when downloading each file)
0:00:00 DEB - argument: download_max_bandwidth= (Maximum bandwidth to consume when downloading files in units mb/s, mbps, kbps, etc.)
0:00:00 DEB - argument: min_feature_size_at_max_zoom=0.0625 (Default value for the minimum size in tile pixels of features to emit at the maximum zoom level to allow for overzooming)
0:00:00 DEB - argument: min_feature_size=1.0 (Default value for the minimum size in tile pixels of features to emit below the maximum zoom level)
0:00:00 DEB - argument: simplify_tolerance_at_max_zoom=0.0625 (Default value for the tile pixel tolerance to use when simplifying features at the maximum zoom level to allow for overzooming)
0:00:00 DEB - argument: simplify_tolerance=0.1 (Default value for the tile pixel tolerance to use when simplifying features below the maximum zoom level)
0:00:00 DEB - argument: osm_lazy_reads=true (Read OSM blocks from disk in worker threads)
0:00:00 DEB - argument: skip_filled_tiles=false (Skip writing tiles containing only polygon fills to the output)
0:00:00 DEB - argument: tile_warning_size_mb=1.0 (Maximum size in megabytes of a tile to emit a warning about)
0:00:00 DEB - argument: color=null (Color the terminal output)
0:00:00 DEB - argument: keep_unzipped=true (keep unzipped sources by default after reading)
0:00:00 DEB - argument: tile_compression=gzip (the tile compression, one of [none, gzip])
0:00:00 DEB - argument: tmpdir=data/tmp (temp directory)
0:00:00 DEB - argument: only_download=false (download source data then exit)
0:00:00 DEB - argument: download=false (download sources)
0:00:00 DEB - argument: temp_nodes=data/tmp/node.db (temp node db location)
0:00:00 DEB - argument: temp_multipolygons=data/tmp/multipolygon.db (temp multipolygon db location)
0:00:00 DEB - argument: temp_features=data/tmp/feature.db (temp feature db location)
0:00:00 DEB - argument: osm_parse_node_bounds=false (parse bounds from OSM nodes instead of header)
0:00:00 DEB - argument: only_fetch_wikidata=false (fetch wikidata translations then quit)
0:00:00 DEB - argument: fetch_wikidata=false (fetch wikidata translations then continue)
0:00:00 DEB - argument: use_wikidata=true (use wikidata translations)
0:00:00 DEB - argument: wikidata_cache=data/sources/wikidata_names.json (wikidata cache file)
0:00:00 DEB - argument: lake_centerlines_path=data/sources/lake_centerline.shp.zip (lake_centerlines shapefile path)
0:00:00 DEB - argument: free_lake_centerlines_after_read=false (delete lake_centerlines input file after reading to make space for output (reduces peak disk usage))
0:00:00 DEB - argument: water_polygons_path=data/sources/water-polygons-split-3857.zip (water_polygons shapefile path)
0:00:00 DEB - argument: free_water_polygons_after_read=false (delete water_polygons input file after reading to make space for output (reduces peak disk usage))
0:00:00 DEB - argument: natural_earth_path=data/sources/natural_earth_vector.sqlite.zip (natural_earth sqlite db path)
0:00:00 DEB - argument: free_natural_earth_after_read=false (delete natural_earth input file after reading to make space for output (reduces peak disk usage))
0:00:00 DEB - argument: natural_earth_keep_unzipped=true (keep unzipped natural_earth after reading)
0:00:00 DEB - argument: osm_path=data/sources/rhode_island.osm.pbf (osm OSM input file path)
0:00:00 DEB - argument: free_osm_after_read=false (delete osm input file after reading to make space for output (reduces peak disk usage))
0:00:00 DEB - argument: output=data/out.mbtiles (output tile archive path)
0:00:00 DEB - argument: version=false (show version then exit)
0:00:00 INF - Planetiler build git hash: fbb6ab52b593c75085dd23abd540f5835bf77ebc
0:00:00 INF - Planetiler build version: 0.6-SNAPSHOT
0:00:00 INF - Planetiler build timestamp: 2023-08-23T14:51:25.428Z
0:00:00 DEB - argument: transliterate=true (attempt to transliterate latin names)
0:00:00 DEB - argument: languages=am,ar,az,be,bg,br,bs,ca,co,cs,cy,da,de,el,en,eo,es,et,eu,fi,fr,fy,ga,gd,he,hi,hr,hu,hy,id,is,it,ja,ja_kana,ja_rm,ja-Latn,ja-Hira,ka,kk,kn,ko,ko-Latn,ku,la,lb,lt,lv,mk,mt,ml,nl,no,oc,pl,pt,rm,ro,ru,sk,sl,sq,sr,sr-Latn,sv,ta,te,th,tr,uk,zh (languages to use)
0:00:00 DEB - argument: only_layers= (Include only certain layers)
0:00:00 DEB - argument: exclude_layers= (Exclude certain layers)
0:00:00 DEB - argument: boundary_country_names=true (boundary layer: add left/right codes of neighboring countries)
0:00:00 DEB - argument: boundary_osm_only=false (boundary layer: only use OSM, even at low zoom levels)
0:00:00 DEB - argument: transportation_z13_paths=false (transportation(_name) layer: show all paths on z13)
0:00:00 DEB - argument: building_merge_z13=true (building layer: merge nearby buildings at z13)
0:00:00 DEB - argument: transportation_name_brunnel=false (transportation_name layer: set to false to omit brunnel and help merge long highways)
0:00:00 DEB - argument: transportation_name_size_for_shield=false (transportation_name layer: allow road names on shorter segments (ie. they will have a shield))
0:00:00 DEB - argument: transportation_name_limit_merge=false (transportation_name layer: limit merge so we don't combine different relations to help merge long highways)
0:00:00 DEB - argument: transportation_name_minor_refs=false (transportation_name layer: include name and refs from minor road networks if not present on a way)
0:00:00 DEB - argument: help=false (show arguments then exit)
0:00:00 INF - Building OpenMapTilesProfile profile into file:///home/runner/work/planetiler/planetiler/data/out.mbtiles in these phases:
0:00:00 INF -   lake_centerlines: Process features in data/sources/lake_centerline.shp.zip
0:00:00 INF -   water_polygons: Process features in data/sources/water-polygons-split-3857.zip
0:00:00 INF -   natural_earth: Process features in data/sources/natural_earth_vector.sqlite.zip
0:00:00 INF -   osm_pass1: Pre-process OpenStreetMap input (store node locations then relation members)
0:00:00 INF -   osm_pass2: Process OpenStreetMap nodes, ways, then relations
0:00:00 INF -   sort: Sort rendered features by tile ID
0:00:00 INF -   archive: Encode each tile and write to TileArchiveConfig[format=MBTILES, scheme=FILE, uri=file:///home/runner/work/planetiler/planetiler/data/out.mbtiles, options={}]
0:00:00 INF - no wikidata translations found, run with --fetch-wikidata to download
0:00:00 DEB - ✓ 197M storage on / (/dev/root) requested for read phase disk, 21G available
0:00:00 DEB -  - 44M used for temporary node location cache
0:00:00 DEB -  - 6.7M used for temporary multipolygon geometry cache
0:00:00 DEB -  - 146M used for temporary feature storage
0:00:00 DEB - ✓ 219M storage on / (/dev/root) requested for write phase disk, 21G available
0:00:00 DEB -  - 146M used for temporary feature storage
0:00:00 DEB -  - 73M used for archive output
0:00:00 DEB - ✓ 313M JVM heap requested for read phase, 4.2G available
0:00:00 DEB -  - 300M used for sparsearray node location in-memory index
0:00:00 DEB -  - 13M used for temporary profile storage
0:00:00 DEB - ✓ 51M storage on / (/dev/root) requested for read phase, 21G available
0:00:00 DEB -  - 44M used for sparsearray node location cache
0:00:00 DEB -  - 6.7M used for multipolygon way geometries
0:00:00 DEB - ✓ 51M temporary files and 2.9G of free memory for OS to cache them
0:00:00 DEB - argument: archive_name=OpenMapTiles ('name' attribute for tileset metadata)
0:00:00 DEB - argument: archive_description=A tileset showcasing all layers in OpenMapTiles. https://openmaptiles.org ('description' attribute for tileset metadata)
0:00:00 DEB - argument: archive_attribution=<a href="https://www.openmaptiles.org/" target="_blank">&copy; OpenMapTiles</a> <a href="https://www.openstreetmap.org/copyright" target="_blank">&copy; OpenStreetMap contributors</a> ('attribution' attribute for tileset metadata)
0:00:00 DEB - argument: archive_version=3.14.0 ('version' attribute for tileset metadata)
0:00:00 DEB - argument: archive_type=baselayer ('type' attribute for tileset metadata)
0:00:00 DEB - argument: archive_format=pbf ('format' attribute for tileset metadata)
0:00:00 DEB - argument: compact=true (mbtiles: reduce the DB size by separating and deduping the tile data)
0:00:00 DEB - argument: no_index=false (mbtiles: skip adding index to sqlite DB)
0:00:00 DEB - argument: vacuum_analyze=false (mbtiles: vacuum analyze sqlite DB after writing)
0:00:00 INF - Using merge sort feature map, chunk size=1431mb max workers=2
0:00:01 INF [lake_centerlines] - 
0:00:01 INF [lake_centerlines] - Starting...
0:00:05 INF [lake_centerlines] -  read: [  59k 100%  29k/s ] write: [    0    0/s ] 0    
    cpus: 1.9 gc:  3% heap: 179M/4.2G direct: 237k postGC: 79M
    ->     (0/3) -> read( -%) ->    (0/1k) -> process( -%  -%) ->   (0/53k) -> write( -%)
0:00:05 INF [lake_centerlines] - Finished in 4s cpu:7s avg:1.8
0:00:05 INF [lake_centerlines] -   read     1x(44% 2s done:2s)
0:00:05 INF [lake_centerlines] -   process  2x(7% 0.3s wait:2s done:2s)
0:00:05 INF [lake_centerlines] -   write    1x(0% 0s wait:2s done:2s)
0:00:05 INF [water_polygons] - 
0:00:05 INF [water_polygons] - Starting...
0:00:15 INF [water_polygons] -  read: [ 1.8k  13%  184/s ] write: [  94k 9.3k/s ] 1.4G 
    cpus: 2 gc: 13% heap: 1G/4.2G direct: 54M postGC: 1G
    ->     (0/3) -> read(51%) ->    (0/1k) -> process(27% 36%) ->  (1k/53k) -> write( 0%)
0:00:25 INF [water_polygons] -  read: [ 4.6k  32%  279/s ] write: [ 268k  17k/s ] 1.4G 
    cpus: 1.7 gc:  6% heap: 3.4G/4.2G direct: 54M postGC: 1.3G
    ->     (0/3) -> read(72%) ->    (0/1k) -> process(12% 25%) ->  (1k/53k) -> write( 0%)
0:00:35 INF [water_polygons] -  read: [ 9.8k  68%  521/s ] write: [ 2.7M 249k/s ] 1.4G 
    cpus: 1.8 gc:  9% heap: 1.6G/4.2G direct: 54M postGC: 416M
    ->     (0/3) -> read(44%) ->   (1k/1k) -> process(38% 41%) -> (1.6k/53k) -> write( 3%)
0:00:36 INF [water_polygons] -  read: [  14k 100% 4.3k/s ] write: [ 4.3M 1.4M/s ] 193M 
    cpus: 1.9 gc:  0% heap: 1.7G/4.2G direct: 54M postGC: 405M
    ->     (0/3) -> read( -%) ->    (0/1k) -> process( -%  -%) ->   (0/53k) -> write(21%)
0:00:36 INF [water_polygons] - Finished in 31s cpu:56s gc:3s avg:1.8
0:00:36 INF [water_polygons] -   read     1x(54% 17s wait:4s)
0:00:36 INF [water_polygons] -   process  2x(31% 10s wait:13s)
0:00:36 INF [water_polygons] -   write    1x(2% 0.6s wait:30s)
0:00:36 INF [natural_earth] - 
0:00:36 INF [natural_earth] - Starting...
0:00:36 INF [natural_earth] - unzipping /home/runner/work/planetiler/planetiler/data/sources/natural_earth_vector.sqlite.zip to data/sources/natural_earth_vector.sqlite.zip-unzipped/%2Fnatural_earth_vector.sqlite%2Fpackages%2Fnatural_earth_vector.sqlite
0:00:54 INF [natural_earth] -  read: [ 349k 100%  35k/s ] write: [  181   18/s ] 193M 
    cpus: 1.5 gc:  0% heap: 511M/4.2G direct: 54M postGC: 409M
    ->     (0/3) -> read( -%) ->    (0/1k) -> process( -%  -%) ->   (0/53k) -> write( -%)
0:00:54 INF [natural_earth] - Finished in 19s cpu:21s avg:1.2
0:00:54 INF [natural_earth] -   read     1x(48% 9s sys:1s done:9s)
0:00:54 INF [natural_earth] -   process  2x(8% 1s wait:9s done:9s)
0:00:54 INF [natural_earth] -   write    1x(0% 0s wait:10s done:9s)
0:00:54 INF [osm_pass1] - 
0:00:54 INF [osm_pass1] - Starting...
0:00:57 INF [osm_pass1:process] - Finished nodes: 4,633,223 (1.5M/s) in 3s cpu:5s avg:1.7
0:00:58 INF [osm_pass1:process] - Finished ways: 344,953 (378k/s) in 0.9s cpu:2s avg:1.8
0:00:58 INF [osm_pass1:process] - Finished relations: 8,097 (80k/s) in 0.1s cpu:0.2s avg:1.8
0:00:58 INF [osm_pass1] -  nodes: [ 4.6M 1.1M/s ] 478M  ways: [ 344k  85k/s ] rels: [   8k   2k/s ] blocks: [  626  155/s ]
    cpus: 1.7 gc:  0% heap: 1.2G/4.2G direct: 54M postGC: 1.1G hppc: 468k
    read( -%) ->     (0/4) -> parse( -%) ->     (0/4) -> process( -%)
0:00:58 DEB [osm_pass1] - Processed 626 blocks:
0:00:58 DEB [osm_pass1] -   nodes: 4,633,223 (1.5M/s) in 3s cpu:5s avg:1.7
0:00:58 DEB [osm_pass1] -   ways: 344,953 (378k/s) in 0.9s cpu:2s avg:1.8
0:00:58 DEB [osm_pass1] -   relations: 8,097 (80k/s) in 0.1s cpu:0.2s avg:1.8
0:00:58 INF [osm_pass1] - Finished in 4s cpu:7s avg:1.7
0:00:58 INF [osm_pass1] -   read     1x(1% 0s wait:4s)
0:00:58 INF [osm_pass1] -   parse    1x(67% 3s)
0:00:58 INF [osm_pass1] -   process  1x(40% 2s wait:2s)
0:00:58 INF [osm_pass2] - 
0:00:58 INF [osm_pass2] - Starting...
0:01:01 DEB [osm_pass2:process] - Sorting long long multimap...
0:01:01 DEB [osm_pass2:process] - Sorted long long multimap 0s cpu:0s avg:2.1
0:01:01 INF [osm_pass2:process] - Finished nodes: 4,633,223 (1.6M/s) in 3s cpu:6s avg:2
0:01:01 WAR [osm_pass2:process] - No GB polygon for inferring route network types
0:01:08 INF [osm_pass2] -  nodes: [ 4.6M 100% 463k/s ] 478M  ways: [ 118k  34%  11k/s ] rels: [    0   0%    0/s ] features: [ 4.8M  49k/s ] 1.6G  blocks: [  593  95%   59/s ]
    cpus: 2 gc:  1% heap: 2.2G/4.2G direct: 54M postGC: 1.1G relInfo: 427k mpGeoms: 424k 
    read( 0%) ->   (11/13) -> process(61% 59%) -> (897/53k) -> write( 2%)
0:01:18 INF [osm_pass2] -  nodes: [ 4.6M 100%    0/s ] 478M  ways: [ 261k  76%  14k/s ] rels: [    0   0%    0/s ] features: [ 5.2M  38k/s ] 1.6G  blocks: [  612  98%    1/s ]
    cpus: 2 gc:  1% heap: 3.6G/4.2G direct: 54M postGC: 1.2G relInfo: 427k mpGeoms: 14M  
    read( 0%) ->   (11/13) -> process(81% 79%) -> (738/53k) -> write( 1%)
0:01:23 INF [osm_pass2:process] - Finished ways: 344,953 (16k/s) in 21s cpu:42s avg:2
0:01:28 INF [osm_pass2] -  nodes: [ 4.6M 100%    0/s ] 478M  ways: [ 344k 100% 8.3k/s ] rels: [ 3.3k  42%  337/s ] features: [ 5.4M  20k/s ] 1.6G  blocks: [  625 100%    1/s ]
    cpus: 2 gc:  1% heap: 3G/4.2G direct: 54M postGC: 1.2G relInfo: 427k mpGeoms: 19M  
    read( -%) ->    (0/13) -> process(83% 80%) -> (1.7k/53k) -> write( 1%)
0:01:36 INF [osm_pass2:process] - Finished relations: 8,097 (588/s) in 14s cpu:27s avg:2
0:01:38 INF [osm_pass2] -  nodes: [ 4.6M 100%    0/s ] 478M  ways: [ 344k 100%    0/s ] rels: [ 7.4k  92%  410/s ] features: [ 5.4M 2.1k/s ] 1.6G  blocks: [  626 100%   <1/s ]
    cpus: 2 gc:  1% heap: 2.3G/4.2G direct: 54M postGC: 1.1G relInfo: 427k mpGeoms: 19M  
    read( -%) ->    (0/13) -> process(87% 86%) -> (1.5k/53k) -> write( 0%)
0:01:39 INF [osm_pass2] -  nodes: [ 4.6M 100%    0/s ] 478M  ways: [ 344k 100%    0/s ] rels: [   8k 100%  616/s ] features: [ 5.4M 4.7k/s ] 283M  blocks: [  626 100%    0/s ]
    cpus: 2 gc:  0% heap: 3.4G/4.2G direct: 54M postGC: 1.1G relInfo: 427k mpGeoms: 19M  
    read( -%) ->    (0/13) -> process( -%  -%) ->   (0/53k) -> write( -%)
0:01:39 DEB [osm_pass2] - Processed 626 blocks:
0:01:39 DEB [osm_pass2] -   nodes: 4,633,223 (1.6M/s) in 3s cpu:6s avg:2
0:01:39 DEB [osm_pass2] -   ways: 344,953 (16k/s) in 21s cpu:42s avg:2
0:01:39 DEB [osm_pass2] -   relations: 8,097 (588/s) in 14s cpu:27s avg:2
0:01:39 INF [osm_pass2] - Finished in 41s cpu:1m21s avg:2
0:01:39 INF [osm_pass2] -   read     1x(0% 0s wait:22s done:19s)
0:01:40 INF [osm_pass2] -   process  2x(77% 32s)
0:01:40 INF [osm_pass2] -   write    1x(1% 0.5s wait:41s)
0:01:40 INF [boundaries] - 
0:01:40 INF [boundaries] - Starting...
0:01:40 INF [boundaries] - Creating polygons for 1 boundaries
0:01:40 WAR [boundaries] - Unable to form closed polygon for OSM relation 148838 (likely missing edges)
0:01:40 INF [boundaries] - Finished creating 0 country polygons
0:01:40 INF [boundaries] - Finished in 0s cpu:0s avg:1.1
0:01:40 INF - Deleting node.db to make room for output file
0:01:40 INF [sort] - 
0:01:40 INF [sort] - Starting...
0:01:40 INF [sort] - Grouped 8 chunks into 1
0:01:42 INF [sort] -  chunks: [   1 /   1 100% ] 283M 
    cpus: 1.4 gc:  7% heap: 1.5G/4.2G direct: 54M postGC: 1.4G
    ->     (0/3) -> worker( -%)
0:01:42 INF [sort] - Finished in 3s cpu:4s avg:1.3
0:01:42 INF [sort] -   worker  1x(71% 2s)
0:01:42 INF - read:1s write:0s sort:1s
0:01:42 INF [archive] - 
0:01:42 INF [archive] - Starting...
0:01:43 DEB [archive:write] - Execute mbtiles: create table metadata (name text, value text);
0:01:43 DEB [archive:write] - Execute mbtiles: create unique index name on metadata (name);
0:01:43 DEB [archive:write] - Execute mbtiles: create table tiles_shallow (
  zoom_level integer,
  tile_column integer,
  tile_row integer,
  tile_data_id integer

  , primary key(zoom_level,tile_column,tile_row)

) without rowid

0:01:43 DEB [archive:write] - Execute mbtiles: create table tiles_data (
  tile_data_id integer primary key,
  tile_data blob
)

0:01:43 DEB [archive:write] - Execute mbtiles: create view tiles AS
select
  tiles_shallow.zoom_level as zoom_level,
  tiles_shallow.tile_column as tile_column,
  tiles_shallow.tile_row as tile_row,
  tiles_data.tile_data as tile_data
from tiles_shallow
join tiles_data on tiles_shallow.tile_data_id = tiles_data.tile_data_id

0:01:43 DEB [archive:write] - Set mbtiles metadata: format=pbf
0:01:43 DEB [archive:write] - Set mbtiles metadata: center=-45.955,32.445,3
0:01:43 DEB [archive:write] - Set mbtiles metadata: bounds=-74.07,21.34,-17.84,43.55
0:01:43 DEB [archive:write] - Set mbtiles metadata: json={"vector_layers":[{"id":"aerodrome_label","fields":{"name_int":"String","iata":"String","ele_ft":"Number","name_de":"String","name":"String","icao":"String","name:en":"String","class":"String","ele":"Number","name_en":"String","name:latin":"String"},"minzoom":10,"maxzoom":14},{"id":"aeroway","fields":{"ref":"String","class":"String"},"minzoom":10,"maxzoom":14},{"id":"boundary","fields":{"disputed":"Number","admin_level":"Number","maritime":"Number"},"minzoom":0,"maxzoom":14},{"id":"building","fields":{"colour":"String","render_height":"Number","render_min_height":"Number","hide_3d":"Boolean"},"minzoom":13,"maxzoom":14},{"id":"housenumber","fields":{"housenumber":"String"},"minzoom":14,"maxzoom":14},{"id":"landcover","fields":{"subclass":"String","class":"String","_numpoints":"Number"},"minzoom":7,"maxzoom":14},{"id":"landuse","fields":{"class":"String"},"minzoom":4,"maxzoom":14},{"id":"mountain_peak","fields":{"name_int":"String","customary_ft":"Number","ele_ft":"Number","name_de":"Str... 2358 more characters
0:01:43 DEB [archive:write] - Set mbtiles metadata: name=OpenMapTiles
0:01:43 DEB [archive:write] - Set mbtiles metadata: description=A tileset showcasing all layers in OpenMapTiles. https://openmaptiles.org
0:01:43 DEB [archive:write] - Set mbtiles metadata: attribution=<a href="https://www.openmaptiles.org/" target="_blank">&copy; OpenMapTiles</a> <a href="https://www.openstreetmap.org/copyright" target="_blank">&copy; OpenStreetMap contributors</a>
0:01:43 DEB [archive:write] - Set mbtiles metadata: version=3.14.0
0:01:43 DEB [archive:write] - Set mbtiles metadata: type=baselayer
0:01:43 DEB [archive:write] - Set mbtiles metadata: minzoom=0
0:01:43 DEB [archive:write] - Set mbtiles metadata: maxzoom=14
0:01:43 DEB [archive:write] - Set mbtiles metadata: compression=gzip
0:01:43 DEB [archive:write] - Set mbtiles metadata: planetiler:version=0.6-SNAPSHOT
0:01:43 DEB [archive:write] - Set mbtiles metadata: planetiler:githash=fbb6ab52b593c75085dd23abd540f5835bf77ebc
0:01:43 DEB [archive:write] - Set mbtiles metadata: planetiler:buildtime=2023-08-23T14:51:25.428Z
0:01:43 DEB [archive:write] - Set mbtiles metadata: planetiler:osm:osmosisreplicationtime=2023-08-22T20:21:53Z
0:01:43 DEB [archive:write] - Set mbtiles metadata: planetiler:osm:osmosisreplicationseq=3799
0:01:43 DEB [archive:write] - Set mbtiles metadata: planetiler:osm:osmosisreplicationurl=http://download.geofabrik.de/north-america/us/rhode-island-updates
0:01:45 INF [archive:write] - Starting z0
0:01:45 INF [archive:write] - Finished z0 in 0s cpu:0s avg:0, now starting z1
0:01:45 INF [archive:write] - Finished z1 in 0s cpu:0s avg:0, now starting z2
0:01:45 INF [archive:write] - Finished z2 in 0s cpu:0s avg:0, now starting z3
0:01:45 INF [archive:write] - Finished z3 in 0s cpu:0s avg:0, now starting z4
0:01:45 INF [archive:write] - Finished z4 in 0s cpu:0s avg:0, now starting z5
0:01:45 INF [archive:write] - Finished z5 in 0s cpu:0s avg:0, now starting z6
0:01:45 INF [archive:write] - Finished z6 in 0s cpu:0s avg:0, now starting z7
0:01:46 INF [archive:write] - Finished z7 in 0.9s cpu:2s avg:2, now starting z8
0:01:49 INF [archive:write] - Finished z8 in 3s cpu:5s avg:1.9, now starting z9
0:01:52 INF [archive:write] - Finished z9 in 3s cpu:6s avg:2, now starting z10
0:01:53 INF [archive] -  features: [ 126k   2%  12k/s ] 283M  tiles: [ 4.7k  477/s ] 1.2M 
    cpus: 2 gc:  2% heap: 2.5G/4.2G direct: 54M postGC: 1.7G
    read( 1%) -> (214/217) -> encode(61% 60%) -> (215/216) -> write( 1%)
    last tile: 10/308/381 (z10 4%) https://www.openstreetmap.org/#map=10/41.77131/-71.71875
0:01:54 INF [archive:write] - Finished z10 in 2s cpu:3s avg:2, now starting z11
0:01:57 INF [archive:write] - Finished z11 in 3s cpu:6s avg:2, now starting z12
0:02:02 INF [archive:write] - Finished z12 in 5s cpu:9s avg:2, now starting z13
0:02:03 INF [archive] -  features: [ 669k  12%  54k/s ] 283M  tiles: [ 288k  28k/s ] 13M  
    cpus: 2 gc:  1% heap: 3.1G/4.2G direct: 54M postGC: 1.8G
    read( 1%) -> (214/217) -> encode(65% 63%) -> (215/216) -> write( 4%)
    last tile: 13/2465/3046 (z13 4%) https://www.openstreetmap.org/#map=13/41.83683/-71.67480
0:02:11 INF [archive:write] - Finished z13 in 10s cpu:19s avg:2, now starting z14
0:02:13 INF [archive] -  features: [ 2.1M  39% 147k/s ] 283M  tiles: [ 1.1M  87k/s ] 44M  
    cpus: 2 gc:  3% heap: 2G/4.2G direct: 54M postGC: 1.8G
    read( 2%) -> (214/217) -> encode(78% 75%) -> (215/216) -> write( 9%)
    last tile: 14/4940/6203 (z14 5%) https://www.openstreetmap.org/#map=14/39.99396/-71.45508
0:02:18 DEB [archive:write] - Shallow tiles written: 4,115,061
0:02:18 DEB [archive:write] - Tile data written: 17,896 (100% omitted)
0:02:18 DEB [archive:write] - Unique tile hashes: 9,017
0:02:18 INF [archive:write] - Finished z14 in 6s cpu:11s avg:1.9
0:02:18 INF [archive] -  features: [ 5.4M 100% 710k/s ] 283M  tiles: [ 4.1M 637k/s ] 109M 
    cpus: 1.8 gc:  3% heap: 2.3G/4.2G direct: 54M postGC: 1.7G
    read( -%) ->   (0/217) -> encode( -%  -%) ->   (0/216) -> write( -%)
    last tile: 14/7380/5985 (z14 100%) https://www.openstreetmap.org/#map=14/43.56447/-17.84180
0:02:18 DEB [archive] - Tile stats:
0:02:18 DEB [archive] - z0 avg:7.9k max:7.9k
0:02:18 DEB [archive] - z1 avg:4k max:4k
0:02:18 DEB [archive] - z2 avg:9.4k max:9.4k
0:02:18 DEB [archive] - z3 avg:4k max:6.4k
0:02:18 DEB [archive] - z4 avg:1.6k max:4.6k
0:02:18 DEB [archive] - z5 avg:1.4k max:7.1k
0:02:18 DEB [archive] - z6 avg:974 max:22k
0:02:18 DEB [archive] - z7 avg:769 max:58k
0:02:18 DEB [archive] - z8 avg:417 max:127k
0:02:18 DEB [archive] - z9 avg:281 max:298k
0:02:18 DEB [archive] - z10 avg:161 max:256k
0:02:18 DEB [archive] - z11 avg:106 max:136k
0:02:18 DEB [archive] - z12 avg:85 max:114k
0:02:18 DEB [archive] - z13 avg:72 max:128k
0:02:18 DEB [archive] - z14 avg:68 max:303k
0:02:18 DEB [archive] - all avg:70 max:303k
0:02:18 DEB [archive] -  # features: 5,437,996
0:02:18 DEB [archive] -     # tiles: 4,115,061
0:02:18 INF [archive] - Finished in 35s cpu:1m8s gc:1s avg:1.9
0:02:18 INF [archive] -   read    1x(2% 0.7s wait:33s)
0:02:18 INF [archive] -   encode  2x(64% 23s wait:1s)
0:02:18 INF [archive] -   write   1x(13% 4s wait:29s)
0:02:18 INF - Finished in 2m18s cpu:4m7s gc:5s avg:1.8
0:02:18 INF - FINISHED!
0:02:18 INF - 
0:02:18 INF - ----------------------------------------
0:02:18 INF - data errors:
0:02:18 INF - 	render_snap_fix_input	16,460
0:02:18 INF - 	osm_boundary_missing_way	63
0:02:18 INF - 	osm_multipolygon_missing_way	57
0:02:18 INF - 	merge_snap_fix_input	14
0:02:18 INF - 	feature_centroid_if_convex_osm_invalid_multipolygon_empty_after_fix	2
0:02:18 INF - 	feature_polygon_osm_invalid_multipolygon_empty_after_fix	2
0:02:18 INF - 	omt_park_area_osm_invalid_multipolygon_empty_after_fix	1
0:02:18 INF - ----------------------------------------
0:02:18 INF - 	overall          2m18s cpu:4m7s gc:5s avg:1.8
0:02:18 INF - 	lake_centerlines 4s cpu:7s avg:1.8
0:02:18 INF - 	  read     1x(44% 2s done:2s)
0:02:18 INF - 	  process  2x(7% 0.3s wait:2s done:2s)
0:02:18 INF - 	  write    1x(0% 0s wait:2s done:2s)
0:02:18 INF - 	water_polygons   31s cpu:56s gc:3s avg:1.8
0:02:18 INF - 	  read     1x(54% 17s wait:4s)
0:02:18 INF - 	  process  2x(31% 10s wait:13s)
0:02:18 INF - 	  write    1x(2% 0.6s wait:30s)
0:02:18 INF - 	natural_earth    19s cpu:21s avg:1.2
0:02:18 INF - 	  read     1x(48% 9s sys:1s done:9s)
0:02:18 INF - 	  process  2x(8% 1s wait:9s done:9s)
0:02:18 INF - 	  write    1x(0% 0s wait:10s done:9s)
0:02:18 INF - 	osm_pass1        4s cpu:7s avg:1.7
0:02:18 INF - 	  read     1x(1% 0s wait:4s)
0:02:18 INF - 	  parse    1x(67% 3s)
0:02:18 INF - 	  process  1x(40% 2s wait:2s)
0:02:18 INF - 	osm_pass2        41s cpu:1m21s avg:2
0:02:18 INF - 	  read     1x(0% 0s wait:22s done:19s)
0:02:18 INF - 	  process  2x(77% 32s)
0:02:18 INF - 	  write    1x(1% 0.5s wait:41s)
0:02:18 INF - 	boundaries       0s cpu:0s avg:1.1
0:02:18 INF - 	sort             3s cpu:4s avg:1.3
0:02:18 INF - 	  worker  1x(71% 2s)
0:02:18 INF - 	archive          35s cpu:1m8s gc:1s avg:1.9
0:02:18 INF - 	  read    1x(2% 0.7s wait:33s)
0:02:18 INF - 	  encode  2x(64% 23s wait:1s)
0:02:18 INF - 	  write   1x(13% 4s wait:29s)
0:02:18 INF - ----------------------------------------
0:02:18 INF - 	archive	109MB
0:02:18 INF - 	features	283MB
-rw-r--r-- 1 runner docker 66M Aug 23 14:53 run.jar

@bbilger bbilger force-pushed the add-file-stream-outputs branch 3 times, most recently from 4a763e0 to c221551 Compare July 30, 2023 23:03
@bbilger
add support for simple file output streams: CSV, JSON, protobuf
5d7bebe 
The primary use case for writing tile data to files using simple formats,
is to provide a simple plugin system. By using named pipes data can be
transformed and processed directly - e.g. to push to s3 or to store in a DB,
without any intermediate file.

This - depending on the use-case - also allows concurrent writing to multiple files.

note: JSON entries are separated by newline, and protobuf messages are length-delimited.
@bbilger bbilger force-pushed the add-file-stream-outputs branch from c221551 to 5d7bebe Compare July 31, 2023 22:25
bbilger added 2 commits August 1, 2023 22:55
@bbilger
add CLI arg tile-compression [gzip, none]
c830714 
to allow to disable tile compression if some usecase doesn't requrie it
@bbilger
implement a ReadableTileArchive for the new formats for tests
e8e08c2 
and run "standard" planetiler test for the new formats
@bbilger bbilger force-pushed the add-file-stream-outputs branch from 4e519dc to e8e08c2 Compare August 1, 2023 22:35
bbilger added 3 commits August 2, 2023 21:46
@bbilger
expose all metadata in json format
c6f6908
@bbilger
improve handling of CLI args for new formats
d49102c 
- generate the list of formats for which append/tile_write_threads
  is allowed, dynamically
- centralize parsing & description for "escaped strings"
  ('\n', ' ', ...) and explain how to pass as query param
- make CSV line separator/ending configurable
@bbilger
add CLI option (csv_)binary_encoding
7d8cdf4 
base64, hex (0001), hex_prefix_start (\x0001), hex_prefix_each (\x00\x01)

hex(_prefix_start) allows a simple import into PostgreSQL, and should be
useful for other scenarios as well.
msbarry
msbarry reviewed Aug 4, 2023
View reviewed changes
Copy link
Contributor

@msbarry msbarry left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very cool! I think that we'd still eventually want to add native support for more popular output formats (s3/file/postgres) as that will most likely end up having the best performance (and support reads/copying between formats), but I do like having the ability to support any kind of output format using this technique!

msbarry
msbarry reviewed Aug 15, 2023
View reviewed changes
Copy link
Contributor

@msbarry msbarry left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking good! A few mostly minor comments.

msbarry
msbarry reviewed Aug 16, 2023
View reviewed changes
Copy link
Contributor

@msbarry msbarry left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! I ran this locally and noticed 2 minor things, after that looks good to merge.

@bbilger bbilger force-pushed the add-file-stream-outputs branch from 091cfcf to da9b304 Compare August 23, 2023 14:50
@bbilger
Copy link
Contributor Author

bbilger commented Aug 23, 2023

I think I addressed everything now
Let me know if I missed something or you want me to change/adjust anything else.

@sonarqubecloud
Copy link

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

75.7% 75.7% Coverage
0.0% 0.0% Duplication

@msbarry
Copy link
Contributor

msbarry commented Aug 24, 2023

Looks great! Thanks for adding this.

@msbarry msbarry merged commit db796e1 into onthegomap:main Aug 24, 2023
@cmahnke cmahnke mentioned this pull request Sep 21, 2023
Open
# for free to join this conversation on GitHub. Already have an account? # to comment
Reviewers

@msbarry msbarry msbarry left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@bbilger @msbarry