Allow easier modification of the OpenStreetMap-carto style #243

galewis2 · 2022-01-25T00:37:16Z

Now, the style isn't built into the image, but is now either hosted in a volume/local directory or worse case, fetched at runtime.

~~It uses a wildcard to find critical style elements (like index.sql, lua file, etc) so there's some uncontrolled flexibility in different file names.~~

Would replace #42 and #97

~~Still need to add some documentation, but here's the existing work.~~

…ure update process is setup correctly

…onal

Better updates

galewis2 · 2022-01-25T19:58:32Z

This is based on the other PR I have, which improves the updating experience.

dbahrdt · 2022-02-10T23:29:53Z

run.sh

@@ -75,13 +91,14 @@ if [ "$1" = "import" ]; then
    fi

    # Import data
-    sudo -u renderer osm2pgsql -d gis --create --slim -G --hstore --tag-transform-script /home/renderer/src/openstreetmap-carto/openstreetmap-carto.lua --number-processes ${THREADS:-4} -S /home/renderer/src/openstreetmap-carto/openstreetmap-carto.style /data.osm.pbf ${OSM2PGSQL_EXTRA_ARGS:-}
+    sudo -u renderer osm2pgsql -d gis --create --slim -G --hstore --tag-transform-script /home/renderer/src/openstreetmap-carto/*.lua --number-processes ${THREADS:-4} -S /home/renderer/src/openstreetmap-carto/*.style /data.osm.pbf ${OSM2PGSQL_EXTRA_ARGS:-}


What happens if I have multiple styles/transform scripts?

I designed this to assume there's only one style/transform script, or just take the first one.

In hindsight, I will make a variable but give it defaults.

Made improvements, now you can specify the style element names as necessary. See the readme

dbahrdt · 2022-02-10T23:31:36Z

As general question:
How do we want to merge request? Do we squash the changes before merging? If not then I would strongly recommend to have useful commits. E.g. no "WIP" commits and the like.

Overv · 2022-02-11T20:56:11Z

@dbahrdt Yeah, I think all PRs should use squash merges as a general rule.

galewis2 · 2022-02-14T13:44:34Z

@dbahrdt Yea, I always do squash commits to keep main tidy. I expect the dev branches to be polluted with non-useful commits as it's not really helpful or necessary to have useful commits on the dev branches.

tommasodelorenzo · 2022-02-21T10:11:53Z

@galewis2 . Ok thanks.
I would bother you with an additional quick question: if I make a small change to my style, is there a simple way to apply those changes without re-running the import?
The fact is that when running the import I get errors such as

+ sudo -u postgres createuser renderer
createuser: error: creation of new role failed: ERROR:  role "renderer" already exists

So I have to remove all data and re-run the import. This takes a lot of time.

nielsole · 2022-02-25T20:11:20Z

run.sh

    exit 1
 fi

 set -x

+if [ ! "$(ls -A /home/renderer/src/openstreetmap-carto)" ]; then
+
+    git clone --single-branch --branch v5.3.1 https://github.com/gravitystorm/openstreetmap-carto.git --depth 1 /home/renderer/src/openstreetmap-carto


I wonder if we should include a copy of these files instead of downloading them.

Especially as we are cloning this repo both on import and on run

run.sh

nielsole · 2022-02-25T22:21:05Z

So I have to remove all data and re-run the import. This takes a lot of time.

The lua script transforms the data during import. The indexes/views might be different which could lead to performance problems if the script is not rerun.
I guess if you just changed the mml you should be able to omit the import step. Either way: best to test your styles on a small dataset :)
EDIT: Ah, this question was already answered in a separate issue: #249 (comment)

Istador · 2022-02-25T22:30:48Z

run.sh

    exit 1
 fi

 set -x

+if [ ! "$(ls -A /home/renderer/src/openstreetmap-carto)" ]; then


if [ ! -d /home/renderer/src/openstreetmap-carto ]; then

Suggested change

if [ ! "$(ls -A /home/renderer/src/openstreetmap-carto)" ]; then

if [ ! -d /home/renderer/src/openstreetmap-carto ]; then

I reverted this back as I need to confirm that openstreetmap-carto is empty, not just the directory exists (like you specify a bad path or something)

nielsole

Great PR!
I haven't setup custom styling myself yet, so my assessments may be off.
I think this could be further improved:

README: Where can I find other styles? (Just one other example would already be great)
better support lack of lua script: e.g. https://github.com/cyclosm/cyclosm-cartocss-style doesn't have a lua script and --tag-transform-script is optional
Better support for multiple SQL scripts e.g.: https://github.com/OpenRailwayMap/OpenRailwayMap-CartoCSS/blob/master/SETUP.md has three SQL scripts that need to be executed in order

I am a big fan of incremental improvement, so maybe we can mark this as "beta"/experimental or something in the README and include 1-2 styles that are known to work and then merge it?

Co-authored-by: Robin C. Ladiges <Istador@users.noreply.github.com>

Dockerfile

galewis2 · 2022-03-15T16:55:38Z

Great PR! I haven't setup custom styling myself yet, so my assessments may be off. I think this could be further improved:
* README: Where can I find other styles? (Just one other example would already be great)

* better support lack of lua script: e.g. https://github.com/cyclosm/cyclosm-cartocss-style doesn't have a lua script and `--tag-transform-script` is optional

* Better support for multiple SQL scripts e.g.: https://github.com/OpenRailwayMap/OpenRailwayMap-CartoCSS/blob/master/SETUP.md has three SQL scripts that need to be executed in order
I am a big fan of incremental improvement, so maybe we can mark this as "beta"/experimental or something in the README and include 1-2 styles that are known to work and then merge it?

I specifically for now targeted this PR to make it easier to modify the existing OpenStreetMap Carto style with small changes, with small in place changes like labelling, colors, etc.

I did add a comment in the readme that openstreetmap-carto and styles like that are supported.

galewis2 · 2022-03-28T13:31:02Z

@nielsole Could I get another approval, just had to do a conflict resolve after #248 , thx!

nielsole

sorry for taking another 3 days for reapproval

galewis2 and others added 12 commits December 14, 2021 08:57

Get Osmium

8750c00

Update run.sh

1421c96

Update openstreetmap-tiles-update-expire

33ea447

You need the openstreetmap-rendered-tiles volume during import to ens…

630e37b

…ure update process is setup correctly

switch to using replication_url

89d238b

update

6d2eaa2

verbose

3deaeda

fix spelling

0ffff3d

provide defaults

9936271

Make it easier to change style

9df02dc

Pull the style out of the build. And, use the latest from zverik/regi…

d2f592e

…onal

use local style and dont build into image

25cb19e

galewis2 marked this pull request as draft January 25, 2022 00:37

galewis2 and others added 4 commits January 24, 2022 19:41

Merge pull request #1 from galewis2/patch-1

eec8034

Better updates

resolve conflict

f1ec6a4

silly slash

8ed995e

Update README.md

f4b6eb0

galewis2 marked this pull request as ready for review January 25, 2022 19:57

galewis2 requested review from Overv, dbahrdt and nielsole February 10, 2022 14:16

dbahrdt reviewed Feb 10, 2022

View reviewed changes

galewis2 mentioned this pull request Feb 14, 2022

Documentation improvement to help users create their own map styles #97

Closed

switch back to defined filenames for style

07d38f6

galewis2 mentioned this pull request Feb 17, 2022

get-external-data.py errors #244

Open

galewis2 marked this pull request as draft February 17, 2022 20:00

galewis2 requested a review from dbahrdt February 18, 2022 14:59

galewis2 marked this pull request as ready for review February 18, 2022 15:48

galewis2 mentioned this pull request Feb 21, 2022

Handle Style Changes #249

Closed

galewis2 closed this Feb 24, 2022

galewis2 deleted the patch-2 branch February 24, 2022 14:12

galewis2 restored the patch-2 branch February 24, 2022 14:12

galewis2 reopened this Feb 24, 2022

nielsole reviewed Feb 25, 2022

View reviewed changes

run.sh Show resolved Hide resolved

Istador reviewed Feb 25, 2022

View reviewed changes

nielsole reviewed Feb 25, 2022

View reviewed changes

Update run.sh

a544a7f

Co-authored-by: Robin C. Ladiges <Istador@users.noreply.github.com>

galewis2 changed the title ~~Allow flexibility in style~~ Allow easier modification of the OpenStreetMap-carto style Feb 28, 2022

dont git clone during run and import

9f09775

Istador reviewed Mar 1, 2022

View reviewed changes

Dockerfile Outdated Show resolved Hide resolved

galewis2 added 2 commits March 15, 2022 10:06

separate the stylesheet fetch back into its own layer

7c325d7

fix path to openstreetmap-carto

b9846e1

revert check for openstreetmap-carto, readme update

43552df

galewis2 requested a review from nielsole March 15, 2022 16:59

nielsole previously approved these changes Mar 27, 2022

View reviewed changes

Merge branch 'master' into patch-2

cc2695f

galewis2 dismissed nielsole’s stale review via cc2695f March 28, 2022 13:29

galewis2 requested a review from nielsole March 28, 2022 13:56

nielsole approved these changes Mar 31, 2022

View reviewed changes

galewis2 merged commit 0f229a0 into Overv:master Mar 31, 2022

galewis2 deleted the patch-2 branch March 31, 2022 11:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow easier modification of the OpenStreetMap-carto style #243

Allow easier modification of the OpenStreetMap-carto style #243

galewis2 commented Jan 25, 2022 •

edited

Loading

galewis2 commented Jan 25, 2022

dbahrdt Feb 10, 2022

galewis2 Feb 16, 2022

galewis2 Feb 18, 2022

dbahrdt commented Feb 10, 2022

Overv commented Feb 11, 2022

galewis2 commented Feb 14, 2022 •

edited

Loading

tommasodelorenzo commented Feb 21, 2022

nielsole Feb 25, 2022

nielsole Feb 25, 2022

galewis2 Mar 15, 2022

nielsole commented Feb 25, 2022 •

edited

Loading

Istador Feb 25, 2022

nielsole Feb 25, 2022

galewis2 Mar 15, 2022

nielsole left a comment •

edited

Loading

galewis2 commented Mar 15, 2022 •

edited

Loading

galewis2 commented Mar 28, 2022

nielsole left a comment

	if [ ! "$(ls -A /home/renderer/src/openstreetmap-carto)" ]; then
	if [ ! -d /home/renderer/src/openstreetmap-carto ]; then

Allow easier modification of the OpenStreetMap-carto style #243

Allow easier modification of the OpenStreetMap-carto style #243

Conversation

galewis2 commented Jan 25, 2022 • edited Loading

galewis2 commented Jan 25, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dbahrdt commented Feb 10, 2022

Overv commented Feb 11, 2022

galewis2 commented Feb 14, 2022 • edited Loading

tommasodelorenzo commented Feb 21, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nielsole commented Feb 25, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nielsole left a comment • edited Loading

Choose a reason for hiding this comment

galewis2 commented Mar 15, 2022 • edited Loading

galewis2 commented Mar 28, 2022

nielsole left a comment

Choose a reason for hiding this comment

galewis2 commented Jan 25, 2022 •

edited

Loading

galewis2 commented Feb 14, 2022 •

edited

Loading

nielsole commented Feb 25, 2022 •

edited

Loading

nielsole left a comment •

edited

Loading

galewis2 commented Mar 15, 2022 •

edited

Loading