osm2pgsql-replication.py updating an extracted (by polygon clip) dataset?

[bjornharrtell]

I used this process to load and prepare my database for updates:

curl -s https://download.geofabrik.de/europe-220116.osm.pbf | \
  osmium extract -p denmark_ext.geojson -F pbf -s simple -f pbf - | \
    osm2pgsql --slim --prefix=denmark_ext -r pbf -
python3 osm2pgsql-replication.py init --prefix denmark_ext --server https://download.geofabrik.de/europe-updates

And then update with:

python3 osm2pgsql-replication.py update --prefix denmark_ext

That will update with all changes in europe, which I now realize is what I have asked it for..

But, I would like to update only within the same polygon (defined by denmark_ext.geojson). Is that possible or would that need new feature development for the osm2pgsql-replication.py helper script?

[lonvia]

osm2pgsql-replication.py can only be used with a predefined replication source.

Updating a custom extract is a lot more involved. It roughly works along those lines:

A. Initial import:
1. Download europe
2. Create extract with osmium extract
3. Import extract with osm2pgsql --slim
4. Keep original europe and extract around.

B. Update:
1. Update europe file using pyosium-up-to-date.
2. Create a new extract with osmium extract
3. Create a diff file between old extract and new extract using osmium derive-changes
4. Apply diff with osm2pgsql -a --slim
5. Keep updated europe file and new extract around.

Repeat B for each update cycle.

[bjornharrtell]

I see, thanks.

I determined that a bbox filter will be enough so I adjusted my method to use bbox filtering on both levels like this:

curl -s https://download.geofabrik.de/europe-220116.osm.pbf | \
  osmium extract -b 6.701660,53.278353,16.018066,58.745407 -F pbf -s simple -f pbf - | \
    osm2pgsql --slim --prefix=denmark_ext -r pbf -
python3 osm2pgsql-replication.py init --prefix denmark_ext --server https://download.geofabrik.de/europe-updates
python3 osm2pgsql-replication.py update --prefix denmark_ext -- -b 6.701660,53.278353,16.018066,58.745407

It does seem to work and avoids the large work involved with a fully custom extract. Does it seem reasonable?

[joto]

The manual has this to say on the subject of the -b option:

Use of the -b, --bbox option is not recommended; it is a crude way of limiting the amount of data loaded into the database and it will not always do what you expect it to do, especially at the boundaries of the bounding box. If you use it, choose a bounding box that is larger than your actual area of interest.

[bjornharrtell]

I'm guessing it will correspond roughly to the simple strategy in osmium, which might work well enough for my purpose.

[stephankn]

You need to extract changes affecting your data from the diff. trim_osc.py can do this.
Below is a section from my update script which keeps an extract updated.
Use a recent version, as there was a nasty bug included.
https://github.com/Zverik/regional/blob/master/trim_osc.py

# start replication and db import
/usr/lib/python3-pyosmium/pyosmium-get-changes -vv --size ${UPDATE_SIZE} -f ${STATE_DIR}/sequence.state -o ${TEMP_DIR}/change.osc.gz

export PGPASSWORD=xxx
./trim_osc.py -d gis --host postgis --use osm -p osm-tools-sea.poly -v -z --password ${TEMP_DIR}/change.osc.gz ${TEMP_DIR}/change-trim.osc.gz
osm2pgsql --multi-geometry --slim --append --hstore --username osm --host postgis --database gis --expire-tiles 3-17 --expire-output ${TEMP_DIR}/expire.list --style openstreetmap-carto/openstreetmap-carto.style --tag-transform-script openstreetmap-carto/openstreetmap-carto.lua ${TEMP_DIR}/change-trim.osc.gz
./zxy2metatile <${TEMP_DIR}//expire.list | sort -u | ./osm-expconv osm 20 6 | sort -u | gzip >"${STATE_DIR}/expire.list.$(date +%s).gz"

[bjornharrtell]

Thanks, that makes sense. :)