Initial commit

Author: Ordoviz

.gitignore

@@ -0,0 +1,6 @@
+commonscats-in-commons.txt
+commonscats-in-osm.xml
+*.gz
+*.geojson
+*.tsv
+*.pbf

LICENSE

@@ -0,0 +1,19 @@
+Copyright (c) 2023 Lennard Hofmann
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

Makefile

@@ -0,0 +1,34 @@
+.PHONY: geojson
+geojson: out.geojson out-maproulette.geojson
+
+.PHONY: tsv
+tsv: out.tsv
+
+planet.pbf:
+	@echo "Missing planet.pbf: See https://wiki.openstreetmap.org/wiki/Planet.osm#Downloading for how to download an Osmium-compatible planet file." >&2; exit 1
+	# You could add a command here to download planet.pbf with a BitTorrent client
+
+commonswiki-all-titles.gz: FORCE
+	curl -z commonswiki-all-titles.gz https://dumps.wikimedia.org/commonswiki/latest/commonswiki-latest-all-titles.gz -o commonswiki-all-titles.gz
+
+commonscats-in-commons.txt: commonswiki-all-titles.gz
+	# namespace 14 is "Category:"
+	zcat commonswiki-all-titles.gz | grep ^14 | cut -f2  > commonscats-in-commons.txt
+
+commonscats-in-osm.xml: planet.pbf
+	osmium tags-filter -R planet.pbf 'nwr/wikimedia_commons=Category:*' -o commonscats-in-osm.xml --overwrite
+
+planet-filtered.geojson: planet.pbf
+	osmium tags-filter -t planet.pbf 'nwr/wikimedia_commons=Category:*' -o planet-filtered.pbf --overwrite
+	osmium export planet-filtered.pbf -c config.json -o planet-filtered.geojson -f jsonseq --overwrite
+
+out.tsv: commonscats-in-osm.xml commonscats-in-commons.txt
+	./main.py out.tsv
+
+out.geojson: planet-filtered.geojson commonscats-in-commons.txt
+	./main.py out.geojson
+
+out-maproulette.geojson: out.geojson
+	./to_maproulette.py out.geojson > out-maproulette.geojson
+
+FORCE: ;

README.md

@@ -0,0 +1,53 @@
+# CommonsCatOSM
+
+Finds OpenStreetMap elements tagged with `wikimedia_commons=Category:*` where the category does not exist on Wikimedia Commons.
+
+## How to use
+
+To minimize false positives, wait until Wikimedia releases a new data dump (usually at the third day of every month). You can get notified by subscribing to [the RSS feed](https://dumps.wikimedia.org/commonswiki/latest/commonswiki-latest-all-titles.gz-rss.xml).
+
+1. Install [Osmium Tool](https://osmcode.org/osmium-tool/)
+2. Clone this repository
+3. Download required files into the same directory:
+   * `planet.pbf`: an Osmium-compatible [OpenStreetMap planet](https://wiki.openstreetmap.org/wiki/Planet.osm)
+   * [commonswiki-latest-all-titles.gz](https://dumps.wikimedia.org/commonswiki/latest/commonswiki-latest-all-titles.gz) (1.1 GB as of 2023)
+4. Choose an output format (see below). If you use a Unix-like operating system, you can run `make tsv` or `make geojson`. Otherwise, run the commands given below.
+
+## Output formats
+
+### Tab-separated values (TSV)
+
+Output contains invalid category names and the OpenStreetMap ID of the node, way, or relation with that category. This is the fastest format to produce. 
+
+```console
+# Make sure you have 10min to spare
+$ osmium tags-filter -R planet.pbf 'nwr/wikimedia_commons=Category:*' -o commonscats-in-osm.xml
+$ python main.py out.tsv
+$ cat out.tsv
+n/1573735855	Conservatoire_National_de_V%C3%A9hicules_Historiques
+n/286133524	ref:sprockhoff No. 465
+n/3022117073	Wildwiesenwarte;Category:Views from the Wildwiesenwarte
+n/306593910	Prince George pub, Brighton Good pictures  Advanced...
+n/6426478285	Dorfkirche_Mechow_(Kyritz)?uselang=de
+w/297069904	https://commons.wikimedia.org/wiki/Category:Gr%C3%BCner_Graben_14_(G%C3%B6rlitz)
+w/320276921	 Ballyellen Upper Lock
+w/474166824	Nages-et-Solorgues#/media/File:Fontaine_Ranquet.jpg
+r/12931220	Brandenburger Straße 36;Riedelsberger Weg 2 (Bayreuth)
+```
+
+### Line-by-Line GeoJSON
+
+[This format](https://learn.maproulette.org/documentation/line-by-line-geojson/) can be used to create a challenge on maproulette.org. It might lack a few categories that are present in the TSV format.
+
+```console
+# Make sure you have 3GB RAM and 25min to spare
+$ osmium tags-filter -t planet.pbf 'nwr/wikimedia_commons=Category:*' -o planet-filtered.pbf
+$ osmium export planet-filtered.pbf -c config.json -o planet-filtered.geojson -f jsonseq
+$ python main.py out.geojson
+$ cat out.geojson
+{"type":"Feature","geometry":{"type":"Point","coordinates":[-2.0835284,53.3600557]},"properties":{"@type":"node","@id":29947059,"wikimedia_commons":"Category:Help Category:Middlewood railway station"}}
+{"type":"Feature","geometry":{"type":"LineString","coordinates":[[10.1212064,54.3247979],[10.120334100000001,54.3242942],[10.1192733,54.3236981],[10.1199922,54.3233703],[10.1204628,54.3231298],[10.1209243,54.3228965],[10.1222211,54.3236181],[10.1212064,54.3247979]]},"properties":{"@type":"way","@id":9408975,"wikimedia_commons":"Category:Wilhelmplatz (Kiel)"}}
+$ python to_maproulette.py out.geojson
+{"type":"FeatureCollection","features":[{"type":"Feature","geometry":{"type":"Point","coordinates":[-2.0835284,53.3600557]},"properties":{"@type":"node","@id":29947059,"wikimedia_commons":"Category:Help Category:Middlewood railway station"}}]}
+{"type":"FeatureCollection","features":[{"type":"Feature","geometry":{"type":"LineString","coordinates":[[10.1212064,54.3247979],[10.120334100000001,54.3242942],[10.1192733,54.3236981],[10.1199922,54.3233703],[10.1204628,54.3231298],[10.1209243,54.3228965],[10.1222211,54.3236181],[10.1212064,54.3247979]]},"properties":{"@type":"way","@id":9408975,"wikimedia_commons":"Category:Wilhelmplatz (Kiel)"}}]}
+```

config.json

@@ -0,0 +1,18 @@
+{
+    "attributes": {
+        "type":      true,
+        "id":        true,
+        "version":   false,
+        "changeset": false,
+        "timestamp": false,
+        "uid":       false,
+        "user":      false,
+        "way_nodes": false
+    },
+    "format_options": {
+    },
+    "linear_tags":  true,
+    "area_tags":    true,
+    "exclude_tags": [],
+    "include_tags": ["wikimedia_commons=Category:*"]
+}

main.py

@@ -0,0 +1,63 @@
+#!/usr/bin/env python3
+import xml.etree.ElementTree as ET
+import gzip
+import sys
+import json
+
+VALID_CATS = "commonscats-in-commons.txt"
+
+def load_valid_categories():
+    cats = set()
+    try:
+        with open(VALID_CATS) as f:
+            for line in f:
+                cats.add(line.rstrip('\n'))
+    except FileNotFoundError:
+        with gzip.open("commonswiki-latest-all-titles.gz", "rt") as f_in, open(VALID_CATS, "w") as f_out:
+            for line in f_in:
+                ns, category_name = line.split('\t', 1)
+                if ns == '14': # "Category:" namespace
+                    cats.add(category_name.rstrip('\n'))
+                    f_out.write(category_name)
+    return cats
+
+
+def tsv(cats, outfile):
+    with open(outfile, "w") as f:
+        tree = ET.parse("commonscats-in-osm.xml")
+        for elem in tree.findall(".//*tag[@k='wikimedia_commons']/.."):
+            wikimedia_commons = elem.find("tag[@k='wikimedia_commons']").get('v').removeprefix("Category:")
+            if wikimedia_commons.replace(' ', '_') not in cats:
+                f.write(f"{elem.tag[:1]}/{elem.get('id')}\t{wikimedia_commons}\n")
+
+
+def geojson(cats, outfile):
+    with open(outfile, "wb") as f:
+        for line in open("planet-filtered.geojson", "rb"):
+            wikimedia_commons = json.loads(line[1:])["properties"]["wikimedia_commons"]
+            if wikimedia_commons.removeprefix("Category:").replace(' ', '_') not in cats:
+                f.write(line)
+
+
+def usage():
+    print("""Usage: ./main.py outfile.geojson
+       ./main.py outfile.tsv""",
+        file=sys.stderr)
+    exit(1)
+
+
+def main():
+    if len(sys.argv) != 2:
+        usage()
+
+    outfile = sys.argv[1]
+    if outfile.endswith(".tsv"):
+        tsv(load_valid_categories(), outfile)
+    elif outfile.endswith(".geojson"):
+        geojson(load_valid_categories(), outfile)
+    else:
+        usage()
+
+
+if __name__ == "__main__":
+    main()

to_maproulette.py

@@ -0,0 +1,8 @@
+#!/usr/bin/env python3
+import sys
+
+infile = sys.argv[1]
+fmtstr = b'\x1e{"type":"FeatureCollection","features":[%s]}\n'
+
+for line in open(infile, "rb"):
+    sys.stdout.buffer.write(fmtstr % line[1:].rstrip(b'\n'))