RFE: ability to output various formats, for use in other software (e.g. apps for surveying)

[Lee-Carre]

Scenario

I use fvasco/pinpoi for various datasets.

Besides PinPoi being an app intended for such purposes, while I'm out surveying I have enough apps running (I collect more than just map data) that I can't also run a Web browser without hitting memory-exhaustion (causing other apps to malfunction).
Thus, using PinPoi means that I can query various (sometimes large) datasets (of features to add / resurvey), and navigate to their PoIs (e.g. a geo URI intent into a map & routing app), with ease. All while using little enough memory that I don't hit resource constraints.

Aside from avoiding technical problems, PinPoi can output a set of matches integrated from multiple different sources; either as a proximity-ordered list, or on a background map (OSM, of course 🙂). Among other features.

So, even if I could run a browser, PinPoi has significant advantages.

Specific (imagined) workflow example

When passing by a notable feature, particularly difficult-to-discover ones (think small & obscure), if I don't have time to stop and map it properly, I'll drop a note (as a reminder, and to easily re-find the feature when I do have time to give it the proper mapping treatment).
Else, when going through descriptive human-readable information (e.g. news announcements of changes to the road network) I'll add notes at those locations as a reference for future attention.

NotesReview is excellent for finding & filtering such specific (topical) notes. Other tools simply show (on a map) the notes within a given bounding box, which must then each be inspected individually (the focus is on location / proximity, not topic).

I would be able to use the output of NotesReview in software of my choosing, to easily identify & navigate to the locations during a survey. And/or during planning of a survey expedition.

For example, all (nearby) notes which request (by mentioning the word) a survey. Else, whatever (topic / keyword) is of interest, as I see fit.

Especially when I'm closing notes once each is solved by a changeset, and querying NotesReview with status=open, then the returned set is (automagically) a self-updating to-do / task list, until the count of matches reaches zero.
Compare using Overpass Turbo with a (wizard-generated) query which includes […] & (check_date!~/^202[2-9]-/ or survey:date!~/^202[2-9]-/) […]; thus as each element is surveyed, it's excluded from the set returned when the query is (next) executed. Thus, only outstanding targets for surveying are shown, rather than every feature of the same type.
This greatly reduces cognitive overhead, when attention should primarily be on mapping what's observed by survey (rather than dataset management; remembering which elements to ignore and mentally filtering them out).

This would all make surveying much more efficient & productive.

Use-cases

Beyond my own specific use-case, I imagine that this feature would be useful to others, in multiple ways

• some editors can show overlays of PoIs from external datasets
• an API for websites
• being part of a processing pipeline of QA tools

User-visible changes

I imagine that adding some extra elements to the share URL UI widget would be a sensible place.

Folks are gonna want a URL to give to whatever other software they're using. The sharing feature is the obvious place to go to get a properly-formatted perma-URL.

In which case, it may be as simple as including a toggle to add

[…&]format=[…]

to the URL string, specifying the desired format as selected by a drop-down list in the same UI (which appears after enabling the non-HTML output option?).

Else, I suppose you could add values to the view parameter, but given its name the semantics don't match. Unless view=(format|export) is the trigger for non-HTML output (instead of the mere presence of format), and the value of format specifies the preferred syntax (else a default is used).

Better might be a mode parameter. Default of mode=view (for current HTML output), but another value (export, output, format, …?) to trigger non-HTML.
Keeps the semantics cleaner (for future feature additions), and gives clues as to other parameters needed (like with OSM tagging: key=category implies that category=type is a sub-tag). It also serves as a

• safety switch; to avoid unintended output type by requiring multiple parameter changes
• master switch; to be able to leave format in the URL and easily switch between HTML and non-HTML, else leaving format blank / unspecified to use the current default format

Regardless of generating mechanism, an example of the resultant URL would be something like

//ent8r.github.io/NotesReview/?[mode=(export|format)&]map=12%2F49.2%2F-2.1&apply-bbox=true&status=open&query=survey&format=gpx

Formats

The common, well-known ones would seem to suffice:

• GPX
• CSV
• GeoJSON
• KML / KMZ
• GeoRSS
• any others which ENT8R or other users think sensible (I'll update this list based on comments)

Other implementations, for inspiration

• Rob Whittaker's #SurveyMe tool can output various combinations of target-PoIs (one of which is OSM Notes that include the string #SurveyMe) in several formats — example: osm.mathmos.net/survey/data.cgi?lat=49.2&lon=-2.1&fmt=gpx (though, N.B., his fmt=kml includes more descriptive info of each target)
• Overpass can (according to the documentation) be queried such that it outputs suitable formats — no example, yet, since I'm still tinkering with this, myself 😋
• Searx (& possibly SearxNG) (a “hackable metasearch engine”; public instances) implements the possibility to add format= to the URL query-string, to output in several non-HTML formats (CSV, JSON, RSS) — example: [searx instance]/search?q=OSM&format=json

Caveats & Considerations

I'm assuming that NotesReview isn't entirely dependent on client-side scripting for even the basics (querying the OSM Notes API & returning a list), and that client-side scripting is only for UI / rendering, processing thumbnails, and other nice-to-have features.
Software for fetching datasets is unlikely to have an HTTP client which executes scripting; it'll simply expect a pre-processed response (in one of a few particular formats).

If this isn't true, then could an exception be made, server-side, for when format is specified in the URL?

PinPoi stores the received set locally (I presume as a cache for performance, and for offline usage), and (at the moment) only issues another HTTP request when manually instructed to. So, the load on the server (and OSM API) should be low. Arguably lower (due to caching) than when using the HTML UI.

I haven't throughly read the OSM Notes API documentation, but I gather that the output is XML-only. However, I could be wrong, and it would be better to query OSM directly rather than through NotesReview. However, NotesReview is a much nicer & easier interface (especially for the less tech-savvy, who would be daunted by querying an API directly).

[ENT8R]

(I moved this issue to this repo, because this will be implemented in the backend not the frontend)

Your use case and the detailed described workflow sound reasonable. Let me just add a few things to your request:

My general idea of the implementation looks like that: the backend currently returns a JSON array (not GeoJSON) containing all notes that match the query of the user. This will stay the same by default, but the backend will also be able to generate other data formats (probably based on the file ending) and the user on the frontend can then download the data as a file using a new dialog. Advanced users could simply copy the URL used to generate this file, so it can be used in apps like PinPoi leading to the workflow you described above.

I'm assuming that NotesReview isn't entirely dependent on client-side scripting for even the basics (querying the OSM Notes API & returning a list), and that client-side scripting is only for UI / rendering, processing thumbnails, and other nice-to-have features.

NotesReview basically consists of a backend with a database containing all OpenStreetMap notes updated every 5 minutes, it acts as an intermediary allowing far more complex queries than it would be possible by using the OSM API alone. The frontend can be seen as a user interface for interacting with the backend to retrieve the correct notes as well as the OSM API to comment on notes. The frontend is a static HTML page written in JavaScript that retrieves all dynamic data from the backend, so there is no other way than implementing this directly in the backend.

I haven't throughly read the OSM Notes API documentation, but I gather that the output is XML-only

The main OSM API does actually support multiple formats for notes (see https://wiki.openstreetmap.org/wiki/API_v0.6#Map_Notes_API). Besides XML its possible to get the same data in JSON, RSS and GPX by appending the correct file ending to the request.

[Lee-Carre]

Now that I have time to respond properly (in no particular order 🤷‍♂️) …

Let me just add a few things to your request:

That all sounds most promising 😀.

generate other data formats (probably based on the file ending)

Probably best for a few reasons

• practicalities: downloader-type UAs seldom enable changing the Accept header; the only variable the user can reliably change is the URL itself
• usability: ‘what's an Accept header? How do I set it? Why do I need to?’ — more users will be familiar with filename extensions, and what they mean, and what they should be set to (as opposed to MIME types)
• users sharing configuration / data sources; when everything is encoded in the URL, that's more reliable than needing the location pointer plus a description of other variables to change
• in cases of actually downloading data (to a file), a filename extension makes things easier for the downloading UA to produce a file which works (e.g. is correctly recognised by type) as expected

Though, (especially) when the URL omits any filename extension, it would seem wise to have the backend respond to the Accept header, too

• sophisticated users implementing scripts which allow setting HTTP headers; maybe the desired format is conditional, and it would help to keep the URL string static
• the Accept header can be an ordered list of preferences (including newer, not-yet-implemented) formats, which is more granular than a singular filename extension

OSM API

I've read (in relevant GH issues)

• the OSM tech-overlords are wanting to deprecate the search endpoint, with a recommendation to use other tools (a subtle reference to NotesReview?); the idea being that if you want to search by keywords, then you should download all notes in a bounding box, and do pattern-matching locally (or use a tool which does this for you)
• there are no plans to fix string matching (e.g. for the ? character) in the API (apparently, it would require reindexing the DB, which they're not keen on)

Though, duly noted about non-XML formats.

the backend currently returns a JSON array

So, those who want to do something complicated could just process that JSON, themselves, as they wish. Interesting.

the user on the frontend can then download the data as a file using a new dialog. Advanced users could simply copy the URL used to generate this file, so it can be used in apps like PinPoi

Exactly, that would do just fine. That's basically what I've done with other sources; it's not like the HTTPd will notice any difference 😄.

the backend will also be able to generate other data formats

I get the impression, from your description, that adding new formats would be relatively painless. That'll be good in future.

NotesReview [queries] a backend with a database containing all OpenStreetMap notes updated every 5 minutes, it acts as an intermediary allowing far more complex queries than it would be possible by using the OSM API alone.

Ooh. Possibilities abound!

+3 for wise architecture.

Though, I gather that you keep the backend deployment / hosting hidden / secret, so if I read the above correctly, this RFE will require exposing it to Joe User. The likes of security come to mind.

However, I suppose you could establish a DB-replication mirror (slave DB) for such queries, keeping the master tucked away.

Especially in light of such implications (on top of the development work required to implement it); thankyou for considering & entertaining this RFE.

Surveying has plenty of challenges (even at the best of times), so having good tools helps a lot. Being able to get more out of precious surveying-time (you'd be surprised how little one gets done even in 12 hours) benefits everyone.

I look forward to testing it, whenever that time comes.