feat(maps): maps marker management#836
Open
kennethbrewer3 wants to merge 88 commits into
Open
Conversation
When the OpenAI-compatible fallback (/v1/models) is used, models are mapped as { name: m.id, size: 0 } with no details field. Accessing model.details.parameter_size throws `TypeError: Cannot read properties of undefined`, which crashes the React render and causes the entire page to go blank.
Use the currently loaded model for chat title generation and query rewrite.
…ogs (Crosstalk-Solutions#741) Corrupted ZIM files cause a native C++ abort (ZimFileFormatError) that bypasses JS try/catch and kills the process. Add magic number validation before passing files to @openzim/libzim so invalid files are skipped gracefully. Also deduplicate Ollama download progress broadcasts — both within a single stream (skip unchanged percentages) and across concurrent callers (share one download promise per model). Co-authored-by: aegisman <aegis@manicode.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Ben Smith <bravosierra99@gmail.com>
Co-Authored-By: Ben Smith <bravosierra99@gmail.com>
…ial files
Downloads are now written to `filepath + '.tmp'` and atomically renamed
to the final path only on successful completion. Kiwix globs for `*.zim`
and ZimService filters `.endsWith('.zim')`, so `.tmp` files are invisible
to both during download. The same staging applies to `.pmtiles` map files.
Ref Crosstalk-Solutions#372
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…on (CWE-918, CWE-209) (Crosstalk-Solutions#552) * fix(security): add SSRF validation to map download URLs from manifest * fix(security): sanitize verbose error in rag controller scan endpoint * fix(security): sanitize verbose errors in benchmark controller * fix(security): sanitize verbose error in system controller version fetch * fix(security): sanitize verbose errors in chats controller (6 instances) * fix(security): sanitize verbose errors in docker service (6 instances) * fix(security): sanitize verbose error in system update service * fix(security): sanitize verbose errors in collection update service --------- Co-authored-by: Jake Turner <52841588+jakeaturner@users.noreply.github.com>
…istent progress UI (Crosstalk-Solutions#701) Adds a cancel button to in-progress Ollama model downloads and unifies the Active Model Downloads card layout with the Active Downloads card used for ZIMs, maps, and pmtiles (byte counts, progress bar, live speed, status indicator). Closes Crosstalk-Solutions#676.
…rosstalk-Solutions#729) Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.15.11 to 1.16.0. - [Release notes](https://github.com/follow-redirects/follow-redirects/releases) - [Commits](follow-redirects/follow-redirects@v1.15.11...v1.16.0) --- updated-dependencies: - dependency-name: follow-redirects dependency-version: 1.16.0 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…lutions#708) Bumps [axios](https://github.com/axios/axios) from 1.13.5 to 1.15.0. - [Release notes](https://github.com/axios/axios/releases) - [Changelog](https://github.com/axios/axios/blob/v1.x/CHANGELOG.md) - [Commits](axios/axios@v1.13.5...v1.15.0) --- updated-dependencies: - dependency-name: axios dependency-version: 1.15.0 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…in (Crosstalk-Solutions#736) Bumps [protocol-buffers-schema](https://github.com/mafintosh/protocol-buffers-schema) from 3.6.0 to 3.6.1. - [Commits](mafintosh/protocol-buffers-schema@v3.6.0...v3.6.1) --- updated-dependencies: - dependency-name: protocol-buffers-schema dependency-version: 3.6.1 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…-Solutions#737) Bumps [protobufjs](https://github.com/protobufjs/protobuf.js) from 7.5.4 to 7.5.5. - [Release notes](https://github.com/protobufjs/protobuf.js/releases) - [Changelog](https://github.com/protobufjs/protobuf.js/blob/master/CHANGELOG.md) - [Commits](protobufjs/protobuf.js@protobufjs-v7.5.4...protobufjs-v7.5.5) --- updated-dependencies: - dependency-name: protobufjs dependency-version: 7.5.5 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…Crosstalk-Solutions#724) Bumps [@adonisjs/http-server](https://github.com/adonisjs/http-server) from 7.8.0 to 7.8.1. - [Release notes](https://github.com/adonisjs/http-server/releases) - [Commits](adonisjs/http-server@v7.8.0...v7.8.1) --- updated-dependencies: - dependency-name: "@adonisjs/http-server" dependency-version: 7.8.1 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…olutions#677) Bumps [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) from 6.4.1 to 6.4.2. - [Release notes](https://github.com/vitejs/vite/releases) - [Changelog](https://github.com/vitejs/vite/blob/v6.4.2/packages/vite/CHANGELOG.md) - [Commits](https://github.com/vitejs/vite/commits/v6.4.2/packages/vite) --- updated-dependencies: - dependency-name: vite dependency-version: 6.4.2 dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…Solutions#643) Bumps [lodash](https://github.com/lodash/lodash) from 4.17.23 to 4.18.1. - [Release notes](https://github.com/lodash/lodash/releases) - [Commits](lodash/lodash@4.17.23...4.18.1) --- updated-dependencies: - dependency-name: lodash dependency-version: 4.18.1 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…Crosstalk-Solutions#753) Introduces a dedicated page listing third-party ZIM content packs built by the community. Launches with the two current add-ons (jrsphoto field manuals, kennethbrewer W3Schools) and explains how to install a ZIM pack and where to submit a new one for inclusion. - New doc at admin/docs/community-add-ons.md - Wired into DocsService DOC_ORDER (slot 4) and TITLE_OVERRIDES so the hyphen in "Add-Ons" is preserved in the sidebar - README gets a link under Community & Resources Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ons#747) Qdrant's upstream default enables anonymous telemetry to telemetry.qdrant.io, which doesn't match NOMAD's offline-first "zero telemetry" posture. Adding QDRANT__TELEMETRY_DISABLED=true to the container environment turns it off for fresh installs and reinstalls. Existing installs keep their current telemetry-enabled container until the Qdrant service is force-reinstalled via the Knowledge Base panel or the next container recreation — Docker bakes Env into containers at create time, so env changes require a new container. Closes Crosstalk-Solutions#742 Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…igured (Crosstalk-Solutions#744) When users set a remote Ollama URL via AI Settings, the local nomad_ollama container continued running and competed with the remote host for port 11434 and GPU access. Now configureRemote stops the local container on set and restores it on clear (if still present). Container and its models volume are preserved so the local install can be re-enabled later. Closes Crosstalk-Solutions#662 Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…orer (Crosstalk-Solutions#746) When many ZIMs are already installed locally, a single Kiwix catalog page (12 items) could return 12 already-installed items, which zim_service would fully filter out client-side. The endpoint returned items: [] with has_more: true, and the frontend's infinite-scroll guard (flatData.length > 0) blocked fetchNextPage — leaving the user with "No records found" despite plenty of uninstalled ZIMs available. Backend now accumulates across up to 5 Kiwix fetches (60 items each) until it has enough post-filter results to return, dedupes by entry id, advances currentStart by actual entries returned (not requested), and returns a next_start cursor. The frontend consumes that cursor instead of computing Kiwix offsets locally, and the flatData.length > 0 guard is removed so the existing on-mount effect drives bounded auto-fetch when a short page lands. The pre-existing has_more off-by-one (compared totalResults against the input start rather than the post-fetch position) is fixed implicitly. Diagnosis credit: @johno10661. Closes Crosstalk-Solutions#731 Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… walk) (Crosstalk-Solutions#745) Three bugs in the RAG embedding pipeline, diagnosed and patched by @sbruschke against v1.31.0 with working before/after chunk counts. All three are root-cause contributors to Crosstalk-Solutions#388. 1. scanAndSyncStorage queued every file under /storage/zim/ for embedding, including Kiwix's generated kiwix-library.xml. EmbedFileJob rejected it with "Unsupported file type" and the default 30-attempt retry policy kept it looping on every sync, flooding nomad_admin logs. Now gated on determineFileType(filePath) !== 'unknown'. 2. hasMoreBatches compared zimChunks.length (section-level chunk count under the 'structured' strategy) against ZIM_BATCH_SIZE (an article limit). Because articles emit multiple sections, the two are never equal for real archives and processing silently stopped after the first 50 articles. Now gated on articlesInBatch >= ZIM_BATCH_SIZE. 3. extractStructuredContent walked only direct children of <body>, so any ZIM that wraps content in a container div (Devdocs, Wikipedia, FreeCodeCamp, React docs, etc.) produced zero sections and silently embedded zero chunks while reporting success. Now walks the full DOM via $('body').find('h2, h3, h4, p, ul, ol, dl, table'), with a whole-body text fallback when the selector walk yields nothing. Before/after chunk counts confirmed by @sbruschke on v1.31.0: devdocs_en_git 0 -> 916 devdocs_en_react 0 -> 481 devdocs_en_node 0 -> 423 libretexts_en_eng 1 -> 35 (climbing) Wikipedia resumed progressing normally through its 6M articles. Closes Crosstalk-Solutions#718 Closes Crosstalk-Solutions#719 Closes Crosstalk-Solutions#720 Closes Crosstalk-Solutions#388 Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…#754) The Dockerfile copied root package.json to /app/version.json, which SystemService.getAppVersion() reads on every render of the app version in the UI. semantic-release only reliably commits that bump back on the main branch; on the rc branch it does not, so v1.31.1-rc.1 and v1.31.1-rc.2 both shipped with a version.json still reading 1.31.0. Result: a user who upgrades to rc.2 sees "1.31.0" in the UI and a persistent "update to v1.31.1-rc.2 available" prompt. The build workflow already passes VERSION as a build-arg (used today only for the OCI image label). Generating version.json from that arg at build time makes the image tag the single source of truth and eliminates the drift, regardless of what the committed-back package.json says. Dev builds (no VERSION override) write "dev", which matches the existing NODE_ENV=development short-circuit in getAppVersion(). Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ursor over the map. Changed the cursor to a crosshairs to make it easier to place map markers.
…ntenance. Enhanced the behavior of the coordinate display on the map to not display when over the on screen controls, and the navigation bar. Added a toggle to turn off the coordinate display if the user doesn't wish to see it. Intentionally left the coordinate display when over a map marker so that the coordinates of the map marker can be estimated. In the future I intend to add the coordinates of a map marker when the map marker is clicked so that behavior may change in the future.
…rkdown in the notes field.
…by name or color. Color sort is lexicographically done via the hex value because I plan to add custom marker colors in the future.
* feat: Updated the map to show the coordinates as the user moves the cursor over the map. Changed the cursor to a crosshairs to make it easier to place map markers. * Moved the scale unit control to its own component file for easier maintenance. Enhanced the behavior of the coordinate display on the map to not display when over the on screen controls, and the navigation bar. Added a toggle to turn off the coordinate display if the user doesn't wish to see it. Intentionally left the coordinate display when over a map marker so that the coordinates of the map marker can be estimated. In the future I intend to add the coordinates of a map marker when the map marker is clicked so that behavior may change in the future. --------- Co-authored-by: Kenneth Brewer <kennethbrewer3@protonmail.com>
The /install/wsl2 community-supported install guide is now live on projectnomad.us. Update the README install section and FAQ to point Windows users at it instead of deflecting WSL2 questions to "see the Debian-only answer." Doesn't change the official-support stance — bare-metal Debian-based Linux remains the supported configuration. Just removes the dead-end deflection so Windows users have a real path forward. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bumps and [picomatch](https://github.com/micromatch/picomatch). These dependencies needed to be updated together. Updates `picomatch` from 4.0.3 to 4.0.4 - [Release notes](https://github.com/micromatch/picomatch/releases) - [Changelog](https://github.com/micromatch/picomatch/blob/master/CHANGELOG.md) - [Commits](micromatch/picomatch@4.0.3...4.0.4) Updates `picomatch` from 2.3.1 to 2.3.2 - [Release notes](https://github.com/micromatch/picomatch/releases) - [Changelog](https://github.com/micromatch/picomatch/blob/master/CHANGELOG.md) - [Commits](micromatch/picomatch@4.0.3...4.0.4) --- updated-dependencies: - dependency-name: picomatch dependency-version: 4.0.4 dependency-type: indirect - dependency-name: picomatch dependency-version: 2.3.2 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com>
…osstalk-Solutions#593) * feat(content): add custom ZIM library sources with pre-seeded mirrors Users reported slow download speeds from the default Kiwix CDN. This adds the ability to browse and download ZIM files from alternative Kiwix mirrors or self-hosted repositories, all through the GUI. - Add "Custom Libraries" button next to "Browse the Kiwix Library" - Source dropdown to switch between Default (Kiwix) and custom libraries - Browsable directory structure with breadcrumb navigation - 5 pre-seeded official Kiwix mirrors (US, DE, DK, UK, Global CDN) - Built-in mirrors protected from deletion - Downloads use existing pipeline (progress, cancel, Kiwix restart) - Source selection persists across page loads via localStorage - Scrollable directory browser (600px max) with sticky header - SSRF protection on all custom library URLs Closes Crosstalk-Solutions#576 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(content): recognize Wikipedia downloads from mirror sources When Wikipedia is downloaded via a custom mirror instead of the default Kiwix server, the completion callback now matches by filename instead of exact URL. This ensures the Wikipedia selector correctly shows "Installed" status and triggers old-version cleanup regardless of which mirror was used. Also handles the case where no Wikipedia selection exists yet (file downloaded before visiting the selector), creating the record automatically. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(ZIM): use cheerio for custom mirror directory parsing * fix(ZIM): use URL constructor for more robust joining --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Jake Turner <jturner@cosmistack.com>
Closes Crosstalk-Solutions#826. 1. Heading and subtext now read from `versionInfo` state (which the Check Again mutation already populates) instead of the server-rendered `props.system`. Previously the card kept showing "System Up to Date / Your system is running the latest version!" alongside the new `Latest Version` row + Start Update button after a successful recheck. Status icon also switched to `versionInfo` for consistency. 2. The pulling-state heading rendered the lowercase status enum (`pulling`, `pulled`, ...) and relied on a Tailwind `capitalize` class for the visible glyph. Screen readers and other accessible-name consumers got the lowercase value with no transform applied. Replaced with a `STAGE_LABELS` map so visual + accessible names match. 3. The sidecar (install/sidecar-updater/update-watcher.sh) writes `complete` for ~5s, then resets the status file to `idle`. The SPA could miss that window across the admin container restart, leaving the page parked on its last observed progress percentage indefinitely while the upgrade was actually finished on disk. A `seenAdvancedStageRef` now records whether the session ever observed an advanced stage; a later poll seeing `idle` is treated as the missed completion, and the page reloads as advertised in step 3 of the on-screen process. Reset on each Start Update. 4. Toggling Enable Early Access now triggers a recheck on success, so the eligible-version list updates immediately instead of requiring a manual Check Again click. Single file touched: admin/inertia/pages/settings/update.tsx. Typecheck (tsc --noEmit) passes; static UI changes verified in source. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes Crosstalk-Solutions#796. The maps API has accepted and persisted `notes` on map markers since PR Crosstalk-Solutions#770, but the marker popup component still rendered name only and ignored the field. Now the popup shows a notes block beneath the name when it's populated, with whitespace preserved and long text wrapped. Threaded `notes` through the read path: - `api.listMapMarkers` / `api.createMapMarker` response types - `MapMarker` interface in `useMapMarkers` and the data.map projection - `MapComponent`'s selectedMarker popup The create/update UI is unchanged — users still set notes via the API or DB directly, matching the issue's stated scope. A marker entry with empty/whitespace-only notes renders the same as before. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…tion Closes Crosstalk-Solutions#810. ## Bug A: HSA_OVERRIDE_GFX_VERSION=11.0.0 was unconditional PR Crosstalk-Solutions#804 set HSA_OVERRIDE_GFX_VERSION=11.0.0 for any AMD GPU. The inline comment claimed this was harmless on supported discrete cards (gfx1030 RX 6800, etc.) — empirically false. With the override, Ollama crashes during GPU discovery on gfx1030 and falls back to CPU silently. Affects every NOMAD user with an RX 6800 or other RDNA 2 discrete card. The correct value depends on the gfx version: - gfx1030, gfx1100, gfx1101, gfx1102: officially supported by ROCm — no override - gfx1031..gfx1036 (RDNA 2 variants + iGPUs like Rembrandt 680M): 10.3.0 - gfx1103, gfx1150, gfx1151 (Phoenix 780M, Strix 890M, Strix Halo): 11.0.0 ### Resolution chain in `_resolveAmdHsaOverride()` 1. KV `ai.amdHsaOverride` — manual override; accepts 'none' to disable, or a semver-style value to force. 2. Marker file `/app/storage/.nomad-amd-gfx` — written by install_nomad.sh based on lspci codename. Mapped to override via `_mapGfxToHsaOverride()`. 3. Default: `11.0.0` — preserves prior behavior so existing iGPU users (780M / 890M, the dominant AMD population today) don't regress on upgrade. Discrete RDNA 2 users on existing installs can opt out via `ai.amdHsaOverride='none'` and force-reinstall AI Assistant, OR re-run install_nomad.sh to refresh the marker file. The helper is used in both `createContainer` (initial install) and `updateContainer` (image update) paths, replacing the unconditional push. ## Bug B: BenchmarkService had no AMD discrete detection path `BenchmarkService.getHardwareInfo()` had three GPU detection fallbacks: 1. `si.graphics()` — empty inside Docker for AMD 2. nvidia-smi — NVIDIA only 3. AMD APU regex from CPU model — integrated only Result: AMD discrete cards (RX 6800, RX 7900 XTX, etc.) showed up as "GPU: Not detected" on the leaderboard despite ROCm working. Corrupts leaderboard data quality for that population. Fix: after the existing fallbacks, call `SystemService.getSystemInfo()` and read `graphics.controllers[0].model`. That path already handles AMD via the marker file + Ollama log probe added in PR Crosstalk-Solutions#804, so we're reusing existing plumbing rather than duplicating detection logic. ## install_nomad.sh changes The existing AMD detection block already runs lspci. Added a codename parse step that maps Navi 21/22/23/24, Rembrandt, Phoenix1/Phoenix2, Strix/Strix Point/Strix Halo, and Navi 31/32/33 to gfx versions, then writes `/opt/project-nomad/storage/.nomad-amd-gfx`. Unknown codenames write nothing (admin handles missing-marker case via the backward-compat default). ## Validation Both bugs were originally surfaced and validated empirically on RX 6800 / gfx1030 / Ubuntu 24.04 + kernel 6.17 + ollama/ollama:rocm during the Crosstalk-Solutions#810 filing. Validation grid from that report: | Run | NOMAD Score | tok/s | GPU detected | |-----------------------------------------------|-------------|-------|-------------------------| | Pre-fix (Bug A active) | n/a | 0 | yes, but library=cpu | | HSA_OVERRIDE removed, Bug B unfixed | 73.8 | 221.6 | "Not detected" | | Both fixes hot-patched (this PR's behavior) | 73.7 | 216.0 | AMD Radeon RX 6800 | Local checks: `npm run typecheck` clean, `npm run build` clean.
Contributor
Author
|
@jakeaturner - I consolidated all my changes into one PR. I will hold off on submitting more code until this has merged into dev to prevent further merge conflicts. There should be fewer now that I've migrated some of the components into their own files. |
6 tasks
kennethbrewer3
changed the title
6 tasks
Collaborator
|
Thanks for consolidating everything into one PR, and for your patience. This is a big change (a full marker management system plus database migrations), so we want to give it a dedicated review rather than a quick pass. It's queued for that and we'll follow up here with feedback. |
Contributor
Author
I totally understand that. Just to give you a heads up, if this gets merged I will have a follow up that adds map zones, and direction arrows to help people chart a course. Would it be preferred that I consolidate those changes into this PR or hold off and keep them separate? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
11 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR introduces major improvements to map marker management, UI organization, and navigation workflows. It refactors key components, enhances marker functionality, and adds new user controls for filtering, sorting, and navigation.
Component Refactoring
Marker Enhancements
Editing & Metadata
Rendering Improvements
Marker Panel Improvements
Search & Sorting
Visibility Controls
Organization (Tree View)
Map Interaction Improvements
Navigation
lat,lngMarker Creation
Database Changes
UX Improvements
Notes / Future Work