Java heap space OOM and recurring DAO resync prompt on 1.10.4:0

[helix-nine]

Tracking a report from #8 that was not resolved before merge. 1.10.4:0 is on community-beta; this should be settled before promotion to community-prod.

Symptom

@martinbarilik reported, after updating to 1.10.4:0:

• Java heap space — a JVM OutOfMemoryError, surfaced through Bisq's error dialog.
• "Your Bisq DAO state needs to be resynced." — recurring, and not fixed by resyncing.
• Cannot invoke "javafx.scene.Node.getScene()" — a NullPointerException.

His words: "can any of this changes cause DAO being not synced and needed resyncing and getting it even after resync? I had this before here and there, but struggling with it ever since this update constantly."

That last part matters — it was intermittent before 1.10.4:0 and became constant after.

Working theory

These are most likely one failure, not three. An OOM during DAO block parsing leaves the persisted DAO state inconsistent with the chain; Bisq detects the hash mismatch and prompts for a resync; the resync re-parses the entire BSQ chain, which is the most memory-intensive thing Bisq does, and OOMs again. That loop explains why resyncing never sticks. The Node.getScene() NPE is plausibly downstream of the same exhaustion — once the heap is gone, half-constructed UI is a normal way for it to surface.

Two candidate root causes, and they are not mutually exclusive:

1. Packaging. Dockerfile sets SELKIES_ENCODER="x264enc,jpeg", so the desktop stream is software-encoded with libx264. That has a materially larger resident footprint than the KasmVNC pipeline it replaced in the same release. On a box where headroom was already marginal, that is the kind of change that turns intermittent into constant. Note that nothing in Update Bisq to 1.10.4 and migrate desktop to Selkies #8 changes the JVM heap ceiling directly — neither master nor the PR branch sets -Xmx — so the mechanism would have to be reduced headroom rather than a smaller ceiling.
2. Upstream. A 1.10.4 regression or increase in DAO parsing memory requirements, independent of packaging. Both @martinbarilik and @BeeJoe leaned this way initially.

Diagnostics needed

From an affected install:

# JVM ceiling and actual usage, plus what else is resident
start-cli package attach bisq -n bisq-sub -- ps -eo pid,rss,args --sort=-rss

# container memory picture
start-cli package attach bisq -n bisq-sub -- free -m

# does the OOM land inside DAO parsing?
start-cli package attach bisq -n bisq-sub -- grep -n -i "OutOfMemory\|heap space" /config/.local/share/Bisq/bisq.log

For the third, the ~50 lines preceding each hit are the useful part. Also needed: total RAM on the host box, which makes the rest interpretable.

What would distinguish the two causes

• If the Selkies/GStreamer processes account for a large share of resident memory and the JVM is hitting a ceiling sized off what's left, that points at (1) and the fix is a packaging change — an explicit -Xmx for Bisq, a lighter encoder setting, or a declared hardwareRequirements.ram.
• If the JVM ceiling is unchanged from 1.10.3:4 and the OOM still occurs at the same point in DAO parsing, that points at (2) and belongs upstream at bisq-network/bisq.

Comparing resident memory between 1.10.3:4 and 1.10.4:0 on the same box would separate these directly, if anyone still has the older revision available.

Impact

If (1), this needs a 1.10.4:1 before promotion to community-prod. If (2), it should be reported upstream and noted in instructions.md as a known limitation with a minimum-RAM recommendation.

cc @BeeJoe @martinbarilik

[martinbarilik]

Providing logs as discussed. Build from current master, straight error after entering username / password

hope this helps

start-cli package attach bisq -n bisq-sub -- ps -eo pid,rss,args --sort=-rss
    PID   RSS COMMAND
    434 3276348 /opt/bisq/bin/Bisq
    274 600776 /usr/bin/Xvfb :1 -screen 0 15360x8640x24 -dpi 96 +extension COMPOSITE +extension DAMAGE +extension GLX +extension RANDR +extension RENDER +extension MIT-SHM +extension XFIXES +extension XTEST +igl
    304 119228 /lsiopy/bin/python3 /lsiopy/bin/selkies --addr=localhost --mode=websockets
    725 112420 /config/.local/share/Bisq/btc_mainnet/tor/tor -f /config/.local/share/Bisq/btc_mainnet/tor/torrc __OwningControllerProcess 434
    402 19088 /usr/bin/openbox --replace --startup /usr/lib/x86_64-linux-gnu/openbox-autostart OPENBOX
    278  8136 nginx: master process /usr/sbin/nginx -g daemon off;
    277  8124 /usr/bin/pulseaudio --log-level=0 --log-target=stderr --exit-idle-time=-1
    895  6420 /usr/lib/x86_64-linux-gnu/xfce4/xfconf/xfconfd
    365  4060 xsettingsd
   1758  3952 ps -eo pid,rss,args --sort=-rss
    294  3904 dbus-daemon --system --nofork --nosyslog
    299  3412 bash ./run svc-de
    279  3404 bash ./run svc-cron
    280  3292 bash ./run svc-docker
    344  3136 nginx: worker process
    378  2572 /usr/bin/dbus-daemon --syslog-only --fork --print-pid 5 --print-address 7 --session
    348  2420 nginx: worker process
    346  2416 nginx: worker process
    345  2404 nginx: worker process
    409  2180 dbus-launch --exit-with-session /usr/bin/openbox-session
    331  1996 sleep infinity
   1733  1948 xclip -selection clipboard -i -t UTF8_STRING
    410  1944 /usr/bin/dbus-daemon --syslog --fork --print-pid 5 --print-address 7 --session
    314  1872 sleep infinity
    273  1828 sleep infinity
    349  1756 nginx: worker process
    347  1752 nginx: worker process
    350  1752 nginx: worker process
    351  1752 nginx: worker process
    352  1752 nginx: worker process
    353  1752 nginx: worker process
    354  1752 nginx: worker process
    355  1752 nginx: worker process
    356  1752 nginx: worker process
    357  1752 nginx: worker process
    358  1752 nginx: worker process
    359  1692 nginx: worker process
    373  1576 dbus-launch --autolaunch 6711c5d2f37a4f10b602a43b111b265e --binary-syntax --close-stderr
      1    76 /package/admin/s6/command/s6-svscan -d4 -- /run/service
     13    60 s6-supervise s6-linux-init-shutdownd
     30    60 s6-supervise s6rc-oneshot-runner
     31    60 s6-supervise s6rc-fdholder
     32    60 s6-supervise svc-cron
     33    60 s6-supervise svc-dbus
     34    60 s6-supervise svc-de
     35    60 s6-supervise svc-docker
     36    60 s6-supervise svc-nginx
     37    60 s6-supervise svc-pulseaudio
     38    60 s6-supervise svc-selkies
     39    60 s6-supervise svc-watchdog
     40    60 s6-supervise svc-xorg
     41    60 s6-supervise svc-xsettingsd
     17    56 /package/admin/s6-linux-init/command/s6-linux-init-shutdownd -d3 -c /run/s6/basedir -g 3000 -C -B
     49    32 /package/admin/s6/command/s6-ipcserverd -1 -- /package/admin/s6/command/s6-ipcserver-access -v0 -E -l0 -i data/rules -- /package/admin/s6/command/s6-sudod -t 30000 -- /package/admin/s6-rc/command/s

start-cli package attach bisq -n bisq-sub -- free -m
               total        used        free      shared  buff/cache   available
Mem:           31492       16886         567         640       15273       14605
Swap:           7872        2739        5133

start-cli package attach bisq -n bisq-sub -- grep -n -i "OutOfMemory\|heap space" /config/.local/share/Bisq/bisq.log
26126:	Exception=java.lang.OutOfMemoryError: Java heap space 
26127:Jul-31 15:31:50.867 [JavaFX Application Thread] ERROR b.c.s.CommonSetup: OutOfMemoryError occurred. We shut down. java.lang.OutOfMemoryError: Java heap space
26167:	Exception=java.lang.OutOfMemoryError: Java heap space 
26169:Jul-31 15:32:00.970 [JavaFX Application Thread] ERROR b.c.s.CommonSetup: OutOfMemoryError occurred. We shut down. java.lang.OutOfMemoryError: Java heap space
26208:	Exception=java.lang.OutOfMemoryError: Java heap space 
26210:Jul-31 15:32:10.815 [JavaFX Application Thread] ERROR b.c.s.CommonSetup: OutOfMemoryError occurred. We shut down. java.lang.OutOfMemoryError: Java heap space

[BeeJoe]

Upstream and independent-runtime findings

This is now independently reproduced and the failure path is clearer.

Upstream reports

• bisq-network/bisq#7989, opened August 1 against v1.10.4, is an exact match: a fresh data directory, Max memory: 2 GB, and an OOM in DaoStateMonitoringService.createHashFromBlock. It documents the same -XX:MaxRAM=8g → approximately 2 GB effective heap and recommends an explicit -Xmx4g on hosts with at least 8 GB RAM. It is still open with no maintainer response or linked fix PR.
• bisq-network/bisq#7896 has another v1.10.4-era trace in CanonicalWriter → CanonicalEncoder → DaoState.encodeCanonicalForStateHashChain.
• bisq-network/bisq#7565 is the older Start9 report on v1.9.21. It already showed the same 2 GB ceiling in DAO state hashing; a Bisq contributor reports that launching with JAVA_TOOL_OPTIONS="-Xmx8g" largely eliminated it.

The canonical DAO encoder was introduced by bisq-network/bisq#7898. bisq-network/bisq#7910 then cached the roughly 300,000-entry spentInfoMap, reducing encoding time, but it did not change the heap ceiling or avoid building the complete canonical byte array.

As of August 2, there is no newer upstream commit or PR addressing this OOM. Upstream master still has -XX:MaxRAM=8g and no explicit -Xmx; none of the commits after v1.10.4 modify canonical DAO encoding or the desktop heap settings.

Independent StartOS reproduction

On a second StartOS host running 1.10.4:0:

• Bisq RSS was approximately 3.26 GiB; the package used about 4.1 GiB overall.
• The host had approximately 35.5 GiB available, no cgroup memory ceiling, and zero cgroup OOM/OOM-kill events.
• Bisq repeatedly reported Max memory: 2 GB, with only 39–78 MiB free after GC; repeated GC reclaimed nothing.
• OOMs occurred in two direct canonical DAO paths:

CanonicalWriter.toByteArray
CanonicalEncoder.encode
DaoState.encodeCanonicalForStateHashChain
DaoStateMonitoringService.createDaoStateBlock

and:

CanonicalWriter.writeRawBytes
CanonicalEncoder.writeMap
DaoState.encodeCanonicalForStateHashChain
DaoStateMonitoringService.processPeersDaoStateHashes

After the first OOM, the DAO monitoring chain remained stuck at block 960171 while new blocks advanced. After restart/resync, Bisq parsed 482 blocks and reached the wallet tip, then OOMed again during peer DAO-state hash processing; the next block immediately reported another monitoring-chain gap. The process stayed alive, so StartOS continued to see a running service despite the broken DAO monitor.

This confirms the OOM → incomplete DAO monitoring hash chain → resync → OOM loop. Selkies and host memory pressure are not the cause.

Recommended test mitigation

Prepare a downstream 1.10.4:1 test build with an explicit 4 GiB Bisq heap and an 8 GiB minimum-RAM requirement, then exercise a full DAO resync against the affected persisted state. Promotion should remain blocked until the resync completes and heap use stabilizes. If 4 GiB still exhausts, this needs an upstream encoder change rather than another heap increase.

[BeeJoe]

Test build and affected-state verification are now complete.

Implementation: BeeJoe:fix/bisq-dao-oom, commit 22c88a2. The downstream package revision is 1.10.4:1; it sets JAVA_TOOL_OPTIONS=-Xmx4g and declares an 8192 MB minimum RAM requirement.

I installed the built x86_64 artifact over the affected 1.10.4:0 instance, preserving its existing Bisq data. Runtime evidence:

• Startup reports Max memory: 4 GB (the prior package reported 2 GB).
• After wallet unlock, Bisq applied the persisted DAO snapshot with 388,425 state hashes.
• DAO parsing completed and DaoStateMonitoringService passed its checkpoint.
• Four seconds after the checkpoint, the profiler reported 2.182 GB total heap, 1.898 GB used, and a 4 GB maximum; GC reduced that to 1.967 GB.
• No OutOfMemoryError occurred through the subsequent stability check. The old 2 GB run had failed eight seconds after this same checkpoint in CanonicalEncoder / createDaoStateBlock.
• The installed version remained 1.10.4:1, the service stayed running at about 4.1 GiB container memory, and no new StartOS notifications appeared.

This confirms the 4 GiB heap clears the reproduced canonical DAO serialization failure on the affected persisted state. A longer soak remains prudent, but the exact path and timing that previously failed now complete successfully.

[martinbarilik]

testing the version ( public beta we previously merged ) with current branch's change

JAVA_TOOL_OPTIONS: '-Xmx4g',

so far, so good, no java heap error since, gonna observe more, will inform

[martinbarilik]

after more testing, DAO got stuck again, but after "DAO sync" everything went well, no java heap error

if you want me to test anything else, just write