opte on `helios/deploy` buildomat worker "somehow failed to get a mblk..." and paniced

[hawkw]

I saw this panic during a helios / deploy CI run (oxidecomputer/omicron#10183 (comment)) on omicron:

sock console login: WARNING: panicked at lib/opte/src/ddi/mblk.rs:291:14:
somehow failed to get an mblk...
panic[cpu18]/thread=fffffe85e6714060: panicked at lib/opte/src/ddi/mblk.rs:291:14:
somehow failed to get an mblk...
fffffe00c6257d40 xde:_RNvCsj0qrnF3AvHz_7___rustc17rust_begin_unwind+16d ()
fffffe00c6257d70 xde:opte_panic_debug+3446024b ()
fffffe00c6257dd0 xde:opte_panic_debug+34460700 ()
fffffe00c6257e80 xde:_ZN4opte6engine6packet8EmitSpec5apply17hea7022e6e50299e3E+e21 ()
fffffe00c62583e0 xde:xde_mc_tx+33ce ()
fffffe00c6258410 mac:mac_ring_tx+50 ()
fffffe00c6258460 mac:mac_provider_tx+6d ()
fffffe00c6258520 mac:mac_tx+28b ()
fffffe00c6258570 dld:str_mdata_fastpath_put+8e ()
fffffe00c6258680 ip:ip_xmit+9ce ()
fffffe00c62588d0 ip:ire_send_wire_v4+3c2 ()
fffffe00c6258950 ip:conn_ip_output+18f ()
fffffe00c6258a30 ip:udp_output_connected+131 ()
fffffe00c6258ad0 ip:udp_send+1bf ()
fffffe00c6258b60 sockfs:so_sendmsg+201 ()
fffffe00c6258bc0 sockfs:socket_sendmsg+65 ()
fffffe00c6258c60 sockfs:sendit+186 ()
fffffe00c6258f00 sockfs:sendmsg+2c6 ()
fffffe00c6258f10 unix:brand_sys_syscall+1fe ()
skipping system dump - no dump device configured
rebooting...

This panic appears to occur when allocb(9F) returns a nullptr, which...sure seems bad:

opte/lib/opte/src/ddi/mblk.rs

Lines 289 to 294 in 04c3d5d

	pub fn new(len: usize) -> Self {
	let inner = NonNull::new(allocb(len))
	.expect("somehow failed to get an mblk...");
	Self(inner)
	}

At the time of the panic, the worker was waiting for Nexus zones to come up. I don't think there's any way to get a core out of this, unfortunately, but perhaps @jclulow has a trick up his sleeve?

[jclulow]

Unfortunately we don't currently collect crash dumps out of there, just the panic stack stuff because it happens to pop out on the console (which we collect for the lab target)

[FelixMcFelix]

We have seen this before, but I don't think anyone has had time to go and retool things yet. My read at the time was that allocb got put in a spot where we'd need to resize the mblk kmem cache, and KM_NOSLEEP causes it to return early.