Erlay

Erlay paper: https://arxiv.org/pdf/1905.10518.pdf

page 1:

Today, Bitcoin disseminates transactions by ensuring that every message received by a node is transmitted to all of its neighbors. This flooding has high fault-tolerance ... However, flooding has poor bandwidth efficiency: every node in the network learns about the transaction multiple times. Our em- pirical measurements demonstrate that transaction announcements account for 30–50% of the overall Bitcoin traffic.

page 2:

The main idea behind our protocol is to reduce the amount of information propagated via flooding and instead use an efficient set reconciliation method

Currently INV/TX messages are used in the flooding (36 bytes) and the actual transactions (a few 100s of bytes) are sent in an efficient way - without flooding/duplication where every node receives a given transaction just once.

So, I try to get an estimate of how much is that flooding/redundant from the overall traffic? That would be the maximum possible theoretical savings Erlay could bring (would be less in practice because it would still do some flooding).

---

Measurements from Nov 2024:

group.js

#!/usr/bin/env node

const fs = require("fs");
const util = require("util");

const input = JSON.parse(fs.readFileSync("/dev/stdin").toString());

let total = 0;
for (const [message_type, stats] of Object.entries(input)) {
    if (message_type != "invtx" && message_type != "invtxredund") {
        total += stats.bytes;
    }
}

let a = [];
for (const [message_type, stats] of Object.entries(input)) {
    a.push({"message_type": message_type, "bytes": stats.bytes, "% of total": (stats.bytes / total * 100).toPrecision(2)});
}
a.push({"message_type": "_total_", "bytes": total});
a.sort((x, y) => x.bytes - y.bytes);
console.log(util.inspect(a, {breakLength: 100}));

Run bitcoind from the counting branch for some time and then:

bitcoin-cli getnetmsgstats '["direction", "network", "connection_type"]' |./group.js

Result on a fully synced listening node with ~124 connections, running for ~20 hours on Nov 4:

[
  { message_type: 'mempool', bytes: 48, '% of total': '1.0e-7' },
  { message_type: 'getblocks', bytes: 93, '% of total': '2.0e-7' },
  { message_type: 'sendtxrcncl', bytes: 135, '% of total': '2.9e-7' },
  { message_type: 'getblocktxn', bytes: 77937, '% of total': '0.00017' },
  { message_type: 'getaddr', bytes: 110718, '% of total': '0.00024' },
  { message_type: 'sendheaders', bytes: 114906, '% of total': '0.00025' },
  { message_type: 'feefilter', bytes: 240663, '% of total': '0.00052' },
  { message_type: 'sendcmpct', bytes: 289257, '% of total': '0.00063' },
  { message_type: 'wtxidrelay', bytes: 321237, '% of total': '0.00070' },
  { message_type: 'sendaddrv2', bytes: 370491, '% of total': '0.00080' },
  { message_type: 'verack', bytes: 426216, '% of total': '0.00092' },
  { message_type: 'version', bytes: 2255913, '% of total': '0.0049' },
  { message_type: 'notfound', bytes: 2433830, '% of total': '0.0053' },
  { message_type: 'pong', bytes: 3994397, '% of total': '0.0087' },
  { message_type: 'ping', bytes: 4077486, '% of total': '0.0088' },
  { message_type: 'getheaders', bytes: 8552589, '% of total': '0.019' },
  { message_type: 'cmpctblock', bytes: 10370096, '% of total': '0.022' },
  { message_type: 'addr', bytes: 11531743, '% of total': '0.025' },
  { message_type: 'blocktxn', bytes: 34915212, '% of total': '0.076' },
  { message_type: 'addrv2', bytes: 56592212, '% of total': '0.12' },
  { message_type: 'getdata', bytes: 75182592, '% of total': '0.16' },
  { message_type: 'headers', bytes: 182977421, '% of total': '0.40' },
  { message_type: 'tx', bytes: 707851131, '% of total': '1.5' },
  { message_type: 'invtxredund', bytes: 1036646091, '% of total': '2.2' }, // <--- redundant INV/TX messages
  { message_type: 'invtx', bytes: 1413006232, '% of total': '3.1' },
  { message_type: 'inv', bytes: 1417615620, '% of total': '3.1' },
  { message_type: 'block', bytes: 43575531277, '% of total': '95' },
  { message_type: '_total_', bytes: 46095833220 }
]

The reason for the low 2.2% is that the node is serving IBD to other nodes. Thus block messages account for 95% of the traffic. How widespread is this? I repeated this test many times and it looks like somebody is doing IBD from me all the time.

Result on a fully synced non-listening node, running for ~20 hours on Nov 4:

[
  { message_type: 'feefilter', bytes: 4379, '% of total': '0.00052' },
  { message_type: 'getaddr', bytes: 4944, '% of total': '0.00059' },
  { message_type: 'sendheaders', bytes: 6312, '% of total': '0.00075' },
  { message_type: 'sendcmpct', bytes: 8046, '% of total': '0.00096' },
  { message_type: 'sendaddrv2', bytes: 13467, '% of total': '0.0016' },
  { message_type: 'verack', bytes: 13602, '% of total': '0.0016' },
  { message_type: 'wtxidrelay', bytes: 13806, '% of total': '0.0016' },
  { message_type: 'getblocktxn', bytes: 31640, '% of total': '0.0038' },
  { message_type: 'version', bytes: 74530, '% of total': '0.0089' },
  { message_type: 'notfound', bytes: 110698, '% of total': '0.013' },
  { message_type: 'addr', bytes: 148252, '% of total': '0.018' },
  { message_type: 'headers', bytes: 219212, '% of total': '0.026' },
  { message_type: 'getheaders', bytes: 309309, '% of total': '0.037' },
  { message_type: 'pong', bytes: 501952, '% of total': '0.060' },
  { message_type: 'ping', bytes: 502045, '% of total': '0.060' },
  { message_type: 'addrv2', bytes: 2294225, '% of total': '0.27' },
  { message_type: 'blocktxn', bytes: 6467471, '% of total': '0.77' },
  { message_type: 'cmpctblock', bytes: 8356433, '% of total': '1.0' },
  { message_type: 'getdata', bytes: 45321664, '% of total': '5.4' },
  { message_type: 'invtxredund', bytes: 148472719, '% of total': '18' }, // <--- redundant INV/TX messages
  { message_type: 'inv', bytes: 208283234, '% of total': '25' },
  { message_type: 'invtx', bytes: 208375729, '% of total': '25' },
  { message_type: 'tx', bytes: 566530335, '% of total': '68' },
  { message_type: '_total_', bytes: 839215556 }
]

note1: invtx is a subset of inv and invtxredund is a subset of invtx

node2: the counting is imperfect because stats for all message types except invtx and invtxredund are collected at the transport level, whereas for invtx and invtxredund at the higher level which is aware of the contents of the messages

---

Measurements from Oct 2022:

With some added counting, after running a listening node with ~100 connections for ~48 hours:

"totalbytesrecv": 1341343657,
"totalbytessent": 56168391051,
"totalbytesrecv_inv_tx": 661329290,
"totalbytesrecv_inv_tx_redundant": 613923732,
"totalbytessent_inv_tx": 706329175,
"totalbytessent_inv_tx_redundant": 111107520,
"totalbytessent_inv_tx_mempool": 0,

Sent+recv INV/TX is 2.4% from the total traffic (.totalbytesrecv_inv_tx + .totalbytessent_inv_tx) / (.totalbytesrecv + .totalbytessent) * 100.

From that 53% is redundant (.totalbytesrecv_inv_tx_redundant + .totalbytessent_inv_tx_redundant) / (.totalbytesrecv_inv_tx + .totalbytessent_inv_tx) * 100.

So, the redundant traffic is 1.2% (53% of 2.4%) of the total traffic.

Which part is Erlay aiming to optimize?

That is just 1.2%?

Repeated run over ~5 days, ending Oct 5 2022:

• redundant / total traffic: 3% (vs 1.2% measured before)
• average traffic speed: 120 KiB/s

On a non-listening node, over ~24h, ending Oct 5 2022:

• redundant / total traffic: 9%
• average traffic speed: 5 KiB/s