Occasional packet metadata not found error in EC2

[bcelenza]

See https://github.com/bcelenza/vesper/blob/main/probes/src/network/main.rs#L148-L150.

Occasionally see an error looking up metadata for packet with ID 0:

2022-10-16T17:06:16.380145Z ERROR vesper::listeners::network: Could not find metadata for packet with ID 0
2022-10-16T17:06:16.380204Z DEBUG vesper::listeners::network: Packet: SlicedPacket { link: Some(Ethernet2(Ethernet2HeaderSlice { slice: [10, 202, 177, 252, 49, 191, 10, 78, 210, 123, 174, 163, 8, 0] })), vlan: None, ip: Some(Ipv4(Ipv4HeaderSlice { slice: [69, 0, 0, 137, 0, 0, 64, 0, 255, 17, 25, 65, 172, 31, 0, 2, 172, 31, 9, 226] }, Ipv4ExtensionsSlice { auth: None })), transport: Some(Udp(UdpHeaderSlice { slice: [0, 53, 207, 94, 0, 117, 55, 105] })), payload: [109, 195, 129, 128, 0, 1, 0, 4, 0, 0, 0, 1, 3, 97, 112, 105, 9, 115, 110, 97, 112, 99, 114, 97, 102, 116, 2, 105, 111, 0, 0, 1, 0, 1, 192, 12, 0, 1, 0, 1, 0, 0, 0, 8, 0, 4, 185, 125, 188, 58, 192, 12, 0, 1, 0, 1, 0, 0, 0, 8, 0, 4, 185, 125, 188, 54, 192, 12, 0, 1, 0, 1, 0, 0, 0, 8, 0, 4, 185, 125, 188, 59, 192, 12, 0, 1, 0, 1, 0, 0, 0, 8, 0, 4, 185, 125, 188, 55, 0, 0, 41, 16, 0, 0, 0, 0, 0, 0, 0] }

DNS responses from the VPC resolver always seem to have IP ID of 0.
Expand metadata map key to include more unique attributes.

Example: Snap Refresh

When refreshing the Ubuntu snap repo via sudo snap refresh

BPF trace:

 systemd-resolve-426     [000] d...1 355702.578997: bpf_trace_printk: Forwarding packet to userspace: protocol=17, class=1, id=45682
          <idle>-0       [000] dNs.1 355702.579141: bpf_trace_printk: Forwarding packet to userspace: protocol=17, class=1, id=0
 systemd-resolve-426     [003] d...1 355702.606286: bpf_trace_printk: Forwarding packet to userspace: protocol=17, class=1, id=39363
 systemd-resolve-426     [003] d...1 355702.606387: bpf_trace_printk: Forwarding packet to userspace: protocol=17, class=1, id=25542
          <idle>-0       [000] d.s.1 355702.606467: bpf_trace_printk: Forwarding packet to userspace: protocol=17, class=1, id=0
          <idle>-0       [000] d.s.1 355702.607063: bpf_trace_printk: Forwarding packet to userspace: protocol=17, class=1, id=0
           snapd-27950   [000] d...1 355702.747783: bpf_trace_printk: Forwarding packet to userspace: protocol=6, class=2, id=32449
          <idle>-0       [003] d.s.1 355702.889026: bpf_trace_printk: Forwarding packet to userspace: protocol=6, class=2, id=24351

Vesper stderr+stdout:

{"time":"2022-10-16T17:06:55.130734282+00:00","type":"TlsServerHello","event":{"TlsServerHello":{"source":{"ip":"185.125.188.58","port":443},"destination":{"ip":"172.31.9.226","port":35924},"version":"TLSv1_2","cipher":"TLS_AES_256_GCM_SHA384"}}}
{"time":"2022-10-16T17:06:55.130910057+00:00","type":"DnsQuery","event":{"DnsQuery":{"source":{"ip":"172.31.9.226","port":59885},"destination":{"ip":"172.31.0.2","port":53},"id":19401,"questions":[{"type":"AAAA","name":"ip-172-31-9-226.us-west-2.compute.internal"}]}}}
{"time":"2022-10-16T17:06:55.157999765+00:00","type":"DnsResponse","event":{"DnsResponse":{"source":{"ip":"172.31.0.2","port":53},"destination":{"ip":"172.31.9.226","port":59885},"id":19401,"status":"NoError","authoritative":false,"recursive":true,"questions":[{"type":"AAAA","name":"ip-172-31-9-226.us-west-2.compute.internal"}],"answers":[]}}}
{"time":"2022-10-16T17:06:55.158167423+00:00","type":"DnsQuery","event":{"DnsQuery":{"source":{"ip":"172.31.9.226","port":52960},"destination":{"ip":"172.31.0.2","port":53},"id":35751,"questions":[{"type":"A","name":"api.snapcraft.io"}]}}}
{"time":"2022-10-16T17:06:55.158659426+00:00","type":"DnsQuery","event":{"DnsQuery":{"source":{"ip":"172.31.9.226","port":37994},"destination":{"ip":"172.31.0.2","port":53},"id":25002,"questions":[{"type":"AAAA","name":"api.snapcraft.io"}]}}}
{"time":"2022-10-16T17:06:55.299438369+00:00","type":"DnsResponse","event":{"DnsResponse":{"source":{"ip":"172.31.0.2","port":53},"destination":{"ip":"172.31.9.226","port":37994},"id":25002,"status":"NoError","authoritative":false,"recursive":true,"questions":[{"type":"AAAA","name":"api.snapcraft.io"}],"answers":[]}}}
2022-10-16T17:06:55.440675Z ERROR vesper::listeners::network: Could not find metadata for packet with ID 0
2022-10-16T17:06:55.440725Z DEBUG vesper::listeners::network: Packet: SlicedPacket { link: Some(Ethernet2(Ethernet2HeaderSlice { slice: [10, 202, 177, 252, 49, 191, 10, 78, 210, 123, 174, 163, 8, 0] })), vlan: None, ip: Some(Ipv4(Ipv4HeaderSlice { slice: [69, 0, 0, 137, 0, 0, 64, 0, 255, 17, 25, 65, 172, 31, 0, 2, 172, 31, 9, 226] }, Ipv4ExtensionsSlice { auth: None })), transport: Some(Udp(UdpHeaderSlice { slice: [0, 53, 206, 224, 0, 117, 25, 199] })), payload: [139, 167, 129, 128, 0, 1, 0, 4, 0, 0, 0, 1, 3, 97, 112, 105, 9, 115, 110, 97, 112, 99, 114, 97, 102, 116, 2, 105, 111, 0, 0, 1, 0, 1, 192, 12, 0, 1, 0, 1, 0, 0, 0, 23, 0, 4, 185, 125, 188, 58, 192, 12, 0, 1, 0, 1, 0, 0, 0, 23, 0, 4, 185, 125, 188, 59, 192, 12, 0, 1, 0, 1, 0, 0, 0, 23, 0, 4, 185, 125, 188, 55, 192, 12, 0, 1, 0, 1, 0, 0, 0, 23, 0, 4, 185, 125, 188, 54, 0, 0, 41, 16, 0, 0, 0, 0, 0, 0, 0] }

TCP dump for port 53:

17:06:55.130539 IP 172.31.9.226.59885 > 172.31.0.2.53: 19401+ [1au] AAAA? ip-172-31-9-226.us-west-2.compute.internal. (71)
17:06:55.130687 IP 172.31.0.2.53 > 172.31.9.226.59885: 19401 0/0/1 (71)
17:06:55.157830 IP 172.31.9.226.52960 > 172.31.0.2.53: 35751+ [1au] A? api.snapcraft.io. (45)
17:06:55.157933 IP 172.31.9.226.37994 > 172.31.0.2.53: 25002+ [1au] AAAA? api.snapcraft.io. (45)
17:06:55.158011 IP 172.31.0.2.53 > 172.31.9.226.37994: 25002 0/0/1 (45)
17:06:55.158609 IP 172.31.0.2.53 > 172.31.9.226.52960: 35751 4/0/1 A 185.125.188.58, A 185.125.188.59, A 185.125.188.55, A 185.125.188.54 (109)