Vectorize base16 (hex) decoding

[k0ekk0ek]

base16 encoding is used quite a lot. i.e. in DS records and to represent rdata for unknown record types (as outlined in RFC3597). Research was done to deserialize more efficiently by Geoff Langdale and Wojciech Muła. A couple of algorithms are outlined in this article. Other research may be available too.

[aqrit]

Base16 (hex) can be decoded similarly to base32hex, except first we’d v = _mm_add_epi8(v, _mm_set1_epi8(-1)) to make the spans line up. We could also get rid of both the multiply instructions.

But.. how many shuffles should be used? Haswell CPUs have only one shuffle execution port. An AVX2 port of the base32hex style decoder would require 4 shuffles per 32-bytes of input. The linked solution by Geoff Langdale uses 2 shuffles, has only 1 more instruction in total, but it would require several more 32-byte constants from memory.

[lemire]

Let us test it out!

[k0ekk0ek]

For base16, we'd need to keep some state as some record types (and generic RDATA representation) allow for spaces. In terms of the zone parser, that means calling lex multiple times, so we'd have to stop at the first non-hexadecimal character.

[lemire]

keep some state as some record types

The standard seems to require that bytes are encoded as pairs and that pairs are held together, so that the input is always an even number of characters. Are you saying that they don't do that? That'd be very strange.

[lemire]

Using GCC11 on an Ice Lake server, I get the following results:

base16hex_simd                 :   2.78 GB/s  253.1 Ma/s   3.95 ns/d   3.20 GHz  12.63 c/d  44.49 i/d    1.2 c/b   4.06 i/b   3.52 i/c 
base16hex_simd_geoff           :   2.00 GB/s  182.6 Ma/s   5.48 ns/d   3.20 GHz  17.50 c/d  65.48 i/d    1.6 c/b   5.97 i/b   3.74 i/c 
base16hex_simple               :   0.85 GB/s   77.4 Ma/s  12.92 ns/d   3.19 GHz  41.27 c/d  85.65 i/d    3.8 c/b   7.81 i/b   2.08 i/c 
base16hex_simdzone_fallback    :   0.76 GB/s   69.6 Ma/s  14.38 ns/d   3.19 GHz  45.91 c/d  88.13 i/d    4.2 c/b   8.04 i/b   1.92 i/c 

My benchmark uses relatively short strings, of random length. We can (and maybe should) change that to more closely match what we have in the actual data. It is important to note that the bulk of the instructions are in the setting up for both base16hex_simd and base16hex_simd_geoff, so if we expect longer strings, then it would be important to know about it. Also if these functions are to be inlined, I should probably change the benchmark as it will make the simd code shine. I'd say my benchmark makes SIMD looks bad... but I prefer to be conservative.

In my table base16hex_simd_geoff is Geoff's version from the article @k0ekk0ek linked above. Whereas base16hex_simd is basically our base32 decoder simplified for base32. So we are significantly faster than Geoff's approach as far as I can tell.

It is possible we can improve our approach further. I did not quite understand @aqrit's description: I am still using a multiplication when combining the bytes. I can do it without a multiplication, but the approach I imagine require several extra instructions (no big deal, but slightly slower...). Maybe there is a clever way that I don't see.

The version in simdzone currently is base16hex_simdzone_fallback (or something close to it)...

Here is the assembly for our version (note that there is a hot loop in the middle):

base16hex_simd(unsigned char*, unsigned char const*):
        movdqu  xmm1, XMMWORD PTR [rsi]
        pcmpeqd xmm5, xmm5
        mov     rax, rsi
        movdqa  xmm7, XMMWORD PTR .LC1[rip]
        paddb   xmm1, xmm5
        movdqa  xmm6, XMMWORD PTR .LC0[rip]
        movdqa  xmm8, XMMWORD PTR .LC2[rip]
        movdqa  xmm2, xmm1
        movdqa  xmm3, xmm7
        psrld   xmm2, 4
        movdqa  xmm0, xmm8
        pand    xmm2, xmm6
        pshufb  xmm3, xmm2
        pshufb  xmm0, xmm2
        paddb   xmm0, xmm1
        paddb   xmm1, xmm3
        pmovmskb        edx, xmm1
        test    edx, edx
        jne     .L4
        movdqa  xmm4, XMMWORD PTR .LC3[rip]
        movdqa  xmm9, XMMWORD PTR .LC4[rip]
.L2:
        add     rax, 16
        pmaddubsw       xmm0, xmm4
        movdqa  xmm3, xmm7
        pshufb  xmm0, xmm9
        movups  XMMWORD PTR [rdi], xmm0
        movdqu  xmm1, XMMWORD PTR [rax]
        movdqa  xmm0, xmm8
        add     rdi, 8
        paddb   xmm1, xmm5
        movdqa  xmm2, xmm1
        psrld   xmm2, 4
        pand    xmm2, xmm6
        pshufb  xmm3, xmm2
        pshufb  xmm0, xmm2
        paddb   xmm0, xmm1
        paddb   xmm1, xmm3
        pmovmskb        edx, xmm1
        test    edx, edx
        je      .L2
.L4:
        bsf     rdx, rdx
        test    edx, edx
        je      .L3
        movsx   rdx, edx
        mov     ecx, 16
        sub     rcx, rdx
        add     rax, rdx
        movdqu  xmm1, XMMWORD PTR base16hex_simd(unsigned char*, unsigned char const*)::zero_masks[rcx]
        pandn   xmm1, xmm0
        movdqa  xmm0, XMMWORD PTR .LC3[rip]
        pmaddubsw       xmm1, xmm0
        pshufb  xmm1, XMMWORD PTR .LC4[rip]
        movups  XMMWORD PTR [rdi], xmm1
.L3:
        sub     rax, rsi
        ret

Here is the assembly for Geoff's version (note the hot loop in the middle):

base16hex_simd_geoff(unsigned char*, unsigned char const*):
        movdqa  xmm3, XMMWORD PTR .LC5[rip]
        mov     rax, rsi
        movdqu  xmm1, XMMWORD PTR [rsi]
        movdqa  xmm4, XMMWORD PTR .LC6[rip]
        movdqa  xmm0, xmm3
        movdqa  xmm5, XMMWORD PTR .LC7[rip]
        paddb   xmm0, xmm1
        movdqa  xmm6, XMMWORD PTR .LC8[rip]
        psubusb xmm0, xmm4
        movdqa  xmm8, XMMWORD PTR .LC9[rip]
        pand    xmm1, xmm5
        movdqa  xmm7, XMMWORD PTR .LC10[rip]
        paddb   xmm1, xmm6
        movdqa  xmm9, XMMWORD PTR .LC11[rip]
        paddusb xmm1, xmm8
        paddb   xmm0, xmm7
        pminub  xmm0, xmm1
        movdqa  xmm1, xmm0
        paddusb xmm0, xmm9
        pmovmskb        edx, xmm0
        test    edx, edx
        jne     .L10
        movdqa  xmm2, XMMWORD PTR .LC3[rip]
        movdqa  xmm10, XMMWORD PTR .LC4[rip]
.L8:
        add     rax, 16
        pmaddubsw       xmm1, xmm2
        movdqa  xmm0, xmm3
        pshufb  xmm1, xmm10
        movups  XMMWORD PTR [rdi], xmm1
        movdqu  xmm1, XMMWORD PTR [rax]
        add     rdi, 8
        paddb   xmm0, xmm1
        pand    xmm1, xmm5
        paddb   xmm1, xmm6
        psubusb xmm0, xmm4
        paddusb xmm1, xmm8
        paddb   xmm0, xmm7
        pminub  xmm0, xmm1
        movdqa  xmm1, xmm0
        paddusb xmm0, xmm9
        pmovmskb        edx, xmm0
        test    edx, edx
        je      .L8
.L10:
        bsf     rdx, rdx
        test    edx, edx
        je      .L9
        movsx   rdx, edx
        mov     ecx, 16
        sub     rcx, rdx
        add     rax, rdx
        movdqu  xmm0, XMMWORD PTR base16hex_simd_geoff(unsigned char*, unsigned char const*)::zero_masks[rcx]
        pandn   xmm0, xmm1
        movdqa  xmm1, XMMWORD PTR .LC3[rip]
        pmaddubsw       xmm0, xmm1
        pshufb  xmm0, XMMWORD PTR .LC4[rip]
        movups  XMMWORD PTR [rdi], xmm0
.L9:
        sub     rax, rsi
        ret

The source code is available at

https://github.com/lemire/Code-used-on-Daniel-Lemire-s-blog/tree/master/2023/07/26

[andrewtj]

The standard seems to require that bytes are encoded as pairs and that pairs are held together, so that the input is always an even number of characters. Are you saying that they don't do that? That'd be very strange.

Quite a few RR presentations don't require that pairs are held together. For instance DS is described in RFC 4034 as:

   The Digest MUST be represented as a sequence of case-insensitive
   hexadecimal digits.  Whitespace is allowed within the hexadecimal
   text.

[lemire]

If you know it is white space, then you can trim it out as a first pass. Super fast if AVX-512 is available, slower but still fast otherwise.

I can easily include that in the benchmark.

Again… what matters is whether I am benchmarking the right problem. We need to make sure that I do.

[lemire]

E.g., when I load the data, I can add a branch that checks for white space and prune it out dynamically as needed.

[k0ekk0ek]

Looks good @lemire!

Note that for simdzone if there's white space we need to stop as for contiguous strings that's a delimiter and the next set of characters may not be complete.

As for @aqrit's comment, I think he means we need to subtract 1 as A-F starts from 0x41, not 0x40, so we can use his range check (please correct me if I'm wrong). And, as for the multiplication, for base16 we can get away with a simple shift(?)

[lemire]

And, as for the multiplication, for base16 we can get away with a simple shift(?)

Last night, I managed to get...

  const __m128i t1 = _mm_srli_epi32(v, 4);
  const __m128i t3 = _mm_or_si128(t0, t1);
  const __m128i t4 = _mm_and_si128(
      t3, _mm_set1_epi16(0x00ff)); // keep just lower bytes in words

as opposed to...

  const __m128i t3 = _mm_maddubs_epi16(v, _mm_set1_epi16(0x0110));

I don't think it is win.