String duplication when using to_string and to_string_pretty.

[maeln]

Description

Hello! This is a weird one. Either there is something I fundamentally didn't understood in the doc, or there is a very weird glitch.
So: I wanted to use simd-json to output a simple json. Here is the code:

use simd_json::json;

fn main() {
    let test = json!({
        "hello": "world",
        "anInt": 123,
    });

    let test_str = simd_json::to_string_pretty(&test).unwrap();
    println!("{test_str}");
}

The expected output should have been:

{
  "hello":  "world",
  "anInt": 213
}

But instead ALL the string (object keys or values) are duplicated. I get this:

{
  "hellohello": "worldworld",
  "anIntanInt": 123
}

I tried with a array of string, and I get duplicated string also:

[
  "testtest",
  "a_valuea_value"
]

Same thing if I just try to encode a single string.

My (ignorant) hunch

I am running this code on a VM inside a server. Running the exact same code on my laptop doesn't show this bug.

So, considering that this makes heavy use of simd instruction and that simd instruction are plateform specific, it seems like there is some f-ery going on with the feature detection / instruction used ? Probably caused by running inside a VM ?

Plateform information

Considering that this bug is very likely plateform specific, I will try to give as much info as I can.
I don't have direct access to the server the VM is running on so I will need to ask the admin for the server info.

OS (VM)

-> % lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 12 (bookworm)
Release:        12
Codename:       bookworm

-> % uname -a
Linux newworld 6.1.0-22-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.94-1 (2024-06-21) x86_64 GNU/Linux

CPU (as seen by the VM)

-> % cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 15
model           : 6
model name      : Common KVM processor
stepping        : 1
microcode       : 0x1
cpu MHz         : 3408.004
cache size      : 16384 KB
physical id     : 0
siblings        : 6
core id         : 0
cpu cores       : 6
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc nopl xtopology cpuid tsc_known_freq pni cx16 x2apic hypervisor lahf_lm cpuid_fault pti
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown bhi
bogomips        : 6816.00
clflush size    : 64
cache_alignment : 128
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 15
model           : 6
model name      : Common KVM processor
stepping        : 1
microcode       : 0x1
cpu MHz         : 3408.004
cache size      : 16384 KB
physical id     : 0
siblings        : 6
core id         : 1
cpu cores       : 6
apicid          : 1
initial apicid  : 1
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc nopl xtopology cpuid tsc_known_freq pni cx16 x2apic hypervisor lahf_lm cpuid_fault pti
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown bhi
bogomips        : 6816.00
clflush size    : 64
cache_alignment : 128
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor       : 2
vendor_id       : GenuineIntel
cpu family      : 15
model           : 6
model name      : Common KVM processor
stepping        : 1
microcode       : 0x1
cpu MHz         : 3408.004
cache size      : 16384 KB
physical id     : 0
siblings        : 6
core id         : 2
cpu cores       : 6
apicid          : 2
initial apicid  : 2
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc nopl xtopology cpuid tsc_known_freq pni cx16 x2apic hypervisor lahf_lm cpuid_fault pti
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown bhi
bogomips        : 6816.00
clflush size    : 64
cache_alignment : 128
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor       : 3
vendor_id       : GenuineIntel
cpu family      : 15
model           : 6
model name      : Common KVM processor
stepping        : 1
microcode       : 0x1
cpu MHz         : 3408.004
cache size      : 16384 KB
physical id     : 0
siblings        : 6
core id         : 3
cpu cores       : 6
apicid          : 3
initial apicid  : 3
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc nopl xtopology cpuid tsc_known_freq pni cx16 x2apic hypervisor lahf_lm cpuid_fault pti
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown bhi
bogomips        : 6816.00
clflush size    : 64
cache_alignment : 128
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor       : 4
vendor_id       : GenuineIntel
cpu family      : 15
model           : 6
model name      : Common KVM processor
stepping        : 1
microcode       : 0x1
cpu MHz         : 3408.004
cache size      : 16384 KB
physical id     : 0
siblings        : 6
core id         : 4
cpu cores       : 6
apicid          : 4
initial apicid  : 4
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc nopl xtopology cpuid tsc_known_freq pni cx16 x2apic hypervisor lahf_lm cpuid_fault pti
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown bhi
bogomips        : 6816.00
clflush size    : 64
cache_alignment : 128
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor       : 5
vendor_id       : GenuineIntel
cpu family      : 15
model           : 6
model name      : Common KVM processor
stepping        : 1
microcode       : 0x1
cpu MHz         : 3408.004
cache size      : 16384 KB
physical id     : 0
siblings        : 6
core id         : 5
cpu cores       : 6
apicid          : 5
initial apicid  : 5
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc nopl xtopology cpuid tsc_known_freq pni cx16 x2apic hypervisor lahf_lm cpuid_fault pti
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown bhi
bogomips        : 6816.00
clflush size    : 64
cache_alignment : 128
address sizes   : 40 bits physical, 48 bits virtual
power management:

Rust

-> % rustup show
Default host: x86_64-unknown-linux-gnu
rustup home:  /home/maeln/.rustup

stable-x86_64-unknown-linux-gnu (default)
rustc 1.82.0 (f6e511eec 2024-10-15)

Step to reproduce

1. Create a new binary project with cargo new test_json
2. Add the simd_json crate with cargo add simd-json
3. In the main.rs use the following code:

use simd_json::json;

fn main() {
    let test = json!({
        "hello": "world",
        "anInt": 123,
    });

    let test_str = simd_json::to_string_pretty(&test).unwrap();
    println!("{test_str}");
}

4. cargo run and get the following result:

{
  "hellohello": "worldworld",
  "anIntanInt": 123
}

[maeln]

Found some of the server info:
Host CPU: 8 x Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz (1 Socket)
Host Kernel: Linux 5.15.107-2-pve #1 SMP PVE 5.15.107-2 (2023-05-10T09:10Z)
VM Management system: Proxmox / pve-manager/7.4-3/9002ab8a

[maeln]

This seems to be indeed caused by the VM. In the proxmox VM hardware option, I changed the processors type from the generic "kvm64" to the more specific "Skylake-Client-noTSX-IBRS" which match the actual host CPU, and wouldn't you know it: The bug is fixed!

Since it does seems to be an issue with feature detection, I could close this ticket, but since I don't know much about feature detection, I will let any contributor decide if this issue should be closed, and/or reported else where (or if it is intended due to the VM cpu being set to kvm64 messing with the FD).

[Licenser]

This is definitely a bug. Thanks for the excellent report 😊. Let's leave it open since it should work without the user needing to change virtualization settings or even think about it. I'll dig into it and try to fix it in a way that doesn't require user actions