Async resume info has unaligned pointers

[MichalStrehovsky]

With runtime async, I'm hitting this assert in the object emitter:

runtime/src/coreclr/tools/Common/Compiler/ObjectWriter/ObjectWriter.cs

Line 446 in 248b9d8

Debug.Assert(reloc.Offset % _nodeFactory.Target.PointerSize == 0);

We guard against unaligned pointer-sized relocs because on Windows things get inefficient when a reloc straddles a page boundary.

RyuJIT is generating read only data blobs that have an unaligned pointer-size reloc in them. Example I'm looking at:

Emitting data sections: 160 total bytes
  section   0, size 16, RWD 0:	46 00 72 00 61 00 6d 00 65 00 20 00 30 00 3a 00 
  section 1, size 44, block relative addr
  BB23: 0x89f
  BB22: 0x86a
  BB25: 0x7a5
  BB28: 0x780
  BB24: 0x758
  BB21: 0x903
  BB26: 0x730
  BB29: 0x708
  BB27: 0x6e0
  BB21: 0x903
  BB30: 0x6b8
  section 2, size 80, async resume info
recordRelocation: 0000021E908D1794 (rw: 0000021E908D1794) => 4000000000952108, type 10 (relocType), delta 0
recordRelocation: 0000021E908D179C (rw: 0000021E908D179C) => 0000021E908D0FFB, type 10 (relocType), delta 0
  Resume=4000000000952108, FinalResumeIP=0000021E908D0FFB
recordRelocation: 0000021E908D17A4 (rw: 0000021E908D17A4) => 4000000000952108, type 10 (relocType), delta 0
recordRelocation: 0000021E908D17AC (rw: 0000021E908D17AC) => 0000021E908D10A5, type 10 (relocType), delta 0
  Resume=4000000000952108, FinalResumeIP=0000021E908D10A5
recordRelocation: 0000021E908D17B4 (rw: 0000021E908D17B4) => 4000000000952108, type 10 (relocType), delta 0
recordRelocation: 0000021E908D17BC (rw: 0000021E908D17BC) => 0000021E908D1159, type 10 (relocType), delta 0
  Resume=4000000000952108, FinalResumeIP=0000021E908D1159
recordRelocation: 0000021E908D17C4 (rw: 0000021E908D17C4) => 4000000000952108, type 10 (relocType), delta 0
recordRelocation: 0000021E908D17CC (rw: 0000021E908D17CC) => 0000021E908D120D, type 10 (relocType), delta 0
  Resume=4000000000952108, FinalResumeIP=0000021E908D120D
recordRelocation: 0000021E908D17D4 (rw: 0000021E908D17D4) => 4000000000952108, type 10 (relocType), delta 0
recordRelocation: 0000021E908D17DC (rw: 0000021E908D17DC) => 0000021E908D12C1, type 10 (relocType), delta 0
  Resume=4000000000952108, FinalResumeIP=0000021E908D12C1
  section 3, size 20, block relative addr
  BB252: 0xd5a
  BB257: 0xdbd
  BB262: 0xe12
  BB267: 0xe6e
  BB272: 0xed5

It's not visible from the JitDump for some reason, but these relocs are just:

• Full pointer relocation to the resumption stub of the method that owns the blob
• Full pointer relocation to the method that owns the blob (with some delta that is different for each entry)

Repeated 5 times.

I wonder if:

1. The pointers could be aligned
2. We track looking into the efficiency of this representation

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[jakobbotsch]

1. The pointers could be aligned

Internally the JIT does try to align these and the JIT should be asking for pointer size alignment for the data blocks. I assume there is a bug here related to the size of section 1 that comes before it.

2. We track looking into the efficiency of this representation

There is just one entry for every suspension point, and compared to the size of the code required for suspension/resumption it is not a lot.
We could store it as 4 byte relatives from the address of the resumption info, but it would mean that NAOT deviates from coreclr in the scheme (can't do that for coreclr).

Is it something we will be able to see the impact of more easily once we compile bigger apps?

[jkotas]

generating read only data blobs

I expect that these blobs will need to be generated into writeable data sections on non-Windows for NAOT. (Absolute pointers have to live in writeable data sections. It is a limitation of the binary formats on non-Windows.)

We track looking into the efficiency of this representation

This representation is a contract for diagnostic tools. It allows the diagnostic tools to capture the async stack using a simple fast algorithm. It is a tradeoff that we have arrived after a long discussion. If we were to diverge the contract between AOT and JIT, the savings would have to be fairly significant. This representation is only needed for runtime with diagnostic enabled. In principle, we can get rid of these blobs in AOT when we are building with diagnostics turned off.

[MichalStrehovsky]

I expect that these blobs will need to be generated into writeable data sections on non-Windows for NAOT. (Absolute pointers have to live in writeable data sections. It is a limitation of the binary formats on non-Windows.)

Yes, it will likely show up as a problem. I haven't looked at non-Windows or x86 yet.

Is it something we will be able to see the impact of more easily once we compile bigger apps?

We can measure it later and just take it from there, yep.

[MichalStrehovsky]

Yes, it will likely show up as a problem. I haven't looked at non-Windows or x86 yet.

Looks like we haven't had relocs in read-only jit-generated data blobs yet. If there's so many pointers in them we'll probably also want to switch them to the dehydrated data plan because pointer sized relocs on ELF have 24 bytes of metadata for each reloc.

[jakobbotsch]

Can you check if #121790 fixes the alignment problem?

[jkotas]

If there's so many pointers in them we'll probably also want to switch them to the dehydrated data plan

We want to keep as much data as possible R/O. We may want to introduce R/W data block in addition to the existing R/O data block on the JIT/EE interface.

[MichalStrehovsky]

Can you check if #121790 fixes the alignment problem?

I no longer see the alignment problem with that.

We want to keep as much data as possible R/O. We may want to introduce R/W data block in addition to the existing R/O data block on the JIT/EE interface.

This would require the JIT to know that ELF needs to place these in a R/W section (but e.g. PE is fine with it as-is). I wonder if we could also address this by placing the data blob in the same section as code everywhere.

[jkotas]

I wonder if we could also address this by placing the data blob in the same section as code everywhere.

It is not good for locality (data in code section are wasting code cache) and it does not work on platforms (like some game consoles) that map code as execution only.