Add support for more than SHN_LORESERVE sections

[filipnavara]

ELF files with more than SHN_LORESERVE (0xff00) sections need special handling for the number of sections and link to section header string table. Both of these fields are unrepresentable using the ushort fields in the ELF header and they are instead written in fields of the NULL section.

---

If the number of sections is greater than or equal to SHN_LORESERVE (0xff00), e_shnum has the value SHN_UNDEF (0) and the actual number of section header table entries is contained in the sh_size field of the section header at index 0 (otherwise, the sh_size member of the initial entry contains 0).

---

e_shstrndx

This member holds the section header table index of the entry associated with the section name string table. If the file has no section name string table, this member holds the value SHN_UNDEF.

If the index of section name string table section is larger than or equal to SHN_LORESERVE (0xff00), this member holds SHN_XINDEX (0xffff) and the real index of the section name string table section is held in the sh_link member of the initial entry in section header table. Otherwise, the sh_link member of the initial entry in section header table contains the value zero.

[filipnavara]

I will try to add a unit test for this, at least some basic round-trip.

(Switched to draft since the reading code is incomplete.)

[filipnavara]

Another broken case that I found is that large section numbers in symbol table as truncated. There's an escape mechanism which uses SHN_XINDEX and another section. I think I'll need to implement that.

[filipnavara]

I have an unpushed fix (+ unit test) for the first two errors but I am not really convinced whether it's going in the good direction...

Do you think it's better to expose this format quirk in ElfNullSection and deal with it in the higher layers? Or should ElfReader/ElfWriter just hide this detail?

[xoofx]

Do you think it's better to expose this format quirk in ElfNullSection and deal with it in the higher layers? Or should ElfReader/ElfWriter just hide this detail?

I haven't looked in the detail, but it might be error prone to expose it so I would think that hiding it would be better, but I would think that would be more in ElfObjectFile than in the basic reader/writers themselves.

There is also the concept of ElfShadowSection and it is used e.g for ElfProgramHeaderTable, and we hide managing this section in ElfObjectFile.

[filipnavara]

There already needs to be a special case in the reader/writer. In the reader I need to read the NULL section header and only then I get the number of sections, and possibly the correct section index of string table. Since both of these are local variables in the reader it's relatively easy to handle it there with no changes to the model itself. That also makes the verification methods easier to reason about. I originally wrote some checks in ElfNullSection.Verify only to realize that they are not really running at the right point. In the test case I add 65k sections and then call Verify followed by UpdateLayout. The checks, as I first wrote them, would fail before ElfObjectFile.UpdateLayout is called.

Similarly, for the writer, I originally started with a design that modified ElfNullSection properties in ElfObjectFile.UpdateLayout. While that does work it's equally easy to do the same transformation right within ElfWriter with no changes in the model.

I feel like this is similar to hiding the fact that sections with 32-bit and 64-bit headers are different in the file format. In this case the quirk is that the large numbers are just written into different place. The place just happens to be part of ElfNullSection section header.

I'll try to get something ready to review this week and explain the design decisions. If necessary I can always rewrite it later. I still need to figure out how to reasonably expose the symbol table with large section numbers.

[filipnavara]

I came up with something that works. Here's the gist:

• The file format for large section numbers / indexes in the ELF header is to put them into sh_link/sh_size fields of the NULL section that is mandatory first section in the file. Read/write that transparently in ElfReader/ElfWriter and pretend the Link/Size fields of ElfNullSection in the model are still zero (makes overlap checks consistent). Enforce the NULL section requirement as additional condition in ElfObjectFile.Verify. VisibleSectionCount represents the real section count, which can now be higher than 0xff00.
• ElfReader.ResolveLink now resolves any non-zero indices, even in the SHN_LORESERVE to SHN_HIRESERVE range. The special values in this range have different meaning in different contexts. In the Info/Link fields the reserved values can still refer to sections with high section indexes. The special meaning like SHN_ABS (abstracted through ElfSectionLink.IsSpecial) is now handled in ElfSymbolTable.
• ElfReader.ResolveLink has a new shortcut code path for resolving section indexes. This avoids O(n^2) complexity when reading ELF file with lot of links (section to section, or symbol table to section).
• ElfSymbolTableSectionHeaderIndices is added to represent the extended 32-bit section index for a symbol table. This has a Link set to point to ElfSymbolTable, and it's automatically populated in UpdateLayout method. Conversely, when reading ELF file it automatically resolves the high section indexes and puts them back into the entries in ElfSymbolTable. Accessing and producing the entries referring to high section indexes in ElfSymbolTable is thus mostly transparent.
• ElfSymbolTable now writes SHN_XINDEX for high section numbers into the file. The real indexes are in ElfSymbolTableSectionHeaderIndices.
• ElfSymbolTable.Verify verifies that corresponding ElfSymbolTableSectionHeaderIndices exists if there are any high section indexes in the table. This ensures that the written structure would not lose the information. The user is still responsible for creating the ElfSymbolTableSectionHeaderIndices which is the only non-transparent thing in the model. With some effort it would likely be possible to make it more transparent for the API users if necessary (in future?).

[xoofx]

Thanks a lot for this!