loader: support ps5 SELF and validate ELF signatures

[anesr5]

Add support for loading Prospero/PS5 SELF executables.

Reference

• etaHEN dumper_utils.cpp

[Chelu97]

Tested this on my Demon's Souls dump (PPSA01341 v01.005) and it checks out.

On current main, loading eboot.bin dies with Input does not contain a valid ELF header because the PS5 magic 54 14 F5 EE isn't recognized. With 0x5414F5EE accepted (I tried an equivalent local patch), the eboot and the sce_module prx files (libc.prx, libSceNpCppWebApi.prx) all load fine: segments map through the SELF segment table, ~173k relocations get processed, 1173 import stubs resolve, and execution reaches libc.prx module init before crashing in the CPU backend — which is expected, that's separate work.

A few things I saw in that dump that back this change:

• The embedded ELF sits at 0x1A0 = 0x20 + segment_count(12) * 0x20, exactly where ParseLayout computes it.
• Ident bytes 4..11 (00 01 01 12 01 01 00 00) match the PS4 layout HasKnownLayout already validates — the 4-byte magic really is the only difference.
• All 12 segments have compressed_size == decompressed_size and no encrypted (& 0x2) or compressed (& 0x8) flags, so the payload is plaintext and nothing here needs decryption.

While checking this I ended up writing a small xunit fixture suite for SelfLoader (synthetic in-memory PS5/PS4/raw-ELF images, encrypted-segment error path, ident validation). Happy to open it as a follow-up PR once this lands so these paths stay covered.

[Kaotic]

Tested this on my Demon's Souls dump (PPSA01341 v01.005) and it checks out.

On current main, loading eboot.bin dies with Input does not contain a valid ELF header because the PS5 magic 54 14 F5 EE isn't recognized. With 0x5414F5EE accepted (I tried an equivalent local patch), the eboot and the sce_module prx files (libc.prx, libSceNpCppWebApi.prx) all load fine: segments map through the SELF segment table, ~173k relocations get processed, 1173 import stubs resolve, and execution reaches libc.prx module init before crashing in the CPU backend — which is expected, that's separate work.

A few things I saw in that dump that back this change:

* The embedded ELF sits at `0x1A0` = `0x20 + segment_count(12) * 0x20`, exactly where `ParseLayout` computes it.

* Ident bytes 4..11 (`00 01 01 12 01 01 00 00`) match the PS4 layout `HasKnownLayout` already validates — the 4-byte magic really is the only difference.

* All 12 segments have `compressed_size == decompressed_size` and no encrypted (`& 0x2`) or compressed (`& 0x8`) flags, so the payload is plaintext and nothing here needs decryption.

While checking this I ended up writing a small xunit fixture suite for SelfLoader (synthetic in-memory PS5/PS4/raw-ELF images, encrypted-segment error path, ident validation). Happy to open it as a follow-up PR once this lands so these paths stay covered.

I have do that in #89 and for loading ELF files :

In SelfLoader -> ParseLayout function

uint magic = BinaryPrimitives.ReadUInt32LittleEndian(imageData);
if (magic == 0x464C457F) // ELF magic
{
    return new LoadContext(
        IsSelf: false,
        ElfOffset: 0,
        SelfFileSize: (ulong)imageData.Length,
        SelfSegments: Array.Empty<SelfSegment>()
    );
}

[anesr5]

Tested this on my Demon's Souls dump (PPSA01341 v01.005) and it checks out.

On current main, loading eboot.bin dies with Input does not contain a valid ELF header because the PS5 magic 54 14 F5 EE isn't recognized. With 0x5414F5EE accepted (I tried an equivalent local patch), the eboot and the sce_module prx files (libc.prx, libSceNpCppWebApi.prx) all load fine: segments map through the SELF segment table, ~173k relocations get processed, 1173 import stubs resolve, and execution reaches libc.prx module init before crashing in the CPU backend — which is expected, that's separate work.

A few things I saw in that dump that back this change:

* The embedded ELF sits at `0x1A0` = `0x20 + segment_count(12) * 0x20`, exactly where `ParseLayout` computes it.

* Ident bytes 4..11 (`00 01 01 12 01 01 00 00`) match the PS4 layout `HasKnownLayout` already validates — the 4-byte magic really is the only difference.

* All 12 segments have `compressed_size == decompressed_size` and no encrypted (`& 0x2`) or compressed (`& 0x8`) flags, so the payload is plaintext and nothing here needs decryption.

While checking this I ended up writing a small xunit fixture suite for SelfLoader (synthetic in-memory PS5/PS4/raw-ELF images, encrypted-segment error path, ident validation). Happy to open it as a follow-up PR once this lands so these paths stay covered.

Thanks for testing this and for the details.

I hit the same issue on a Space Adventure Cobra dump (PPSA17337), which is what led me to this fix in the first place. Once the PS5 SELF magic was accepted, it started loading correctly.