NativeAOT: Export scalar methods as LLVM IR for LTO

[hez2010]

LLVM supports available_externally function definitions, allowing their bodies to participate in optimization without emitting an out-of-line implementation. This allows native LTO callers to inline managed code while retaining the original RyuJIT codegen.

This PR adds --llvm-lto option to export eligible methods as LLVM IR using a simple translator. The translator shares scalar semantics with the type-preinitialization interpreter and checks native relocations for unmodeled runtime behavior.

This covers scalar methods on x64 and ARM64. The translator supports integer and floating-point operations, conversions, branches, switches, local addresses, and calls with known targets, including function pointers. EH, GC-dependent operations, potentially throwing operations, backedges, recursion, and unknown indirect calls are rejected for now because the necessary managed runtime semantics are not modeled.

Most code in the translator is carried from the existing type preinitializer.

Inlining native functions into managed callers are not supported for now, because this would require either importing native IR into RyuJIT before codegen, or emitting replacement bodies with the required managed ABI, GC, and unwind support, which is a complex topic. Maybe in the future we can make use of the outcome from NativeAOT-LLVM to extend the support for both directions.

For UCO support, the translator allocates the reverse P/Invoke frame and calls RhpReversePInvoke on entry, and calls RhpReversePInvokeReturn before every return. When translating a managed call inside a UCO body, it copies the IR as __llvm_lto_inline_<method> which uses available_externally alwaysinline, as well as its nested calls, for example:

declare void @RhpReversePInvoke(ptr)
declare void @RhpReversePInvokeReturn(ptr)

define available_externally i32 @ManagedEntry(i32 %x) {
  %frame = alloca [2 x ptr], align 8
  call void @RhpReversePInvoke(ptr %frame)
  %result = call i32 @__llvm_lto_inline_Helper(i32 %x)
  call void @RhpReversePInvokeReturn(ptr %frame)
  ret i32 %result
}

define available_externally i32 @__llvm_lto_inline_Helper(i32 %x) alwaysinline {
  %result = add i32 %x, 1
  ret i32 %result
}

Since available_externally doesn't emit the out-of-line definition, if the __llvm_lto_inline_ helper doesn't get inlined by LLVM, the linking will fail due to unresolved symbols. This ensures that we won't leave an unrecognized native frame during GC stack walk.

Also added some smoke tests.

Example:

In C#:

using System.Runtime.CompilerServices;
using System.Runtime.InteropServices;

internal static class Program
{
    [UnmanagedCallersOnly(
        EntryPoint = "ManagedAdd",
        CallConvs = new[] { typeof(CallConvCdecl) })]
    public static int Add(int x, int y) => unchecked(x + y);

    [DllImport("native",
        EntryPoint = "NativeFold",
        CallingConvention = CallingConvention.Cdecl)]
    private static extern int NativeFold(int x);

    private static int Main()
    {
        return NativeFold(41) == (42 ^ 0x1234) ? 100 : 1;
    }
}

In C:

extern int ManagedAdd(int x, int y);

int NativeFold(int x)
{
    return ManagedAdd(x, 1) ^ 0x1234;
}

In csproj:

<ItemGroup>
  <DirectPInvoke Include="native" />
  <NativeLibrary Include="native.lib" />
</ItemGroup>

On Windows x64, linking the same NativeAOT object and native bitcode archive:

Before:

NativeFold:
    sub     rsp, 0x28
    mov     edx, 1
    call    ManagedAdd
    xor     eax, 0x1234
    add     rsp, 0x28
    ret

ManagedAdd:
    push    rsi
    push    rbx
    sub     rsp, 0x38
    mov     ebx, ecx
    mov     esi, edx
    lea     rcx, [rsp + 0x28]
    call    RhpReversePInvoke
    add     ebx, esi
    lea     rcx, [rsp + 0x28]
    call    RhpReversePInvokeReturn
    mov     eax, ebx
    add     rsp, 0x38
    pop     rbx
    pop     rsi
    ret

After:

NativeFold:
    push    rsi
    push    rdi
    sub     rsp, 0x38
    mov     esi, ecx
    lea     rdi, [rsp + 0x28]
    mov     rcx, rdi
    call    RhpReversePInvoke
    inc     esi
    mov     rcx, rdi
    call    RhpReversePInvokeReturn
    xor     esi, 0x1234
    mov     eax, esi
    add     rsp, 0x38
    pop     rdi
    pop     rsi
    ret

[azure-pipelines]

Azure Pipelines:
Successfully started running 3 pipeline(s).
13 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[dotnet-policy-service]

Tagging subscribers to this area: @agocke, @dotnet/ilc-contrib
See info in area-owners.md if you want to be subscribed.

[jkotas]

What's the practical use for this?

translator allocates the reverse P/Invoke frame and calls RhpReversePInvoke on entry, and calls RhpReversePInvokeReturn

This will produce code that does not follow runtime ABI - no unwind/GC info for GC stackwalks, etc. I do not see how this can work. This feels like an experiment that won't be ever accepted into main.

[hez2010]

What's the practical use for this?

I'm trying to integrate NAOT into a game engine, so I would like to see if LTO that allows managed code (mostly reverse P/Invokes) to be inlined into the native callers can reduce the interop overhead and make things faster, as well as potentially enable whole program optimizations provided by the native linker.