Simplify ImageBuilder and container image publishing infrastructure #2279

This epic tracks high-level objectives for simplifying ImageBuilder's container image publishing infrastructure.

Primary goal: make image builds and publishing as fast as possible, so that:

• Security fixes can be published as quickly as possible.
• Coding agents can iterate as quickly as possible.

I am proposing that we accomplish this by making ImageBuilder simpler and more flexible - essentially offering more composable pieces of build/publishing infrastructure rather than one monolithic pipeline setup like we have today.

This will allow downstream repos to tailor their image publishing pipelines to their specific needs/requirements.
In turn, this will reduce the amount of time required to ship security fixes.

High-level ideas

Remove build/test matrix generation

Instead of generating a one-size-fits-all build matrix in ImageBuilder, simply allow downstream repos to construct their own pipelines with whatever build/test matrix they need.

• Issue: ⚠️ needs issue
• Why:
  • Makes publishing faster by removing two sequential jobs (build/test matrix gen) from each pipeline.
  • Simplifies manifest model by removing the need for awkward build/test matrix declarations/workarounds for things like distroless images.
  • Allows pipelines in downstream repos to be constructed much more flexibly. For example, a repo with just one or two images might decide to run build / test in the same job instead of separate stages.
• Related: Simplify build matrices #1178
• Follow-up: Simplify pipelines/pipeline templates.

Simplify / improve manifest and image info models

• Issue: ⚠️ needs issue
• Why:
  • Reduce duplication and chance of mistakes/typos in large manifests like dotnet-docker
  • Make references to images in manifest and image info unambiguous.
• Related: Editing the manifest.json file by hand is time-consuming and error-prone dotnet-docker#4704

Combine publishing steps into one command

• Issue: ⚠️ needs issue
• Why:
  • Simplifies testing / validation by moving logic out of Azure Pipelines and into C#
  • Will make publishing faster

Create an MSTest helper library for testing container images

• Issue: Use manifest.json/image-info.json to determine which tests to run. dotnet-docker#5337
• Why:
  • Make it easier to write reliable tests for images built with ImageBuilder. Less chance of images silently skipping tests.
  • Consume ImageBuilder's build outputs to decide which tests to run.
  • Delete custom code from downstream repos.
• Depends on Refactor common logic out of ImageBuilder into a shared library. #314

Remove templating from ImageBuilder

• Issue: Dockerfile templating should be removed from ImageBuilder #1748
• Why:
  • Makes ImageBuilder simpler, because not all consuming repos need templates.
  • Opportunity to make templating (much) faster and more flexible in consuming repos.