RFC: Distribute native modules for easy installation

Description

One of MicroPython's superpowers is the ability to install and use additional libraries without needing to rebuild and flash the device. The native module support in MicroPython makes installing at runtime also possible for modules that are implemented in C.

There are a couple of C modules in MicroPython repo, and in micropython-lib that can or would be very useful to distribute as native modules. To make that happen, we would need build them (for relevant architectures), and to publish them on HTTP server. It would then be possible to install them using mip - and this could be documented.

The immediate candidates would be btree, deflate, re from examples/natmod/. But in general this mechanism is useful for any module which is useful to have optional, and which can be implemented within the restrictions of natmod.

I propose adding the build and distribution setup to Github Actions. Such that on every new tag of MicroPython, natmod .mpy files would be placed onto HTTP file-server (Github Pages etc), in a versioned manner. Might also want to have "latest" (latest release) and "nightly" folders.

If this addition is considered welcome, then I can provide a pull request.

Related: PRs for making mip install for natmods more ergonomic:

Code Size

This feature could be used to enable smaller code size of default images, by having more modules be installable at runtime instead of being baked into firmware image.

Implementation

I intend to implement this feature and would submit a Pull Request if desirable

Code of Conduct

Yes, I agree

I hope that this will make more MicroPython users aware of native modules and how they can be useful. And that more MicroPython library developers consider creating third-party libraries using natmods (especially instead of / in addition to extmod). And that more dogfooding will trigger us to improve the developer UX when it comes to developing, distributing and using native modules.

Related to #1140 / #19478: URL templating ({MPY_ARCH}_{MPY_VERSION}) solves single-file installs, but native modules are frequently not single-file - a natmod .mpy often ships alongside a pure-Python wrapper module (e.g. this is exactly the layout of deflate/btree-style bindings, and of third-party natmods like a7p/bclibc), so a whole-package install needs multiple files resolved per-arch, not one.

package.json already has hashes/urls for multi-file packages (both flat top-level lists of [target_path, source]) - it just has no per-arch/per-mpy-version axis. Two ways to add one, instead of requiring a separate package.json per arch (which duplicates the whole file for a one-line URL change):

A. Auto-resolve - let {MPY_ARCH}/{MPY_VERSION} substitution apply inside urls values too, not just to raw one-shot file installs. mip does a blind string substitution, no matching logic involved:

{
  "urls": [
    ["mymodule.py", "mymodule.py"],
    ["mymodule.mpy", "{MPY_ARCH}_{MPY_VERSION}/mymodule.mpy"]
  ]
}

Minimal change (reuses #1140 's substitution as-is), but only works for urls - hashes entries aren't URLs, nothing to substitute into. Requires the server-side layout to actually mirror the {arch}_{version} path convention.

B. Filter format - an optional field per entry that mip evaluates against the device, so one manifest mixes arch-independent and arch-gated sources explicitly:

{
  "urls": [
    ["mymodule.py", "mymodule.py"],
    ["mymodule.mpy", "https://.../armv7m/mymodule.mpy", {"arch": "armv7m", "mpy_version": "6.3"}],
    ["mymodule.mpy", "https://.../armv6m/mymodule.mpy", {"arch": "armv6m", "mpy_version": "6.3"}]
  ]
}

Entries without the field install unconditionally (backward compatible); entries with it only install if arch/mpy_version match sys.implementation._mpy. Covers hashes entries the same way, and doesn't require the source URLs to follow any particular naming/path convention - the manifest is the single source of truth for what matches what.

(A) is smaller and reuses existing machinery, at the cost of coupling to a URL convention. (B) is more general - explicit matching, not string-shape - but is a real schema/logic addition, not just a substitution rule. Both are orthogonal to #1140, not a replacement - that one covers "one URL, no manifest"; either of these covers "one manifest, several arch-gated sources."

Happy to draft the _install_json change for whichever direction looks preferable.

A GitHub Pages-shaped distribution (versioned folders, {MPY_ARCH}_{MPY_VERSION} path convention) isn't generally applicable - it only covers the raw-URL install path. Packages published through the actual index protocol (micropython.org/pi/v2) address files by content hash ({index}/file/{hash[:2]}/{hash}), not by a human-readable arch/version path - there's nothing for a folder-naming convention to hook into there, so this only ever helps third-party projects willing to stand up and maintain their own static site with that exact layout, not packages going through the index mip already knows how to talk to.

I'd rather see this solved at the protocol/schema level: extend package.json itself to support native precompiled variants as first-class metadata (see my comment above for two concrete shapes), so it works uniformly whether a package resolves via hashes (index-hosted) or urls (self-hosted), instead of depending on which static host a given project happens to use.

@jonnor

@jonnor @agatti

Simpler variant of (B): instead of separate arch/mpy_version fields, match on the raw sys.implementation._mpy integer directly. It already packs version + sub-version + native arch (+ arch-flags, when present) into one value, so an exact-match check covers every compatibility dimension the loader itself cares about, with no decoding on either side:

{
  "urls": [
    ["mymodule.py", "mymodule.py"],
    ["mymodule.mpy", "https://.../mymodule.mpy", 1543]
  ]
}

for entry in package_json.get("urls", ()):
    target_path, url = entry[0], entry[1]
    tag = entry[2] if len(entry) > 2 else None
    if tag is not None and tag != sys.implementation._mpy:
        continue

Same idea applies to hashes entries. Trade-off vs named arch/mpy_version fields: the manifest is less human-readable (an opaque int instead of "armv7m"/"6.3"), but the check itself is ~3-4 lines and there's no arch-table to keep in sync with mpyfiles.rst.

Simpler variant of (B): instead of separate arch/mpy_version fields, match on the raw sys.implementation._mpy integer directly. It already packs version + sub-version + native arch (+ arch-flags, when present) into one value, so an exact-match check covers every compatibility dimension the loader itself cares about

An exact match won't work for arch flags. See #19478 (comment)

Moreover, I believe doing this is also going to lock you with MPY v6.3 and the current sys.implementation._mpy scheme. The former may appear some time in the future, but the latter does not have any guarantees of staying the same going forward either. You may version your own schema, and now module authors have to maintain two sets of manifests, obsoleting existing modules that aren't immediately updated.

Maybe moving all the complexity into mpremote and keeping the overhead on module authors as minimal as it can be is the way to go here. Let authors publish once, and then act behind the scenes to work with the current interpreter if possible: a MPYv7 interpreter maybe won't load a v6.3 binary, but a v6.3 binary with a future architecture and custom arch flags encoding can be handled by mpremote much more easily.

@agatti

An exact match won't work for arch flags. See #19478 (comment)

@agatti

Re: MPY_ARCH/MPY_VERSION feeling "magical" (@Josverl) — a couple of data points that might help ground this:

_mpy is only exposed when MICROPY_PERSISTENT_CODE_LOAD is set, and that's the same flag gating the entire .mpy load path in persistentcode.c#L72-L551 — MICROPY_PERSISTENT_CODE_LOAD_NATIVE has no independent entry point, it's only checked inside that same loader. So any port capable of loading a natmod necessarily exposes _mpy with a correct, self-consistent arch nibble (derived from the same #ifdef cascade as the native emitter itself, see persistentcode.h#L37-L90) — there's no config where natmod loads but _mpy arch is wrong/absent.
For disambiguating boards that share arch+abi but differ in libc/toolchain, sys.implementation._build (1.25+, gated on MICROPY_BOARD_BUILD_NAME) and the platform module (extmod/modplatform.c, gated on MICROPY_PY_PLATFORM) already expose platform.libc_ver() and, on RISC-V, platform.processor() (reads MISA CSR directly). Neither is universal, but both are usable as optional extra keys beyond MPY_ARCH/MPY_VERSION for distributors needing finer-grained variants — worth a mention in the docs alongside the two placeholders.

@agatti

Moreover, I believe doing this is also going to lock you with MPY v6.3 and the current sys.implementation._mpy scheme. The former may appear some time in the future, but the latter does not have any guarantees of staying the same going forward either. You may version your own schema, and now module authors have to maintain two sets of manifests, obsoleting existing modules that aren't immediately updated.

Maybe moving all the complexity into mpremote and keeping the overhead on module authors as minimal as it can be is the way to go here. Let authors publish once, and then act behind the scenes to work with the current interpreter if possible: a MPYv7 interpreter maybe won't load a v6.3 binary, but a v6.3 binary with a future architecture and custom arch flags encoding can be handled by mpremote much more easily.

@agatti

Today mip parses both lists via fixed-arity tuple unpacking:

That's the only reason a 3rd element would be breaking. Swap both to explicit indexing —

for entry in package_json.get("urls", ()):
    target_path, url = entry[0], entry[1]
    tag = entry[2] if len(entry) > 2 else None

@agatti

@agatti

So I did make patched version and it seems like works.

Given that any changes to the mip can get at least in the next release, older versions would still not allow the use of the updated scheme proposed by @jonnor. Therefore, my offer seems to be no worse

>>> import emip
>>> emip.install('urequests')
Target: arch=x86 abi=6.3 byteorder=little build=standard
Copying: /root/.micropython/lib/requests/__init__.mpy
Copying: /root/.micropython/lib/urequests.mpy
Done
True
>>>

# emip - Extended MicroPython package installer
# Adds: per-arch / per-mpy-abi variant resolution with explicit _mpy match
# checking, on top of the same wire protocol shape as micropython-lib's mip.
#
# Reference (structure/behaviour this is modeled on):
# https://github.com/micropython/micropython-lib/blob/master/micropython/mip/mip/__init__.py
#
# package.json shape expected by emip (superset of plain mip's):
#
# {
#   "variants": [
#     {
#       "arch": "rv32imc",              # required - must match MPY_ARCH
#       "abi": "6.3",                    # required - "major.minor", minor optional
#       "byteorder": "little",           # optional - must match sys.byteorder if present
#       "hashes": [["_foo.mpy", "sha256prefix..."]],
#       "urls":   [["_foo.mpy", "https://.../foo-rv32imc-6.3.mpy"]]
#     },
#     ...
#   ],
#   "deps": [["some-pure-py-dep", ""]]   # optional, installed regardless of variant
# }
#
# MPY_ARCH naming follows py/persistentcode.h's MP_NATIVE_ARCH_* table.

import sys

_PACKAGE_INDEX = "https://micropython.org/pi/v2"
_CHUNK_SIZE = 128

_HOSTS = {
    "codeberg:": "codeberg.org/api/v1/repos/{}/{}/raw/{p}?ref={}",
    "github:": "raw.githubusercontent.com/{}/{}/{}/{p}",
    "gitlab:": "gitlab.com/{}/{}/-/raw/{}/{p}",
}
_ALLOWED_URL_PREFIXES = ("http://", "https://", "codeberg:", "github:", "gitlab:")

# Index must match py/persistentcode.h MP_NATIVE_ARCH_* ordering (0 = NONE).
_ARCH = (
    None, "x86", "x64", "armv6", "armv6m", "armv7m", "armv7em",
    "armv7emsp", "armv7emdp", "xtensa", "xtensawin", "rv32imc", "rv64imc",
)


# --------------------------------------------------------------------------
# Local target resolution
# --------------------------------------------------------------------------

def local_info():
    """Return a dict describing this device's native-code compatibility.

    Raises RuntimeError if this build cannot load persistent .mpy at all
    (sys.implementation has no _mpy field) - in which case natmod
    distribution is not applicable here in the first place.
    """
    mpy = getattr(sys.implementation, "_mpy", None)
    if mpy is None:
        raise RuntimeError(
            "sys.implementation._mpy is absent - this build has no "
            "MICROPY_PERSISTENT_CODE_LOAD support, natmod install is not possible"
        )

    info = {
        "arch": _ARCH[(mpy >> 10) & 0x0F],
        "abi_major": mpy & 0xFF,
        "abi_minor": (mpy >> 8) & 3,
        "byteorder": sys.byteorder,
        "build": getattr(sys.implementation, "_build", None),
    }
    info["abi"] = "{}.{}".format(info["abi_major"], info["abi_minor"])
    return info


def _abi_matches(required, local_major, local_minor):
    # Accept "6" (major only) or "6.3" (major.minor, exact).
    parts = required.split(".")
    req_major = int(parts[0])
    if req_major != local_major:
        return False
    if len(parts) > 1:
        return int(parts[1]) == local_minor
    return True


def variant_matches(variant, local=None):
    """The core _mpy match check: does this manifest variant run here?"""
    local = local or local_info()

    if local["arch"] is None:
        # This build has no native emitter at all - only bytecode-only
        # variants (arch omitted/None in the manifest) could ever apply.
        return variant.get("arch") is None

    if variant.get("arch") != local["arch"]:
        return False
    if not _abi_matches(variant.get("abi", "0"), local["abi_major"], local["abi_minor"]):
        return False
    if "byteorder" in variant and variant["byteorder"] != local["byteorder"]:
        return False
    if "build" in variant:
        wanted = variant["build"]
        got = local["build"]
        if got is None or not got.startswith(wanted):
            return False
    return True


def select_variant(package_json, local=None):
    local = local or local_info()
    for variant in package_json.get("variants", ()):
        if variant_matches(variant, local):
            return variant
    return None


# --------------------------------------------------------------------------
# Transfer primitives (same shape as mip's, kept local so emip has no hard
# dependency on mip's private internals, which are not a stable API)
# --------------------------------------------------------------------------

def _ensure_path_exists(path):
    import os

    split = path.split("/")
    if not split[0]:
        split.pop(0)
        split[0] = "/" + split[0]

    prefix = ""
    for i in range(len(split) - 1):
        prefix += split[i]
        try:
            os.stat(prefix)
        except OSError:
            os.mkdir(prefix)
        prefix += "/"


def _chunk(src, dest):
    buf = memoryview(bytearray(_CHUNK_SIZE))
    while True:
        n = src.readinto(buf)
        if n == 0:
            break
        dest(buf if n == _CHUNK_SIZE else buf[:n])


def _check_exists(path, short_hash):
    import binascii
    import hashlib
    import os

    try:
        with open(path, "rb") as f:
            hs256 = hashlib.sha256()
            _chunk(f, hs256.update)
            existing = str(binascii.hexlify(hs256.digest())[: len(short_hash)], "utf-8")
            return existing == short_hash
    except OSError:
        return False


def _rewrite_url(url, ref=None):
    for provider, url_format in _HOSTS.items():
        if not url.startswith(provider):
            continue
        components = url[len(provider):].split("/")
        return _ALLOWED_URL_PREFIXES[1] + url_format.format(
            components[0], components[1], ref or "HEAD", p="/".join(components[2:])
        )
    return url


def _download_file(url, dest):
    import requests

    response = requests.get(url)
    try:
        if response.status_code != 200:
            print("Error", response.status_code, "requesting", url)
            return False
        print("Copying:", dest)
        _ensure_path_exists(dest)
        with open(dest, "wb") as f:
            _chunk(response.raw, f.write)
        return True
    finally:
        response.close()


# --------------------------------------------------------------------------
# Install logic
# --------------------------------------------------------------------------

def _install_entries(entries, index, target, ref, key):
    ok = True
    for target_path, value in entries.get(key, ()):
        fs_target_path = target + "/" + target_path
        if key == "hashes":
            if _check_exists(fs_target_path, value):
                print("Exists:", fs_target_path)
                continue
            url = "{}/file/{}/{}".format(index, value[:2], value)
        else:  # "urls"
            url = value
            is_full = any(url.startswith(p) for p in _ALLOWED_URL_PREFIXES)
            if not is_full:
                url = value  # relative urls are not supported for variant blocks
        if not _download_file(_rewrite_url(url, ref), fs_target_path):
            print("File not found:", target_path, value)
            ok = False
    return ok


def _fetch_package_json(package, index, ref, local):
    """Resolve `package` (bare name or URL) to a parsed package.json dict."""
    import requests

    if any(package.startswith(p) for p in _ALLOWED_URL_PREFIXES):
        url = package
        if not url.endswith(".json"):
            url = url.rstrip("/") + "/package.json"
    else:
        # bare name -> official mip index convention:
        # {index}/package/{mpy_version}/{package}/{version}.json
        mpy_version = local["abi_major"] if local["arch"] is not None else "py"
        version = ref or "latest"
        url = "{}/package/{}/{}/{}.json".format(index, mpy_version, package, version)

    response = requests.get(_rewrite_url(url, ref))
    try:
        if response.status_code != 200:
            print("Package not found:", url)
            return None
        return response.json()
    finally:
        response.close()


def install(package, index=None, target=None, ref=None, deps=True):
    """Install a variant-aware package.

    package - bare name, or a URL/`github:`-style ref pointing at a
              variant-schema package.json (see module docstring)
    ref     - git ref/branch/tag when `package` is a URL form; unused
              (but harmless) for bare index-style names
    """
    if not target:
        for p in sys.path:
            if not p.startswith("/rom") and p.endswith("/lib"):
                target = p
                break
        else:
            print("Unable to find lib dir in sys.path")
            return False

    index = (index or _PACKAGE_INDEX).rstrip("/")

    local = local_info()
    print("Target: arch={arch} abi={abi} byteorder={byteorder} build={build}".format(**local))

    package_json = _fetch_package_json(package, index, ref, local)
    if package_json is None:
        return False

    if "variants" in package_json:
        variant = select_variant(package_json, local)
        if variant is None:
            print("No matching variant for arch={arch} abi={abi} in {pkg}".format(
                pkg=package, **local
            ))
            return False
    else:
        # Plain mip-style package.json (no per-arch variants) - install as-is.
        variant = package_json

    ok = _install_entries(variant, index, target, ref, "hashes")
    ok = _install_entries(variant, index, target, ref, "urls") and ok

    if deps:
        for dep, dep_ref in package_json.get("deps", ()):
            if not install(dep, index, target, dep_ref, deps):
                ok = False

    print("Done" if ok else "Package may be partially installed")
    return ok

@agatti

Re: exact match not working for arch flags (@agatti) — fair catch, and worth being precise about why: the flags byte is additive capability, not identity, so == over the whole int is the wrong operation regardless of which field-shape (named vs raw) wins. Fix is just splitting the comparison, not abandoning the raw-int approach:

_MPY_BASE_MASK = 0xFFFF  # version + sub-version + arch (bits 0-15)

def mpy_tag_matches(required, local_mpy):
    if (required & _MPY_BASE_MASK) != (local_mpy & _MPY_BASE_MASK):
        return False
    req_flags = required >> 16
    loc_flags = local_mpy >> 16
    return (req_flags & loc_flags) == req_flags  # required ⊆ available

For disambiguating boards that share arch+abi but differ in libc/toolchain, sys.implementation._build (1.25+, gated on MICROPY_BOARD_BUILD_NAME) and the platform module (extmod/modplatform.c, gated on MICROPY_PY_PLATFORM) already expose platform.libc_ver() and, on RISC-V, platform.processor() (reads MISA CSR directly). Neither is universal, but both are usable as optional extra keys beyond MPY_ARCH/MPY_VERSION for distributors needing finer-grained variants — worth a mention in the docs alongside the two placeholders.

I'm afraid it's not entirely like you mention here.

If you look at platform.processor you'll see that it will read up to 26 bits, as per the MISA specification. That's great and all, but then you should look at which extensions are currently supported right now: Zba, Zcmp, and hopefully Zcmt (see #19472 for the latter, I'll sort the cache issue out as soon as customs find my ESP32P4), with Zcb to follow. They're not in MISA, and there's no ratified method for sub-extensions discovery - the only one that went past the initial drafting stage was abandoned years ago as far as I remember, and it only targeted RVA profiles anyway.

So, to support the current state of things, what you propose won't work. Let alone adding support for F and D extensions along with their associated sub-extensions (v7, maybe), which you can look up in MISA, but then you'll quickly find out that Zcmp and Zcmt are not compatible with Zcd (they share opcodes).

The fact that Zba, Zcmp, and others are not enabled by default does not mean that nobody will use them. We have to support the lowest common denominator here, which is ESP32C3-class cores (C3/C5/C6/C61). The RP2350 supports Zba, Zcmp, and not Zcmt, while the ESP32P4/ESP32S31 supports Zcmp, Zcmt, but not Zba.

RISC-V extensions are a mess, but since there are no ratified profiles for RV32 that's the way things are.

@agatti

@agatti @jonnor
I propose you to try my sollution by yourself. It is simple, uses expected APIs, works fine and adds only ~20 lines of code.
Also it is compatible with mip indexes, github releases, github pages or any other distribution layout.

PoC simple test with real-world natmod package

Clone, checkout patched branches and build

git clone --branch feat/mip-natmod-support https://github.com/o-murphy/micropython.git
cd micropython/ports/unix
make -C ../../mpy-cross
make submodules
# Checkout patched micropython-lib
git -C ../../lib/micropython-lib fetch https://github.com/o-murphy/micropython-lib.git feat/mip-natmod-support
git -C ../../lib/micropython-lib checkout FETCH_HEAD
make

Test suite

./build-standard/micropython

>>> import mip
>>> mip.install("https://github.com/ballistics-lab/micropython-bclibc/releases/download/v1.2.0/package.json")
Installing https://github.com/ballistics-lab/micropython-bclibc/releases/download/v1.2.0/package.json to /home/murphy/.micropython/lib
Copying: /home/murphy/.micropython/lib/tiny_bclibc.mpy
Done
>>> import tiny_bclibc; tiny_bclibc.version()
'1.2.0-dp'
>>>

Simple test for pre-built firmware on board via bootstrap

>>> import mip
>>> mip.install("github:ballistics-lab/micropython-bclibc/tools/nmip.py")
Downloading https://raw.githubusercontent.com/ballistics-lab/micropython-bclibc/refs/heads/main/tools/nmip.py to /home/murphy/.micropython/lib
Copying: /home/murphy/.micropython/lib/nmip.py
Done
>>> import nmip as mip
>>> mip.install("https://github.com/ballistics-lab/micropython-bclibc/releases/download/v1.2.0/package.json")
Installing https://github.com/ballistics-lab/micropython-bclibc/releases/download/v1.2.0/package.json to /home/murphy/.micropython/lib
Copying: /home/murphy/.micropython/lib/tiny_bclibc.mpy
Done
>>>
>>> import tiny_bclibc; tiny_bclibc.version()
'1.2.0-dp'
>>>

Building `package.json`

To compile package.json from .mpy build artifacts I used this simple script

@agatti

@agatti

Everything you wrote about platform.processor()/MISA is correct, but I think it's answering a narrower reading than what I meant - that line was about an optional, additional disambiguator for libc/toolchain differences on top of the base scheme, not a proposal to use MISA for RISC-V extension discovery. Worth separating the two, because the extension problem you're describing doesn't actually apply to the _mpy-tag scheme itself.

The arch_flags bits (16+) in sys.implementation._mpy never come from MISA or any runtime CSR read. They're a straight copy of a build-time constant:

c // py/asmrv32.h #define MICROPY_RV32_EXTENSIONS \ ((MICROPY_EMIT_RV32_ZBA ? RV32_EXT_ZBA : 0) | \ (MICROPY_EMIT_RV32_ZCMP ? RV32_EXT_ZCMP : 0))

i.e. whatever the firmware was actually compiled to require, not something discovered from the running CPU. So:
No MISA involved anywhere, so "Zba/Zcmp/Zcmt aren't in MISA and there's no ratified sub-extension discovery" doesn't bite here - the firmware already knows exactly what it was built with.
RP2350 (Zba+Zcmp, no Zcmt) vs ESP32P4 (Zcmp+Zcmt, no Zba) are just two different bitmasks. The match is a subset check (tag_flags & device_flags == tag_flags), so each board's own _mpy value already disambiguates them correctly today, independent of any discovery mechanism.
When Zcmt lands (py/asmrv32: Add support for Zcmt opcodes. #19472) and presumably gets its own RV32_EXT_ZCMT bit next to RV32_EXT_ZBA/RV32_EXT_ZCMP in that same enum, the matching logic doesn't need to change at all - it's a generic bitmask-subset comparison, not a hardcoded list of extensions to check.

So I'd push back on "what you propose won't work" as applied to the core _mpy scheme specifically - it was never relying on runtime extension discovery in the first place, RISC-V or otherwise. The platform.processor() mention was a separate, optional idea for a problem the _mpy tag genuinely can't solve (same arch+abi+extensions, different libc/toolchain) - happy to drop it from the discussion entirely if it's a distraction, since it's not load-bearing for the actual proposal.

Pushed a small follow-up: _mpy_tag_ok now only requires an exact sub-version match when the tag's arch nibble is non-zero, matching mp_raw_code_load()'s own behavior in py/persistentcode.c (sub-version is only checked for native code; ignored for plain bytecode).

Side effect worth calling out: this also covers plain (non-native) .mpy in a single manifest, across mpy major versions. A tag doesn't have to be a full native variant - a bare major-version integer is a valid tag too (arch nibble is 0), e.g.:

{
  "urls": [
    ["mymodule.mpy", "mymodule_mpy5.mpy", 5],
    ["mymodule.mpy", "mymodule_mpy6.mpy", 6],
    ["mymodule.mpy", "mymodule_armv7m.native.mpy", 5894]
  ]
}

One manifest, no separate per-version paths: a board with an old mpy-5 build gets the first entry, a current mpy-6 build gets the second, an armv7m board with native support gets the third - same mechanism, same field, no special-casing needed on the mip side for bytecode vs. native entries.

@Josverl

Side effect worth calling out: this also covers plain (non-native) .mpy in a single manifest, across mpy major versions.

If I recall correctly, py/persistentcode.c checks both major and minor versions when it comes to load/reject a file - so if that's indeed what it does you'll have to encode the version differently (and there's no guarantee that MPYv7 will restrict the minor version number to just two bits).

My main concern in relying on sys.implementation._mpy this way is that such a value has no guarantees of stability in its format or encoding. It wouldn't surprise me if it would get a overhaul in MicroPython v2. It is true that it hasn't changed in a while, but as the name suggests it is both implementation-specific and a (documented) private implementation detail.

I think making the platform information metadata encoding more explicit and as far as possible from any internal representation would shield you from potential internal changes, and possibly prevent having to version the format schema (although I'd version that anyway just in case). But a larger mpremote mip may involve getting a larger on-device mip installer!

So, this brings up a few interesting questions regarding whether to maintain compatibility (and if so, how) with existing on-device installations of the mip module or with code that uses its API.

@Josverl not sure if you've been following this, but it may be of interest :)

(Also, I wish I could have port board definitions loaded from a manifest but that's probably just me!)

@agatti

Side effect worth calling out: this also covers plain (non-native) .mpy in a single manifest, across mpy major versions.

If I recall correctly, py/persistentcode.c checks both major and minor versions when it comes to load/reject a file - so if that's indeed what it does you'll have to encode the version differently (and there's no guarantee that MPYv7 will restrict the minor version number to just two bits).

My main concern in relying on sys.implementation._mpy this way is that such a value has no guarantees of stability in its format or encoding. It wouldn't surprise me if it would get a overhaul in MicroPython v2. It is true that it hasn't changed in a while, but as the name suggests it is both implementation-specific and a (documented) private implementation detail.

I think making the platform information metadata encoding more explicit and as far as possible from any internal representation would shield you from potential internal changes, and possibly prevent having to version the format schema (although I'd version that anyway just in case). But a larger mpremote mip may involve getting a larger on-device mip installer!

So, this brings up a few interesting questions regarding whether to maintain compatibility (and if so, how) with existing on-device installations of the mip module or with code that uses its API.

@Josverl not sure if you've been following this, but it may be of interest :)

(Also, I wish I could have port board definitions loaded from a manifest but that's probably just me!)

@agatti

py/persistentcode.c checks both major and minor versions when it comes to load/reject a file

That's not quite right for bytecode-only files - straight from the source, right next to the version #defines:

micropython/py/persistentcode.h

Lines 33 to 38 in 5aaa6d1

    
           // The current version of .mpy files. A bytecode-only .mpy file can be loaded 
        
           // as long as MPY_VERSION matches, but a native .mpy (i.e. one with an arch 
        
           // set) must also match MPY_SUB_VERSION. This allows 3 additional updates to 
        
           // the native ABI per bytecode revision. 
        
           #define MPY_VERSION 6 
        
           #define MPY_SUB_VERSION 3

// The current version of .mpy files. A bytecode-only .mpy file can be loaded
// as long as MPY_VERSION matches, but a native .mpy (i.e. one with an arch
// set) must also match MPY_SUB_VERSION.

And the actual enforcement, matching the comment exactly (sub-version check gated on arch != MP_NATIVE_ARCH_NONE):

micropython/py/persistentcode.c

Lines 472 to 477 in 5aaa6d1

    
           if (header[0] != 'M' 
        
               || header[1] != MPY_VERSION 
        
               || (arch != MP_NATIVE_ARCH_NONE && MPY_FEATURE_DECODE_SUB_VERSION(header[2]) != MPY_SUB_VERSION) 
        
               || header[3] > MP_SMALL_INT_BITS) { 
        
               mp_raise_ValueError(MP_ERROR_TEXT("incompatible .mpy file")); 
        
           }

Sub-version is only enforced when the file has native code. That's the actual behavior _mpy_tag_ok mirrors - not a new assumption.

On stability: _mpy's bit layout isn't a NEW dependency this introduces - every third-party natmod published today already depends on the .mpy file format's version contract completely, tag or no tag. If v2 redoes that encoding, all existing native .mpy files stop loading regardless of whether mip reads the field at all. mip shipping alongside micropython-lib/MicroPython releases and updating when the format does is the same story pip already has with CPython - not a special new risk for this specific field.

An explicit versioned schema is a reasonable design to weigh, but it's a real trade-off you already named: bigger on-device mip for every install, to hedge against a hypothetical v2 encoding change, when the underlying .mpy format itself gives no such guarantee either. Happy to let that be a maintainer call either way.

@agatti

@agatti
More generally: we can't predict or design around every future change a project this size might make - that's true of any API, internal or public. What this PR tries to do is add a solution to a problem that exists today, with minimal changes, no impact on backward compatibility, and built entirely on mechanisms the project already has (sys.implementation._mpy, the existing .mpy version/arch contract) rather than inventing a new one. If and when the underlying format changes, that's a problem for whatever depends on the format at that point - this proposal included, same as everything else.

That's not quite right for bytecode-only files - straight from the source, right next to the version [...]

Thanks for checking, I don't think I ever looked into bytecode-only MPY files loading.

If v2 redoes that encoding, all existing native .mpy files stop loading

That will also happen for MPYv7, or even with the current source tree and a MPY file generated with MicroPython 1.22.0 (MPYv6.2). That doesn't bother me as it is expected given the past events on previous version updates.

Tying metadata representation with internal structures/values is, IMHO, something else entirely. Anyway, I guess we need more input from other people who know more about this, I can only provide insights on a limited set of things and this is quickly getting out of my area of expertise :)

Tying metadata representation with internal structures/values is, IMHO, something else entirely. Anyway, I guess we need more input from other people who know more about this, I can only provide insights on a limited set of things and this is quickly getting out of my area of expertise :)

However, perhaps you could point out here those maintainers who could provide a broader representation.

@o-murphy

привіт @o-murphy! Thanks for providing a suggestion on an alternative approach. I think that putting the .mpy file with associated metadata into package.json files would also work ok. It has some advantages over a URL-template based approach, in particular it supports a N-to-1 matches. This could be useful with subtypes down the line, because there might be multiple .mpy modules that are usable on a given architecture. And that way the logic for selection could be in mip, potentially with overrides by user. Say to default to all (supported) RISC-V extensions, but allow specifying to use the "generic".

It seems that mip already supports having multiple package.json for a single git repository. So in the case of emlearn-micropython, which ships multiple independently installable .mpy files, I would generate one package.json for each .mpy file. And optionally some "bundles" that would provide all/multiple .mpy files. That should be no problem.

I dont have particular opinions on how to encode the binary .mpy versions into package.json. Changing the URL items to be 3 tuple instead of 2 seems a bit risky (no way to know how other software than mip uses that). But if maintainers are OK with that, no objection on my part.

As for input - I believe that @jimmo originally wrote mip, so he might have some input. Maybe @vshymanskyy has some inputs since he worked on native module install (ref vshymanskyy/ViperIDE#42 ) - there was a prototype in ViperID which was based on template substitution inside the package.json (example: https://github.com/emlearn/emlearn-micropython/blob/gh-pages/builds/master/package.json)

I think the approach I experimented with in ViperIDE worked - or at least it covered emlearn needs.

Although it is not directly related, I also addressed the question of installing examples using a special package.json.
For example this will install the example emlearn xor_trees example code (along with the required dependencies):

That approach works, but afaik it has not seen broad adoption yet.

I need a bit more time - and a shift in focus - to read through this discussion before I can provide constructive feedback.
In any case, I’d be happy to adopt any standardized approach in ViperIDE.

@agatti

@agatti @jonnor @vshymanskyy

I dont have particular opinions on how to encode the binary .mpy versions into package.json. Changing the URL items to be 3 tuple instead of 2 seems a bit risky (no way to know how other software than mip uses that). But if maintainers are OK with that, no objection on my part.

I've been working with my 3-item version for a while now and I don't see any regressions. However, we could consider a few alternatives cause adding metadata via positional items to the tuples possibly not good idea.

🏆 1. Encode mip version directly to `file suffix` - my winner so far

The approach similar to topicstarter's but instead of using url slugs we can use file suffix with some specific format
I select column symbol : cause it is not valid for most filesystems and to avoid old mip be able to setup new package.json format. The presence of the suffix indicates that you need to check the _mpy version when installing

{
  "urls": [
    ["mymodule.mpy:5", "mymodule_mpy5.mpy"],
    ["mymodule.mpy:5894", "mymodule_armv7m.native.mpy"]
  ]
}

Advantages:

Full backward compatibility (old mip will just try to load a non-existent file)
Does not increase the size of the mip module
: guaranteed to be safe as a token

Tradeoffs:

A slightly "hacker" approach
Can be unintuitive for new developers

2. Download and read 4 bytes chunk of the `.mpy` file

Advantages:

The most "correct" technically
No need to change package.jsonformat

Tradeoffs:

Slow for the large packages
Needs much more IO operations

3. Extra root level field

It would probably be more correct to allow mappings instead of list urls: {} | [] because key-based access for structured data is much better than indexes, but this approach will definitely create regressions for existing packages, but then you can extend the format with any metadata.
An alternative could be to introduce additional top-level fields that will extend but not overlap the existing ones

{
  "mpy_urls": [
    ["mymodule.mpy", "mymodule_mpy5.mpy", 5],
    ["mymodule.mpy", "mymodule_armv7m.native.mpy", 5894]
  ]
}

Advantages:

Clean and explicit approach
Easy to extend

Tradeoffs:

Increases the size of the mip module
Requires support for two fields

4. Introduce `mip_min_version`

The version of the format, as entities and fields in package.json - but this is even worse, because it will force you to agree on that too.

Tradeoffs:

Most complex approach
Creates fragmentation
Requires version reconciliation

I select column symbol : cause it is not valid for most filesystems and to avoid old mip be able to setup new package.json format. The presence of the suffix indicates that you need to check the _mpy version when installing

It can be anything like mymodule.mpy:5 or even much wide metadata mymodule.mpy:{ ..some extra json metadata.. }
I add it to filename to not affect on url item

Another shape worth considering, that sidesteps the 3-tuple compatibility question entirely: keep urls/hashes entries 2-element always, but let the second element be either the current string (unchanged meaning) or a dict for tag-gated variants:

{
  "urls": [
    ["mymodule.py", "mymodule.py"],
    ["mymodule.mpy", {"mpy": 5894, "src": "https://.../mymodule_armv7m.mpy"}]
  ],
  "hashes": [
    ["othermodule.mpy", {"mpy": 5894, "sha256": "ab12..."}]
  ]
}

Old mip does target_path, url = entry (fixed 2-arity, unchanged), then immediately:

https://github.com/micropython/micropython-lib/blob/master/micropython/mip/mip/__init__.py#L125-L127

for target_path, url in package_json.get("urls", ()):
    fs_target_path = target + "/" + target_path
    is_full_url = any(url.startswith(p) for p in _ALLOWED_MIP_URL_PREFIXES)

dict has no .startswith - AttributeError right there, immediately, on the first tagged entry. Old mip never gets far enough to write a wrong/partial file anywhere. Same fail-loud property the 3-tuple has, but with a smaller footprint: the shape (2-tuple) never changes, so any other consumer that unpacks path, val = entry and doesn't immediately call string methods on val (e.g. something that just lists/displays manifest contents) doesn't break at all - only code that actually tries to use the value as a URL string does, which is arguably the code that needs updating anyway.

Match logic is the same _mpy_tag_ok scheme already in #19532 / micropython-lib#1144, just reading entry[1]["mpy"] (or whatever key we settle on, happy to bikeshed the name) instead of entry[2] - the tag-comparison code doesn't change, only where it reads the tag from.

For symmetry, hashes entries need the same str-or-dict treatment ({"mpy": ..., "sha256": ...}), not just urls - otherwise native variants would only be usable for self-hosted urls packages, not ones going through the content-addressed index.

Happy to update the PR to this shape if it looks preferable to the 3-tuple.

@agatti

Everything proposed so far in this thread solves the same problem: given one package.json, pick the right file among its urls/hashes entries. That's one level. There's a level above it that nobody's touched yet: a project supporting more than one MicroPython ABI generation doesn't have variants of one package, it has several separate packages — a .mpy built for one ABI just won't load on another, no matter how the entries inside one manifest are tagged. A real build matrix for one module looks like this:

mpy5-v1.18-x86      mpy5-v1.18-x64      mpy5-v1.18-armv6m      ...  (8 archs)
mpy6-v1.19.1-x86     mpy6-v1.19.1-x64    mpy6-v1.19.1-armv6m    ...  (8 archs)
mpy6.1-v1.21.0-x86   mpy6.1-v1.21.0-x64  mpy6.1-v1.21.0-armv6m  ...  (8 archs)
mpy6.2-v1.22.2-x86   mpy6.2-v1.22.2-x64  mpy6.2-v1.22.2-armv6m  ...  (8 archs)
mpy6.3-v1.29.0-x86   mpy6.3-v1.29.0-x64  mpy6.3-v1.29.0-armv6m  ...  (8 archs + rv32imc + rv64imc)

42 builds, one module, one release, each its own package.json. Nothing tells a device or a script which of those 42 URLs is theirs — today that's a human reading a release page. (real output from cibuildmp, a cibuildwheel-shaped build tool for natmod/usermod — mentioned only as where this example comes from, not load-bearing for the proposal itself.)

Idea: put the tag in the filename itself, instead of in any manifest. That's literally what PEP 600/656 are — not a resolution protocol, just a grammar for embedding a compatibility tag in the name of the file you're about to download (-manylinux_2_17_x86_64.whl sitting right there, pip never has to fetch anything extra to know what a wheel is compatible with). package.json's own shape doesn't change at all — same plain two-element urls list it already has — only the filename in the second element grows a platform tag: <filename>.<platform-tag>.mpy instead of a bare name. For a build targeting mpy6.3/x64:

{
  "urls": [
    ["mymod.mpy", "mymod.mpy6.3-x64.mpy"]
  ],
  "version": "0.1.0"
}

and for rv64imc, same shape, just a different tag in the same spot:

{
  "urls": [
    ["mymod.mpy", "mymod.mpy6.3-rv64imc.mpy"]
  ],
  "version": "0.1.0"
}

A resolver that knows its own (abi, arch[, arch_flags]) just formats the same string and either builds the URL directly, if the base path is predictable, or picks it straight out of a flat asset listing (a GitHub Release's asset list, say) — no manifest to fetch first, nothing new to parse beyond a filename.

Doesn't force splitting into one package.json per platform either — a combined manifest covering a whole release still works the same way, tagged and untagged entries side by side:

{
  "urls": [
    ["mymod.mpy", "mymod.mpy6.3-x64.mpy"],
    ["mymod.mpy", "mymod.mpy6.3-rv64imc.mpy"],
    ["bytecode.mpy", "bytecode.mpy"],
    ["bytecode.mpy", "bytecode_explicit_tag.any-none.mpy"],
    ["facade.py", "facade.py"]
  ],
  "version": "0.1.0"
}

facade.py and the bare bytecode.mpy are exactly what stock mip already does today, unchanged — no tag at all means "installs everywhere," same as always. bytecode.any-none.mpy says the same thing more explicitly, borrowing the actual PEP wording instead of inventing a new marker — none/any are already what a pure-Python wheel's own trailing tags say (py3-none-any.whl: no ABI requirement, any platform), so reusing them here rather than making up a bespoke "any" keeps it recognizable to anyone who already reads wheel filenames. Purely cosmetic once nothing gates on it, but keeps every entry readable the same way once tagged ones are sitting next to it. The two mymod.mpy rows are the real case this needs help with: today's mip would just install both, one overwriting the other, since it loops over every entry unconditionally — that's exactly the gap the schemes already on this thread (raw int, suffix, dict-shaped) are for. The point here isn't that this filename convention replaces those — it's that whichever one wins wouldn't need its own separate tag encoding at all, since the tag is already sitting in the filename it would read src/url from anyway.

Where does the literal MicroPython tag go, then — nowhere resolvable, and on purpose. v1.28.0 and v1.29.0 both produce ABI 6.3, and at runtime the literal tag carries no meaning at all, only (abi, arch) does — I checked this rather than assumed it: built the same source against both, got different bytes (the native emitter gained a shorter jump encoding between them, no MPY_VERSION bump), then cross-loaded every combination of binary × .mpy from both — all four ran fine. So mymod.mpy6.3-rv64imc.mpy is right whether it came from v1.28.0 or v1.29.0 — rebuilding against a newer tag in the same ABI just republishes over that same name, no need to keep both around under separate filenames. If the literal tag is worth keeping for provenance, it can ride along as non-resolving build metadata (mymod.mpy6.3-rv64imc+v1.29.0.mpy, the same way semver keeps build metadata out of what version comparison looks at) — nice to know, never part of matching.

That doesn't mean the tag segment can never carry the literal build tag, either — a project that wants that extra precision can just make the tag itself more specific: mymod.mpy6.3-v1.28.0-x64.mpy instead of mymod.mpy6.3-x64.mpy, published alongside or instead of the coarser one. Real reason to want it: not compatibility, just saying plainly which dynruntime checkout actually built this file — "functionally interchangeable" is what makes it safe to leave the tag out, not a reason nobody would ever want it recorded. The GOT-jump encoding change above is exactly why that fact is worth having on hand even though nothing loads differently because of it. Same idea PEP 600 already uses when a wheel claims more than one floor at once — nothing about the grammar forces one level of specificity. The point above isn't "the literal tag can never appear in a tag," it's that nothing about _mpy or the loader ever requires it to — a resolver that only cares about (abi, arch) matches the coarse form fine, one that wants exact-tag precision can look for the longer one.

Thinking about it more, I'd actually put this up as an alternative to my own three proposals above, not just something extra — and it answers @agatti's stability worry better than any of them. Every version I proposed (raw int, the .mpy:5 suffix, the dict-shaped entries) makes package.json/mip understand something about _mpy's internal encoding, forever, on every device. This doesn't touch package.json or mip at all — no new field, no new file even. All the _mpy-reading stays in whatever resolves the filename, off-device, before mip.install() ever runs with a plain URL. If the encoding changes down the line, only that resolver needs updating — not the manifest format, not mip, not any device that already installed something. Closer to what @agatti was gesturing at: push the complexity out to mpremote-side tooling, not into the protocol.

Not decided: the exact grammar (this is roughly what natmod build output already looks like, just not written down as a spec anywhere), and whether resolving belongs in mpremote or stays a separate opt-in helper.

push the complexity out to mpremote-side tooling

There also may be a need for a tools ( as part of mpremote - or a stand-alone tool) to create and/or lint the packages as it will be harder to just eye-ball a package.json

And I think there is no escaping that the package.json (schema) itself will need a version number.

@o-murphy

@o-murphy check out our work with @jonnor , which resulted in a package.json like this:
https://github.com/emlearn/emlearn-micropython/blob/gh-pages/builds/master/package.json

It actually addresses most (all?) of the questions (mpy abi version, architecture, etc), while allowing the author of the module to define the exact supported targets build matrix, and not restricting the build system to use a single output directory layout.

@o-murphyознайомтеся з нашою роботою з@jonnor, що призвело до створення package.json, подібного до цього: https://github.com/emlearn/emlearn-micropython/blob/gh-pages/builds/master/package.json

Фактично, це вирішує більшість (або всі?) питань (версія mpy abi, архітектура тощо), водночас дозволяючи автору модуля визначити точну матрицю збірки підтримуваних цілей та не обмежуючи систему збірки використанням єдиного макета вихідного каталогу.

IMO any of the options presented in this thread has the right to life. The difference between our options and yours is only in the presentation and the amount of additional code to mip required to resolve it. The question is how the maintainers see it.

Navigation Menu

Repository navigation

RFC: Distribute native modules for easy installation #19479

Description

Code Size

Implementation

Code of Conduct

PoC simple test with real-world natmod package

Simple test for pre-built firmware on board via bootstrap

Building `package.json`

3 remaining items

🏆 1. Encode mip version directly to `file suffix` - my winner so far

2. Download and read 4 bytes chunk of the `.mpy` file

3. Extra root level field

4. Introduce `mip_min_version`

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	// The current version of .mpy files. A bytecode-only .mpy file can be loaded
	// as long as MPY_VERSION matches, but a native .mpy (i.e. one with an arch
	// set) must also match MPY_SUB_VERSION. This allows 3 additional updates to
	// the native ABI per bytecode revision.
	#define MPY_VERSION 6
	#define MPY_SUB_VERSION 3

	if (header[0] != 'M'
	\|\| header[1] != MPY_VERSION
	\|\| (arch != MP_NATIVE_ARCH_NONE && MPY_FEATURE_DECODE_SUB_VERSION(header[2]) != MPY_SUB_VERSION)
	\|\| header[3] > MP_SMALL_INT_BITS) {
	mp_raise_ValueError(MP_ERROR_TEXT("incompatible .mpy file"));
	}

RFC: Distribute native modules for easy installation #19479

Description

Description

Code Size

Implementation

Code of Conduct

Activity

jonnor commented on Jul 16, 2026

o-murphy commented on Jul 27, 2026

o-murphy commented on Jul 27, 2026

o-murphy commented on Jul 27, 2026

agatti commented on Jul 27, 2026

agatti commented on Jul 27, 2026

o-murphy commented on Jul 27, 2026

o-murphy commented on Jul 27, 2026

o-murphy commented on Jul 27, 2026

o-murphy commented on Jul 27, 2026

agatti commented on Jul 27, 2026

o-murphy commented on Jul 28, 2026

PoC simple test with real-world natmod package

Simple test for pre-built firmware on board via bootstrap

Building package.json

o-murphy commented on Jul 28, 2026

3 remaining items

o-murphy commented on Jul 28, 2026

agatti commented on Jul 28, 2026

o-murphy commented on Jul 28, 2026

o-murphy commented on Jul 28, 2026

agatti commented on Jul 28, 2026

o-murphy commented on Jul 28, 2026

jonnor commented on Jul 29, 2026

vshymanskyy commented on Jul 30, 2026

o-murphy commented on Jul 31, 2026

🏆 1. Encode mip version directly to file suffix - my winner so far

2. Download and read 4 bytes chunk of the .mpy file

3. Extra root level field

4. Introduce mip_min_version

o-murphy commented on Jul 31, 2026

o-murphy commented on Jul 31, 2026

o-murphy commented on Aug 31, 2026

Josverl commented on Sep 1, 2026

vshymanskyy commented on Sep 2, 2026

o-murphy commented on Sep 2, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Building `package.json`

🏆 1. Encode mip version directly to `file suffix` - my winner so far

2. Download and read 4 bytes chunk of the `.mpy` file

4. Introduce `mip_min_version`