Skip to content

Build stdlibs with rewatch or bsb and integrate Core #6826

Closed
Closed
Build stdlibs with rewatch or bsb and integrate Core#6826
Assignees
cknitt
Labels
infrastructure
Milestone
v12

Description

@cknitt
cknitt
opened on Jun 23, 2024

Before integrating Core into the compiler repo, we should sort out the standard library build procedure. We should get rid of the ninja scripts and build everything with rewatch (or maybe even still bsb at first).

It turns out to be difficult to set this up because of (among other things) cyclic dependencies that need to be resolved. A lot things need to be shuffled around. It would be easier to iterate on this outside of the compiler repo IMHO.

My proposal is therefore to create a separate repo rescript-stdlib to work on a monorepo structure where all standard libs can be built with a single build command and without relying on any pre-existing artifacts in lib/ocaml. For this, we do not even need the v12 compiler, it should work fine with v11, too. This rescript-stdlib repo would be a temporary one, allowing us to iterate quickly on the standard library build infrastructure. Once we everything is working, it can be taken over into the compiler repo.

A rough plan could look like this:

  • 1. Re-add the -nostdlib option, see Re-add -nostdlib option and make it actually do something #6824.
  • Creeate separate repo for the work on the standard libraries
  • Add a "stdlib mini" (the absolutely essential stuff to get the rest to build), runtime (caml_*.res) and belt and get everything to compile with "-nostdlib -nopervasives" in curried mode
  • Switch to uncurried mode and get runtime to compile
  • Add Core
  • Add (part of?) the old Js namespace as a compatibility package
  • Add (part of?) the old OCaml 4.06 stdlib as a compatibility package
  • Clean up

Activity

  1. cknitt commented on Jun 23, 2024

    @cknitt
    cknitt
    on Jun 23, 2024
    MemberAuthor

    I think the resulting standard library packages could then even be published as separate npm packages (see #6183) independent of a v12 compiler release. They could be used/tested in any ReScript project that sets the -nostdlib compiler option and adds them as dependencies.

  2. cknitt commented on Jun 23, 2024

    @cknitt
    cknitt
    on Jun 23, 2024
    MemberAuthor

    @zth @cristianoc @jfrolich @rolandpeelen what do you think?

  3. jfrolich commented on Jun 25, 2024

    @jfrolich
    jfrolich
    on Jun 25, 2024
    Collaborator

    I think it's probably easier to get everything compiled the same way using rewatch in the compiler repo, and refactor from there. This allows us to have the two options side-by-side. Feels like the above approach takes too many steps at once. Sorry I didn't get to finalizing the PR where we compile the stdlib with rewatch but I don't have the impression that it's hard. We just have to determine if we keep the cyclic deps, or refactor them out, we went for the last and that seemed to work. Just need to spend some time to make sure everything is properly compiling.

  4. cristianoc commented on Jun 26, 2024

    @cristianoc
    cristianoc
    on Jun 26, 2024
    Collaborator

    I think the stdlib mini would be pretty nice to have, as it isolates all the special cases. If I understand correctly, everything else can be built user-level on top of it without tricks/surprises. So it seems doable and independent from the details of the other steps.

  5. cknitt commented on Jun 26, 2024

    @cknitt
    cknitt
    on Jun 26, 2024
    MemberAuthor

    Feels like the above approach takes too many steps at once.

    Thanks for the feedback. My feeling was the opposite, that doing it in the compiler would mean taking too many steps at once. IMHO it is hard to refactor all this until it has the proper structure while still keeping the old build process working. (Unless we copy everything to a new dir in the compiler repo, but then we may as well put it into a separate repo temporarily to be able to iterate quickly, like we did for Core itself.)

    I think the stdlib mini would be pretty nice to have, as it isolates all the special cases. If I understand correctly, everything else can be built user-level on top of it without tricks/surprises. So it seems doable and independent from the details of the other steps.

    Definitely! Everything can be built from scratch in user level, starting with the stdlib mini. (Even with bsb, and even with the v11 compiler with -nostdlib). I'll put up a repo tomorrow with what I have now (basically step 1-4).

  6. cknitt commented on Jun 27, 2024

    @cknitt
    cknitt
    on Jun 27, 2024
    MemberAuthor

    Ok, so here is my repo: https://github.com/cknitt/rescript-stdlib

    It uses compiler v11.1.3-rc.1 with -nostdlib -nopervasives. This ensures that everything is indeed built from scratch.

    You can clone it and run npx rewatch to build with rewatch, or npx rescript to build with bsb.

    It will then build in one go:

    • a mini stdlib (not quite mini enough yet, I had to add some part of Js, too)
    • belt, depending only on the mini stdlib
    • runtime (caml_*), depending only on the mini stdlib

    This is based on #6772, but further modified to get it to build in this setup.

    Next up would be adding Core.

  7. cknitt commented on Jun 28, 2024

    @cknitt
    cknitt
    on Jun 28, 2024
    MemberAuthor

    Moved my repo to https://github.com/rescript-lang/experimental-rescript-stdlib-build

    I have also added Core in the meantime and adapted it a bit to fit into the structure.

  8. cknitt commented on Jun 28, 2024

    @cknitt
    cknitt
    on Jun 28, 2024
    MemberAuthor

    Some questions:

    1. I want to be able to build runtime, belt, Js, and the OCaml stdlib without relying on Core being present (and am already doing so now for the first two). A consequence of this is that common type definitions cannot have Core as their source. I have collected these type definitions in https://github.com/rescript-lang/experimental-rescript-stdlib-build/blob/main/stdlib-mini/src/js_types.res for now. Maybe at least some of these should actually be builtin types?
    2. What we need to be able to rely on though when compiling ReScript code is runtime (caml_*) to be present, see Compiler should not rely on Js or Belt module being present #6836. Therefore the question is, shouldn't the basic types and the "stdlib mini" actually be part of the runtime as caml_types and caml_pervasives or something like that?

    What do you think @cristianoc @zth?

  9. cknitt
    added this to the v12 milestone on Aug 25, 2024
  10. cknitt
    self-assigned this
    on Aug 25, 2024
  11. cknitt
    added
    infrastructure
    on Aug 28, 2024
  12. cknitt
    changed the title [-]Proposed plan for building all stdlibs with rewatch and integrating Core[/-] [+]Build stdlibs with rewatch or bsb and integrate Core[/+] on Aug 29, 2024
  13. cknitt
    added this to ReScript developmenton Aug 29, 2024
  14. cknitt
    moved this to In progress in ReScript developmenton Aug 29, 2024

  15. cknitt commented on Oct 22, 2024

    @cknitt
    cknitt
    on Oct 22, 2024
    MemberAuthor

    Done in #7108 and #7116 (with a simpler approach, no monorepo like in https://github.com/rescript-lang/experimental-rescript-stdlib-build).

  16. cknitt
    closed this as completedon Oct 22, 2024
  17. github-project-automation
    moved this from In progress to Done in ReScript developmenton Oct 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

infrastructure

Type

No type

Projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions