Including files added via meson.add_dist_script into the wheel

[WillAyd]

I am trying to set up meson-python in pyarrow, which is included as part of the larger Arrow monorepo. PyArrow must include both a LICENSE.txt and NOTICE.txt file that exist in the monorepo source, technically outside of the Python package itself.

Including these files into the sdist seems trivial - I can just add a dist_script that looks like:

import os
import pathlib
import shutil


def main():
    src_dir = pathlib.Path(os.environ["MESON_SOURCE_ROOT"])
    parent_dir = src_dir.parent.resolve()
    dist_dir = pathlib.Path(os.environ["MESON_DIST_ROOT"]).resolve()

    license_file = parent_dir / 'LICENSE.txt'
    shutil.copy(license_file, dist_dir)
    notice_file = parent_dir / 'NOTICE.txt'
    shutil.copy(notice_file, dist_dir)


if __name__ == "__main__":
    main()

While that includes the files into the generated sdist, I'm not sure how to install those files into the wheel thereafter.

I'm guessing meson.add_install_script can help, but is there a more idiomatic way that meson-python can detect these files and place them into the dist-info directory of the wheel?

[WillAyd]

Another alternative to the scripts is to run something like:

fs = import('fs')
foreach file : ['LICENSE.txt', 'NOTICE.txt']
    fs.copyfile(
        meson.current_source_dir() / '..' / file,
        file,
        install: true,
        install_dir: py.get_install_dir() / 'pyarrow',
    )
endforeach

although I am not sure how to signal that these files should be placed into the dist.info directory of the wheel

[dnicolodi]

These are two different issues:

1. including the files in the sdist
2. including the files in the right location in the wheel

For 1 a Meson's dist script is most likely the right approach. For 2 you can declare these files as license-files in the project metadata in pyproject.toml, see https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#license-files

Note, however, that having an installation from an sdist and an installation from a git checkout producing different results (in this case having or not the license files) is annoying.

[dnicolodi]

Well, I would go as far as to say that it is kind of obvious that you cannot include in the wheel files that do not exist, thus the error makes perfect sense. This is exactly what I meant with having the build from a source distribution and from a git checkout having to produce different results: a build from a git checkout cannot include these files.

This implies that you need to find a way to patch the license-files metadata entry appropriately. It is possible to declare license-files as dynamic in pyproject.toml in which case meson-python substitutes the value passed to the license_files argument to the project() declaration in meson.build but AFAIK that canno be filled conditionally either.

The easiest solution I see is to have the git repository contain a LICENSE.txt to ../LICENSE.txt symlink and replace it with the real file in the dist script.

I cannot avoid stating once more that this project structure organization IMHO introduces three problems for every apparent problem it solves.

[WillAyd]

When you refer to the git checkout you are referring to the Arrow monorepo right? There is no git repo just for PyArrow, so the files would still be available from source, just in a different location than the Python sdist

Noted on the symlink suggestion. I was initially hesitant regarding that given the history of symlink support on Windows, but maybe enough time has passed so that is no longer an issue

[dnicolodi]

When you refer to the git checkout you are referring to the Arrow monorepo right?

Yes. You are getting the above mentioned error while compiling from the PyArrow source directory in the Arrow repository checkout, not compiling from an sdist, aren't you?

so the files would still be available from source, just in a different location than the Python sdist

I don't understand why you think this is a relevant distinction. These files cannot be found unless they are at the expected location.

I was initially hesitant regarding that given the history of symlink support on Windows, but maybe enough time has passed so that is no longer an issue

AFAIK symlinks are not really a thing on Windows (they are available only in some very specific setups). AFAIK, on Windows, git turns symbolic links into regular files on checkout.

[WillAyd]

Yes. You are getting the above mentioned error while compiling from the PyArrow source directory in the Arrow repository checkout, not compiling from an sdist, aren't you?

That's right. Its not even possible to create an sdist with that configuration, because the failure prevents you from getting so far.

I don't understand why you think this is a relevant distinction. These files cannot be found unless they are at the expected location.

I think its relevant because Meson supports the functionality. Its translating that expectation into an sdist where things get murky

AFAIK symlinks are not really a thing on Windows (they are available only in some very specific setups). AFAIK, on Windows, git turns symbolic links into regular files on checkout.

This seems like the most authoritative resource I can find on the subject:

https://gitforwindows.org/symbolic-links.html

Seems like the Windows 10 requirement should be commonplace at this point, since that went EOL last month. Not sure how commonplace it is for Developer Settings to be enabled

[eli-schwartz]

It is impossible one way or the other, to reference the file ../anything.txt from the root of meson.build inside an sdist uploaded to PyPI , because sdists have to be the root of the project and can't provide files in the next directory up.

But it is theoretically possible for a build backend to support doing precisely that, "as long as the file actually exists", and then just say "that will obviously fail if you upload an sdist without further changes but it will at least work when building from git".

It would then be up to the project itself to make sdists work as well, by including a dist script that:

• copies the file out of the parent and into the sdist root
• rewrites whichever file contains the ../ (either pyproject.toml or else meson.build) to strip the ../ from the filename and reference the new location. meson rewrite may help you there though sed is a possibility too.

It sounds like the current meson-python policy is to assume the project won't use a dist script and therefore it will pre-emptively error out before even trying to run meson for the first time in the git checkout.