pip install overrides existing build configuration

[Alvaro-Kothe]

When using meson-python with editable installs, running pip install -e a second time, reusing the build directory, but without repeating the buildtype argument overwrites the existing build configuration.

Reproduction

#!/usr/bin/env bash

mkdir repro

# Create files for meson-python

cat <<EOF > repro/pyproject.toml
[project]
name = "foo"
version = "0.0.1"

[build-system]
build-backend = "mesonpy"
requires = ["meson-python"]
EOF

cat <<EOF > repro/meson.build
project('foo', 'c', version: '0.1.0')
EOF

python -m pip install meson-python

# Create a debug build
pip install --no-build-isolation -e repro -Csetup-args=-Dbuildtype=debug -Cbuild-dir=debug

grep buildtype repro/debug/meson-logs/meson-setup.txt
# buildtype   : debug

# If I decide to reuse this build directory,
# and I call `pip install` without `-Csetup-args=-Dbuildtype=debug`
# it reconfigures to have `buildtype=release` instead of reusing the current configuration.
pip install --no-build-isolation -e repro -Cbuild-dir=debug

grep buildtype repro/debug/meson-logs/meson-setup.txt
# buildtype   : release

[rgommers]

This is by design; you never have to re-run the editable install command to keep working on code (including compiled code), so if you do and you're changing the build options, you are asking for a new build configuration.

Do you see a problem with that?

This is where the reconfigure happens:

meson-python/mesonpy/__init__.py

Lines 814 to 819 in 4977b7d

	# reconfigure if we have a valid Meson build directory. Meson
	# uses the presence of the 'meson-private/coredata.dat' file
	# in the build directory as indication that the build
	# directory has already been configured and arranges this file
	# to be created as late as possible or deleted if something
	# goes wrong during setup.

[Alvaro-Kothe]

Do you see a problem with that?

The main issue arises when managing multiple build directories for different configurations (e.g., debug, release, ASAN). I want to alternate between them without accidentally wiping out existing compiled objects or unintentionally altering the build type.

Right now, if I run pip install and explicitly pass the matching buildtype setup argument, everything works as expected (no reconfigure). However, if I omit that option on a subsequent run, meson-python defaults to a release build. This triggers a destructive reconfigure on an active debug directory, wiping out the existing binaries. I then have to abort the build and re-run pip install with the explicit -Csetup-args=-Dbuildtype=debug flag to restore it.

[eli-schwartz]

so if you do and you're changing the build options, you are asking for a new build configuration.

As far as I understand, the problem is when one does not change the build options but uses -Cbuild-dir=debug (this is a big giveaway) to reinstall the editable loader while toggling between different build directories.

Despite this, meson-python blows away the existing build options by passing buildtype as a command line argument even when also passing --reconfigure, which some might call a bit opinionated. :P

This could also be solvable if it was possible to save an editable install and reinstall the editable bits, but I'm not sure that the pypa ecosystem supports this use case generically. If it was just a matter of Meson, I could have suggested using meson devenv inside each build directory, but that lacks the editable loader, which sort of renders the whole topic moot!

[rgommers]

The main issue arises when managing multiple build directories for different configurations (e.g., debug, release, ASAN). I want to alternate between them

Ah, rerunning the install command makes sense if you alternate within the same environment. I've found it's usually better to have one environment per type of build - which you'd need anyway for ASan, because you need to build your dependencies with ASan as well - that works quite effectively, and this reconfigure issue doesn't come up then.

If you do want to alternate within a single environment, why not define some aliases? It's a very long invocation anyway, so something like this would help and avoid the problem of missing an argument:

alias pip-debug='pip install -e . --no-build-isolation --Csetup-args=-Dbuildtype=debug -Cbuild-dir=build-debug

Despite this, meson-python blows away the existing build options by passing buildtype as a command line argument even when also passing --reconfigure, which some might call a bit opinionated.

We didn't do that from the start, but not passing --reconfigure gives surprising behavior in other circumstances, that more people seemed to run into and get surprised by. In the end there's no single way to make everyone happy and make this fully intuitive, for a user base whose intuition level is both fairly low/inconsistent and shaped by stuff setuptools and pip used to do.

[eli-schwartz]

Shower thought: maybe you could save and restore any config-settings inside a custom -Cbuild-dir under the rationale that custom build directories implies users want persistence.

[Alvaro-Kothe]

I opened #852 with the idea of not providing any default option if the build directory already exists. From what I've tested, it preserved the original configuration.

I want to point out that every alternative that @rgommers provided in #851 (comment) is valid, but I think that it's more practical to call pip install -e . --no-build-isolation -Cbuild-dir=<builddir> to be able to alternate between build directories, while also preserving existing configurations.