[SPARK-58664][PYTHON] Factor out repeated paths in PySpark Classic's setup.py

[nchammas]

What changes were proposed in this pull request?

Factor out repeated paths into a single list of install paths that is used throughout the PySpark Classic install script.

During cleanup, change the check to look at the actual filesystem object -- is it a symlink or not? -- rather than repeat the general check on OS capabilities.

I preserved the order in which the paths are installed and then cleaned up.

Why are the changes needed?

General code hygiene. We shouldn't have to manually touch multiple blocks of code that are supposed to always be in sync.

Checking the file's type directly during cleanup is also more robust than checking a general capability.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

dev/make-distribution.sh --pip
dev/run-pip-tests

Was this patch authored or co-authored using generative AI tooling?

I wrote this with assistance from GitHub Copilot.

[dongjoon-hyun]

LGTM. Nice cleanup — I verified that the install/cleanup ordering is preserved and that the removed variables have no remaining usages (JARS_TARGET/SCRIPTS_TARGET, which are still used elsewhere, are correctly kept). The cleanup change is a real robustness improvement: since it runs in the finally: block, the old code could mask the original exception with a FileNotFoundError when staging failed partway, while the new per-target islink check lets the root cause propagate.

A couple of minor, non-blocking notes:

• In the sdist-install scenario (in_spark=False with deps/ present), JARS_PATH remains an empty list [], which gets passed into InstallPath(source: str, ...). Harmless at runtime since INSTALL_PATHS is only iterated when in_spark is true, but it contradicts the declared type.
• Silently skipping missing targets also drops the signal that staging was partial; a one-line note to stderr when skipping could be worth considering.

[nchammas]

• In the sdist-install scenario (in_spark=False with deps/ present), JARS_PATH remains an empty list [], which gets passed into InstallPath(source: str, ...). Harmless at runtime since INSTALL_PATHS is only iterated when in_spark is true, but it contradicts the declared type.

I think we should refactor how we build JARS_PATH (maybe move the logic into a helper function), but I'll leave that for a future PR since it's independent of the cleanup here. The type mismatch is unfortunate but it's harmless in this case.

• Silently skipping missing targets also drops the signal that staging was partial; a one-line note to stderr when skipping could be worth considering.

That's fair. I reverted the behavior change to just fail as it did before. I've updated the PR description accordingly.