#65/TrafficLayer and VirCarlaEnv build and run on Linux, with a 20.04+22.04+24.04 CI gate - #229
Open
yunlishao-vibe wants to merge 26 commits into
Open
#65/TrafficLayer and VirCarlaEnv build and run on Linux, with a 20.04+22.04+24.04 CI gate#229yunlishao-vibe wants to merge 26 commits into
yunlishao-vibe wants to merge 26 commits into
Conversation
Adds a CMake build for TrafficLayer on Linux alongside the existing MSBuild build, which is untouched and remains the Windows build of record. Migrating Windows onto CMake is deliberately left as follow-up work. Verified end to end on Ubuntu 20.04 (glibc 2.31, gcc 9.4, cmake 3.16): the binary builds, links against libtracicpp, parses a real repo config.yaml and exits cleanly. Windows MSBuild re-verified afterwards -- TrafficLayer Debug+Release and both DriverModel DLLs still build, and the exe still reports v0.9.0 from generate_version.ps1. The portability defects were enumerated by compiling, not by grepping for the Win32 surface, which is why three of them were missed by the audit in #65: * Timer_t / QueryPerformanceCounter -- dead code (declared, frequency queried, never read). Deleted; the live timing path now uses steady_clock. * TrafficHelper's `const double M_PI` member -- legal on MSVC, which leaves M_PI undefined without _USE_MATH_DEFINES; a syntax error against glibc, which always defines it. Renamed to kPi. * ConfigHelper's unscoped enum value `link` -- collides with POSIX link(2). Made the enum scoped so this class of collision cannot recur. Also fixed: SocketHelper's plain-POSIX tier declared clientAddrLen as size_t where accept() needs socklen_t*, proving that branch had never been compiled by anyone. RS_DSPACE and plain POSIX now share one declaration. New CommonLib/PlatformCompat.h is header-only and C++14 on purpose. Five .vcxproj files compile ConfigHelper.cpp, two of them inside the private ProprietaryFiles submodule, so a new .cpp TU would have forced a submodule PR; an inline header changes no project file at all. C++14 because CommonLib is consumed by the CarMaker/dSPACE RT targets (#65 Q4) -- the constraint is kept, not relaxed. VISSIM on Linux is rejected at config-parse time with an explicit diagnostic rather than left to hang on a connect() that can never succeed; the two Windows-only DSProxy TUs drop out of the Linux source list. Because the TU list now exists twice, scripts/check_source_lists.py compares the .vcxproj and CMake lists and CI fails on divergence.
Correctness fixes found by compiling the shared socket/traffic layer with gcc, which warns where MSVC is silent. All of these are pre-existing and affect the Windows build equally -- none are Linux-only. * SocketHelper leaked a file descriptor on every POSIX error path. close() was guarded on RS_DSPACE at all 10 sites, so a plain-POSIX build called shutdown() and then never closed the fd. close() is POSIX and unistd.h is already included there, so the guard was never needed; RS_DSPACE and plain POSIX now share one path (#65 Q7). * Every POSIX socket failure logged a bare "error: " with no value -- the WSAGetLastError() calls had no errno counterpart wired, at 18 sites. They now report strerror(errno) through the platform helper, so a socket failure on Linux is diagnosable instead of anonymous. * TrafficHelper::checkIfEgoExist fell off the end of a non-void function when nothing was subscribed yet, returning garbage (UB). It gates SimulationMode 1 ("wait until the ego enters the network"), so a garbage non-zero could declare the ego present before it exists. Now returns 0. * SocketHelper printed a size_t through %d, reading the wrong width out of the varargs. * ConfigHelper.h had four 'typedef struct X {...};' declarations whose typedef is silently discarded. Dropped the keyword; meaning is unchanged. FIXS code now compiles warning-free on gcc 9 (the 3 remaining warnings are inside vendored yaml-cpp). Windows re-verified after these changes: TrafficLayer Debug+Release and both DriverModel DLLs build, via both the dispatch scripts and msbuild directly.
Both matrix legs built and linked TrafficLayer successfully, then the job failed at 'file build/TrafficLayer' with exit 127 -- file(1) is not installed in the bare ubuntu:20.04/22.04 container images. Use ls instead of adding a package for a one-line diagnostic.
3 tasks
VehDataMsgDefs.h declares uint8_t/uint16_t/uint32_t members but included <cstdint> only inside #ifdef RS_DSPACE. MSVC and gcc <= 12 pull the header in transitively so nobody noticed; gcc 13 (Ubuntu 24.04) does not, and the types fail to parse. The members are then silently absent from the structs, so the real failure surfaces 24 lines later as "VehFullData_t has no member named 'color'" in MsgHelper -- a cascade with five root causes. Same shape as the RS_DSPACE-guarded close(): a need that is general got scoped to the one toolchain that happened to notice it first. Also added explicit <cstdint> to MsgHelper.h and SocketHelper.h, which use the fixed-width types and were relying on a transitive include from VehDataMsgDefs.h. Verified: all 6 TUs compile clean on Ubuntu 24.04 (gcc 13.3, glibc 2.39); 20.04 still builds, links and passes the smoke tests; Windows TrafficLayer Debug+Release and both DriverModel DLLs still build.
24.04 (gcc 13, glibc 2.39) catches what 20.04 and 22.04 structurally cannot: gcc 13 dropped many transitive includes, so a header using uint32_t without including <cstdint> builds fine on gcc 9/11 and MSVC and fails only there. That is not hypothetical -- it is the VehDataMsgDefs.h bug fixed in the previous commit, which this leg found on its first run. Verified locally before adding: SUMO 1.22's libtracicpp builds on gcc 13, and FIXS builds, links and passes both smoke checks on 24.04. This is compile coverage, not reach: glibc is forward-compatible, so 24.04 users can already run the focal artifact.
One conflict, in ConfigHelper.cpp's subscription switch: dev_v0.9.0 taught the 'intersection' case to accept 'all' alongside id/name (a silent-drop bug fix), while #65 scoped the enum to stop 'link' colliding with POSIX link(2). The two changes are orthogonal -- resolved by keeping both.
…ndored dev_v0.9.0 dropped the vendored CommonLib/libsumo from git (#238) -- on Windows it is restored by fetch_native_deps.ps1, which has no Linux counterpart. The CMake build hard-coded CommonLib/libsumo as an include dir, so after merging dev_v0.9.0 the Linux build failed with "libsumo/libtraci.h: No such file". Now the headers are located: the fetched CommonLib/libsumo when present, otherwise the SUMO source checkout we already build libtracicpp from, which carries the headers for exactly the version being linked so the two cannot drift. CI stages just src/libsumo/*.h next to the library, keeping the cache small rather than holding all of SUMO's src/. Verified on the merged tree: Ubuntu 20.04 builds, links, and passes --help, the VISSIM-rejection check and a real repo config parse, with 0 warnings in FIXS code. Windows TrafficLayer Debug+Release build after fetch_native_deps.
#238 moved libsumo out of git, but only the PowerShell fetcher was written, so Linux had no supported way to obtain it -- the CI workflow open-coded a SUMO build inline and a developer had to copy those steps by hand. The modes are deliberately NOT symmetric with the .ps1, because the rolling 'fixs-native-deps' release carries Windows .lib/.dll only: --mode source (default) build libtracicpp from SUMO at the version pinned in dependencies.yaml, install into CommonLib/libsumo. --mode prebuilt fails with an explicit message until Linux assets exist, rather than downloading binaries that cannot load here. Install layout matches what the .ps1 produces, so after running it a plain `cmake -S . -B build` finds everything with no -D flags, exactly as on Windows. CMake also learned to look in CommonLib/libsumo/bin, so FIXS_SUMO_HOME is now only needed for a pre-existing SUMO build elsewhere. CI calls this script instead of duplicating its steps, so the documented developer path is the tested one: if it breaks, CI goes red. The cache now holds the installed CommonLib/libsumo, which is the script's own sentinel, so a cache hit skips the SUMO build entirely. Verified on Ubuntu 20.04: builds and installs, is idempotent on a second run, rejects --component carla and --mode prebuilt with actionable messages, and runs the loadability check that the .ps1 does (an unresolved NEEDED entry is the Linux equivalent of the missing geos_c.dll in #70). Windows TrafficLayer still builds afterwards -- both platforms' artifacts coexist in CommonLib/libsumo since they are the same SUMO version.
Counterpart of dispatch.bat, deliberately narrower because most of what the .bat builds cannot exist on Linux -- VISSIM DriverModel DLLs, CarMaker, dSPACE and the MATLAB MEX are all licensed Windows toolchains. What remains is: toolchain check -> fetch native deps -> configure -> build -> smoke -> summary, with the same per-step OK/FAILED reporting shape. scripts/dispatch/dispatch.sh [--clean] [--debug] [--no-deps] [--no-smoke] [--jobs N] Builds INTO the repo on purpose: TrafficLayer locates the SUMO runtime by walking up from its own executable directory, so a binary parked outside the tree cannot find CommonLib/libsumo/bin at run time. Logs land in build/, which is already gitignored, so a build leaves nothing behind. Also fixes an include-order trap: generate_version.ps1 leaves a CommonLib/RealSimVersion.h behind on a machine that also builds with MSBuild. Because the include is quoted, CommonLib won the search and a Linux build could silently embed the version from the last Windows run. The CMake-generated header now comes first. CI never saw this -- that file is gitignored, so it only bites on a dual-toolchain machine.
…ld path
Windows and Linux bind SUMO at different times, and the Linux side was relying
on an accident.
Windows DELAY-loads libsumocpp.dll, so ConfigureSumoLibraryPath's
SetDllDirectoryA() at startup is early enough to steer the search. On Linux
libtracicpp.so is a DT_NEEDED entry that the loader resolves BEFORE main()
runs, so the same function's setenv(LD_LIBRARY_PATH) cannot affect this
process at all -- by the time it executes, the library is already bound. (It
still matters for a child SUMO process, so it stays.)
What was actually making the binary work was CMake's default: it baked the
BUILD MACHINE's absolute library directory into RUNPATH. That runs where it
was built and nowhere else -- the CI artifact carried /__w/... and would have
failed to start on any developer's machine that downloaded it.
Now RUNPATH leads with $ORIGIN/../CommonLib/libsumo/bin, the direct analogue
of the exe-relative DLL search on Windows. Verified by copying the binary and
the .so into an unrelated directory tree: it runs there with LD_LIBRARY_PATH
unset.
Follow-on changes so this cannot regress unnoticed:
* the CI smoke steps no longer export LD_LIBRARY_PATH, so a regression to an
absolute RUNPATH fails in CI instead of in a user's hands, and the step
prints the RUNPATH it is relying on.
* the artifact now ships libtracicpp.so alongside the binary with the
relative layout preserved, so the extracted tree runs as-is. Previously it
shipped an executable with no library to load.
* dispatch.sh drops its LD_LIBRARY_PATH export for the same reason.
…erge
Three contexts see three different layouts around the executable, and the
previous single relative RUNPATH only served one of them:
cloned repo / CI build/TrafficLayer + <repo>/CommonLib/libsumo/bin
released bundle TrafficLayer + libraries beside the exe
(8_create_zip.ps1 stages executables at the zip root)
staged install bin/TrafficLayer + lib/
RUNPATH now lists all of them as $ORIGIN-relative candidates, resolved by the
loader at run time. Verified by reconstructing each layout in an unrelated
directory and running with LD_LIBRARY_PATH unset: all three work from one
binary.
Nothing about SUMO is hardcoded any more. Version, upstream URL and install
location all come from dependencies.yaml -- fetch_native_deps.sh parses the
sumo block, and CMake reads the same 'location' key, so the two cannot
disagree and moving the vendor directory is a one-line yaml edit.
One residue, deliberately left: CMake also appends the absolute directory of
the library it linked, as the LAST RUNPATH entry. SKIP_BUILD_RPATH would
remove it but also drops the explicit list, leaving nothing that resolves
(measured -- all three layouts then fail). It is harmless: the $ORIGIN
candidates are tried first and a non-existent path is skipped. INSTALL_RPATH,
which `cmake --install` uses, carries only the relative list.
…ob does
release.yml keys its libsumo cache on hashFiles('dependencies.yaml'). The Linux
job keyed on the parsed version alone, which would reuse a stale cache if the
'source' URL changed while the version did not -- and fetch_native_deps.sh now
reads that URL from the same file. Same key shape on both platforms.
Linux now takes the same path Windows already does: download a SHA-256-verified
asset from the public rolling 'fixs-native-deps' release instead of building
SUMO. Fetching is seconds rather than ~7 minutes, and needs no SUMO build
dependencies at all.
Nothing upstream could be reused. SUMO publishes no Linux binary for a pinned
version -- ppa:sumo/stable is the only prebuilt Linux distribution and it tracks
a different version (1.27.1 today, against the 1.22.0 pin). So the asset is
built from the pinned source and published by us:
scripts/pack_native_deps.sh [--publish]
-> libsumo-1.22.0-linux-x86_64.zip (+ .sha256, same sidecar format as
the .ps1), containing libsumo/*.h and libsumo/bin/libtracicpp.so --
the same layout as the Windows asset, so one fetch script and one
CMake search serve both platforms.
ONE asset covers the whole matrix. glibc is forward-compatible only, so it is
built on the oldest supported distro and the packer REFUSES to publish from a
newer one unless overridden. Verified rather than assumed: the focal-built
asset was consumed on 24.04 (glibc 2.39, gcc 13) -- checksum verifies, the .so
resolves, FIXS builds against it with 0 warnings and runs with LD_LIBRARY_PATH
unset.
397 KB, against 127 MB for the Windows asset, because only libtracicpp is
shipped -- see #300 on whether Windows needs the other 100 DLLs either.
--mode source remains, and is the escape hatch for bumping the SUMO version
before an asset exists. It also grew a fix: a build directory is bound to the
source tree it was configured from, so switching between --sumo-src and the
default checkout used to fail with a CMake cache mismatch; the stale cache is
now dropped automatically.
…osed
SUMO <-> TrafficLayer <-> Python echo client now runs end to end on Linux:
the client receives vehicle data ("Step 20000 | Time: 2000.00s | Vehicles: 1")
while SUMO steps and TrafficLayer relays. Nothing before this had exercised a
running simulation -- CI only proved compile, link and config-parse -- and
running it immediately surfaced two real defects that no amount of building
would have shown.
1. SO_REUSEADDR was never set (only TCP_NODELAY). Linux refuses to bind a port
that still holds a TIME_WAIT socket, so restarting TrafficLayer inside the
~60 s window failed with EADDRINUSE. Windows allows that rebind by default,
which is why this never appeared there. Set on POSIX only and deliberately
NOT on Windows, where SO_REUSEADDR has different, unsafe semantics (it lets
a second process bind a port already in use).
2. The POSIX signal handler could not stop the process. glibc's signal()
installs handlers with SA_RESTART, so a blocking call that a signal
interrupts is RESTARTED. The handler only raises a flag that the main loop
reads, so a TrafficLayer parked in accept() waiting for a client never
returned to the loop and ignored SIGTERM permanently -- a `timeout 15` probe
from Aug 07 was found still alive days later, holding port 2444 and blocking
every subsequent run. Replaced with sigaction() and sa_flags = 0, so the
blocking call returns EINTR, which the recv/select paths already check for.
Supporting pieces:
* fetch_native_deps.sh --with-server builds the PINNED sumo server from the
same checkout. libtraci is only a client; a co-sim needs a server, and
neither platform's release ships one. Building it guarantees client and
server are the same version, which `apt install sumo` cannot (focal ships
1.4.0, ppa:sumo/stable is on 1.27.1, against our 1.22.0 pin).
* tests/Python/SimpleEchoClient/run_simple_echo_client.sh, the counterpart of
the .bat. It takes the .bat's manual-launch path rather than the auto-launch
one, because auto-launch spawns sumo-gui -- fox/OpenGL and a display -- so it
starts a headless sumo itself and stages a config with EnableAutoLaunch:false
(the committed config.yaml is never modified). --gui opts back in.
It invokes the client by ABSOLUTE path: simple_echo_client.py finds CommonLib
via Path(__file__).parents[3], and __file__ is only guaranteed absolute from
Python 3.9. Focal ships 3.8, where the relative path raises IndexError -- so
this would have failed for any 3.8 user, Windows included.
Teardown escalates SIGTERM -> SIGKILL, since bug 2 above means a TrafficLayer
waiting for a client still cannot be stopped politely.
…ebuilt The --with-server refactor let control fall through to the source-build path even when nothing needed building. On a cache hit (or after the prebuilt download) the script still ran the toolchain/xerces/proj check and died with "SUMO's build needs packages that are not installed" -- which is exactly the case CI hits, since the runners deliberately no longer install those. The guard that was supposed to stop this was written but silently never applied: that particular string replacement did not match, and unlike the others in the same change it was not asserted. Broke all three Linux legs. Reproduced the CI conditions before and after, on a distro with no SUMO build dependencies installed: cache hit -> exit 0, cache miss -> download, verify, extract, exit 0. Neither path now touches the source-build machinery.
CARLA's GitHub releases carry no assets at all, so the C++ client SDK is
published nowhere: it exists only inside a built CARLA source tree, under
PythonAPI/carla/dependencies. Building it needs UE4's clang -- Setup.sh derives
CC/CXX from $UE4_ROOT -- which no CI runner has. So the asset is produced on a
box that does have one and consumed everywhere else, which is the same shape as
the libsumo asset and needs the same machinery.
scripts/pack_native_deps.sh --component carla
-> libcarla-0.9.15-linux-x86_64.zip (+ .sha256), 30 MB
30 MB, against the 468 MB the Windows -Mode source path copies wholesale,
because only what VirCarlaEnv actually compiles and links against is shipped:
* Headers: include/carla plus include/system/{boost,rpc,recast}. Established
with `g++ -M` over the CARLA API VirCarlaEnv includes, not by eye. Left out:
libpng16, png*.h, moodycamel, pugixml, odrSpiral, OSM2ODR.h -- nothing
reaches them; they belong to the server/osm2odr side of that directory.
* Archives: the set VirCarlaEnv.vcxproj links, minus the ones with no Linux
counterpart (zlib/zlibstatic are the system libz here; boost_python310 and
carla_client_debug are vestigial on Windows too -- no TU references them).
--strip-debug takes them from 240 MB to 19 MB: libcarla_client.a alone is
217 MB unstripped and 12.6 MB after, and still links.
The verification step is the point of the script, not an afterthought. libsumo
load-tests its .so because a vendored copy silently lacked geos_c.dll for
months (#70); the equivalent exposure here is an ABI mismatch, since the SDK is
built by UE4's clang while every consumer uses the distro's g++. So the packer
compiles AND links a probe against the STAGED tree using the same CARLA API
surface VirCarlaEnv does, and separately refuses an archive carrying libc++
symbols -- the silent std::string corruption #65 Q1 warned about. An asset that
cannot build VirCarlaEnv cannot be published.
dependencies.yaml: carla.location was "VirCarlaEnv", documentation that never
matched anything -- fetch_native_deps.ps1 has always installed to
CommonLib/libcarla. Corrected, and now actually read (only carla.version was,
by 7_build_info.ps1). Also records the boost version the Linux SDK carries:
0.9.15 pins 1.80.0 but needs a one-line Setup.sh bump to 1.81.0 to build on a
modern toolchain, and since the asset ships its own boost headers it stays
self-consistent even though the Windows SDK is on 1.80.
fetch_native_deps.sh refused --component carla by design, citing the unresolved
client ABI. That question is now answered (the SDK is libstdc++, and the packer
refuses an archive carrying libc++ symbols), so the refusal becomes an
acquisition:
scripts/fetch_native_deps.sh --component carla # 3.7 s
scripts/fetch_native_deps.sh --component carla --mode source \
--carla-root ~/carla_0.9.15 # 1 s
Both modes mirror the Windows .ps1, and both leave the SAME 19 MB tree at
CommonLib/libcarla -- source mode strips its copy of the archives (never the
CARLA tree) rather than dropping 213 MB of debug info into the checkout. One
layout means CMake needs no per-mode branch.
libcarla stays OPTIONAL, unlike libsumo: TrafficLayer does not use it, and a
machine without it just does not build VirCarlaEnv.
Two things that only look like refactors:
* The download+verify+extract block is now get_asset(), shared with libsumo.
There were about to be two copies of "curl, refuse without a .sha256 sidecar,
verify, extract", and the sidecar rule is the one thing in this script that
must not diverge between components.
* yaml_sumo_key() became yaml_key(<block> <key>), because carla needs the same
three lookups. Same guarantee as before: version and install location come
from dependencies.yaml, and since pack_native_deps.sh reads it with identical
code the packer and the fetcher cannot disagree about what an asset is called
or where it lands.
Verification is asymmetric on purpose. The expensive compile+link gate lives in
the packer, where it runs once before publishing; the fetcher only checks the
structure and that carla/Version.h matches the pinned version, so a mismatched
or truncated asset is caught without making every clone pay for a probe build.
Rerunning the SUMO path after the refactor still downloads, verifies and
load-tests libtracicpp unchanged.
All eleven TUs VirCarlaEnv compiles now build on Linux with g++ -std=c++17. The entire source cost was three lines in two files, which is what the #174 backend-agnostic refactor bought: VirEnvCore, EgoDriver, CarlaBackend, BridgeHelper and DebugHelper were already Win32-free as written. * DataLogger.cpp included <windows.h> for one CreateDirectoryA call. That include is why the file could not be compiled anywhere else; PlatformCompat's createDirectory() has the same best-effort semantics on both platforms (already-exists is success), and the header is the reason it stays C++14 -- DataLogger is compiled by the CarMaker/dSPACE targets too. * mainVirCarla.cpp tested WSAGetLastError() twice, on the recv/step and send paths, to distinguish a real socket failure from an interrupted call. Both now read FIXS::Platform::socketErrorCode() ONCE into a local and compare against kSocketErrInterrupted/kSocketErrFault -- the same constants SocketHelper and TrafficLayer use. Reading the error once is not cosmetic: the old form called WSAGetLastError() twice per branch, and errno can be clobbered by anything between the two reads. Compile-checked TU by TU against the real SDK rather than by grep, which is what caught that these were the only sites -- the #65 audit's search-based count of the same files missed defects that were not Win32 API calls.
VirCarlaEnv compiles and links on Linux, so the second of the two targets #65 set out to port is done. 11 TUs, ~10 s from clean, zero warnings from FIXS code, and the binary needs no CARLA shared library at run time -- the client SDK is static, so ldd shows only libstdc++/libm/libgcc_s/libpthread/libc. The target is OPTIONAL, which TrafficLayer is not. It needs the CARLA client SDK, which no distro packages and CARLA publishes nowhere, so a machine without it must still build TrafficLayer: a missing SDK is a configure-time skip with the fetch command in the message, never an error. Where the SDK lives comes from dependencies.yaml, read the same way libsumo's location is, so this file and fetch_native_deps.sh cannot disagree. Three details worth stating: * The archives are listed dependency-first because a static link is order-sensitive -- carla_client pulls rpc, boost::filesystem and the Recast/Detour navigation code. * Four entries in VirCarlaEnv.vcxproj are deliberately NOT carried over: libsumocpp/libtracicpp (no TU in this target references libsumo, libtraci or TrafficHelper), boost_python310 (nothing uses the Python bindings), and zlib/zlibstatic (the system libz here). They are vestigial on Windows too; dropping them there is a separate change to a file #65 does not touch. * include/system is included as SYSTEM. boost 1.81 and rpclib do not compile warning-free under this project's flags and are not ours to fix; without this, their warnings would bury ours. check_source_lists.py now guards both targets instead of hardcoding TrafficLayer. Conditional-block handling became per-target data rather than one regex, since VirCarlaEnv has no Windows-only TUs to exclude, and a .vcxproj that has gone missing is now a failure rather than a silent pass -- a guard that quietly stops guarding is the failure mode it exists to prevent. Verified by deleting a TU from the CMake list and watching it fail with that TU named.
scripts/dispatch/dispatch.sh --with-carla fetches the CARLA client SDK and builds both targets; without the flag the script behaves exactly as before. Opt-in because the SDK is a 30 MB download that only Carla users need and TrafficLayer neither links nor requires -- but once it is present the target builds on every later run without the flag, since CMake just finds it. A failed libcarla fetch is a warning, not an abort, mirroring how the Windows .ps1 treats it: nothing else in the build depends on it. And a missing SDK is reported as SKIPPED rather than passing silently, so "VirCarlaEnv did not get built" is never something you have to notice yourself. Smoke check is --help only. Carla needs a GPU and this script has to run on a headless runner, so what is verified is that the binary and its statically linked CARLA client load at all -- which is the part that can break from a bad asset. Actually driving Carla is a dev-box exercise. Warning count now excludes /libcarla/ alongside yaml-cpp: boost 1.81 and rpclib arrive inside the SDK and are no more ours to fix than the vendored yaml-cpp is. FIXS-side warnings stay at 0 on this build.
…rove it The first libcarla asset was unusable on Ubuntu 20.04 -- the one distro it exists to serve. It linked here, and on 22.04 and 24.04, and failed on stock focal with undefined references to std::__exception_ptr::exception_ptr::_M_addref (gcc 12+, from carla_client's boost::asio) and std::__throw_bad_array_new_length (gcc 11+, from rpclib). The cause is a gap the existing distro guard cannot see. This box IS Ubuntu 20.04, so /etc/os-release passes, but it has libstdc++6 13.1.0 installed -- normal on a box that builds CARLA. UE4's clang-10 then compiles against GCC 13's libstdc++ headers, emitting references focal's stock libstdc++ (10.5) does not export. "Built on the oldest distro" and "links against the oldest distro's libraries" are not the same claim, and only the first was being checked. Found by running the CI legs locally in containers before pushing them. 22.04 and 24.04 were green; stock 20.04 was not. Nothing on this machine could have found it -- which is the point. Fix, in two halves: 1. The payload. libcarla_client.a and librpc.a are rebuilt with focal's stock g++-9 and replace the clang-10 ones; the other eight archives are clean as built (C code, or no post-gcc-10 libstdc++ entry points). Verified on stock 20.04, 22.04 and 24.04, and the corrected asset is republished. Worth recording: that rebuild needs NO UE4 and no `make setup`. UE4's clang is required for the SERVER half (libc++); the client is libstdc++ and builds with the distro compiler once Build/ has the third-party deps. #65's Q2 asked whether a client-only build works without UE4_ROOT -- for the part we ship, it does. 2. The gate, so this cannot be published again. The check has to differ by artifact kind, and the difference is the whole lesson: * A SHARED LIBRARY records the symbol VERSION NODES it needs and the loader demands exactly those, so reading them off the .so answers the question completely. Applied to libtracicpp.so, which had the same exposure and passes (max GLIBCXX_3.4.22 <= 3.4.28). * A STATIC ARCHIVE records unversioned names, so the version is whatever the final link picks. Probing on THIS box over-reports (condition_variable::wait binds to GLIBCXX_3.4.30 here and to an older node on focal -- both correct) and under-reports the failure that matters. So the authoritative check is a link inside a stock ubuntu:20.04 container, plus a cheap name screen for the three symbols already known to break, which is what a box with no container runtime falls back to (loudly). Verified both directions: the packer now rejects the original payload naming the offending symbols, and accepts the rebuilt one after the container link. doc/Carla_Linux_building.md gains Part Four: how the C++ client SDK is packed and consumed, why it needs no UE4, and the rebuild recipe the error messages point at.
The Linux job now fetches the prebuilt CARLA client SDK and builds, links and smoke-tests VirCarlaEnv alongside TrafficLayer, on 20.04, 22.04 and 24.04. The distro spread earns its keep twice here. The SDK is built on focal and consumed by gcc 9, 11 and 13, so a libstdc++ ABI break shows up as a link failure on the newer legs -- and the focal leg is the one that catches an asset built against a newer libstdc++ than focal ships, which is exactly how the first asset was found to be broken. Two deliberate choices: * A skip is a failure here. Configure skips VirCarlaEnv when the SDK is missing, which is right for a developer and wrong for CI: the fetch step is supposed to have provided it, so a skip means the asset is broken and must fail rather than quietly shrink the job's scope to what still works. * ldd is asserted, not just printed. VirCarlaEnv links the CARLA client statically, so its dynamic dependencies must stay libstdc++/libm/libgcc_s/ libpthread/libc. Anything CARLA-shaped appearing there means the artifact stopped being self-contained. Carla itself is never started: it needs a GPU. What CI proves is compile, link, load and main(); anything about Carla RUNNING on Linux needs a dev box. All three legs were run locally in ubuntu:20.04/22.04/24.04 containers before this was pushed -- the same habit that caught the ABI break.
…exposed SUMO -> TrafficLayer -> VirCarlaEnv -> CARLA now runs end to end on Linux. On the SimpleLoop scenario against a CARLA 0.9.15 server with simple_loop.xodr loaded: the bridge connects, spawns 43 SUMO vehicles as CARLA actors, poses them for the whole run, and despawns every one at shutdown (0 actors left in the world afterwards). TrafficLayer logged no errors; the bridge logged two "failed to spawn" warnings out of 45, which is ordinary CARLA spawn-point contention. run_sumo_carla_demo.sh is the counterpart of the .bat, and differs from it twice, both so it can run unattended on a headless box: * it starts HEADLESS sumo and stages EnableAutoLaunch:false, because the auto-launch path spawns sumo-gui -- fox/OpenGL and a display. Same approach run_simple_echo_client.sh already takes; the committed config is never modified. --gui opts back in. * it does NOT start or stop CARLA. Bringing the server up is machine-specific (package vs source, GPU, RenderOffScreen), and a script that killed a server someone else was using would be worse than one that asks. It checks the RPC port is listening before starting anything, so a wrong port fails in a second instead of leaving SUMO and TrafficLayer running against nothing. THE BUG. The committed config publishes on port 440. Windows binds that happily; Linux does not -- ports below 1024 need root or CAP_NET_BIND_SERVICE, so TrafficLayer died immediately with "bind() failed: Permission denied" and the run went nowhere. Running a co-simulation as root is not an answer, so the staged config shifts any sub-1024 port up by 4000, in BOTH places -- the ApplicationSetup port and CarlaSetup.CarlaClientPort are the two ends of one socket, and moving only one produces a bridge that connects to nothing. This is the second time building proved nothing about running: CI had been green for the whole port, and the privileged-port failure was in the first second of the first real run. Two things this run leaves open, neither blocking: * The pinned SUMO 1.22 server does not build on this box -- SUMO picks up the system libfmt 6 headers and fails to link (fmt::v6::format_error). The run used the distro's sumo 1.20 against the 1.22 libtraci client, which TraCI accepted. Worth pinning down before anyone depends on --with-server here. * The bridge prints a CARLA version-mismatch warning: "client 0.9.15-dev-dirty / server 0.9.15". Harmless -- the stamp comes from the modified CARLA tree the SDK was built from -- but it will keep appearing until the asset is built from a clean checkout.
cjffly
changed the title
Contributor
Update: VirCarlaEnv now builds and runs on Linux tooPicking up the handoff's "next task". The PR title, body and scope are updated — this comment is what changed since the last review pass, in case the earlier body was already read. Six commits, smallest first:
The two findings worth a reviewer's attention (both detailed in the body):
Evidence: 0 actors left in the world after shutdown; 0 errors in Answers two of the issue's open questions. Q1 (client ABI): libstdc++, verified on the artifact, and the packer refuses an archive carrying libc++ symbols. Q2 (client-only build without Known-open, none blocking:
|
It needed three things done by hand first -- a built tree, a running CARLA, and simple_loop.xodr loaded into it -- so it was a script you had to read before you could use it. Now `./run_sumo_carla_demo.sh` with no arguments brings CARLA up, loads the map, runs the co-simulation and takes it all down again. The reason it did not start CARLA before was real, and is still honoured: a box that develops Carla usually has a session on the conventional port 2000, and a script that spawned 40 vehicles into it -- or killed it at teardown -- would be a nasty surprise. So this starts its OWN server on 2100 and stops only what it started. --carla-port at something already listening ATTACHES instead: no map load (that would replace the world you are looking at) and no shutdown. Verified both ways, including that an attached server still holds its own map and no leftover actors afterwards. Three things this shook out, none of which the first version would have survived in someone else's hands: * $CARLA_ROOT outranked ~/.fixs/carla.json, and on this box pointed at a 0.9.14 tree while carla.json named the pinned 0.9.15 -- so the demo launched the wrong server against a 0.9.15 client. carla.json now wins, and every candidate must carry a VERSION matching dependencies.yaml. A wrong server is worth refusing loudly: unrefused it shows up as a connect timeout, which reads like a bug in the bridge. * Readiness was "is $CARLA_PID alive", but CarlaUE4.sh hands off to CarlaUE4-Linux-Shipping and the launcher can exit first. That read as a crash and aborted a server that was starting normally. Liveness is now "is a CarlaUE4 serving OUR port". * Teardown left the server running. UE4 acknowledges SIGTERM (RequestExit appears in carla.log) and can then outlive the shell, so the kill escalates like the other three processes already did. A survivor holds the RPC port and the GPU, and the NEXT run would silently attach to it instead of starting clean. And a trap worth naming: `pkill -f -- -carla-rpc-port=N` matches the command line of any shell that typed that string, so it kills the shell running the teardown. Observed while testing this. The script resolves explicit pids and excludes itself instead.
…isplay
The first version ran sumo headless and CARLA -RenderOffScreen on every box, so
a co-simulation that really did run -- 43 vehicles spawned, posed and despawned
-- produced nothing on screen but the word PASS. Those defaults are right for
ssh and CI and wrong for a demo, and the .bat shows both windows.
So the default now follows the environment: windows when $DISPLAY (or
$WAYLAND_DISPLAY) is set, headless when it is not, which keeps the same script
usable over ssh and on a runner. --headless forces it off, --gui/--render force
the visible side on. sumo-gui being absent downgrades just that half rather
than failing the run, since it is a separate package from sumo.
Verified with xwininfo while running: the SUMO window is real and mapped
("simple_loop_ego.sumocfg - SUMO 1.20.0", 913x902). CARLA launched without
-RenderOffScreen still produced no window in this environment -- and neither
has the long-running CARLA already on port 2000 -- so that half is reported,
not claimed.
Also fixes a precedence bug in the CARLA root pick that the version gate had
been masking: --carla-root and the ambient $CARLA_ROOT were the same variable,
so the documented order (flag > carla.json > env) was really (flag-or-env >
carla.json). They are separate variables now, and the third loop entry is no
longer a duplicate of the first.
…lready does The .sh had grown to 331 lines against the .bat's 101, which was a fair thing to be asked about. The .bat is not doing less -- it is SPLIT: CARLA bring-up lives in Carla\launch_carla.bat, waiting for the RPC port in Carla\wait_for_rpc.ps1, and the self-check in verify_sumo_carla.py. Linux had no counterpart to the first two, so I had inlined both into the demo instead of adding the missing helper. Carla/launch_carla.sh is that helper: start a server and return when its RPC port answers (launch_carla.bat + wait_for_rpc.ps1 in one file, since on Linux the wait is four lines rather than a separate PowerShell script), plus --stop, which Windows has no counterpart for because the .bat leaves its windows for the user to close. Any other Linux Carla probe can now use it. run_sumo_carla_demo.sh 331 -> 173 lines (114 code) Carla/launch_carla.sh 108 lines (64 code), reusable The demo file now reads like the .bat: check the binaries, stage the config, start three processes, report. What is left that the .bat does not have is platform-forced, not gold-plating: the privileged-port remap (Linux refuses to bind 440), SIGTERM->SIGKILL escalation (TrafficLayer in accept() ignores SIGTERM, and UE4 outlives a polite kill), display detection (Windows gets a window per process free from `start`), and own-port isolation so a CARLA someone else is using is never touched. Verified after the split: 43 vehicles spawned and posed, clean teardown, no leftover CARLA or sumo-gui, and the pre-existing server on 2000 untouched.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes TrafficLayer and VirCarlaEnv compile, link and run on Linux, and adds a CI gate for both on Ubuntu 20.04, 22.04 and 24.04.
The MSBuild/
.vcxprojbuild is untouched and remains the Windows build of record. Migrating Windows onto CMake is deliberately left as follow-up work — this PR only adds a CMake path beside it, written so that migration is later a deletion rather than a rewrite.Both targets are verified beyond compiling: SUMO ↔ TrafficLayer ↔ Python echo client, and SUMO ↔ TrafficLayer ↔ VirCarlaEnv ↔ CARLA, both run end to end on Ubuntu 20.04. Windows was re-verified after the TrafficLayer work (Debug+Release plus both DriverModel DLLs); the VirCarlaEnv changes touch no
.vcxprojand noProprietaryFiles.Scope
libtracipath) and VirCarlaEnv (CARLA C++ client path).connect()that can never succeed. The two Windows-only DSProxy TUs drop out of the Linux source list.--helponly. The runtime evidence below comes from a dev box.The defects were found by compiling, not by grepping
The feasibility audit in #65 enumerated the Win32 surface by search. That method structurally cannot find portability defects that aren't Win32 API calls, and three such defects turned up immediately:
const double M_PImember inTrafficHelper.hM_PIundefined without_USE_MATH_DEFINES; glibc always defines it, so the declaration becomes a syntax errorlinkinConfigHelper.cpplink(2). Made the enumenum classso the collision class cannot recurTimer_t/QueryPerformanceCountersteady_clockSharpest confirmation of the audit's main thesis:
SocketHelper's plain-POSIX tier declaredclientAddrLenassize_twhereaccept()needssocklen_t*— 64-bit vs 32-bit, so that branch could never have compiled. Nobody had ever built it.Correctness fixes (second commit)
All pre-existing and affecting Windows equally — none are Linux-only:
close()was guarded onRS_DSPACEat all 10 sites, so a plain-POSIX build calledshutdown()and never closed the fd."error: "with no value, at 18 sites — now reportsstrerror(errno), so a Linux socket failure is diagnosable instead of anonymous.TrafficHelper::checkIfEgoExistreturned garbage (UB) when nothing was subscribed yet — control fell off the end of a non-void function. It gatesSimulationMode: 1("wait until the ego enters the network"), so a garbage non-zero could declare the ego present before it exists.size_tprinted through%d— wrong width out of the varargs.FIXS code now compiles warning-free on gcc 9 (3 remaining warnings are inside vendored yaml-cpp).
Two design decisions worth reviewing
CommonLib/PlatformCompat.his header-only, and C++14. Five.vcxprojfiles compileConfigHelper.cpp, two of them inside the privateProprietaryFilessubmodule — a new.cppTU would have forced a companion submodule PR and edits to five project files. An inline header changes no project file anywhere. C++14 because CommonLib is consumed by the CarMaker/dSPACE RT targets, which is whyConfigHelperhand-rolled these helpers onshlwapioriginally (#65 Q4); the constraint is kept, not relaxed. The Win32 bodies are the originals moved verbatim, so Windows behaviour is unchanged by construction.The TU list now exists twice. That is the cost of keeping MSBuild authoritative on Windows.
scripts/check_source_lists.pydiffs the.vcxprojand CMake lists and CI fails on divergence, so "added a TU on Windows, forgot Linux" — which would fail nothing on Windows and silently never compile on Linux — cannot pass review.VirCarlaEnv: the source cost was three lines; the dependency was everything
Nine of the eleven TUs were already Win32-free — what the #174 backend-agnostic refactor bought. The whole source change is
DataLogger's oneCreateDirectoryAand twoWSAGetLastErrorsites inmainVirCarla, all routed through the existingPlatformCompat.h. Everything else was the SDK.Acquisition. CARLA publishes no release assets at all, so the C++ client SDK exists only inside a built CARLA source tree. It is now packed and published the same way
libsumoalready is:30 MB, against the 468 MB the Windows
-Mode sourcepath copies wholesale: only the headersg++ -Msays are reachable and the archives that actually link,--strip-debug'd (libcarla_client.aalone goes 217 MB → 12.6 MB and still links).The trap, which cost a published asset. The first one linked here, and on 22.04 and 24.04, and failed to link on stock 20.04 — the one distro it exists to serve. This box is focal, so the existing distro guard passed, but it has
libstdc++613.1.0 installed (normal on a box that builds CARLA), and UE4's clang-10 compiled the client against GCC 13's headers:exception_ptr::_M_addref/_M_releaselibcarla_client.a(boost::asio)std::__throw_bad_array_new_lengthlibrpc.a"Built on the oldest distro" and "links against the oldest distro's libraries" are different claims, and only the first was being checked. Both archives are rebuilt with focal's stock
g++-9and the asset republished.The gate now differs by artifact kind, which is the part worth reviewing:
.sois the whole answer (applied tolibtracicpp.so, which passes: maxGLIBCXX_3.4.22≤3.4.28).condition_variable::waitbinds toGLIBCXX_3.4.30here and an older node on focal — both correct) and misses the failure that matters. So the authoritative check is a link inside a stockubuntu:20.04container, with a cheap name screen as the fallback when no container runtime exists.Q1 and Q2 from the issue are answered. Q1 (libc++ or libstdc++): libstdc++, verified on the artifact (
0_ZNSt3__1symbols), and the packer refuses an archive carrying libc++ symbols. Q2 (does a client-only build needUE4_ROOT): no — UE4's clang is the server half's requirement; the client is libstdc++ and rebuilds with the distro compiler onceBuild/has the third-party deps. That rebuild recipe is nowdoc/Carla_Linux_building.mdPart Four.Running it found what building could not
tests/Sumo/Probes/TrafficLayer_SUMO_Carla/run_sumo_carla_demo.sh(counterpart of the.bat) drives SUMO → TrafficLayer → VirCarlaEnv → CARLA on the SimpleLoop scenario. It is click-to-run: with no arguments it starts its own CARLA on port 2100, loadssimple_loop.xodr, runs the co-simulation and takes everything down. Deliberately not the conventional port 2000 — a box that develops Carla usually has a session there, and this must never spawn 40 vehicles into it or kill it at teardown.--carla-port <listening port>attaches to a server you started instead: no map load, no shutdown. On a CARLA 0.9.15 server withsimple_loop.xodrloaded: the bridge connects, spawns 43 SUMO vehicles as CARLA actors, poses them for the run, and despawns every one at shutdown (0 actors left afterwards).The first attempt died in its first second: the committed config publishes on port 440, which Windows binds happily and Linux refuses without root —
bind() failed: Permission denied. The staged config shifts any sub-1024 port up by 4000, in both places, sinceApplicationSetup.portandCarlaSetup.CarlaClientPortare the two ends of one socket. CI had been green throughout.Notes for reviewers
cmake_minimum_required(VERSION 3.5), so focal's stock cmake 3.16 clears the floor.libtracicppneeds onlyxerces/proj/gdal/zlib/python3-dev, and building just that target avoids the gdal/fox/osg chainlibsumocppwould drag in.RealSimVersion.hwas generated only bygenerate_version.ps1(PowerShell), so a Linux build had no version header at all.cmake/FixsVersion.cmakemirrors its semantics — including thev[0-9]*tag match that ignores the rollinglatest/alpha_*tags ([MAINTENANCE] Automated single-canonical release CI (source-visibility split) + FIXS_VERSION/compat fixes #191) — and generates into the build tree, never the source tree.yaml-cpp/dll.his a generated artifact (generate_export_header), not in git. That's why anything includingyaml.hfails until yaml-cpp is configured; the CMake target depends on it properly.client 0.9.15-dev-dirty / server 0.9.15). Harmless — the stamp comes from the modified CARLA tree the SDK was built from — but it persists until the asset is built from a clean checkout.fmt::v6::format_error). The co-sim ran against the distro's sumo 1.20 with the 1.22libtraciclient, which TraCI accepted. Worth pinning down before anyone depends on--with-serverhere.container:images rather thanruns-on:—ubuntu-20.04was retired from hosted runners in 2025 and is only reachable as a container, and pinning all three the same way keeps glibc/gcc determined by the image. 20.04 is the artifact-worthy one: glibc is forward-incompatible only, so a focal binary runs on jammy, not the reverse.Related Issues / Tasks
Relates to #65. Both targets the issue scoped are now done; the Windows→CMake migration remains, as does #300 (the 104 Windows libsumo DLLs).
Type of Change
Affected Modules / Components
CommonLib/—PlatformCompat.h(new),ConfigHelper.{cpp,h},SocketHelper.{cpp,h},TrafficHelper.{cpp,h}TrafficLayer/TrafficLayer/mainTrafficLayer.cppCMakeLists.txt,cmake/FixsVersion.cmake(new)scripts/check_source_lists.py(new).github/workflows/linux-build.yml(new)No
.vcxprojfile was modified.ProprietaryFilesis untouched — no companion submodule PR is needed.Test Cases
Linux (Ubuntu 20.04, WSL2) — every CI step rehearsed locally before the workflow was committed:
Windows regression:
Not covered: a live SUMO co-sim on Linux (needs a SUMO runtime configured via
SumoSetup.RuntimeLibraryPath), and anything Carla.Environment
libtraci;ENABLE_LIBSUMOremains commented out)libcarla-0.9.15-linux-x86_64.zip; boost 1.81)Checklist
Additional Notes
doc/BUILD.mdis still not touched — it is Windows-only and orphaned from the RTD TOC (#137), so a Linux section belongs there once #137 settles the TOC.doc/Carla_Linux_building.mdis updated (Part Four), because the CARLA SDK rebuild recipe is what the packer's error messages point at and it cannot live in a commit message.