Skip to content

Automate Update Site verification#1397

Merged
AndriiFilippov merged 1 commit intomasterfrom
IEP-1685
Mar 24, 2026
Merged

Automate Update Site verification#1397
AndriiFilippov merged 1 commit intomasterfrom
IEP-1685

Conversation

@AndriiFilippov
Copy link
Collaborator

@AndriiFilippov AndriiFilippov commented Feb 12, 2026

Description

This PR introduces automated update-path testing for Espressif-IDE and Eclipse-based setups using the p2 director on Linux runner.
It validates installation and update flows across multiple scenarios, including stable and nightly update paths.

Fixes # (IEP-1685)

│ └── update-site-tests/
│ │
│ ├── variables.sh
│ │ # root config:
│ │ # - URLs (Eclipse, Espressif)
│ │ # - update sites
│ │
│ ├── detect-latest-eclipse-steams.sh
│ │ # - parses release.xml
│ │ # - detects latest R / M / RC
│ │ # - forms URLs
│ │ # - exports variables
│ │
│ ├── run-all-cases.sh
│ │ # orchestrator
│ │ # - runs Case 1–4
│ │ # - prepares env
│ │ # - download Espressif-IDE
│ │
│ ├── run-single-case.sh
│ │ # case executor
│ │ # - install Eclipse / IDE
│ │ # - executes installIU / update
│ │ # - writes run.log
│ │
│ ├── out/
│ │ └── case*/
│ │ └── logs/
│ │ ├── run.log
│ │ ├── installed-ius.txt
│ │ └── installed-roots.txt

Scheduled daily execution at 00:00 UTC.

Case 1: Eclipse (latest R) → install stable plugin
Case 2: Eclipse (latest R) → install stable → update to nightly
Case 3: Eclipse (latest pre-release: M/RC) → install stable → update to nightly. ⚠️ optional test case (based on Milestone / Release Candidate availability) ⚠️
Case 4: Espressif-IDE (bundled) → update to nightly

Results:
Each test case produces:

run.log – full execution log
installed-ius.txt
installed-roots.txt
https://github.com/espressif/idf-eclipse-plugin/actions/runs/23256402453?pr=1397

Type of change

Please delete options that are not relevant.

  • New feature (non-breaking change which adds functionality)

How has this been tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

  • Test A
  • Test B

Test Configuration:

  • ESP-IDF Version:
  • OS (Windows,Linux and macOS):

Dependent components impacted by this PR:

  • Component 1
  • Component 2

Checklist

  • PR Self Reviewed
  • Applied Code formatting
  • Added Documentation
  • Added Unit Test
  • Verified on all platforms - Windows,Linux and macOS

Summary by CodeRabbit

  • Tests

    • Added an automated Eclipse plugin upgrade test that installs stable and RC builds, checks for conflicts, verifies startup, and writes a structured success/failure report.
  • Chores

    • CI Linux builds now run the upgrade test and always archive detailed logs and a summary report.
    • The active Maven build step in the Linux CI sequence has been disabled.

@coderabbitai
Copy link

coderabbitai bot commented Feb 12, 2026

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Adds a CI step that runs a new Eclipse upgrade test script which downloads/extracts Eclipse, installs a stable and an RC plugin via the Eclipse director, performs log/conflict checks, writes a structured report, and always uploads logs and the report as an artifact. (34 words)

Changes

Cohort / File(s) Summary
CI Workflow
​.github/workflows/ci.yml
Comments out the active Maven build step for Linux, adds a step to run releng/update-site-tests/test-update.sh (exports LOGDIR/REPORT_FILE), then an always-run artifact upload step that uploads logs and report.txt as eclipse-update-site-test-logs.
Eclipse Update Test Script
releng/update-site-tests/test-update.sh
New executable shell script that downloads/extracts Eclipse, prepares work/log/report dirs, installs the stable plugin and then the RC via the Eclipse director with per-step logs and error handling, scans logs for conflicts, lists installed roots, performs a headless startup check, and writes a structured report.txt.

Sequence Diagram(s)

sequenceDiagram
    autonumber
    actor Developer
    participant GitHubActions as "GitHub Actions (CI)"
    participant Script as "releng/update-site-tests/test-update.sh"
    participant Eclipse as "Eclipse (download / director / headless)"
    participant ArtifactStore as "actions/upload-artifact"

    Developer->>GitHubActions: push / open PR
    GitHubActions->>Script: run test-update.sh (env: LOGDIR, REPORT_FILE)
    Script->>Eclipse: download & extract Eclipse
    Script->>Eclipse: install stable plugin (director) -> stable-install.log
    Script->>Eclipse: install RC plugin (director) -> rc-install.log
    Script->>Script: scan logs, list installed roots, create report.txt
    Script->>Eclipse: headless startup check -> headless.log
    Script->>GitHubActions: exit and place REPORT_FILE
    GitHubActions->>ArtifactStore: upload logs + report (always)
    ArtifactStore-->>GitHubActions: artifact stored
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Suggested reviewers

  • kolipakakondal
  • alirana01

Poem

🐰 I hopped into CI at dawn's first light,

I fetched Eclipse and logged each install rite;
Stable then RC, I checked every line,
Scanned for conflicts, stamped the report fine;
Logs bundled up — the upgrade's alright.

🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Merge Conflict Detection ✅ Passed ✅ No merge conflicts detected when merging into master
Title check ✅ Passed The title 'WIP: Automate Update Site verification' directly relates to the main changes, which add automation for update site verification through new CI workflow steps and test scripts.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch IEP-1685

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Fix all issues with AI agents
In `@releng/update-site-tests/test-update.sh`:
- Around line 135-148: The script launches Eclipse with -vmargs
-Djava.awt.headless=false which will fail on headless CI; change the invocation
to either run under Xvfb (wrap "$ECLIPSE_HOME/eclipse" with xvfb-run) or set
-Djava.awt.headless=true, and replace the fragile sleep+kill logic that uses PID
and sleep 20 with a timeout utility (e.g., use timeout to limit launch duration)
and add a robust startup check (verify $PID is running and/or scan
"$LOGDIR/eclipse-launch.log" for a successful startup marker) so the test
reliably times out or confirms a running process instead of blindly killing
after 20s.
- Around line 1-2: Add `set -o pipefail` after the existing `set -e` in
test-update.sh to ensure pipe failures aren't masked, then update the two piped
commands guarded by `PIPESTATUS` (the two places currently checking
PIPESTATUS[0], referenced in the comment as lines with PIPESTATUS checks) so
they won't be prematurely aborted: either temporarily disable `set -e` around
each pipeline or append `|| true` to the pipeline and continue to use the
existing `PIPESTATUS` checks to detect failures; ensure you apply the same
pattern to the RC install pipeline as well so both pipelines reliably surface
errors.
- Around line 111-115: Change the broad grep that searches for "conflict" in
$LOGDIR/*.log to first ensure there are log files (e.g., test if any
"$LOGDIR"/*.log exist and skip check if none), then grep only for p2-specific
error patterns (use grep -iqE -- 'Cannot satisfy|conflicting
requirements|Conflicts detected|Resolution failed' "$LOGDIR"/*.log) so you avoid
benign matches; if grep returns success, print the failure message and exit 1.
Ensure you reference LOGDIR and replace the existing grep -iq "conflict"
"$LOGDIR"/*.log block with this guarded, pattern-specific check.
🧹 Nitpick comments (4)
releng/update-site-tests/test-update.sh (3)

8-18: Hardcoded URLs and versions limit the stated goal of testing against multiple IDE/Eclipse versions.

The PR objective is to verify compatibility "against previous Espressif-IDE versions and recent Eclipse CDT releases," but the script pins a single Eclipse release (2025-09) and a single stable plugin version (v4.0.0). Consider parameterizing these as script arguments or environment variables so the CI matrix (or future callers) can iterate over multiple version combinations.

Example
-ECLIPSE_URL="https://ftp.osuosl.org/pub/eclipse/technology/epp/downloads/release/2025-09/R/eclipse-cpp-2025-09-R-linux-gtk-x86_64.tar.gz"
-STABLE_ZIP_URL="https://dl.espressif.com/dl/idf-eclipse-plugin/updates/com.espressif.idf.update-v4.0.0.zip"
-RC_REPO="https://dl.espressif.com/dl/idf-eclipse-plugin/updates/latest/"
+ECLIPSE_URL="${ECLIPSE_URL:?Please set ECLIPSE_URL}"
+STABLE_ZIP_URL="${STABLE_ZIP_URL:?Please set STABLE_ZIP_URL}"
+RC_REPO="${RC_REPO:?Please set RC_REPO}"

27-27: Dead assignment — ECLIPSE_HOME is overwritten on line 36.

Line 27 sets ECLIPSE_HOME="$WORKDIR/eclipse", but line 36 immediately reassigns it via find. Remove the line 27 assignment to avoid confusion.


33-36: No integrity verification of downloaded artifacts.

wget downloads over HTTPS but there's no checksum verification for the Eclipse tarball or the plugin zip. A corrupted or tampered download would produce confusing failures downstream. Consider adding SHA-256 checks.

.github/workflows/ci.yml (1)

49-55: Consider whether update site test failures should block the entire CI pipeline.

The test downloads external resources (ftp.osuosl.org, dl.espressif.com) that are outside your control. If either host is temporarily unavailable, the build will fail. Consider adding continue-on-error: true if this test is intended as informational, or at minimum add retry logic in the shell script for the wget calls.

Comment on lines +111 to +115
echo "Checking logs for conflicts..."
if grep -iq "conflict" "$LOGDIR"/*.log; then
echo "❌ Conflict detected in installation logs!"
exit 1
fi
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

grep "conflict" is overly broad and may produce false positives or false negatives.

The word "conflict" could appear in benign log messages (e.g., "No conflicts found", class names containing "conflict"). Consider grepping for p2-specific conflict markers or error patterns instead. Also, if no .log files exist the glob will fail under set -e.

Suggestion
-if grep -iq "conflict" "$LOGDIR"/*.log; then
+shopt -s nullglob
+if grep -iqP '(?<!no )conflicts?\b' "$LOGDIR"/*.log; then
     echo "❌ Conflict detected in installation logs!"
     exit 1
 fi

Or better, grep for the specific p2 error messages that indicate real conflicts.

🤖 Prompt for AI Agents
In `@releng/update-site-tests/test-update.sh` around lines 111 - 115, Change the
broad grep that searches for "conflict" in $LOGDIR/*.log to first ensure there
are log files (e.g., test if any "$LOGDIR"/*.log exist and skip check if none),
then grep only for p2-specific error patterns (use grep -iqE -- 'Cannot
satisfy|conflicting requirements|Conflicts detected|Resolution failed'
"$LOGDIR"/*.log) so you avoid benign matches; if grep returns success, print the
failure message and exit 1. Ensure you reference LOGDIR and replace the existing
grep -iq "conflict" "$LOGDIR"/*.log block with this guarded, pattern-specific
check.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🤖 Fix all issues with AI agents
In `@releng/update-site-tests/test-update.sh`:
- Around line 117-124: The -listInstalledRoots invocation in test-update.sh is
currently restricted to -profile SDKProfile so it only reports the RC install;
update the script to list the default profile (where the stable plugin was
installed) as well — either run the org.eclipse.equinox.p2.director
-listInstalledRoots command once without -profile to capture the default profile
or run it twice (once with -profile SDKProfile and once without) and write both
outputs (e.g., to installed-roots-default.txt and installed-roots-sdk.txt) so
the report reflects both stable and RC plugins.
- Line 50: STABLE_REPO is hardcoded to
"file://$WORKDIR/stable-repo/artifacts/update" which breaks if the zip layout
changes; modify test-update.sh to discover the extracted subdirectory
dynamically by searching under "$WORKDIR/stable-repo" for a unique marker file
(e.g., content.jar or artifacts.jar) and set STABLE_REPO to the file:// URL of
the directory that contains that marker. In other words, replace the fixed
STABLE_REPO assignment with logic that uses WORKDIR and a find/lookup of
content.jar or artifacts.jar, assigns the containing directory to STABLE_REPO,
and emits a clear error if no marker is found.
- Around line 56-100: The stable and RC installs are using different p2 profiles
so the RC run is not upgrading the stable install; make both invocations target
the same p2 profile. Update the stable-install block (the
"$ECLIPSE_HOME/eclipse" call that uses "$STABLE_REPO") to include the same flags
used in the RC block: -profile SDKProfile -destination "$ECLIPSE_HOME"
-bundlepool "$WORKDIR/p2" (so FEATURE_ID is installed into the same
profile/bundlepool), or alternatively remove -profile
SDKProfile/-bundlepool/-destination from the RC block if you prefer default
profile behavior — but ensure both invocations use identical
profile/destination/bundlepool settings and remove any stray/duplicate mentions
of -profile SDKProfile elsewhere so the test actually performs an in-place
upgrade.
- Around line 36-37: The current assignment to ECLIPSE_HOME via the find command
can yield an empty string; guard against that by checking ECLIPSE_HOME after the
find (the variable set by ECLIPSE_HOME=$(find "$WORKDIR" -maxdepth 1 -type d
-name "eclipse*" | head -n1")) and if empty, print a clear error message
(including WORKDIR context) and exit non‑zero; otherwise continue and echo the
path as now. Ensure subsequent references use the validated $ECLIPSE_HOME.
🧹 Nitpick comments (2)
releng/update-site-tests/test-update.sh (2)

8-15: Hardcoded URLs will silently break when versions rotate.

ECLIPSE_URL and STABLE_ZIP_URL point to specific release versions. When these URLs are retired or updated, the script fails with no clear indication of why. Consider parameterizing these (via environment variables with defaults) so CI can override them, and add a check after each wget that verifies the downloaded file is valid (e.g., non-zero size or correct content type).

Proposed approach
-ECLIPSE_URL="https://ftp.osuosl.org/pub/eclipse/technology/epp/downloads/release/2025-09/R/eclipse-cpp-2025-09-R-linux-gtk-x86_64.tar.gz"
-STABLE_ZIP_URL="https://dl.espressif.com/dl/idf-eclipse-plugin/updates/com.espressif.idf.update-v4.0.0.zip"
+ECLIPSE_URL="${ECLIPSE_URL:-https://ftp.osuosl.org/pub/eclipse/technology/epp/downloads/release/2025-09/R/eclipse-cpp-2025-09-R-linux-gtk-x86_64.tar.gz}"
+STABLE_ZIP_URL="${STABLE_ZIP_URL:-https://dl.espressif.com/dl/idf-eclipse-plugin/updates/com.espressif.idf.update-v4.0.0.zip}"

149-163: Report always says "Conflict check: OK" — consider making it dynamic.

Line 160 unconditionally writes "Conflict check: OK". While this is technically correct today (the script exits on conflict), it's brittle if the flow is ever changed to be non-fatal. A small improvement:

-    echo "Conflict check: OK"
+    echo "Conflict check: ${CONFLICT_STATUS:-OK}"

This is a minor maintainability nit.

mkdir -p "$WORKDIR/stable-repo"
unzip -q "$WORKDIR/stable.zip" -d "$WORKDIR/stable-repo"

STABLE_REPO="file://$WORKDIR/stable-repo/artifacts/update"
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Hardcoded zip internal path artifacts/update is fragile.

STABLE_REPO assumes the zip extracts into artifacts/update/. If the zip structure changes across plugin versions, this silently breaks. Consider discovering the path dynamically (e.g., by locating content.jar or artifacts.jar inside the extracted directory).

🤖 Prompt for AI Agents
In `@releng/update-site-tests/test-update.sh` at line 50, STABLE_REPO is hardcoded
to "file://$WORKDIR/stable-repo/artifacts/update" which breaks if the zip layout
changes; modify test-update.sh to discover the extracted subdirectory
dynamically by searching under "$WORKDIR/stable-repo" for a unique marker file
(e.g., content.jar or artifacts.jar) and set STABLE_REPO to the file:// URL of
the directory that contains that marker. In other words, replace the fixed
STABLE_REPO assignment with logic that uses WORKDIR and a find/lookup of
content.jar or artifacts.jar, assigns the containing directory to STABLE_REPO,
and emits a clear error if no marker is found.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Fix all issues with AI agents
In `@releng/update-site-tests/test-update.sh`:
- Around line 137-150: The unconditional "✅ Eclipse launched successfully"
should be gated by an actual check: after launching "$ECLIPSE_HOME/eclipse" to
background and capturing PID, wait and then inspect the process exit status
and/or the "$LOGDIR/eclipse-launch.log" contents for a known startup marker
before printing success; treat kill returning 137/143 as expected and consider
them not-a-success for immediate crash, e.g., check `kill -0 $PID` or `ps` to
see if the PID is still running and/or grep the log for a startup line, and only
echo the success message when those checks confirm Eclipse started (otherwise
print an error and exit non‑zero).
🧹 Nitpick comments (2)
releng/update-site-tests/test-update.sh (2)

28-28: Dead assignment — ECLIPSE_HOME is overwritten on line 37.

This initial assignment is never used because find on line 37 replaces it unconditionally. Remove it to avoid confusion about which value is authoritative.

-ECLIPSE_HOME="$WORKDIR/eclipse"

53-58: Capture the pre-install roots to a file for the report.

The output of listInstalledRoots here goes to stdout only (lost once CI scrollback is trimmed). Redirect to a log file so it can be compared with the post-install listing in step 6.

 echo "Listing installed roots before stable:"
 "$ECLIPSE_HOME/eclipse" \
   -nosplash \
   -application org.eclipse.equinox.p2.director \
   -listInstalledRoots \
-  -consoleLog
+  -consoleLog \
+  | tee "$LOGDIR/installed-roots-before.txt"

Comment on lines +137 to +150
echo "Launching Eclipse once..."
"$ECLIPSE_HOME/eclipse" \
-nosplash \
-clean \
-data "$WORKDIR/workspace" \
-configuration "$WORKDIR/configuration" \
-consoleLog \
-vmargs -Djava.awt.headless=false \
> "$LOGDIR/eclipse-launch.log" 2>&1 &

PID=$!
sleep 20
kill $PID || true
echo "✅ Eclipse launched successfully"
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Headless launch issues already flagged; additionally, the success message on line 150 is unconditional.

The prior review covered headless=false and the fragile sleep/kill pattern. On top of that: "✅ Eclipse launched successfully" is printed regardless of whether Eclipse actually started or crashed instantly. Check the process exit status (or at least verify the log isn't empty / contains a startup marker) before declaring success.

 PID=$!
 sleep 20
 kill $PID || true
-echo "✅ Eclipse launched successfully"
+wait $PID 2>/dev/null
+LAUNCH_EXIT=$?
+if [ $LAUNCH_EXIT -ne 0 ] && [ $LAUNCH_EXIT -ne 137 ] && [ $LAUNCH_EXIT -ne 143 ]; then
+    echo "❌ Eclipse launch failed with exit code $LAUNCH_EXIT"
+    exit 1
+fi
+echo "✅ Eclipse launched successfully"

Exit codes 137 (SIGKILL) and 143 (SIGTERM) are expected from the kill.

🤖 Prompt for AI Agents
In `@releng/update-site-tests/test-update.sh` around lines 137 - 150, The
unconditional "✅ Eclipse launched successfully" should be gated by an actual
check: after launching "$ECLIPSE_HOME/eclipse" to background and capturing PID,
wait and then inspect the process exit status and/or the
"$LOGDIR/eclipse-launch.log" contents for a known startup marker before printing
success; treat kill returning 137/143 as expected and consider them
not-a-success for immediate crash, e.g., check `kill -0 $PID` or `ps` to see if
the PID is still running and/or grep the log for a startup line, and only echo
the success message when those checks confirm Eclipse started (otherwise print
an error and exit non‑zero).

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Fix all issues with AI agents
In `@releng/update-site-tests/test-update.sh`:
- Around line 5-8: The cleanup block uses $WORKDIR and $LOGDIR before their
defaults are set, causing rm/mkdir to run with empty paths; move the default
assignments (the parameter expansions ${WORKDIR:-...} and ${LOGDIR:-...}) so
they occur before the cleanup block or alternatively move the entire cleanup
block after the configuration section where WORKDIR and LOGDIR are set; ensure
only one mkdir -p is kept (remove the redundant second mkdir -p) and update
references to WORKDIR and LOGDIR accordingly.
🧹 Nitpick comments (4)
releng/update-site-tests/test-update.sh (4)

162-179: Report generation looks correct but always says "Conflict check: OK".

Because the script exits on line 121 if a conflict is detected, line 172 will only ever print "OK". This is technically accurate but means the report is never generated in the failure case. Consider trapping on exit or deferring the conflict-check exit so the report is always produced (it could be useful for debugging failed runs, especially since CI uploads it as an artifact).

Sketch: generate report even on failure using a trap
# Near the top of the script, after REPORT is defined:
generate_report() {
    echo "Generating report..."
    {
        echo "Espressif IDE Upgrade Test Report"
        echo "================================"
        echo ""
        echo "Stable install exit code: ${STABLE_EXIT:-N/A}"
        echo "RC install exit code: ${RC_EXIT:-N/A}"
        echo ""
        echo "Installed Roots:"
        cat "$LOGDIR/installed-roots.txt" 2>/dev/null || echo "(not available)"
        echo ""
        echo "Conflict check: ${CONFLICT_STATUS:-UNKNOWN}"
        echo ""
        echo "Logs directory: $LOGDIR"
    } > "$REPORT"
}
trap generate_report EXIT

Then set CONFLICT_STATUS="OK" or CONFLICT_STATUS="FAILED" accordingly before exiting.


14-15: Hardcoded Eclipse version 2025-09 will become stale.

The PR objective is to test compatibility with "recent Eclipse CDT releases." Consider parameterizing ECLIPSE_URL (or at least the release identifier) so the CI workflow can matrix-test against multiple Eclipse versions without editing the script.

Sketch
-ECLIPSE_URL="https://ftp.osuosl.org/pub/eclipse/technology/epp/downloads/release/2025-09/R/eclipse-cpp-2025-09-R-linux-gtk-x86_64.tar.gz"
+ECLIPSE_VERSION="${ECLIPSE_VERSION:-2025-09}"
+ECLIPSE_URL="https://ftp.osuosl.org/pub/eclipse/technology/epp/downloads/release/${ECLIPSE_VERSION}/R/eclipse-cpp-${ECLIPSE_VERSION}-R-linux-gtk-x86_64.tar.gz"

17-18: Hardcoded stable plugin version v4.0.0 will become stale.

Same concern — consider parameterizing STABLE_ZIP_URL or at least the version component so it can be overridden from CI or updated easily.


58-63: Pre-install listInstalledRoots output is not captured.

The output of this invocation goes to stdout (interleaved with set -x trace output) and is not written to a file. If the intent is to compare before/after, capture it to a log file for the report.

Proposed fix
 echo "Listing installed roots before stable:"
 "$ECLIPSE_HOME/eclipse" \
   -nosplash \
   -application org.eclipse.equinox.p2.director \
   -listInstalledRoots \
-  -consoleLog
+  -consoleLog \
+  > "$LOGDIR/installed-roots-before.txt"
+cat "$LOGDIR/installed-roots-before.txt"

@AndriiFilippov AndriiFilippov changed the title Automate Update Site verification WIP: Automate Update Site verification Feb 13, 2026
@AndriiFilippov AndriiFilippov self-assigned this Feb 17, 2026
@AndriiFilippov
Copy link
Collaborator Author

AndriiFilippov commented Feb 18, 2026

@kolipakakondal , @sigmaaa hi !
Please, take a look.

We must have 2 tests.

Test 1: installs the release candidate on the latest stable Eclipse release.

Test 2: takes Eclipse-1, installs the latest stable IDE release on it, and then updates it to the release candidate.

At this moment I have 1 hybrid test to test both cases.
Please review this minimum working test.

Test logic:

  1. rc-update-site-test.yml depends on the "Java CI with Maven" workflow to ensure that the artifact files have been uploaded.
    At the moment, I am using the artifact file from this PR to simulate “release candidate update behavior” but downloading the .zip, unzipping it, and installing the archive is a bit flaky.
    So, question is:

Since this is an RC test case, what is the best approach to trigger this test? I don’t see that we use -beta or -rc tags very often. Should we introduce specific tags, or should we rely on the beta update site link instead?

https://dl.espressif.com/dl/idf-eclipse-plugin/updates/beta/](https://dl.espressif.com/dl/idf-eclipse-plugin/updates/beta/
  1. The main purpose of detect-latest-eclipse.sh is to automatically detect the current latest stable Eclipse release, Eclipse-1 and provide correct URL. (LATEST_ECLIPSE_RELEASE0``LATEST_ECLIPSE_RELEASE1, ECLIPSE_URL).

  2. test-update.sh uses these parameters and performs the following actions:

    • Downloads the latest Eclipse release
    • Installs the latest stable version of the plugin
    • Updates Eclipse to the Release Candidate (RC)

But this approach works only if we follow the pattern of 1 IDE release per 1 Eclipse release.

However, to make this script fully independent of manual intervention, we should consider adding an “Eclipse version support” tag to our Release Notes to ensure that we test against the correct Eclipse version.
Example:
Let’s assume Eclipse release 26-03 is available today, and we release IDE 4.2.0 (with Eclipse 26-03 support) and IDE 4.3.0 in the same quarter.
In this case, the script logic breaks:

  • It installs the latest stable IDE 4.2.0 on Eclipse 25-12.
  • Then it attempts to update to IDE 4.3.0.

This mismatch can cause incorrect or inconsistent test behavior.

@AndriiFilippov AndriiFilippov requested review from kolipakakondal and removed request for alirana01 February 18, 2026 21:15
@sigmaaa
Copy link
Collaborator

sigmaaa commented Feb 25, 2026

Hi @AndriiFilippov,

Thanks for the PR.

In this case, the script logic breaks:
It installs the latest stable IDE 4.2.0 on Eclipse 25-12.
Then it attempts to update to IDE 4.3.0.

Ideally, we should provide dependencies that allow using 4.3.0 on Eclipse 25-12 even if it is written for 26-03. This is important because we need to support updating Espressif-IDE, which does not have an Eclipse platform update site. We also agreed not to include such a site, since users could accidentally break the IDE by installing incompatible dependencies.

If a problem occurs in this scenario, it will indicate that we need to add additional dependencies, and the test will help us understand that something is missing.

@kolipakakondal
Copy link
Collaborator

kolipakakondal commented Feb 27, 2026

Hi @AndriiFilippov @sigmaaa I have created a detailed specification and goals we need to achieve with this task in the Athena project, please have a look.

Adding here for your reference as well.

Update Site Test — Specification


1. Goals

  • Whenever a new Eclipse CDT version is available, ensure that:
    • The current stable IEP update site works on that Eclipse.
    • The nightly / in-development IEP update site (e.g. v4.2.0) works on that Eclipse.
  • On the latest stable Espressif-IDE, ensure that:
    • We can use our nightly update site to update the plugin (i.e. nightly builds work on the latest stable Espressif-IDE).
    • Users on the latest stable Espressif-IDE can safely update to the in-development plugin via the nightly update site.
  • Avoid manual verification of these update paths; run them automatically in CI.
  • Schedule: All cases run daily, once (e.g. via scheduled workflow on master), not on every push/PR.
  • Cover both:
    • Plain Eclipse CDT (install plugin from update site).
    • Espressif-IDE (branded IDE; update from stable to dev/nightly).

2. Test Cases

Case Base product Step 1 Step 2 Schedule / Trigger
Case 1 Eclipse CDT 2025-12 Install/update with latest stable IEP (e.g. v4.1.0) from official update site. Verify it works. Daily, once (e.g. on master).
Case 2 Eclipse CDT 2025-12 Same as Case 1 (stable, e.g. v4.1.0). Update to v4.2.0 from the update site we are working on (e.g. PR build or master build). Verify update path works. Daily, once (e.g. on master). Ensures master's update site always works with current Eclipse.
Case 3 Eclipse CDT 2026-03 (nightly / next release) Install/update with latest stable IEP (e.g. v4.1.0). Verify it works. Update to v4.2.0 from nightly update site. Verify it works. Daily, once. Ensures compatibility with next Eclipse before release.
Case 4 Espressif-IDE v4.1.0 (full branded IDE) Update to v4.2.0 from the update site we are working on (e.g. nightly). Verify update path works. Daily, once. Validates the branded IDE upgrade path.

3. What We Achieve With This

  • Case 1: Confidence that "Eclipse 2025-12 + latest stable IEP" works (fresh install / update to stable).
  • Case 2: Confidence that "Eclipse 2025-12 + stable → v4.2.0" works when using the master/PR update site; run daily on master so the public update site is continuously validated.
  • Case 3: Early signal that both stable and nightly IEP work on the next Eclipse (e.g. 2026-03), before that Eclipse is GA.
  • Case 4: Confidence that Espressif-IDE users on latest stable (e.g. v4.1.0) can safely update to the in-development plugin (e.g. v4.2.0) via the nightly update site — i.e. our nightly builds work on the latest stable Espressif-IDE.

Overall: automated coverage of stable + dev update paths on current and next Eclipse, plus Espressif-IDE (latest stable → nightly update site), without manual testing.


4. Variables

All URLs and version-like values used by the update-site tests are kept in one file: releng/update-site-tests/variables.env. This keeps the spec and scripts in sync with Eclipse and Espressif-IDE releases.

Purpose: A single place to update whenever an external dependency changes, so we don't have to hunt through scripts or workflows.

What you need to do:

  • When the Eclipse update site for RC builds changes (e.g. a new integration stream like 4.40-I-builds), update ECLIPSE_RC_UPDATES_URL in variables.env. Optionally set ECLIPSE_RC_BUILD_ID to a specific I-build if required.
  • When we publish a new Espressif-IDE or change the "latest" path, update ESPRESSIF_IDE_BASE_URL and ESPRESSIF_IDE_PLATFORM_CI in variables.env if the pattern changes (the /latest redirect is usually updated by the release process).
  • When our IEP update site paths change (stable/beta/nightly), update IEP_UPDATE_SITE_* in variables.env.

Scripts and workflows that run the update-site tests should source this file so they use the same values.

@kolipakakondal
Copy link
Collaborator

@AndriiFilippov As I could PR address some of these changes already (case 2 and case 3), please take a look at the spec and goals again and make the PR changes accordingly.

@AndriiFilippov AndriiFilippov force-pushed the IEP-1685 branch 2 times, most recently from 5820fec to 30ee99c Compare March 5, 2026 17:20
@AndriiFilippov
Copy link
Collaborator Author

@kolipakakondal , @sigmaaa hi !
please, review

@kolipakakondal
Copy link
Collaborator

Hi @AndriiFilippov Can you update the PR description and publish test results what all cases addressed with this PR?

Copy link
Collaborator

@sigmaaa sigmaaa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @AndriiFilippov,

Once again, thanks for the PR. LGTM! Before merging, do not forget to revert ci.yml and ci_windows.yml to run swtbot tests

@AndriiFilippov AndriiFilippov force-pushed the IEP-1685 branch 2 times, most recently from 5f47040 to 25be21d Compare March 24, 2026 09:16
- move all URLs into variables.sh
- run workflow daily at 00:00 UTC
- add workflow summary
@AndriiFilippov AndriiFilippov merged commit 5862981 into master Mar 24, 2026
9 checks passed
@AndriiFilippov AndriiFilippov changed the title WIP: Automate Update Site verification Automate Update Site verification Mar 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants