[MNT] Dockerized tests for CI runs using localhost by satvshr · Pull Request #1629 · openml/openml-python

satvshr · 2026-01-29T11:04:10Z

Metadata

Reference Issue: fixes [MNT] Intermediate test plan #1614, stacks on [ENH] Allow using a local test server #1630
New Tests Added: No
Documentation Updated: No

Details

What does this PR implement/fix? Explain your changes.
This PR implements the setting up of the v1 and v2 test servers in CI using docker via localhost.

Locally, MinIO already has more parquet files than on the test server.

Note that the previously strategy didn't work anymore if the server returned a parquet file, which is the case for the new local setup.

This means it is not reliant on the evaluation engine processing the dataset. Interestingly, the database state purposely seems to keep the last task's dataset in preparation explicitly (by having processing marked as done but having to dataset_status entry).

codecov-commenter · 2026-01-29T21:26:14Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 67.55%. Comparing base (7feb2a3) to head (5f079ba).

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #1629       +/-   ##
===========================================
+ Coverage   52.82%   67.55%   +14.73%     
===========================================
  Files          37       37               
  Lines        4371     4371               
===========================================
+ Hits         2309     2953      +644     
+ Misses       2062     1418      -644

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

PGijsbers · 2026-02-13T11:26:05Z

I didn't look at it too closely but considering it looks like the local evaluations go wrong its not likely to do with any server connection issues https://github.com/openml/openml-python/actions/runs/21919012927/job/63293911025?pr=1629#logs . The error message about datatypes immediately makes me think of pandas, and this PR does not contain the fixes from #1628. I have to assume that is the underlying issue for that error.

The other error you sent is strange https://github.com/openml/openml-python/actions/runs/21940621497/job/63365126160?pr=1629. I'll have a closer look after my next meeting.

update: I can't quickly find a reason for the error. I added it on my list to check later.

Co-authored-by: Armaghan Shakir <raoarmaghanshakir040@gmail.com>

geetu040 · 2026-02-18T06:53:24Z

.github/workflows/test.yml

+        # sed -i 's|/minio/|/data/|g' config/database/update.sh
+
+        # echo "=== Patched Update Script ==="
+        # cat config/database/update.sh | grep "nginx"


why extra work here? locally just running the services is enough

Kindly ignore these, the pr isnt ready for review yet as tests are still failing and I was trying to debug tests.

geetu040 · 2026-02-18T06:53:31Z

openml/config.py

+if sys.platform.startswith("win"):
+    TEST_SERVER_URL = "http://localhost"
+else:
+    TEST_SERVER_URL = "http://localhost:8000"
+


we should actually use an env variable here, please see https://github.com/openml/openml-python/pull/1629/changes#r2797509441
should be controlled by that env variable, which if not set, should default to use https://test.openml.org/

This is not how I plan to resolve this either, just a temporary fix to the windows issue.

geetu040 · 2026-02-18T08:37:07Z

The tests are taking too long because connection_n_retries is set to 5, you can set it to 1 for this PR, to avoid delays in CI.

satvshr · 2026-02-18T08:41:30Z

The tests are taking too long because connection_n_retries is set to 5, you can set it to 1 for this PR, to avoid delays in CI.

Will do that to prevent hold ups for other CIs in the repo, for my branch it is noticeable if a run is going to fail if it has been stuck on a single test for more than a minute.

geetu040 · 2026-02-18T09:37:16Z

Will do that to prevent hold ups for other CIs in the repo, for my branch it is noticeable if a run is going to fail if it has been stuck on a single test for more than a minute.

yeah but each job in this PR still takes full 150 minutes

geetu040 · 2026-02-23T06:46:10Z

openml/config.py

    "avoid_duplicate_runs": False,
    "retry_policy": "human",
-    "connection_n_retries": 5,
+    "connection_n_retries": 1,


I don't think this would work, since we change this again in conftest.py.
To be completely sure that this works, you can temporarily set n_retries = 1 in _api_calls.py::_send_request

geetu040 · 2026-02-23T06:46:45Z

.github/workflows/test.yml

+      run: |
+        git clone --depth 1 https://github.com/openml/services.git
+        cd services
+


you are not running these services yet.

DId not realise I accidentally removed it

geetu040 · 2026-02-23T09:13:53Z

tests/test_flows/test_flow.py

            f"collected from {__file__.split('/')[-1]}: {flow.flow_id}",
        )

+    @pytest.mark.skip(reason="Pending resolution of #1657")


skip these only if OPENML_USE_LOCAL_SERVICES is set to True

geetu040 · 2026-02-23T09:14:31Z

tests/test_flows/test_flow.py

some tests are skipped though they are not mentioned in #1657, why is that?

I was using the failures from here

PGijsbers and others added 11 commits January 20, 2026 12:35

Use the correct path to the cache directory for the task

83e1531

Push configuration of test server URL exclusively to config.py

f90036d

Update the test to use a dataset which does not have a parquet file

3a257ab

Locally, MinIO already has more parquet files than on the test server.

Replace hard-coded cache directory by configured one

3b79017

Update test to use dataset file that is already in cache

f524d75

Note that the previously strategy didn't work anymore if the server returned a parquet file, which is the case for the new local setup.

Windows test

7ef12c2

relax assumptions on local file structure

a5601e3

Do not use static cache directory

d862be2

bug fixing

7c14c68

merge main

78b2038

geetu040 assigned satvshr Jan 29, 2026

satvshr added 4 commits January 30, 2026 02:06

remove db refresh every test

16ceeaa

bug fixing

015acf4

bug fixing

937fc77

bug fixing

30972f8

PGijsbers and others added 9 commits January 30, 2026 10:30

Add symlink to regular test cache directory

775dcf7

Skip test for 1.8 since expected results differ too much

319cb35

Simplify path to static cache directory

a680ebe

Update symbolic link to be relative

b161b3b

Fix typo

0b989d1

trying ot fix multiple threads issue

892ea6c

removed test file

ae3befb

removed unnecessary code (?)

5f396a0

Trigger Build

8a319cd

satvshr marked this pull request as ready for review January 31, 2026 16:13

satvshr marked this pull request as draft January 31, 2026 16:14

satvshr added 2 commits February 1, 2026 17:18

Clean up code

4ba4239

comment fixing

0292404

satvshr and others added 6 commits February 13, 2026 02:18

test

4086730

windows fix?

fecebbc

windows fix?

4845a1e

windows fix?

a247050

Do not include ports in cache path. ':' not supported by windows

e4a6807

Merge branch 'main' into update-tests-for-local

03bf396

satvshr and others added 7 commits February 13, 2026 21:07

merge 1630

4c6bd2f

revert test.yml changes

ba0e480

Update openml/config.py

519d5cb

Co-authored-by: Armaghan Shakir <raoarmaghanshakir040@gmail.com>

Merge branch 'main' into update-tests-for-local

30fd44d

Merge branch 'main' into update-tests-for-local

c908993

Keep port as part of cache directory path

06b9741

merge update-tests-for-local

015cca3

geetu040 suggested changes Feb 18, 2026

View reviewed changes

geetu040 mentioned this pull request Feb 18, 2026

[MNT] Failures when running tests against locally replicated test-server #1657

Open

Sandipmandal25 mentioned this pull request Feb 19, 2026

[ENH] Make TEST_SERVER_URL configurable via environment variable #1663

Open

satvshr added 3 commits February 23, 2026 01:25

merge main

8de78af

req changes

4482a2c

changing retries

278b546

geetu040 suggested changes Feb 23, 2026

View reviewed changes

satvshr added 5 commits February 23, 2026 13:04

fixes

cf30367

testing replacement

b41a9b2

bug fix

f737cb1

added skip tests

b40d702

final touches

5f079ba

geetu040 suggested changes Feb 23, 2026

View reviewed changes

Uh oh!

Comments

Conversation

satvshr commented Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Metadata

Details

Uh oh!

codecov-commenter commented Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

PGijsbers commented Feb 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

geetu040 commented Feb 18, 2026

Uh oh!

satvshr commented Feb 18, 2026

Uh oh!

geetu040 commented Feb 18, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

satvshr commented Jan 29, 2026 •

edited

Loading

codecov-commenter commented Jan 29, 2026 •

edited

Loading

PGijsbers commented Feb 13, 2026 •

edited

Loading