Define the get_instance_status and get_job_status() methods of the BenchmarkLauncher

[R-Palazzo]

Resolve #570
CU-86b8tmh3y

If you’d like to test the feature end to end, here’s a code snippet you can use. You only need to update the credentials and the output_destination so it doesn’t point to the internal benchmark destination.
To use SDV Enterprise synthesizers, you’ll need to have sdv-enterprise installed in the local environment where you run the code.

from sdgym._benchmark_launcher import BenchmarkLauncher
from sdgym.run_benchmark.run_benchmark import _get_config
import os
import json

service_account = {
    "type": "service_account",
    "project_id": "sdgym-337614",
    "private_key_id": "",
    "private_key": "",
    "client_email": "",
    "client_id": "",
    "auth_uri": "",
    "token_uri": "",
    "auth_provider_x509_cert_url": "",
    "client_x509_cert_url": "",
    "universe_domain": "",
    "gcp_project": "sdgym-337614",
    "gcp_zone": "us-central1-a"
}

os.environ["GCP_SERVICE_ACCOUNT_JSON"] = json.dumps(service_account)
os.environ["AWS_ACCESS_KEY_ID"] = ''
os.environ["AWS_SECRET_ACCESS_KEY"] = ''
os.environ['SDV_ENTERPRISE_USERNAME'] = ''
os.environ['SDV_ENTERPRISE_LICENSE_KEY'] = ''

aws_access_key_id = os.getenv('AWS_ACCESS_KEY_ID')
aws_secret_access_key = os.getenv('AWS_SECRET_ACCESS_KEY')
modality = 'single_table'

config = _get_config(modality)
launcher = BenchmarkLauncher(config)
launcher.launch()
job_status = launcher.get_job_status()
instance_status = launcher.get_instance_status()

[sdv-team]

Task linked: CU-86b8tmh3y SDGym - Define the get_instance_status and get_job_status() methods of the BenchmarkLauncher #570

[codecov]

Codecov Report

❌ Patch coverage is 98.76543% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.15%. Comparing base (7b3ff6a) to head (40e0b20).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
sdgym/_benchmark_launcher/benchmark_launcher.py	97.77%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #583      +/-   ##
==========================================
+ Coverage   84.58%   85.15%   +0.57%     
==========================================
  Files          40       41       +1     
  Lines        3418     3563     +145     
==========================================
+ Hits         2891     3034     +143     
- Misses        527      529       +2     

Flag	Coverage Δ
integration	44.87% <0.00%> (-1.91%)	⬇️
unit	80.83% <98.76%> (+0.75%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[amontanez24]

don't we just need to do this once in init?

[R-Palazzo]

Good point. Since we don’t expose any method to update the config after init, validating it once there should be enough. Updated in c33a1e5

[amontanez24]

what's the scenario where invalid instance names are being provided?

[R-Palazzo]

We raise an error for invalid instance names currently:

SDGym/sdgym/_benchmark_launcher/benchmark_launcher.py

Line 166 in 92f8165

raise ValueError(

[R-Palazzo]

I am including this change in the current PR so tomorrow’s runs already save the BenchmarkLauncher automatically. That gives a few extra days to finish and test the upload work in #584, which will be used to generate the results of tomorrow’s runs.

[amontanez24]

One small comment