idc-index-data is a Python package providing index files to query and download
data hosted by the
NCI Imaging Data Commons (IDC).
The PyPI package bundles a core set of index files (Parquet, JSON schemas, SQL queries). Supplementary indices that are too large for PyPI distribution are published as release artifacts on GitHub and uploaded to a public Google Cloud Storage bucket on each release.
The package provides metadata for the following indices via the INDEX_METADATA
dictionary:
| Index | In PyPI package | Description |
|---|---|---|
idc_index |
yes | Core IDC DICOM study-level index |
prior_versions_index |
yes | Historical version tracking |
collections_index |
yes | Collection-level metadata |
analysis_results_index |
yes | Analysis results metadata |
clinical_index |
- | Clinical data (large) |
sm_index |
- | Slide microscopy index (large) |
sm_instance_index |
- | Slide microscopy instance-level index (large) |
seg_index |
- | Segmentation index |
ann_index |
- | Annotation index |
ann_group_index |
- | Annotation group index |
contrast_index |
- | Contrast agent index |
Additionally, the following supplementary parquet files are generated and published alongside the index files (not included in the PyPI package):
| File | Description |
|---|---|
gdc_idc_mapping.parquet |
Mapping of IDC patients to GDC cases |
tcia_idc_subset.parquet |
Subset of IDC index columns for TCIA workflows |
All index files (including supplementary ones) are available from:
- GitHub Releases: attached as release assets
- Google Cloud Storage: publicly readable via HTTPS
Artifacts are uploaded to the idc-index-data-artifacts bucket on each release.
Two paths are maintained:
| Path | Description |
|---|---|
gs://idc-index-data-artifacts/<version>/release_artifacts/ |
Artifacts for a specific release (e.g. 23.5.0) |
gs://idc-index-data-artifacts/current/release_artifacts/ |
Always points to the latest release |
Individual files can be accessed via HTTPS at:
https://storage.googleapis.com/idc-index-data-artifacts/current/release_artifacts/<filename>
For example:
Replace current with a specific version tag (e.g. 23.5.0) to pin to a
particular release.
This package is intended to be used by the idc-index Python package.
import idc_index_data
# Access core index file paths
idc_index_data.IDC_INDEX_PARQUET_FILEPATH
idc_index_data.PRIOR_VERSIONS_INDEX_PARQUET_FILEPATH
# Access unified metadata for all indices
idc_index_data.INDEX_METADATA["idc_index"]["parquet_filepath"]
idc_index_data.INDEX_METADATA["idc_index"]["schema"] # pre-loaded dict
idc_index_data.INDEX_METADATA["idc_index"]["sql"] # pre-loaded stringThis software is maintained by the IDC team, which has been funded in whole or in part with Federal funds from the NCI, NIH, under task order no. HHSN26110071 under contract no. HHSN261201500003l.
If this package helped your research, we would appreciate if you could cite IDC paper below.
Fedorov, A., Longabaugh, W. J. R., Pot, D., Clunie, D. A., Pieper, S. D., Gibbs, D. L., Bridge, C., Herrmann, M. D., Homeyer, A., Lewis, R., Aerts, H. J. W., Krishnaswamy, D., Thiriveedhi, V. K., Ciausu, C., Schacherer, D. P., Bontempi, D., Pihl, T., Wagner, U., Farahani, K., Kim, E. & Kikinis, R. National Cancer Institute Imaging Data Commons: Toward Transparency, Reproducibility, and Scalability in Imaging Artificial Intelligence. RadioGraphics (2023). https://doi.org/10.1148/rg.230180