Repository navigation
Eliminate S3 bucket access #112
Description
Activity
[bot] Comment posted by a Claude Code agent on behalf of @kltm.
Aggregate data from the bbop-sqlite access logger (~6 days, 2026-07-09 → 07-14), to help scope the straggler drain before removing public access:
- ~85% of egress bytes still hit the raw S3 URL; ~15% now goes through the CDN (
semanticsql.berkeleybop.io) at 87–90% edge-cache-hit. The fronting works; consumers largely have not migrated yet. - Consumer fingerprint: dominated by Python stdlib
urllib(multiple 3.x versions) with 0% conditional GETs (every fetch is a full re-download), from hundreds of ephemeral IPs, ~half Microsoft/Azure — i.e. GitHub-Actions / cloud CI on OAK ≤0.7.1 (raw-S3 + urllib path). Matches the pre-0.7.2 code exactly. - Biggest objects (raw URL):
go.db.gzis the single largest, thenchebi,mondo,rhea,pr,ncit,hp,ncbitaxon— each repeat-pulled by hundreds of distinct IPs (exactly what the CDN's shared cache collapses). - Encouragingly, a large consumer is already on the CDN, so ≥0.7.2 / env-override adoption is starting.
- Separately, a Meta
facebookexternalhitcrawler is pulling multi-GB.db.gzfiles — not a real consumer; blockable independent of this effort.
Implication for lockdown: the raw URL still carries the majority with a live, identifiable OAK-ecosystem-CI consumer set, so removing
AllUsersnow would break them. The drain should accelerate as OAK ≥0.7.2 propagates — chiefly the ODK image bump (INCATools/ontology-development-kit#1354 / #1355) — and the direct hardcoders in #115 repoint. Suggest gating removal on this logger showing raw-URL object GETs fall off.— Posted by Claude Code agent on behalf of @kltm.
- ~85% of egress bytes still hit the raw S3 URL; ~15% now goes through the CDN (
- added this to Software essential and proactive maintenance and removed this from Software essential and proactive maintenance
on Aug 18, 2026 [bot] Posted by a Claude Code agent on behalf of @kltm.
Done — raw public access to
bbop-sqlitewas retired 2026-08-28 ~23:30 UTC. The bucket now has a full Public Access Block; CloudFront reaches it via Origin Access Control with a distribution-scoped policy grantings3:GetObjectands3:ListBucket.https://semanticsql.berkeleybop.iois the sole supported endpoint — object downloads and V1 root listing both verified end-to-end post-flip. Raws3.amazonaws.com/bbop-sqliteURLs and unsigned SDK/CLI access now return AccessDenied.Listing decision, as implemented: V1 root listing is preserved through the CDN (the ListBucket grant). SDK/ListObjectsV2 and
aws s3 lsagainst the bucket do not work — affected consumers (see #115) should read the root XML or await a manifest file if one is published later.Stragglers: denied raw requests are logged; anyone broken by this should swap to the CDN URL (drop-in, same paths) — or comment here.
— Posted by Claude Code agent on behalf of @kltm.
This has been executed.
- added a commit that references this issue
on Aug 31, 2026
Once the mechanisms have switched over and enough traffic is on https://semanticsql.berkeleybop.io/{db}.gz , we will turn off public access to the underlying S3 bucket.
As well, we have now setup a logger on the bbop-sqlite bucket so that we can see who the stragglers are in more detail.
See: #110