Serving software packages from the registry
Step 0 has two of its four items done: binaries are re-evaluated on every wave and seed binaries refresh (bioc-registry#51, merged 2026-09-29). The dependency gate rule and keeping bioc-build output out of the software index are open. Steps 1–5 have not started, and none of the decisions below is made.
A plan for making bioconductor.org’s software repositories come from bioc-registry, which is built from r-universe, instead of from the legacy build system (BBS). Every step and every test below cites the measurement that makes it necessary. Measurements are from 2026-09-28 unless another date is given, and are reproducible with the scripts in evidence/registry-switch/. Tracked in bioc-infrastructure#35.
Where software packages come from today
bioconductor.org is served by the new edge, but its package repositories are still an hourly mirror of the legacy docroot. The Worker serves /packages/... from R2, which the edge’s sync job fills from that docroot. It never reads from the registry.
On bioconductor.org |
Sample | Built by |
|---|---|---|
| Release (3.23) source tarballs | 20 of 20 | BBS (packaged by biocbuild) |
| Devel (3.24) source tarballs | 10 of 10 | BBS (biocbuild) |
| Release Windows binaries | 15 of 15 | r-universe (carry r-universe’s RemoteSha) |
| Release macOS binary (limma) | 1 | r-universe |
So Windows and macOS users already install r-universe builds, reaching bioconductor.org through the core team’s propagation, which only publishes an r-universe build whose version and commit match what the BBS host nebbiolo1 built (legacy propagation). Anyone installing from source (Linux, most CI, containers) gets BBS tarballs.
Separately, the registry already runs a complete r-universe-built repository, gated by its own propagation gate, at https://bioc-registry.seandavi.workers.dev/repo/{bioc-release,bioc}. Nothing on bioconductor.org uses it yet. Switching bioconductor.org to it is the step that takes BBS out of the path for software packages.
What the switch covers
Path under /packages/<ver>/bioc/ |
After the switch | Why |
|---|---|---|
src/contrib/ tarballs and PACKAGES, .gz, .rds |
registry | the install path |
bin/windows/…, bin/macosx/… binaries and their indexes |
registry | the install path |
VIEWS |
registry | read by BiocManager-adjacent tools and the site build |
src/contrib/Archive/ |
stays on the mirror at first | old versions; the registry keeps none (see finding 5) |
vignettes/, news/, manuals/, licenses/, citations/, readmes/ |
stays on the mirror | produced by BBS; the registry doesn’t produce them |
html/ |
unchanged (the site build) |
Only the current release (3.23) and devel (3.24). Older releases are archival (ADR 0006) and stay as they are; they are 77% of package traffic, so most installs are unaffected either way. Annotation, experiment and workflow repositories stay on the mirror until bioc-build covers them.
Findings: what would regress if we switched today
Compared file by file, the registry and bioconductor.org agree on almost everything: for release, 2,374 of 2,384 source packages at the same version. The differences below are the ones that matter. Request counts are from about 70 minutes of post-cutover access logs, with a blocked crawler excluded.
1. No PACKAGES.rds. The registry serves PACKAGES and PACKAGES.gz but answers 404 for PACKAGES.rds, for source and every binary directory. It’s the index R asks for first, and the most requested: 2,784 requests against about 220 for .gz and plain PACKAGES. R falls back to the next format, so installs keep working, but every repository lookup pays an extra round trip. Blocking.
2. Packages that can’t be installed. The registry serves 6 release and 13 devel packages that bioconductor.org doesn’t, such as Ularcirc, SNPhood, progeny and sparrow. They are still in Bioconductor’s manifest and in r-universe, but a hard dependency was removed from Bioconductor (decoupleR, BiocSet, Organism.dplyr, BPRMeth), so BBS stopped propagating them. Checked with tools::package_dependencies over the repository plus CRAN and the data repositories, the registry adds 4 release and 7 devel packages whose hard dependencies can’t be resolved, on top of bioconductor.org’s own (9 release, 2 devel, all CRAN packages that were archived). A user asking for one gets a confusing failure instead of “not available”. Blocking. The fix is a gate rule, not a list: don’t publish a package whose hard dependencies aren’t resolvable.
3. Versions that fail in r-universe lag. 153 release and 124 devel entries were seeded once from Bioconductor’s own builds for packages that fail in r-universe’s environment. A seed is replaced only when r-universe builds a newer version that passes the gate. Today 3 release and 8 devel packages are behind bioconductor.org, including biomaRt, DOSE and CNVRanger. Asked through the registry’s own POST /gate, every one of them is blocked by a rule, which is what the gate is for: the newer version fails to build in r-universe (biomaRt 2.69.4, bedbaser, CNVRanger, knowYourCG, alabaster.vcf; SwathXtend, whose 346 MB tarball has no successful build), or it builds but its checks ERROR on every platform (DOSE 4.7.3, MsBackendMetaboLights 1.7.8). PIPETS is the same case one step removed: r-universe’s last successful build is 1.7.0, and its 1.9.0 build failed. BBS publishes these versions because they pass in BBS’s environment. The gate is stricter than BBS here, and after a switch those packages’ bug-fix releases would wait until they pass in r-universe. A decision, not a fix: accept the lag (and chase the failures with the maintainers), or bridge by re-seeding from BBS, which keeps BBS in the path for those packages.
4. Missing binaries. These counts compare the registry’s repository with bioconductor.org; bioconductor.org itself is not missing these binaries. On 2026-09-28, before the fix described below, for release the registry lacked 61 Windows, 49 macOS arm64 and 76 macOS Intel binaries that bioconductor.org serves (devel: 48, 57, 72), 363 in all. The gate publishes a binary only for a platform whose r-universe check has no ERROR or FAIL on the gating R line, and seeds copy a binary only if bioconductor.org’s binary matches the source version. Classifying every one of the 363 against those rules (binwhy.py, 2026-09-28):
| Cause | Count | Verdict |
|---|---|---|
The platform’s check fails in r-universe (e.g. KEGGREST on Windows, BiocParallel on macOS) |
114 | by design |
Not in the source repository at all; bioconductor.org serves a binary for a package it no longer lists as source (e.g. BiRewire) |
26 | by design |
Seed, and bioconductor.org’s binary is older than its own source (e.g. Windows AnnotationForge 1.53.0 next to source 1.54.0) |
58 | by design |
Seed, and bioconductor.org’s binary has since caught up to the seeded version (seeds never refresh) |
12 | gap |
Seed whose version lags bioconductor.org |
2 | follows finding 3 |
Withheld when first published; r-universe has rebuilt the same version since, and it now passes (e.g. ChIPpeakAnno on macOS, OGRE on Windows) |
107 | bug |
The platform passed, but the binary wasn’t built yet when the source was published, and is never added later (e.g. SparseArray: published Sept 23, binaries built Sept 24) |
30 | bug |
| Withheld; r-universe has since moved to a newer version that isn’t published | 14 | follows finding 3 |
The two bugs have one cause: the gate decides a package’s binaries once, when its version is first published, and never looks again at that version. So 137 binaries that the registry’s own rules would admit today are missing (bioc-registry#50). Fixed and backfilled in bioc-registry#51: re-running the comparison, the missing binaries fell from 363 to 222, the bug rows from 137 to 8, and the seed-refresh row by 12. None of the 8 left has an r-universe binary to add (a macOS Intel build failed or doesn’t exist, or the binary hasn’t been built yet for a version published an hour earlier). Was blocking: re-evaluate the binaries of each published version on every r-universe wave, adding any whose platform now passes (the source and the version stay as they are).
The “by design” rows change what users get: in 88 of the 363 (2026-09-28; not re-counted after the fix), bioconductor.org serves a binary older than its source (BBS keeps the last binary that built). After a switch, a Windows or macOS user gets a source install of the current version instead of an older binary, which fails for a package with compiled code on a machine without a toolchain. That is a policy choice to make explicitly (see decisions).
5. No Archive/. The registry keeps only the current version. src/contrib/Archive/ on bioconductor.org served 886 requests (renv restores and remotes::install_version) in the sample. At first it stays on the mirror; once BBS stops producing it, superseded registry tarballs have to be archived by whatever writes the repository. Blocking for detaching BBS, not for the switch.
6. Data and workflow packages in the software repository. The registry has two universes, bioc and bioc-release, and bioc-build publishes into them, so msdata (experiment data), rnaseqDTU (a workflow) and ARRmData appear in the software PACKAGES. Blocking. They belong in data/experiment and workflows.
7. Missing index fields. PACKAGES lacks MD5sum, Archs, License_is_FOSS and License_restricts_use; VIEWS lacks those plus dependencyCount, Rfiles, hasNEWS, organism, PackageStatus and others. No consumer of MD5sum is confirmed. Writing the indexes with R’s tools::write_PACKAGES() from the actual tarballs produces the PACKAGES fields for free. Not blocking; the VIEWS fields are checked against the site build.
8. Same version, different build. For 33 release packages both sides have the same version but different contents: bioconductor.org declares R (>= 4.6), the registry R (>= 4.5). The maintainer changed DESCRIPTION without a version bump, and the registry kept the first build of that version, as its gate intends. Installing on R 4.6 is unaffected. Expected; the tests must allow it.
9. The registry is not built to serve at edge speed. Its indexes are generated per request with cache-control: no-store and take 250–550 ms; bioconductor.org serves the same files from the edge cache in about 60–85 ms (2026-09-28). Re-measured 2026-09-30 from onclappc02 with a new connection per request: 210–240 ms for a cached PACKAGES from bioconductor.org, 440–480 ms from the registry. The index changes a few times a day for release and about ten times a day for devel. This decides the mechanism below.
Mechanism: write the registry’s repository into the edge’s storage
Two ways to serve it:
- Proxy: the Worker forwards
/packages/3.2x/bioc/src/…to the registry at request time. Every index request would take the registry’s latency or need a new caching and purge scheme, andbioconductor.orginstalls would depend on the registry Worker being up. - Materialize (recommended): a job writes the registry’s repository into the same R2 keys the mirror writes today (
packages/3.24/bioc/src/contrib/…), generatingPACKAGES,.gzand.rdswithtools::write_PACKAGES()from the tarballs, and purges the changed keys with the existingcf_purge_keys. The Worker, its cache rules, 404 caching, logs, and the mirror manifest need no change. The hourly sync stops writing those paths (anrsync-filterexclusion), and turning that exclusion off again is the rollback.
This follows the rule the estate already has: producers publish into storage, and the edge decides what to serve (contract 3 on the overview).
Plan
Each step names its rollback. Nothing moves to the next step until its tests pass.
0. Fix the blocking gaps in the registry.
- Gate rule: a package is published only if its hard dependencies resolve within the repository, CRAN and the Bioconductor data repositories (finding 2).
- Binaries: re-evaluate the binaries of each published version on every wave, so a platform that passes later, or a binary built after the source was published, is added (finding 4: 137 binaries; bioc-registry#50). Done.
- Repository type: route bioc-build output into data and workflow repositories, never the software index (finding 6).
- Seeds: refresh a seed’s binaries when
bioconductor.org’s catch up to the seeded version (finding 4: 12 binaries; also bioc-registry#50). Done. Whether to re-seed newer versions from BBS is a decision (finding 3).
1. Build the materializer, writing to a staging prefix. It writes the full software tree for 3.23 and 3.24 under a prefix the Worker doesn’t serve (for example staging/registry/), on every registry index change. Rollback: nothing is served from it.
2. Serve the staging tree on the rehearsal host. next.bioconductor.org gets a route that maps /packages/3.2x/bioc/{src,bin,VIEWS} to the staging prefix, the same mechanism as the strangler route table. Run every test below against next. for at least a week of registry updates. Rollback: remove the route.
3. Flip devel (3.24) on bioconductor.org. Devel is 2% of package traffic and its users expect movement. The materializer writes the production keys for 3.24; the hourly sync excludes them. Watch the logs for 404s and for installs of packages that used to exist. Hold for a week. Rollback: remove the exclusion, run the sync, purge.
4. Flip release (3.23). Same steps, after devel has been clean for a week. Rollback: same.
5. Archive. Before BBS stops producing Archive/, the materializer moves superseded tarballs into Archive/<pkg>/ (finding 5).
After step 4, BBS is out of the install path for software packages, except for any seeds that are still bridged (finding 3). The legacy docroot still supplies documentation files and older releases until M2 retires it.
Regression tests
Each test exists because of a finding above. The comparison tests use bioconductor.org as the baseline while BBS still runs; after the flip their baseline becomes the previous materialized state.
| # | Test | Guards against | Runs |
|---|---|---|---|
| T1 | Every package in the baseline is present at the same or a newer version | missing or older packages (findings 3, 6) | on every materialize; blocks publishing |
| T2 | No package whose hard dependencies are unresolvable, beyond the baseline’s own | uninstallable packages (finding 2) | on every materialize; blocks |
| T3 | PACKAGES, .gz and .rds exist in every directory and parse to the same package set |
missing index formats (finding 1) | on every materialize; blocks |
| T4 | Every published package has a binary for each platform its gate verdict passes, at the published version; baseline binaries the registry lacks are reported by cause | binaries the rules allow but that are missing (finding 4) | on every materialize; blocks on the first part |
| T5 | Every tarball’s name matches its DESCRIPTION’s Package and Version, and its hash matches the registry index |
corrupt or mislabeled artifacts | on every materialize; blocks |
| T6 | No data or workflow package in the software index | repository-type mix-up (finding 6) | on every materialize; blocks |
| T7 | BiocManager::install() for release and devel, from source on Linux and from binaries on Windows and macOS, for a fixed sample including packages with compiled code |
an install path that parses but doesn’t work | CI matrix, daily and before each flip |
| T8 | A package’s previous version installs from Archive/ |
breaking renv and install_version (finding 5) |
daily after step 5 |
| T9 | After a registry propagation, bioconductor.org serves the new PACKAGES within one materialize cycle |
a stale edge cache (s-maxage is a year) | on every materialize |
| T10 | Same-version differences (finding 8) are reported, not failed | noise that would train everyone to ignore T1 | reported daily |
T1–T6 and T9 are checks on files the materializer just wrote, so they run inside it and stop it from publishing a broken tree. T7 already exists in part (test-biocmanager.R in bioc-edge, which passes for 3.23 and 3.24 against bioconductor.org today) and needs Windows and macOS runners.
Decisions needed
Neither this page nor #35 names who makes these decisions or by when (checked 2026-09-30).
- Lagging versions (finding 3): accept that versions failing in r-universe wait, or bridge them from BBS for now?
- Stale binaries (finding 4): for a package whose current version fails on a platform,
bioconductor.orgserves an older binary today. After the switch, users on that platform get a source install of the current version. Accept, or also publish the last binary that passed? - Mirror operators keep syncing from the legacy host, so after the flip they would serve BBS builds while
bioconductor.orgserves registry builds. Point them at the manifest-based mirror guide first, or accept the divergence for a while? - Documentation files (
vignettes/and the rest) stay on BBS’s output, so for a package where the two builds differ, its vignette can come from a different build than its tarball. Acceptable until the site build produces them?