Bioconductor Infrastructure

This site documents the infrastructure behind bioconductor.org: what runs today, what is replacing it, and the contracts between the pieces. It exists because none of it was written down anywhere a newcomer — or a future maintainer — could find in one place.

TL;DR

The replacement estate is four repos that meet at data contracts rather than shared machines, which is why each can proceed — or be replaced — without stopping the others.

Repo What it is Retires Status
bioc-registry The data plane: an r-universe observer plus the propagation gate that decides which build is fit to publish and what is true about a package, exposed as an installable repository and a Parquet archive The three builders and the legacy build system freshness
bioc-website The builder: a Python pipeline and an Astro static build that render package truth into ~93,000 pages, published immutably by CI The master and staging webservers site
bioc-edge The serving side: a Cloudflare Worker and the R2 mirror behind bioc-dev.cancerdatasci.org, owner of the route table that decides what is served where CloudFront, the S3 bucket, the Open Storage Network buckets health
bioc-infrastructure This documentation site Nothing. It is the map. docs

The status badges are daily black-box probes of the live system, not just test suites — red means something is actually down or stale. Where this is all going, milestone by milestone: the roadmap.

TipTry it now

The replacement site is live at bioc-dev.cancerdatasci.org — identical content, verified against production — and two lines of R point your installs at it:

options(BioC_mirror = "https://bioc-dev.cancerdatasci.org")
BiocManager::install("GEOquery")

See Installing packages from the new site for verification, persistence, and what to expect. Real usage, and reports of anything that behaves differently, are the most useful validation we can get.

Where to file issues

How the pieces meet

flowchart TB
  m(["maintainers"]) --> ru["r-universe<br/>builds packages"]
  ru -- "polled every 15 min" --> reg["bioc-registry<br/>the propagation gate"]
  reg -. "install.packages()" .-> cran["installable R repository"]
  reg -. "DuckDB" .-> pq["Parquet archive"]
  reg -- "build-time fetch" --> site["bioc-website<br/>static build, CI, ~60 s"]
  site --> r2[("R2<br/>immutable builds")]
  r2 --> wk["bioc-edge Worker<br/>owns the route table"]
  wk --> v(["visitor"])

The organising principle is one idea applied at two scales: define where the data comes from once, and let every consumer render it. Inside bioc-website that shows up as the astro/data/ seam between its pipeline and its renderer. Across the estate it shows up as the table above. The website is one consumer of package truth. It is deliberately not the place where “what is true about a package” gets decided.

The contracts

Everything above holds together at five seams. They are worth stating in one place, because each is a place where a change in one repo can quietly break another.

1. astro/data/ — pipeline to renderer, inside bioc-website. JSON written to disk by a Python pipeline, read by an Astro build. Nothing downstream knows where the data came from; nothing upstream knows how it is rendered. That seam is what let the same component render both the legacy-parity pages and the new data-plane-fed pages unchanged.

2. The registry’s HTTP API — data plane to website, fetched at build time from /data/prop/{universe}/index.json, /data/state/{universe}/latest, and /prop/{universe}/cas/{sha256}. This is what the /next/ pages render: what propagated, not whatever r-universe last built.

3. The R2 object layout — build to serving. The website’s CI writes immutable site/<sha>/ builds, a site/latest pointer, and preview/pr-<n>/ for every pull request. The Worker reads them. The build side publishes; the serving side decides what is served — write access to the website repo never implies the power to change production.

4. The route table — _routes.json, an object in R2, not code. Flipping a route from mirrored legacy content to a static build is a data write; rolling it back is the same write in reverse. No deploy, no rebuild. An empty table means the mirror serves everything, which is the safe default.

5. Request logs — serving to statistics. The Worker emits one structured record per request in the same field vocabulary as the CloudFront logs it replaces, shipped to object storage and queried from BigQuery, so both eras are one query rather than a migration project. The record keeps the raw client address; de-identification is a view, and nothing published carries the address (ADR 0012).

Where this stands

The serving replacement is functionally complete and running at bioc-dev.cancerdatasci.org: the mirror is verified drift-free, logging and its gap alerting are live, and the remaining step is the production cutover decision. Package landing pages rebuild from primary sources for 35 releases, measured against the legacy build rather than assumed to match. The build-system track is earlier — r-universe covers software packages; where annotation, experiment, and workflow packages get built is the open workstream, and the last thing standing between the estate and three fewer machines.

Numbers throughout this site are point-in-time measurements, not live dashboards — check the date on each page before relying on specifics.

Two costs, not one

Two figures appear across these pages and they are not competing estimates of the same thing:

  • ~$5,000/month is the AWS estate being retired — CloudFront, the S3 bucket, and the master and staging webservers — before counting the builder hardware or the staff time to keep three operating systems patched.
  • ~$80/month is the Cloudflare stack replacing it, dominated by R2 storage. R2 charges no egress fees, which matters disproportionately for a package repository, where serving tarballs to the world is precisely what a CDN bills most aggressively for.

Known weak points

Stated here rather than discovered later:

  • The website reads the registry’s stored JSON shape directly, not a versioned API. An unreachable endpoint fails the build loudly, which is right; a renamed field renders a blank section on a page and fails nothing. The convention is “never change a shape in place, add fields and tolerate absences” — it is a convention, with no mechanism behind it.
  • astro/data/ has no written schema. Its shape is implied by the code that writes it and the code that reads it, and by nothing else.
  • The 15-minute poll has no alerting. A silent stop is currently detected by someone remembering to look.
  • Some facts are written in several places — release numbers, repository lists, display labels — and at least one set of display labels already disagrees with itself.

How this site is organised

The pages fall into two kinds, and the sidebar separates them:

If you want the reasoning rather than the description, the ADRs are the honest record — including the decisions that were later reversed.

Note

This site is deliberately public, so that the people affected by this work can read it without needing an account or an invitation. Findings are shared with the Bioconductor core team as they land, so read these pages as work in progress rather than settled conclusions.