What the new site has changed
These are the stories of what moving bioconductor.org onto the new stack has made better, and what it has made possible that wasn’t before. Each one says how far along it is. Some are done and measured, some are measured but not yet realized, and some are just started. Where a number is a projection rather than a measurement, it says so. New stories go at the top.
Watching the logs caught a crawler eating three quarters of the site
Status: fixed 2026-09-28, the same evening: the loop broken at 20:58 UTC, the rest of the crawler blocked at the edge at 22:34 UTC (bioc-edge#32). The daily savings are still projected from minutes of data.
bioconductor.org moved onto the new edge on the afternoon of 2026-09-28. Within the hour we were reading the request logs, which the new edge writes one structured record per request, and one thing stood out immediately. In the first 19 minutes, 84,414 of 112,632 requests (75%) came from a single crawler. It ran on two cloud-hosting networks in Singapore, spread across 1,130 addresses, and rotated three Mac Chrome user agents so it looked like ordinary browsers.
It was requesting paths like /talks/2016/BioC2016/ConcurrentWorkshops1/Sauteraud/2016/..., up to 294 characters long, and the reason is a loop we were feeding. The old site redirects anything under /talks to the course-materials index, and the new edge copied that rule. The index has 392 relative links, such as 2022/CSAMA/. The crawler resolved them against the address it had asked for instead of the page it landed on, so every redirect produced 392 new /talks/... addresses, each of which redirected back to the same index. The redirect never helped a person either. It discarded the rest of the path, so an old link to a talk landed on a list of every talk, and in the logs nobody but the crawler asked for /talks.
We removed the rule, so those paths now return 404, and wrote the general policy down as ADR 0013: keep redirects that land a reader on the page they wanted, keep anything package clients depend on, and drop catch-alls to index pages. In the first minutes after the change, the crawler’s fetches of the course-materials page went from 41,873 to zero and its request rate fell from about 8,300 to about 3,400 a minute. What remained was the crawler working through addresses it had already queued, and we expected that to run dry, since a 404 page gives it nothing new.
It didn’t. The loop had given the crawler an enormous queue, and it kept working through it at about 3,700 requests a minute, still well over half of everything the site received. Worse, each of those 404s now cost the edge a few storage lookups, where the redirects had cost none. Caching 404s helped every other client but not this one, because every address it asked for was new. So we added one firewall rule in Cloudflare, scoped as narrowly as we could make it: requests from those two networks, and only for paths under /talks/, are blocked before they reach our code. The site has nothing under /talks/, so no real content can be caught by it, and the same networks’ other requests still get through. It blocked 7,500 requests in its first few minutes. The crawler’s traffic reaching the site went from 5,000 to 8,000 requests a minute to between none and fifteen, and blocked requests cost us nothing: no compute, no storage reads, no log volume. Without request-level logs we would have seen a busy site, not one crawler, and blocking whole cloud networks to stop it would have caught real users too.
The old site almost certainly had the same crawler, since production has the identical rule. We couldn’t see it: the old logs fed statistics jobs, and nobody was reading them request by request. The same afternoon’s logs turned up two smaller problems, which were fixed the same day. 774 requests for old links with a doubled slash (/packages//release/...) got a 404 where the old server quietly served the page. A download resuming past the end of a file got an error instead of the right “nothing left to send” answer.
Taken together, that is somewhere between 5 and 12 million fewer requests a day, depending on which stretch of the first evening you take as the crawler’s normal rate: well over half of the edge’s request and log volume. Those figures are projected from minutes of data, and a full day’s comparison will replace them.
Costs: about $5,000 a month down to a couple of hundred
Status: projected, and re-estimated 2026-09-28 from the first hours of production traffic. The Cloudflare side is running now; the AWS side retires at milestone M2, so today we pay for both. Prices are Cloudflare’s published list prices, not a bill.
The legacy estate costs about $5,000 a month in AWS: CloudFront, the S3 bucket, and the master and staging web servers. That is before the builder hardware and the staff time to keep three operating systems patched. The difference is mostly egress. A package repository’s job is sending tarballs to the world, which is exactly what a CDN usually bills for most, and R2 doesn’t charge for it.
We used to quote about $80 a month for the replacement. That is the storage alone: the bucket holds 5.1 TB in 4.2 million objects, about $77 a month. Serving traffic adds per-request charges on top, and the first hours in production let us estimate them from real numbers instead of guessing. At today’s rate of about 6.8 million requests a day, with the crawler from the story above still working through its queue, the whole stack comes to roughly $500 a month. Most of that is R2 reads, about 800 million a month, and the crawler causes most of those: its 404s each cost a few storage lookups. Without the crawler it is roughly $240 a month. Caching 404s, which are now the largest source of storage reads, should bring it to around $170, still falling as the edge caches fill. So the honest figure is a couple of hundred dollars a month, not $80, against $5,000. The overview has the AWS breakdown.
Since 2026-10-02 we also record what both providers actually bill, day by day, so this story can move from projections to measured costs over time. A daily job collects the Bioconductor AWS account’s costs from Cost Explorer, back to 2025-09-01, and the Cloudflare account’s usage charges, back to 2026-06-17, which is as far as Cloudflare’s billing API goes. The Cloudflare account also hosts unrelated projects, so its figures are an upper bound for Bioconductor. The code is in the private infra-costs repository.
Faster, smaller downloads, and a regression we caught
Status: measured 2026-09-28 from one well-connected host in Denver, before and after fixing the regression described below.
The old site answers only over HTTP/1.1. The July 2026 performance audit measured an 11-second first paint on the homepage over a slow mobile connection, mostly from fetching dozens of small files one after another over a handful of connections. The new edge speaks HTTP/2 and compresses everything it serves, including the repository indexes that every install.packages() call reads, which the old site sends uncompressed. PACKAGES for the software repository goes over the wire as 236 KB instead of 944 KB, and VIEWS as 1.2 MB instead of 5.3 MB, about a quarter of the size each time. That matters most for exactly the people the audit worried about: users on slow or metered connections.
The same measurement found a regression, and it’s fixed. Package files (tarballs, PACKAGES, VIEWS, config.yaml) were never being served from the edge cache: a rule that protects newly built pages from stale copies had ended up covering the whole site, so every package download went back to storage. bioc-edge#36, the fix for #35, let the cache lookup include files only the mirror can hold, and those files now come from the edge cache, still expiring as soon as a package changes. From Denver, PACKAGES went from 447 to 85 ms, VIEWS from 550 to 205 ms, a tarball from 227 to 72 ms and config.yaml, which BiocManager reads on every call, from 232 to 58 ms. CloudFront is still faster from this one host, by 30 to 50 ms. About a third of that is network distance from this particular machine, and the rest is time spent in the Worker on every request, cached or not. That gap is the next thing to look at.
Download statistics that can tell people from bots
Status: in progress. The logs are being collected now. The classification is bioc-traffic#5: a first version (v0) is written and in use for analysis, and the issue is still open on 2026-10-01. bioc-registry#26 tracks generating statistics from the de-identified access log; it leaves the definition of a download to the classification. Published statistics still come from the legacy pipelines.
Bioconductor’s download statistics have never filtered bots. The legacy code has a list of known crawlers and build-node addresses, and the checks that use them are commented out; the newer system never reads the user agent at all. The download statistics page shows what that costs. In August and September 2025, downloads roughly quadrupled while the number of distinct clients didn’t move: CI and container builds reinstalling the same base packages. Those two months are still inflated in every published table and in the ranking behind each package’s Rank. Separately, a quarter of June 2026 went missing from the published series without anyone noticing.
The new edge keeps every request, with fields that help tell traffic apart: the client’s network (ASN), Cloudflare’s category for verified bots, and hashes of the TLS client hello. It has no bot score and no JA4 fingerprint; those need Cloudflare’s Enterprise Bot Management. The crawler above is a good example of why the network field matters. Its user agents claim to be browsers, so a user-agent filter would have counted it as people; its two networks give it away. The plan is a classification we build and maintain ourselves, with versioned rules so history can be recomputed when they change, and statistics generated from it as static files. Web usage and package usage are reported separately, rather than as one number that mixes people, installs and crawlers.
The build system migration has a path forward
Status: early. A working proof of concept in bioc-build, with a handful of successful builds as of early September 2026.
The move off the legacy build machines stalled on one gap: r-universe builds Bioconductor’s software packages, but not the experiment-data and workflow packages. Some are far bigger than its size limit, and until something else builds them, the old machines can’t be turned off. We measured that corpus instead of guessing at it. It is 434 experiment-data packages totalling 28 GiB, with 67 of them holding three quarters of the bytes, and 25 workflow packages that together fit in 319 MiB. The data almost never change: across the last release bump, 425 of 433 experiment packages shipped a tarball whose size moved by less than 1%.
bioc-build is a build pipeline on GitHub Actions that starts from a pinned copy of r-universe’s own workflow and diverges only where it has to: a configurable size limit, packages authorized by a reviewed manifest, and dependencies resolved from the new registry. Staying that close to r-universe means a package both systems build should produce the same artifact and the same check result, which is what makes a comparison possible. It is early, but the part that was stuck now has a route.