Running a Bioconductor mirror

Published

September 30, 2026

This replaces the rsync-over-SSH instructions at bioconductor.org/about/mirrors/mirror-how-to/. Those instructions depend on master.bioconductor.org, which milestone M2 retires. The new site has no rsync server. Instead it publishes a manifest: a list of every object in the package repositories, with sizes and MD5s, that any HTTP client can sync from.

No credentials, no sign-up form, no SSH key. Fetch the manifest, fetch the files, recreate the symlinks, serve the tree.

Read this before running any sync command

packages/release/ and packages/devel/ are not directories on the new site. They are symlinks resolved per request, and object storage has no symlinks. A sync scoped to packages/release/ sees an empty listing, not an error. With rclone sync or aws s3 sync --delete, an empty remote means your entire existing mirror is deleted to match it. Always sync from the manifest, always copy rather than sync, and always dry-run first and check the object count.

What changed

	Old	New
Source	`bioc-rsync@master.bioconductor.org` over SSH	`https://bioconductor.org/api/v1/manifest/index.json` over HTTPS
Access	SSH key, requested by form	none
Tool	`rsync -zrtlv --delete`	`rclone copy --files-from`
Symlinks	created by hand: `release` and `devel`	listed in the manifest index (202 on 2026-09-30); the script recreates `release`, `devel` and the R-version aliases
Current version numbers	in the rsync module names	in `index.json`, read every run
Cadence	release monthly-ish, devel at most weekly	manifest regenerated on every hourly sync; syncing daily is enough

The manifest

Endpoint	Contents
`GET /api/v1/manifest/index.json`	which numeric versions `release` and `devel` point at, the symlink map, per-manifest object counts, generation timestamp
`GET /api/v1/manifest/<version>/<repo>.tsv.gz`	`path` · `size` · `md5`, one object per line, tab-separated

Repos are bioc, data-annotation, data-experiment, workflows and books, for the current release and devel only. Only the manifest names use a hyphen: the paths inside them, on the site and in your mirror are the usual packages/<version>/data/annotation/ and .../data/experiment/, so options(repos=) and R’s etc/repositories work unchanged. Older releases are not published; ask if you need them (reporting a problem).

The MD5 is a real MD5 of the object, not a composite ETag. An empty third column means no usable hash exists for that object and it should be checked by size alone. Today that column is populated for every object.

Choosing scope

Decide explicitly rather than mirroring everything.

Scope	Notes
`bioc` (`src/contrib`, `bin/windows`, `bin/macosx/*`)	Every mirror needs source. Binaries are comparable in size; skip if your users build from source.
`data-annotation`, `data-experiment`	Large, changes rarely.
`workflows`, `books`	Small.
Releases before the current one	Not in the manifest.
The historical archive (4.66 TB)	Not in the manifest. Never carried by any mirror.

Release and devel together are roughly 320 GB.

Running the sync

mirror-sync.sh is a short wrapper around rclone copy. It needs bash, curl, gunzip, jq and rclone, and checks for them before doing anything. It runs on Linux and macOS (brew install jq rclone).

curl -fsSLO https://seandavi.github.io/bioc-infrastructure/mirror-sync.sh
chmod +x mirror-sync.sh
DEST=/srv/bioc ./mirror-sync.sh          # DEST is the directory you serve as the site root

From cron, daily is plenty:

17 3 * * *  DEST=/srv/bioc /usr/local/bin/mirror-sync.sh >> /var/log/bioc-mirror-sync.log 2>&1

What it does

Reads index.json for the current release and devel versions, every run. Nothing is pinned: at a release roll the script follows the new numbers by itself.
Lists every file in those versions’ manifests for the repos you carry, then applies your INCLUDE/EXCLUDE patterns. A failed manifest download stops the run.
Copies with rclone copy --files-from. Only new and changed files are fetched, judged by size and modification time. rclone writes each file under a temporary name and renames it when complete, so your server never serves a half-written file. rclone doesn’t check the manifest’s MD5s; they’re there if you want to verify a copy yourself.
Recreates the symlinks (release, devel and the R-version aliases inside contrib/). rclone can’t make them, and without them, URLs that go through them return 404 on your mirror.

It never deletes. rclone’s http backend can’t list directories on this site, so rclone sync can’t tell “removed upstream” from “not listed” (see the warning above). Old package versions stay on disk until you remove them. At a release roll, delete the retired version yourself, e.g. rm -rf /srv/bioc/packages/3.22. That also clears the older tarballs that piled up while it was current.

Settings

Variable	Default
`DEST`	(required)	Directory you serve as the site root.
`REPOS`	`bioc data-annotation data-experiment workflows books`	Which repositories to carry (see Choosing scope).
`INCLUDE`	everything	Extended regex; copy only paths that match.
`EXCLUDE`	nothing	Extended regex; skip paths that match.
`RCLONE_*`		Any rclone setting, e.g. `RCLONE_TRANSFERS=16` for more parallel downloads.

Patterns match the full path, e.g. packages/3.23/bioc/src/contrib/limma_3.66.0.tar.gz. Source packages only, no data repos:

DEST=/srv/bioc REPOS="bioc workflows books" INCLUDE='/src/contrib/' ./mirror-sync.sh

Exit status is 0 when done, and non-zero with a message on stderr otherwise. A failed run has deleted nothing; rerun it.

Without the script

The script is the recipe; you can run its steps by hand or adapt them. The part that matters is the file list:

curl -fsS https://bioconductor.org/api/v1/manifest/3.23/bioc.tsv.gz | gunzip | cut -f1 \
  | grep '/src/contrib/' > files.txt
test -s files.txt
rclone copy --http-url https://bioconductor.org :http: /srv/bioc --files-from files.txt --no-traverse

A bare rclone sync against a directory URL, the way rsync used to work, isn’t possible yet: the site serves no directory listings for rclone to read. That’s tracked in bioc-edge#61.

Serving

Apache and nginx serve the tree as-is. Two things to get right:

Follow symlinks. Apache needs Options +FollowSymLinks on the directory, or every release/ URL returns 403.
Do not copy .htaccess from the old origin. It redirects old-release traffic out to the archive, which would send your users away from your own mirror.

Content types come from your server’s mime.types, so extensionless files such as PACKAGES and VIEWS behave as they always have.

Users point at your mirror exactly as before:

options(BioC_mirror = "https://mirror.example.org")
BiocManager::install("GEOquery")

Migrating an existing rsync mirror

Point the script at the DEST you already serve and run it by hand the first time. rsync kept the old origin’s timestamps, which differ from the new site’s, so the first run may download much of the tree again; later runs fetch only what changed. If packages/release or packages/devel is a real directory rather than a symlink, move it aside first, or the script’s link lands inside it. Nothing is deleted, so files the old tree has that the manifest doesn’t stay until you remove them. Tell the Bioconductor team when you have switched, through the site problem form, so the rsync account can be closed on your say-so rather than on log silence.

Listing your mirror

The mirrors page and chooseBioCmirror() are still the registry. That process has not changed; the request form on the mirrors page still applies.

Source

The generator and the reasoning behind the design, including why read-only storage credentials and a custom client were rejected, live in the serving repo’s MIRRORS.md.