Package: HTAN 0.99.1 (source).
HTAN’s metadata answer three questions about every data file: what assay and processing step produced it, which biospecimen it was measured on, and which participant that biospecimen came from. This vignette shows how the Phase 1 data model records the answers, how to follow them with HTAN, and where the records are messier than the model suggests. It assumes you have read the introduction.
All output below was computed from the Release 7 snapshot:
The data model in brief
HTAN’s Phase 1 data model is a set of metadata templates, one per kind of record. Each template is called a component: Demographics, Diagnosis and Therapy for clinical data, Biospecimen for samples, and one component per assay and level for files, such as ScRNA-seqLevel3 or ImagingLevel2. Centers filled in a manifest (a spreadsheet) for each component, and the DCC validated it against the model before release. The model was maintained as a CSV in ncihtan/data-models, now archived, and documented in the HTAN manual (metadata standards, file standards, relationships).
ISB-CGC loads each component into its own BigQuery table, and the snapshot keeps those tables as they are. The model itself is in the snapshot too, as the schema table: one row per attribute, with its description, valid values and the component it belongs to. Here is the attribute that links a file to the files it came from:
schema <- htan_tbl("schema", con) |> collect()
nrow(schema)
#> [1] 1072
schema |>
filter(Attribute == "HTAN Parent Data File ID") |>
select(Attribute, Parent, Description) |>
as.list()
#> $Attribute
#> [1] "HTAN Parent Data File ID"
#>
#> $Parent
#> [1] "File"
#>
#> $Description
#> [1] "HTAN Data File Identifier indicating the file(s) from which these files were derived"Clinical data
Phase 1 clinical data are organised in three tiers (metadata standards). Tier 1 follows the NCI Genomic Data Commons clinical model, in seven components: demographics, diagnosis, exposure, family history, follow-up, molecular test and therapy. Tier 2 adds disease-agnostic HTAN extensions, and tier 3 adds disease-specific ones (breast, colorectal, lung, melanoma, brain and others).
Each clinical record carries the HTAN_Participant_ID it describes. To keep records de-identified, dates are never calendar dates. They are recorded as a number of days from the participant’s index date, which the center chooses and records.
The main tables
Row counts are for Release 7:
| table | rows | description |
|---|---|---|
| clinical_tier1_demographics | 2,805 | Sex, race, ethnicity, vital status; one row per participant |
| clinical_tier1_diagnosis | 2,300 | Primary diagnosis, site, stage, grade; one or more rows per participant |
| clinical_tier1_therapy | 3,337 | Treatments, with days from index |
| clinical_tier1_followup | 2,167 | Follow-up visits, progression, comorbidities |
| clinical_tier1_exposure | 2,004 | Smoking, alcohol and other exposures |
| clinical_tier1_familyhistory | 2,284 | Family history of cancer |
| clinical_tier1_moleculartest | 4,700 | Clinical molecular tests (e.g. receptor status, mutations) |
| clinical_tier2 | 359 | HTAN disease-agnostic clinical extensions |
| biospecimen | 13,105 | Every biospecimen and its parent, type, site, preservation |
| id_provenance | 464,304 | One row per file-parent-biospecimen link: the lineage table |
| gc_drs_map | 66,891 | HTAN file ID to CRDC General Commons DRS URI |
| imaging_channel_metadata | 23,972 | Channel (antibody/marker) lists for imaging files |
| tissueOrOrganOfOrigin_to_wholeOrgan_mapping | 332 | Fine-grained site of origin to whole organ |
| phase1_longitudinal_data | 7,620 | Per-participant timeline of biopsies, treatment and vital status |
| schema | 1,072 | The Phase 1 data model: attributes, descriptions, valid values |
The other tables are assay metadata, one per component, named after the assay and level (scRNAseq_level4_metadata, imaging_level2_metadata, 10xvisium_spatialtranscriptomics_scRNAseq_level3_metadata, …). The tier 3 tables are clinical_tier3_<disease>, and the Trans-Network Project SRRS has its own srrs_biospecimen, srrs_clinical_tier2 and srrs_imaging_level2_metadata. Use htan_tables() for the full list.
File components
Every assay table shares a core set of columns (HTAN_Data_File_ID, Filename, File_Format, File_Size, md5, Component, the Synapse entityId, HTAN_Center and Data_Release), plus assay-specific fields: workflow and reference genome for sequencing, channels and microscope for imaging, capture area and slide for Visium. htan_files() stacks the core columns of all of them. Here are the file components in Release 7, each with the model’s own description from the schema table:
| Component | level | files | Description |
|---|---|---|---|
| 10XGenomicsXeniumISSExperiment | NA | 33 | All data pertaining to the 10X Genomics Xenium In-Situ Hybridization experiment |
| 10xVisiumSpatialTranscriptomics-AuxiliaryFiles | NA | 465 | Auxiliary data associated with spot/slide analysis (aligned Images, quality control files, etc) from Spatial Transcriptomics. |
| 10xVisiumSpatialTranscriptomics-RNA-seqLevel1 | 1 | 794 | Files contain raw RNA-seq data associated with spot/slide data. |
| 10xVisiumSpatialTranscriptomics-RNA-seqLevel2 | 2 | 223 | Alignment workflows downstream of Spatial Transcriptomics RNA-seq Level 1. |
| 10xVisiumSpatialTranscriptomics-RNA-seqLevel3 | 3 | 1,264 | Processed data files based on Spatial Transcriptomics RNA-seq Level 2 and Spatial Transcriptomics Auxiliary files. |
| 10xVisiumSpatialTranscriptomics-RNA-seqLevel4 | 4 | 175 | Processed data files based on Spatial Transcriptomics RNA-seq Level 3. |
| BulkMethylation-seqLevel1 | 1 | 80 | Raw data for bulk methylation sequencing, such as FASTQs and unaligned BAMs |
| BulkMethylation-seqLevel2 | 2 | 51 | Aligned primary data for bulk methylation sequencing, such as gene expression matrix files, VCFs, etc. |
| BulkRNA-seqLevel1 | 1 | 4,134 | Bulk RNA-seq [EFO_0003738] |
| BulkRNA-seqLevel2 | 2 | 1,922 | Bulk RNA-seq alignment protocol description |
| BulkRNA-seqLevel3 | 3 | 640 | Bulk RNA-seq gene expression matrices |
| BulkWESLevel1 | 1 | 7,033 | Bulk Whole Exome Sequencing raw files |
| BulkWESLevel2 | 2 | 1,613 | Bulk Whole Exome Sequencing aligned files and QC |
| BulkWESLevel3 | 3 | 1,014 | Bulk Whole Exome Sequencing called variants |
| ElectronMicroscopyLevel1 | 1 | 74,252 | Raw electron microscopy data as one TIFF file per plane for a 3D image stack or per tile for a 2D large area montage |
| ElectronMicroscopyLevel2 | 2 | 36,073 | Processed electron microscopy data as one OME-TIFF image per plane or montage |
| ElectronMicroscopyLevel3 | 3 | 32 | Segmented electron microscopy data as .am or .tiff formats |
| ElectronMicroscopyLevel4 | 4 | 41 | Movies or other derived files from electron microscopy data |
| ExSeqMinimal | NA | 21,153 | Minimal metadata for the ExSeq assay |
| HI-C-seqLevel1 | 1 | 66 | Unaligned sequence data |
| HI-C-seqLevel2 | 2 | 34 | Aligned read pairs, contact matrix |
| HI-C-seqLevel3 | 3 | 2 | Summary data for the HI-C-seq assay. |
| ImagingLevel1 | 1 | 16 | Raw imaging data |
| ImagingLevel2 | 2 | 27,079 | Raw and pre-processed image data |
| ImagingLevel3Segmentation | 3 | 4,628 | Object segmentations |
| ImagingLevel4 | 4 | 3,539 | Derived imaging data: Object-by-feature array |
| MassSpectrometryLevel1 | 1 | 1,187 | Mass Spectrometry derived data that includes proteomics, metabolomics, and lipidomics, level 1 |
| MassSpectrometryLevel3 | 3 | 28 | Mass Spectrometry derived data that includes proteomics, metabolomics, and lipidomics, level 3 |
| MassSpectrometryLevel4 | 4 | 28 | Mass Spectrometry derived data that includes proteomics, metabolomics, and lipidomics, level 4 |
| NanoStringGeoMxDSPSpatialTranscriptomicsLevel1 | 1 | 239 | Files contain raw data output from the NanoString GeoMx DSP Pipeline. These can include RCC or DCC Files. |
| NanoStringGeoMxDSPSpatialTranscriptomicsLevel3 | 3 | 53 | Files contain processed data from the NanoString GeoMx DSP Pipeline. This level depends on GeoMx Level 1 and Imaging Level 2. |
| OtherAssay | NA | 514 | Metadata applying to any assay without standard descriptors. Can be used as a placeholder for minimal amount of metadata until the assay descriptors are standardized |
| RPPALevel2 | 2 | 26 | Array based protemics. Each dilution curve of spot intensities is fitted using the monotone increasing B-spline model in the SuperCurve R package. This fits a single curve using all the samples on a slide with the signal intensity as the response variable and the dilution steps as independent variables. The fitted curve is plotted with the signal intensities on the y-axis and the log2-concentration of proteins on the x-axis for diagnostic purposes. |
| RPPALevel3 | 3 | 26 | Level 3 Reverse Phase Protein Array (RPPA) data contains intra-batch normalized intensities. |
| RPPALevel4 | 4 | 26 | Level 4 Reverse Phase Protein Array (RPPA) data contains intra-batch corrected intensities. |
| SRRSImagingLevel2 | 2 | 143 | SRRS-specific HTAN raw and pre-processed image data |
| ScATAC-seqLevel1 | 1 | 2,548 | scATAC-seq files containing sequence read information, with or without alignment, as FASTQ or BAM files |
| ScATAC-seqLevel2 | 2 | 344 | scATAC-seq files containing aligned sequence data, as a BAM file |
| ScATAC-seqLevel3 | 3 | 1,006 | Processed data files containing peak information for cells |
| ScATAC-seqLevel4 | 4 | 218 | Data represents the relationships between cells derived from Level 3 expression data and shown as tSNE or UMAP coordinates per cell, plus all other cell-specific meta information (e.g., cell type) |
| ScDNA-seqLevel1 | 1 | 1,774 | Single-cell DNA-seq |
| ScDNA-seqLevel2 | 2 | 1,271 | Alignment workflows downstream of scDNA-seq Level 1 |
| ScRNA-seqLevel1 | 1 | 15,201 | Single-cell RNA-seq [EFO_0008913] |
| ScRNA-seqLevel2 | 2 | 5,755 | Alignment workflows downstream of scRNA-seq Level 1 |
| ScRNA-seqLevel3 | 3 | 3,705 | Gene and Isoform expression files |
| ScRNA-seqLevel4 | 4 | 857 | Data represents the relationships between cells derived from Level 3 expression data and shown as tSNE or UMAP coordinates per cell, plus all other cell-specific meta information (e.g., cell type) |
| Slide-seqLevel1 | 1 | 7 | Raw sequencing files for the Slide-seq assay. |
| Slide-seqLevel2 | 2 | 47 | Aligned sequencing files and QC for the Slide-seq assay. |
| Slide-seqLevel3 | 3 | 329 | Gene matrices with features and barcodes for Slide-seq as well as spatial information (bead location files). |
Identifiers
The grammar
Every participant, biospecimen and file has an HTAN identifier, and all of them start with the center’s prefix (HTA1 to HTA16 in Phase 1; see the introduction for the list). The HTAN manual gives the Phase 1 grammar as (identifiers):
<participant_id> ::= <htan_center_id>_<integer>
<derivative_entity_id> ::= <participant_id>_<integer>So HTA11_7862 is participant 7862 of center HTA11 (Vanderbilt), and anything derived from that participant, whether a biospecimen or a file, is HTA11_7862_<integer>. Phase 1 identifiers do not say whether an ID is a biospecimen or a file; the table it appears in does. (Phase 2 adds B and D markers, as in HTA209_1_B1 and HTA209_1_D1.) A participant number of 0000 marks data pooled from several participants.
A regular expression splits an ID into its parts:
parse_htan_id <- function(id) {
m <- regmatches(id, regexec("^(HTA[0-9]+)_([0-9]+)(?:_(.+))?$", id))
tibble::tibble(
id = id,
center = vapply(m, `[`, "", 2L),
participant = vapply(m, function(x) paste(x[2:3], collapse = "_"),
""),
rest = vapply(m, `[`, "", 4L)
)
}
parse_htan_id(c("HTA11_7862", "HTA11_7862_2000001011",
"HTA11_7862_200000101134111", "HTA5_0000_1003255"))
#> # A tibble: 4 × 4
#> id center participant rest
#> <chr> <chr> <chr> <chr>
#> 1 HTA11_7862 HTA11 HTA11_7862 ""
#> 2 HTA11_7862_2000001011 HTA11 HTA11_7862 "2000001011"
#> 3 HTA11_7862_200000101134111 HTA11 HTA11_7862 "200000101134111"
#> 4 HTA5_0000_1003255 HTA5 HTA5_0000 "1003255"The prefix names the center that submitted the data. You can check that against the HTAN_Center column:
htan_files(con = con) |>
mutate(prefix = regexp_extract(HTAN_Data_File_ID, "^HTA[0-9]+")) |>
count(prefix, HTAN_Center) |>
arrange(as.integer(substr(prefix, 4L, 6L))) |>
collect()
#> # A tibble: 16 × 3
#> prefix HTAN_Center n
#> <chr> <chr> <dbl>
#> 1 HTA1 HTAN HTAPP 41626
#> 2 HTA2 HTAN BU 87
#> 3 HTA3 HTAN BU 5232
#> 4 HTA4 HTAN CHOP 4739
#> 5 HTA5 HTAN DFCI 17560
#> 6 HTA5 HTAN SRRS 11
#> 7 HTA6 HTAN Duke 7035
#> 8 HTA7 HTAN HMS 1690
#> 9 HTA8 HTAN MSK 5381
#> 10 HTA9 HTAN OHSU 116657
#> 11 HTA10 HTAN Stanford 6321
#> 12 HTA11 HTAN Vanderbilt 3340
#> 13 HTA12 HTAN WUSTL 7176
#> 14 HTA13 HTAN TNP SARDANA 301
#> 15 HTA14 HTAN TNP - TMA 4281
#> 16 HTA15 HTAN SRRS 251Two quirks show up. Files with the PCAPP prefix HTA2 were submitted under HTAN BU, and a few HTA5 files under HTAN SRRS. And there are no HTA16 (CASI) files in Release 7 at all. To group by center, use HTAN_Center; to group by the atlas a participant was enrolled in, use the prefix.
regexp_extract() is a DuckDB function. dbplyr passes functions it does not know straight through to SQL, so you can use DuckDB’s string and regex functions inside dplyr verbs.
Do not parse the participant out of a file ID
It is tempting to read a file’s participant off its ID. That is wrong often enough to matter:
f <- htan_files(con = con) |>
select(HTAN_Data_File_ID, HTAN_Participant_ID, n_participants) |>
collect() |>
mutate(id_participant = sub("^(HTA[0-9]+_[0-9]+).*$", "\\1",
HTAN_Data_File_ID))
single <- filter(f, n_participants == 1)
mismatch <- filter(single, id_participant != HTAN_Participant_ID)
nrow(mismatch)
#> [1] 2101
mismatch |>
count(prefix = sub("_.*", "", HTAN_Data_File_ID), sort = TRUE)
#> # A tibble: 10 × 2
#> prefix n
#> <chr> <int>
#> 1 HTA4 1508
#> 2 HTA14 190
#> 3 HTA2 87
#> 4 HTA12 74
#> 5 HTA6 55
#> 6 HTA5 52
#> 7 HTA9 48
#> 8 HTA3 42
#> 9 HTA10 32
#> 10 HTA1 13
head(mismatch, 3)
#> # A tibble: 3 × 4
#> HTAN_Data_File_ID HTAN_Participant_ID n_participants id_participant
#> <chr> <chr> <dbl> <chr>
#> 1 HTA4_1001_1424 HTA4_1022 1 HTA4_1001
#> 2 HTA4_1001_1744 HTA4_1009 1 HTA4_1001
#> 3 HTA4_1001_1995 HTA4_1010 1 HTA4_1001Of the 217,475 files that come from one participant, 2,101 (1.0%) carry another participant’s number in their own ID, most of them from HTA4 (CHOP). Across all files, 1,236 have the pooled participant number 0000, and 1,942 are attributed to more than one participant. A file’s participant is whatever id_provenance says, and that is where htan_files() gets it.
The hierarchy: participant, biospecimen, file
Biospecimens
The biospecimen table holds 13,105 rows. Each has an HTAN_Parent_ID. For an originating biospecimen (the biopsy or resection itself) that is the participant; for a derived biospecimen (a block, a section, a cell suspension, an extracted analyte) it is the biospecimen it was cut or extracted from. Follow those links and you have the biospecimen’s ancestry. Counting what each parent ID points to shows that not every one is a participant or a biospecimen:
bs <- htan_tbl("biospecimen", con) |>
select(HTAN_Biospecimen_ID, HTAN_Parent_ID, Biospecimen_Type,
Acquisition_Method_Type, Preservation_Method, Tumor_Tissue_Type,
Site_of_Resection_or_Biopsy) |>
collect()
participants <- htan_tbl("clinical_tier1_demographics", con) |>
distinct(HTAN_Participant_ID) |>
pull()
bs |>
mutate(parent_is = case_when(
HTAN_Parent_ID == HTAN_Biospecimen_ID ~ "itself",
HTAN_Parent_ID %in% participants ~ "a participant",
HTAN_Parent_ID %in% bs$HTAN_Biospecimen_ID ~ "a biospecimen",
.default = "an ID in neither table"
)) |>
count(parent_is)
#> # A tibble: 4 × 2
#> parent_is n
#> <chr> <int>
#> 1 a biospecimen 6807
#> 2 a participant 6004
#> 3 an ID in neither table 288
#> 4 itself 6A small function walks a biospecimen up to its participant. It stops at a participant, an unknown ID, or a cycle:
biospecimen_path <- function(id, bs) {
parent <- setNames(bs$HTAN_Parent_ID, bs$HTAN_Biospecimen_ID)
path <- id
while (id %in% names(parent) && !parent[[id]] %in% path) {
id <- parent[[id]]
path <- c(path, id)
}
path
}
biospecimen_path("HTA5_1430_1022", bs)
#> [1] "HTA5_1430_1022" "HTA5_1430_1020" "HTA5_1430_1001" "HTA5_1430_1000"
#> [5] "HTA5_1430"Participant HTA5_1430, from the Dana-Farber melanoma atlas, had a lymph node metastasis resected (every biospecimen in the chain has Tumor_Tissue_Type “Metastatic”). The fresh resection (_1000) was divided, snap frozen and divided again before an assay was run on _1022. Here is the path with each biospecimen’s annotation:
bs |>
filter(HTAN_Biospecimen_ID %in% biospecimen_path("HTA5_1430_1022", bs)) |>
arrange(HTAN_Biospecimen_ID) |>
select(HTAN_Biospecimen_ID, HTAN_Parent_ID, Preservation_Method,
Site = Site_of_Resection_or_Biopsy)
#> # A tibble: 4 × 4
#> HTAN_Biospecimen_ID HTAN_Parent_ID Preservation_Method Site
#> <chr> <chr> <chr> <chr>
#> 1 HTA5_1430_1000 HTA5_1430 Fresh Lymph nodes of axilla …
#> 2 HTA5_1430_1001 HTA5_1430_1000 Snap Frozen Lymph nodes of axilla …
#> 3 HTA5_1430_1020 HTA5_1430_1001 Snap Frozen Lymph nodes of axilla …
#> 4 HTA5_1430_1022 HTA5_1430_1020 Snap Frozen Lymph nodes of axilla …Annotations such as tissue type and site are often repeated on every biospecimen in a chain, but not always; when they differ, the one closest to the assayed biospecimen describes what was measured.
Files
Files connect to biospecimens and to each other through two columns of the assay tables. Level 1 files, and other files measured directly on a sample, have an HTAN_Parent_Biospecimen_ID: the biospecimen that went into the instrument. Level 2 to 4 files have an HTAN_Parent_Data_File_ID, naming the file or files they were computed from.
So a Level 4 single-cell object points at its Level 3 count matrices, which point at Level 2 alignments, which point at Level 1 FASTQs, which point at a biospecimen. Reconstructing that chain from the assay tables means joining across many tables and splitting comma-separated lists. The DCC did it once, for every file, and published the result as id_provenance.
id_provenance, the lineage table
id_provenance has one row per combination of a file, one of its parent files, and one of the biospecimens it was derived from, with the participant attached. Its columns, with the descriptions cut to 70 characters:
htan_columns("id_provenance", con) |>
select(column, description) |>
mutate(description = substr(description, 1, 70))
#> # A tibble: 12 × 2
#> column description
#> <chr> <chr>
#> 1 HTAN_Data_File_ID "Self-identifier for this data file - HTAN I…
#> 2 Filename "Name of a file"
#> 3 entityId "Synapse ID of entity"
#> 4 Component "Category of metadata (e.g. Diagnosis, Biosp…
#> 5 Data_Release "Inclusion in HTAN Data Release e.g. Release…
#> 6 GC_Release "Released to CRDC’s General Commons (GC) as …
#> 7 HTAN_Center "Name of HTAN Center e.g. HTAN HTAPP"
#> 8 HTAN_Assayed_Biospecimen_ID "HTAN ID of the biospecimen directly assayed…
#> 9 HTAN_Parent_Data_File_ID "HTAN Data File Identifier indicating the fi…
#> 10 HTAN_Originating_Biospecimen_ID "HTAN ID of the biopsied or resected biospec…
#> 11 HTAN_Participant_ID "HTAN ID associated with a patient based on …
#> 12 Biospecimen_Path "Full Biospecimen ancestry path from HTAN_As…The biospecimen terms come from the HTAN manual (relationships). The originating biospecimen is the biopsy or resection taken from the participant, the assayed biospecimen is the one put into the instrument, and the biospecimen path is the comma-separated chain from the first to the second. In Release 7 the table holds:
prov <- htan_tbl("id_provenance", con)
prov |>
summarise(
rows = n(),
files = n_distinct(HTAN_Data_File_ID),
participants = n_distinct(HTAN_Participant_ID),
biospecimens = n_distinct(HTAN_Assayed_Biospecimen_ID)
) |>
collect()
#> # A tibble: 1 × 4
#> rows files participants biospecimens
#> <dbl> <dbl> <dbl> <dbl>
#> 1 464304 221167 2416 10895A file has more than one row whenever it has several parents or several biospecimens. Here are the counts per file, in bins:
per_file <- prov |>
group_by(HTAN_Data_File_ID) |>
summarise(
parents = n_distinct(HTAN_Parent_Data_File_ID),
biospecimens = n_distinct(HTAN_Assayed_Biospecimen_ID),
participants = n_distinct(HTAN_Participant_ID)
) |>
collect()
bucket <- function(x) {
cut(x, c(-Inf, 0, 1, 2, 10, 100, Inf),
labels = c("0", "1", "2", "3-10", "11-100", ">100"))
}
tibble::tibble(
count = levels(bucket(1)),
parents = as.vector(table(bucket(per_file$parents))),
biospecimens = as.vector(table(bucket(per_file$biospecimens))),
participants = as.vector(table(bucket(per_file$participants)))
)
#> # A tibble: 6 × 4
#> count parents biospecimens participants
#> <chr> <int> <int> <int>
#> 1 0 0 0 0
#> 2 1 210451 219178 219224
#> 3 2 8048 98 87
#> 4 3-10 2031 707 1520
#> 5 11-100 245 1175 331
#> 6 >100 392 9 5Most files have one parent, one biospecimen and one participant. A file with no parent is a Level 1 file, or a file whose parent was not recorded. The long tail is merged objects: a Level 4 single-cell atlas built from dozens of samples has dozens of parents, biospecimens and participants.
htan_lineage() walks the parent links recursively in DuckDB and returns one row per (query file, related file), with depth counting generations (0 for the query, negative for ancestors, positive for descendants).
Worked lineage walks
A Visium spatial transcriptomics object, back to the patient
Start with a Level 4 Visium object from the Vanderbilt colorectal atlas:
visium_id <- "HTA11_7862_200000101134111"
htan_files(HTAN_Data_File_ID == visium_id, con = con) |>
select(Component, Filename, File_Size, HTAN_Participant_ID,
HTAN_Assayed_Biospecimen_ID, access) |>
collect() |>
glimpse()
#> Rows: 1
#> Columns: 6
#> $ Component <chr> "10xVisiumSpatialTranscriptomics-RNA-seqLe…
#> $ Filename <chr> "visium_level_4/8270_AS_1_unfiltered.h5ad"
#> $ File_Size <dbl> 13075894
#> $ HTAN_Participant_ID <chr> "HTA11_7862"
#> $ HTAN_Assayed_Biospecimen_ID <chr> "HTA11_7862_2000001011"
#> $ access <chr> "open-synapse"Its ancestors:
htan_lineage(visium_id, con = con) |>
arrange(depth, related_file_id) |>
collect() |>
transmute(depth, related_file_id, file = basename(Filename))
#> # A tibble: 8 × 3
#> depth related_file_id file
#> <int> <chr> <chr>
#> 1 -3 HTA11_7862_200000101131111 8270-AS-0001_S1_L005_R1_001.fastq.gz
#> 2 -3 HTA11_7862_200000101131121 8270-AS-0001_S1_L005_R2_001.fastq.gz
#> 3 -2 HTA11_7862_200000101132111 8270_AS_1_possorted_genome_bam.bam
#> 4 -1 HTA11_7862_200000101133111 8270_AS_1_barcodes.tsv.gz
#> 5 -1 HTA11_7862_200000101133121 8270_AS_1_features.tsv.gz
#> 6 -1 HTA11_7862_200000101133131 8270_AS_1_matrix.mtx.gz
#> 7 -1 HTA11_7862_200000101133141 8270_AS_1_tissue_positions_list.csv
#> 8 0 HTA11_7862_200000101134111 8270_AS_1_unfiltered.h5adThe chain is FASTQ (Level 1) to aligned BAM (Level 2) to the four pieces of the Space Ranger output (Level 3: barcodes, features, the count matrix and spot positions) to the processed object (Level 4). Every file in it was measured on the same biospecimen. That biospecimen leads to the participant:
biospecimen_path("HTA11_7862_2000001011", bs)
#> [1] "HTA11_7862_2000001011" "HTA11_7862"
bs |>
filter(HTAN_Biospecimen_ID == "HTA11_7862_2000001011") |>
select(-HTAN_Parent_ID) |>
glimpse()
#> Rows: 1
#> Columns: 6
#> $ HTAN_Biospecimen_ID <chr> "HTA11_7862_2000001011"
#> $ Biospecimen_Type <chr> "Tissue Biospecimen Type"
#> $ Acquisition_Method_Type <chr> "Biopsy"
#> $ Preservation_Method <chr> "Fresh"
#> $ Tumor_Tissue_Type <chr> "Premalignant"
#> $ Site_of_Resection_or_Biopsy <chr> "Sigmoid colon"The biospecimen is a fresh biopsy of a premalignant lesion of the sigmoid colon. Its Histologic_Morphology_Code (not shown above) is M82110, the ICD-O-3 code for tubular adenoma, the common precancerous colorectal polyp. The participant’s diagnosis record tells us almost nothing:
htan_tbl("clinical_tier1_diagnosis", con) |>
filter(HTAN_Participant_ID == "HTA11_7862") |>
select(Primary_Diagnosis, Tissue_or_Organ_of_Origin, Age_at_Diagnosis) |>
collect()
#> # A tibble: 1 × 3
#> Primary_Diagnosis Tissue_or_Organ_of_Origin Age_at_Diagnosis
#> <chr> <chr> <chr>
#> 1 Not Reported Not Reported 0This is common in the pre-cancer atlases: the biospecimen carries the histology, and the diagnosis table, built around cancer diagnoses, says “Not Reported”. Look in both.
Comma-separated parents
The assay table records the Visium object’s parents in a single field:
while id_provenance gives them one row each:
prov |>
filter(HTAN_Data_File_ID == visium_id) |>
select(HTAN_Parent_Data_File_ID, HTAN_Assayed_Biospecimen_ID) |>
collect()
#> # A tibble: 4 × 2
#> HTAN_Parent_Data_File_ID HTAN_Assayed_Biospecimen_ID
#> <chr> <chr>
#> 1 HTA11_7862_200000101133111 HTA11_7862_2000001011
#> 2 HTA11_7862_200000101133121 HTA11_7862_2000001011
#> 3 HTA11_7862_200000101133131 HTA11_7862_2000001011
#> 4 HTA11_7862_200000101133141 HTA11_7862_2000001011The comma-separated form is common. Here, for four assay tables, is the number of files whose parent field holds a list:
count_commas <- function(table) {
htan_tbl(table, con) |>
summarise(
files = n(),
several_parents = sum(as.integer(
grepl(",", HTAN_Parent_Data_File_ID)), na.rm = TRUE)
) |>
collect() |>
mutate(table = table, .before = 1)
}
bind_rows(lapply(c("scRNAseq_level3_metadata", "scRNAseq_level4_metadata",
"imaging_level4_metadata",
"10xvisium_spatialtranscriptomics_scRNAseq_level4_metadata"),
count_commas))
#> # A tibble: 4 × 3
#> table files several_parents
#> <chr> <dbl> <dbl>
#> 1 scRNAseq_level3_metadata 3705 57
#> 2 scRNAseq_level4_metadata 857 604
#> 3 imaging_level4_metadata 3539 383
#> 4 10xvisium_spatialtranscriptomics_scRNAseq_level4_metada… 175 93A filter such as HTAN_Parent_Data_File_ID == "HTA11_7862_200000101133111" on the assay table misses the Visium object. Query id_provenance instead, or split the field first; in DuckDB, unnest(string_split(x, ',')) does it.
Descendants of a raw file
Walking down from a Level 1 file finds everything computed from it. Take a single-cell RNA-seq FASTQ from Washington University:
htan_lineage("HTA12_251_1000", direction = "descendants", con = con) |>
arrange(depth, related_file_id) |>
select(depth, related_file_id, Component) |>
collect()
#> # A tibble: 8 × 3
#> depth related_file_id Component
#> <int> <chr> <chr>
#> 1 0 HTA12_251_1000 ScRNA-seqLevel1
#> 2 1 HTA12_251_2000 ScRNA-seqLevel2
#> 3 2 HTA12_251_3000 ScATAC-seqLevel3
#> 4 2 HTA12_251_3001 ScATAC-seqLevel3
#> 5 2 HTA12_251_3002 ScATAC-seqLevel3
#> 6 2 HTA12_251_3003 ScATAC-seqLevel3
#> 7 3 HTA12_251_4000 ScRNA-seqLevel4
#> 8 3 HTA12_251_4001 ScATAC-seqLevel4The descendants mix two assays. The scRNA-seq alignment is recorded as a parent of scATAC-seq Level 3 files, and those are parents of both the RNA and the ATAC Level 4 objects. This is presumably a multiome (joint RNA and ATAC) experiment processed together, though the metadata do not say so directly. Either way, lineage is a graph rather than a tree, and the assay can change along a path.
A multiplexed imaging cell table
Imaging follows the same pattern. Here is a Level 4 cell-by-marker table from an Orion whole-slide image (Harvard Medical School):
htan_lineage("HTA7_1000_1003", con = con) |>
arrange(depth) |>
collect() |>
transmute(depth, related_file_id, Component, file = basename(Filename))
#> # A tibble: 3 × 4
#> depth related_file_id Component file
#> <int> <chr> <chr> <chr>
#> 1 -2 HTA7_1000_1000 ImagingLevel2 LSP15300_P54_A31_C100_HMS_Ori…
#> 2 -1 HTA7_1000_1002 ImagingLevel3Segmentation LSP15300-CRC68-mask.ome.tif
#> 3 0 HTA7_1000_1003 ImagingLevel4 LSP15300-CRC68.csvThe table was computed from a segmentation mask (Level 3), and the mask from the stitched, registered image (Level 2). For the channels (markers) of an imaging file, see imaging_channel_metadata.
A file from several participants
Some single-cell objects pool cells from several participants. This Level 4 AnnData file from the Dana-Farber melanoma atlas is named for one participant, but id_provenance attributes it to four:
pooled_id <- "HTA5_0000_1003255"
htan_files(HTAN_Data_File_ID == pooled_id, con = con) |>
select(Filename, n_participants, HTAN_Participant_ID) |>
collect() |>
glimpse()
#> Rows: 1
#> Columns: 3
#> $ Filename <chr> "HTAN DFCI/single_cell_RNAseq_level_4_melanoma/all…
#> $ n_participants <dbl> 4
#> $ HTAN_Participant_ID <chr> "HTA5_1470,HTA5_1536,HTA5_1765,HTA5_1950"
prov |>
filter(HTAN_Data_File_ID == pooled_id) |>
distinct(HTAN_Parent_Data_File_ID, HTAN_Participant_ID,
HTAN_Assayed_Biospecimen_ID) |>
collect()
#> # A tibble: 4 × 3
#> HTAN_Parent_Data_File_ID HTAN_Participant_ID HTAN_Assayed_Biospecimen_ID
#> <chr> <chr> <chr>
#> 1 HTA5_0000_1003231 HTA5_1470 HTA5_1470_1004
#> 2 HTA5_0000_1003231 HTA5_1536 HTA5_1536_1004
#> 3 HTA5_0000_1003231 HTA5_1765 HTA5_1765_1004
#> 4 HTA5_0000_1003231 HTA5_1950 HTA5_1950_1004The attribution comes from its parent, a Level 3 file that was itself pooled from four participants’ samples (the 0000 in its ID). The metadata cannot tell you which cells in the object came from which participant. If the object records that at all, it does so in its own cell annotations. So when a file has more than one participant, htan_experiment() asks for a participant_col: the column of colData that names each cell’s participant or biospecimen.
Pitfalls
The data model is consistent; the data entered into it, by more than a dozen centers over seven releases, are less so. These are the problems most likely to give silently wrong answers.
Comma-separated lists. HTAN_Parent_Data_File_ID in the assay tables, Biospecimen_Path in id_provenance and Adjacent_Biospecimen_IDs in biospecimen hold comma-separated lists. In htan_files(), HTAN_Participant_ID and HTAN_Assayed_Biospecimen_ID are comma-separated for multi-participant files. Never compare these fields with ==. Use id_provenance or the participants argument of htan_files(), or split them first.
Several parents, several participants. A file can have many parents and many participants (see the multiplicity table above), so joining id_provenance to anything multiplies rows. Use distinct() or aggregate to one row per file before joining, as htan_files() does.
Participants inferred from IDs. As shown above, 1.0% of single-participant files carry another participant’s number in their ID. Take participants from id_provenance.
Biospecimen parents that lead nowhere. Most biospecimens point to a participant or another biospecimen. Some point to themselves, a cycle that a naive recursive walk never leaves, and some to participant IDs with no demographics record. The biospecimen_path() function above stops at both.
Duplicate records. Some participants and biospecimens appear more than once:
dup_count <- function(table, id) {
htan_tbl(table, con) |>
summarise(rows = n(), distinct_ids = n_distinct(.data[[id]])) |>
collect() |>
mutate(table = table, .before = 1)
}
bind_rows(
dup_count("clinical_tier1_demographics", "HTAN_Participant_ID"),
dup_count("clinical_tier1_diagnosis", "HTAN_Participant_ID"),
dup_count("biospecimen", "HTAN_Biospecimen_ID")
)
#> # A tibble: 3 × 3
#> table rows distinct_ids
#> <chr> <dbl> <dbl>
#> 1 clinical_tier1_demographics 2805 2797
#> 2 clinical_tier1_diagnosis 2300 2273
#> 3 biospecimen 13105 13095A participant can legitimately have several diagnoses, but duplicated demographics and biospecimen rows are most likely entry artefacts. Decide which row to keep before joining; htan_experiment() keeps the first row per participant, ordered by ID.
Spelling and missingness. Categorical values vary in case and form (Female and female, Primary and primary, Not Recorded and Not recorded), and missing values appear as NA, "Not Reported", "unknown", "Not Applicable" and sometimes "0". Before you tabulate, normalise case and decide what counts as missing. Here are the values of Tumor_Tissue_Type:
htan_tbl("biospecimen", con) |>
count(Tumor_Tissue_Type) |>
arrange(desc(n)) |>
collect()
#> # A tibble: 15 × 2
#> Tumor_Tissue_Type n
#> <chr> <dbl>
#> 1 Primary 5520
#> 2 Metastatic 2340
#> 3 Not Otherwise Specified 2259
#> 4 Premalignant 894
#> 5 Normal 399
#> 6 Atypia - hyperplasia 343
#> 7 Normal adjacent 343
#> 8 <NA> 273
#> 9 Premalignant - in situ 237
#> 10 Recurrent 230
#> 11 Additional Primary 101
#> 12 primary 92
#> 13 Post therapy neoadjuvant 58
#> 14 Local recurrence 13
#> 15 Normal distant 3Numbers stored as text. Clinical fields such as Age_at_Diagnosis and Days_to_Last_Follow_up are character columns, because they mix numbers with codes like "Not Applicable". Convert them with as.numeric(), which turns the codes into NA, and treat a value of 0 with suspicion. In the Vanderbilt diagnosis record above, Age_at_Diagnosis is "0" while every other field is “Not Reported”; that looks more like a placeholder than an age.
The same file in several tables. A few files appear in more than one assay table (for example, mass spectrometry files in both a Release 7 table and a legacy Release 2 table). htan_files() keeps one row per file, from the newest table.
Files without provenance. Some files have no id_provenance rows, and therefore no participant. By component:
htan_files(n_participants == 0, con = con) |>
count(Component) |>
arrange(desc(n)) |>
collect()
#> # A tibble: 8 × 2
#> Component n
#> <chr> <dbl>
#> 1 ElectronMicroscopyLevel2 1461
#> 2 OtherAssay 392
#> 3 BulkWESLevel3 294
#> 4 ScRNA-seqLevel4 100
#> 5 ImagingLevel1 16
#> 6 ScRNA-seqLevel3 4
#> 7 BulkWESLevel2 3
#> 8 ImagingLevel2 1Summary
- Identifiers start with a center prefix and, in Phase 1, do not distinguish biospecimens from files. The prefix identifies the center reliably; the participant number does not reliably identify the participant.
- The hierarchy is participant, then a chain of biospecimens linked by
HTAN_Parent_ID, then a graph of files linked byHTAN_Parent_Biospecimen_ID(Level 1) andHTAN_Parent_Data_File_ID(Levels 2 to 4). -
id_provenanceflattens all of that into one table, one row per file, parent and biospecimen;htan_files()andhtan_lineage()are built on it. - Expect lists in single fields, many-to-many links, duplicates and inconsistent spelling, and check before you join or count.
The atlas overview uses these tables to describe the whole release, and the analysis vignette follows one file from query to analysis.
Session information
sessionInfo()
#> R version 4.6.1 (2026-06-24)
#> Platform: x86_64-pc-linux-gnu
#> Running under: Ubuntu 24.04.5 LTS
#>
#> Matrix products: default
#> BLAS: /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
#> LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.26.so; LAPACK version 3.12.0
#>
#> locale:
#> [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
#> [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
#> [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
#> [7] LC_PAPER=en_US.UTF-8 LC_NAME=C
#> [9] LC_ADDRESS=C LC_TELEPHONE=C
#> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
#>
#> time zone: Etc/UTC
#> tzcode source: system (glibc)
#>
#> attached base packages:
#> [1] stats graphics grDevices utils datasets methods base
#>
#> other attached packages:
#> [1] dplyr_1.2.1 HTAN_0.99.1
#>
#> loaded via a namespace (and not attached):
#> [1] bit_4.6.0 jsonlite_2.0.0 compiler_4.6.1
#> [4] filelock_1.0.3 tidyselect_1.2.1 blob_1.3.0
#> [7] yaml_2.3.12 fastmap_1.2.0 R6_2.6.1
#> [10] generics_0.1.4 curl_8.1.0 httr2_1.3.0
#> [13] knitr_1.52 tibble_3.3.1 DBI_1.3.0
#> [16] pillar_1.11.1 rlang_1.3.0 utf8_1.2.6
#> [19] cachem_1.1.0 xfun_0.61 bit64_4.8.6
#> [22] otel_0.2.0 RSQLite_3.53.3 memoise_2.0.1
#> [25] cli_3.6.6 withr_3.0.3 magrittr_2.0.5
#> [28] digest_0.6.39 dbplyr_2.6.0 lifecycle_1.0.5
#> [31] BiocFileCache_3.3.0 vctrs_0.7.3 evaluate_1.0.5
#> [34] glue_1.8.1 duckdb_1.5.6 rmarkdown_2.32
#> [37] purrr_1.2.2 tools_4.6.1 pkgconfig_2.0.3
#> [40] htmltools_0.5.9