Find HTAN files — htan

One row per HTAN data file, combining every file-level metadata table in the snapshot (the tables with HTAN_Data_File_ID, Filename, File_Format and Component columns) with the participants and biospecimens from id_provenance and the CRDC General Commons DRS URI from gc_drs_map. The result is lazy: filter it with dplyr, then pass it to htan_download().

Usage

htan_files(..., participants = NULL, con = htan_con())

Arguments

...: Filter conditions passed to dplyr::filter(), e.g. assay == "ScRNA-seq", level == 4.
participants: character() or NULL. Keep files derived from any of these HTAN_Participant_IDs, including multi-participant files.
con: A connection from htan_con().

Value

A lazy tbl_duckdb_connection.

Details

Columns:

HTAN_Data_File_ID, Filename, File_Format, File_Size, md5, Component, entityId, HTAN_Center, Data_Release: From the file's metadata table. File_Format is coarse (hdf5 covers .h5ad and 10x .h5; RData covers .rds), so see also extension.
table: The snapshot table the row came from. A file listed in several tables appears once, from the most recently released one (in r7, 260 mass-spectrometry files are also in the legacy r2_v2 lipidomics_metadata and metabolomics_metadata tables).
assay, level: Component split into the assay and its level (1-4; NA for auxiliary and other files).
extension: Lower-case file extension from Filename, e.g. .h5ad, .rds, .csv, .tsv.gz.
HTAN_Participant_ID, HTAN_Assayed_Biospecimen_ID: From id_provenance. A file derived from several participants (about 1.6 percent of files in r7) lists them comma-separated; use the participants argument, which handles that case, rather than ==.
n_participants: Number of distinct participants.
drs_uri: CRDC General Commons DRS ID, if the file is there.
access: Where the file can be fetched: "open-gc" (open Level 2 imaging, anonymous DRS), "open-synapse" (Level 3/4, auxiliary and other open files, Synapse with a token), or "controlled" (Level 1/2 sequencing; dbGaP phs002371, not supported).

Examples

fixture <- system.file("extdata", "snapshot", package = "HTAN")
con <- htan_con("r7", base_url = fixture)
htan_files(con = con)
#> # A query:  ?? x 18
#> # Database: DuckDB 1.5.6 [unknown@Linux 7.0.0-1017-azure:R 4.6.1/:memory:]
#>    table        HTAN_Data_File_ID Filename File_Format File_Size md5   Component
#>    <chr>        <chr>             <chr>    <chr>           <dbl> <chr> <chr>    
#>  1 bulkWES_lev… HTA12_15_1025     whole_e… fastq         1.19e 9 9385… BulkWESL…
#>  2 imaging_lev… HTA12_15_1105     imc_lev… OME-TIFF      2.55e 5 a649… ImagingL…
#>  3 scATACseq_l… HTA12_251_2002    snATAC-… bam           3.37e10 c6ac… ScATAC-s…
#>  4 10xvisium_s… HTA12_253_1008    V10X_le… gzip          7.46e 9 d4d6… 10xVisiu…
#>  5 10xvisium_s… HTA12_253_1012    V10X_le… gzip          3.79e 9 ac1f… 10xVisiu…
#>  6 bulkWES_lev… HTA12_253_1022    WES_lev… gzip          3.16e 9 97ca… BulkWESL…
#>  7 bulkWES_lev… HTA12_253_1025    WES_lev… gzip          1.49e 9 c311… BulkWESL…
#>  8 imaging_lev… HTA12_253_2002    CODEX_I… OME-TIFF      2.33e 9 7b40… ImagingL…
#>  9 scRNAseq_le… HTA12_15_1003     sc_RNAs… fastq         8.71e 9 1005… ScRNA-se…
#> 10 bulkRNAseq_… HTA12_15_1011     bulk_RN… fastq         8.99e 8 af3d… BulkRNA-…
#> # ℹ more rows
#> # ℹ 11 more variables: entityId <chr>, HTAN_Center <chr>, Data_Release <chr>,
#> #   assay <chr>, level <int>, extension <chr>, HTAN_Participant_ID <chr>,
#> #   HTAN_Assayed_Biospecimen_ID <chr>, n_participants <dbl>, drs_uri <chr>,
#> #   access <chr>
htan_files(level == 4, extension == ".h5ad", con = con)
#> # A query:  ?? x 18
#> # Database: DuckDB 1.5.6 [unknown@Linux 7.0.0-1017-azure:R 4.6.1/:memory:]
#>   table         HTAN_Data_File_ID Filename File_Format File_Size md5   Component
#>   <chr>         <chr>             <chr>    <chr>           <dbl> <chr> <chr>    
#> 1 scRNAseq_lev… HTA5_0000_1003297 HTAN DF… hdf5        132312198 7c55… ScRNA-se…
#> 2 scRNAseq_lev… HTA5_0000_1003255 HTAN DF… hdf5         11599590 ac4f… ScRNA-se…
#> 3 scRNAseq_lev… HTA5_0000_1003257 HTAN DF… hdf5         40003858 0834… ScRNA-se…
#> 4 scRNAseq_lev… HTA5_0000_1003276 HTAN DF… hdf5        109992026 483f… ScRNA-se…
#> # ℹ 11 more variables: entityId <chr>, HTAN_Center <chr>, Data_Release <chr>,
#> #   assay <chr>, level <int>, extension <chr>, HTAN_Participant_ID <chr>,
#> #   HTAN_Assayed_Biospecimen_ID <chr>, n_participants <dbl>, drs_uri <chr>,
#> #   access <chr>
htan_files(participants = "HTA12_15", access == "open-gc", con = con)
#> # A query:  ?? x 18
#> # Database: DuckDB 1.5.6 [unknown@Linux 7.0.0-1017-azure:R 4.6.1/:memory:]
#>    table        HTAN_Data_File_ID Filename File_Format File_Size md5   Component
#>    <chr>        <chr>             <chr>    <chr>           <dbl> <chr> <chr>    
#>  1 imaging_lev… HTA12_15_1102     imc_lev… OME-TIFF     53859994 cf07… ImagingL…
#>  2 imaging_lev… HTA12_15_1106     imc_lev… OME-TIFF     49945197 c010… ImagingL…
#>  3 imaging_lev… HTA12_15_1107     imc_lev… OME-TIFF     50665773 e167… ImagingL…
#>  4 imaging_lev… HTA12_15_1104     h_and_e… svs         100726792 9ab4… ImagingL…
#>  5 imaging_lev… HTA12_15_1101     h_and_e… svs         133291818 294e… ImagingL…
#>  6 imaging_lev… HTA12_15_1105     imc_lev… OME-TIFF       254833 a649… ImagingL…
#>  7 imaging_lev… HTA12_15_1110     imc_lev… OME-TIFF     49593562 b237… ImagingL…
#>  8 imaging_lev… HTA12_15_1109     imc_lev… OME-TIFF     53524752 d5eb… ImagingL…
#>  9 imaging_lev… HTA12_15_1108     h_and_e… svs         105028040 0de0… ImagingL…
#> 10 imaging_lev… HTA12_15_1103     imc_lev… OME-TIFF     56245062 78a0… ImagingL…
#> # ℹ 11 more variables: entityId <chr>, HTAN_Center <chr>, Data_Release <chr>,
#> #   assay <chr>, level <int>, extension <chr>, HTAN_Participant_ID <chr>,
#> #   HTAN_Assayed_Biospecimen_ID <chr>, n_participants <dbl>, drs_uri <chr>,
#> #   access <chr>