Sync up development with main

.github/workflows/spell-check.yml

@@ -1,7 +1,6 @@
 name: Spell check Markdown files
 
 # Controls when the action will run.
-# Pull requests to master only.
 on:
   pull_request:
     branches:

components/dictionary.txt

@@ -19,6 +19,7 @@ cellhash
 cellhashing
 CELLxGENE
 CHANGELOG
+CNV
 confounders
 CZI
 CZI's
@@ -47,13 +48,15 @@ hashedDrops
 HDF
 hexamer
 HBC
+Heimberg
 Hemberg
 HTO
 HTODemux
 HTOs
 Hippen
 HVG
 HVGs
+inferCNV
 intronic
 introns
 isotype
@@ -65,12 +68,16 @@ Lun
 Marioni
 miQC
 MLA
+MuData
+Neuroblastoma
 oligo
 oligos
+OpenScPCA
 Pearson
 Prasad
 pre
 preprint
+programmatically
 pseudocount
 pseudogenes
 Looney

docs/CHANGELOG.md

@@ -12,6 +12,27 @@ For more information about `AlexsLemonade/scpca-nf` versions, please see [the re
 <!-- PUT THE NEW CHANGELOG ENTRY RIGHT BELOW THIS -->
 <!-------------------------------------------------->
 
+## 2025.12.04
+
+All data on the Portal has been updated to include a number of new features.
+
+* Doublet detection was run on all samples using [`scDblFinder`](https://bioconductor.org/packages/release/bioc/html/scDblFinder.html). 
+No doublets were filtered, but the results from `scDblFinder` are present in the filtered and processed objects.
+* Updated cell type annotations
+  * All samples include cell type annotations obtained from [`SCimilarity`](https://genentech.github.io/scimilarity/), in addition to the existing cell type annotations from `SingleR` and `CellAssign`.
+  * Consensus cell types have been updated to incorporate `SCimilarity` results. 
+  If two of the three automated methods agree using an ontology-based approach, a consensus cell type is assigned. 
+  * See our {ref}`documentation on cell type annotation<processing_information:cell type annotation>` for more information on these updated cell types.
+* Cell types were annotated as part of the ongoing [OpenScPCA project](https://openscpca.readthedocs.io) for `SCPCP000004` (Neuroblastoma) and `SCPCP000015` (Ewing sarcoma).
+These cell types are now included in all objects for those samples.  
+  * For more information see our {ref}`documentation on OpenScPCA cell types<processing_information:cell type annotations from the OpenScPCA project>`.
+* CNV inference was performed using [`InferCNV`](https://github.com/broadinstitute/infercnv) on all samples with at least 100 non-malignant reference cells, as identified by the consensus cell types. 
+  * See our {ref}`documentation on CNV inference <processing_information:CNV inference>`
+  * For more information on where to find the inferCNV results in the downloaded objects see {ref}`the single-cell gene expression file contents page<sce_file_contents:singlecellexperiment cell metrics>` and {ref}`the merged object file contents page<merged_objects:singlecellexperiment cell metrics>`.
+
+In addition to these new features, data from the Portal can now be downloaded programmatically using the new [`ScPCAr` package](https://alexslemonade.github.io/ScPCAr). 
+See an example in {ref}`our documentation <download_files:programmatic downloads from the ScPCA Portal>`.
+
 ## 2025.07.25
 
 * Previously, the `cell_id` column in the cell metadata for merged objects was incorrectly formatted.  

docs/download_files.md

@@ -16,9 +16,9 @@ Note that multiplexed sample libraries are only available as `SingleCellExperime
 See the {ref}`FAQ section about samples and libraries <faq:What is the difference between samples and libraries?>` for more information.
 
 The files shown below will be included with each library (example shown for a library with ID `SCPCL000000`):
-- An unfiltered counts file: `SCPCL000000_unfiltered.rds` or `SCPCL00000_unfiltered_rna.h5ad`,
-- A filtered counts file: `SCPCL000000_filtered.rds` or `SCPCL00000_filtered_rna.h5ad`,
-- A processed counts file: `SCPCL000000_processed.rds` or `SCPCL00000_processed_rna.h5ad`,
+- An unfiltered counts file: `SCPCL000000_unfiltered.rds` or `SCPCL000000_unfiltered_rna.h5ad`,
+- A filtered counts file: `SCPCL000000_filtered.rds` or `SCPCL000000_filtered_rna.h5ad`,
+- A processed counts file: `SCPCL000000_processed.rds` or `SCPCL000000_processed_rna.h5ad`,
 - A quality control report: `SCPCL000000_qc.html`,
 - A supplemental cell type report: `SCPCL000000_celltype-report.html`
 
@@ -188,7 +188,7 @@ This file contains the raw and normalized counts data for cell barcodes that hav
 In addition to the counts matrices, the `SingleCellExperiment` or `AnnData` object stored in the file includes the results of dimensionality reduction using both principal component analysis (PCA) and UMAP.
 
 See {ref}`Single-cell gene expression file contents <sce_file_contents:Single-cell gene expression file contents>` for more information about the contents of the `SingleCellExperiment` and `AnnData` objects and the included statistics and metadata.
-See also {ref}`Using the provided RDS files in R <faq:how do i use the provided RDS files in r?>` and {ref}`Using the provided H5AD files in Python <faq:how do i use the provided H5AD files in python?>`.
+See also {ref}`Getting started with an ScPCA dataset <getting_started:Getting started with an scpca dataset>`.
 
 ## QC report
 
@@ -325,3 +325,40 @@ The `SCPCP000000_bulk_quant.tsv` file contains a gene by sample matrix (each row
 The `SCPCP000000_bulk_metadata.tsv` file contains associated metadata for all samples with bulk RNA-seq data.
 This file will contain fields equivalent to those found in the `single-cell_metadata.tsv` related to processing the sample, but will not contain patient or disease specific metadata (e.g. `age`, `sex`, `diagnosis`, `subdiagnosis`, `tissue_location`, or `disease_timing`).
 See also {ref}`processing bulk RNA samples <processing_information:Bulk RNA samples>`.
+
+## Programmatic downloads from the ScPCA Portal
+
+We provide an R package, [`ScPCAr`](https://alexslemonade.github.io/ScPCAr/), to facilitate programmatic access to the ScPCA Portal.
+This package allows you to search for and download data from the ScPCA Portal directly within R.
+
+An example of basic usage of the `ScPCAr` package follows:
+
+```r
+library(ScPCAr)
+
+# First, look at the terms of use
+view_terms()
+
+# Get an authentication token for use with the ScPCA Portal
+auth_token <- get_auth(email = "[email protected]", agree = TRUE)
+
+# Get the sample metadata for a project
+sample_metadata <- get_sample_metadata(project_id = "SCPCP000001")
+
+# Download data for a sample
+# this function returns a vector of the downloaded file paths
+file_paths <- download_sample(
+  sample_id = "SCPCS000001",
+  auth_token = auth_token,
+  destination = "scpca_data",
+  format = "sce"
+)
+
+# select and read in the processed SingleCellExperiment object
+processed_data <- grep("_processed.rds$", file_paths)
+sce <- readRDS(processed_data)
+```
+
+Please see the [package documentation](https://alexslemonade.github.io/ScPCAr/) for more details about installation and usage.
+Source code for the package can be found on [GitHub](https://github.com/AlexsLemonade/ScPCAr).
+Information about working with downloaded files can be found in our {ref}`Getting started with an ScPCA dataset <getting_started:Getting started with an scpca dataset>` guide.

docs/faq.md

@@ -44,11 +44,11 @@ This download includes H5AD files that can be directly read into Python.
 
 _Note: You will need to install the [`AnnData` package](https://anndata.readthedocs.io/en/latest/index.html) to work with the provided files._
 
-To read in the H5AD files you can use the `readh5ad` function from the `AnnData` package.
+To read in the H5AD files you can use the `read_h5ad` function from the `AnnData` package.
 
 ```python
 import anndata
-scpca_sample = anndata.readh5ad(file = "SCPCL000000_processed_rna.h5ad")
+scpca_sample = anndata.read_h5ad(file = "SCPCL000000_processed_rna.h5ad")
 ```
 
 A full description of the contents of the `AnnData` object can be found in the section on {ref}`Components of an AnnData object <sce_file_contents:Components of an anndata object>`.
@@ -69,6 +69,30 @@ There are two types of samples where `AnnData` objects are not available:
     Therefore, we do not currently provide any multiplexed libraries as `AnnData` objects.
     - In addition, providing multiplexed data in this form is not compliant with the standards for [CZI's CELLxGENE](https://cellxgene.cziscience.com), which we have tried to match as closely as possible.
 
+
+## What if I want to use MuData instead of AnnData objects?
+
+[`MuData` objects](https://mudata.readthedocs.io/en/latest/index.html) are Python objects built on top of `AnnData` objects that are specifically used to store multimodal data.
+Currently, we provide RNA counts and ADT counts, if present, as separate `AnnData` objects in their own H5AD files, as described in {ref}`the file contents documentation<sce_file_contents:Additional AnnData components for CITE-seq libraries (with ADT tags)>`.
+However, these objects can be combined into a `MuData` object if desired for a multimodal analysis.
+
+_Note: You will need to install the [`MuData` package](https://mudata.readthedocs.io/en/latest/index.html) to generate and work with `MuData` objects._
+
+```python
+import anndata
+import mudata
+
+# Read individual AnnData files
+rna_object = anndata.read_h5ad(file = "SCPCL000000_processed_rna.h5ad")
+adt_object = anndata.read_h5ad(file = "SCPCL000000_processed_adt.h5ad")
+
+# Combine into a MuData object, using keys "RNA" and "ADT" to distinguish modalities
+mdata_object = mudata.MuData({"RNA": rna_object, "ADT": adt_object})
+```
+
+For more information on working with `AnnData` objects, see {ref}`Getting started with an ScPCA dataset <getting_started:Getting started with an scpca dataset>`.
+
+
 ## What is the difference between samples and libraries?
 
 A sample ID, labeled as `scpca_sample_id` and indicated by the prefix `SCPCS`, represents a unique tissue that was collected from a participant.
@@ -132,12 +156,29 @@ You can find the [function for generating a QC report](https://github.com/AlexsL
 ## Which libraries include cell type annotations?
 
 Most single-cell and single-nuclei RNA-seq libraries available on the portal will have cell type annotations included in the processed `SingleCellExperiment` or `AnnData` object.
-For more information on where to find the cell type annotations, refer to section(s) describing {ref}`SingleCellExperiment file contents <sce_file_contents:singlecellexperiment sample metadata>` and/or {ref}`AnnData file contents <sce_file_contents:anndata cell metrics>`.
+For more information on where to find the cell type annotations, refer to section(s) describing {ref}`SingleCellExperiment file contents <sce_file_contents:singlecellexperiment cell metrics>` and/or {ref}`AnnData file contents <sce_file_contents:anndata cell metrics>`.
 If cell type annotation was performed, a supplemental cell type report (`SCPCL000000_celltype-report.html`) will be included in the download.
 
 Cell type annotation is not performed on samples derived from cell lines.
 This means processed objects will not include cell type annotations, and the download will not include a cell type report.
 
+## Which libraries include CNV inferences?
+
+As with cell type annotation, most single-cell and single-nuclei RNA-seq libraries available on the portal will have {ref}`CNV inferences<processing_information:cnv inference>` in the processed `SingleCellExperiment` or `AnnData` object.
+For more information on where to find these results, refer to sections describing {ref}`SingleCellExperiment cell metrics <sce_file_contents:singlecellexperiment cell metrics>` and {ref}`SingleCellExperiment metadata <sce_file_contents:singlecellexperiment experiment metadata>`, and/or {ref}`AnnData cell metrics <sce_file_contents:anndata cell metrics>` and {ref}`AnnData metadata <sce_file_contents:anndata experiment metadata>`.
+
+There are several circumstances when CNV results are not available:
+
+* CNV inference is not performed on libraries which do not have enough cells to include in a normal reference, as described in the {ref}`CNV inference processing documentation<processing_information:cnv inference>`
+* CNV inference is not performed on libraries derived from cell line samples
+* If `inferCNV` experienced a failure while running, there will not be any associated results in the processed objects
+
+## Where can I find the inferCNV heatmap?
+
+For libraries that underwent CNV inference, the [`inferCNV` heatmap depicting expression across genomic regions](https://github.com/broadinstitute/inferCNV/wiki/Interpreting-the-figure) is embedded in the final QC report.
+You can directly copy the figure from the QC report file for use in other contexts.
+
+
 ## What if I want to use Seurat instead of Bioconductor?
 
 The RDS files available for download contain [`SingleCellExperiment` objects](http://bioconductor.org/books/3.13/OSCA.intro/the-singlecellexperiment-class.html).
@@ -249,8 +290,14 @@ Download links expire in 7 days, but you can generate a new link on the ScPCA Po
 
 Download links are only available for projects (i.e., not for downloading individual samples).
 
+## Can I download data from the Portal programmatically?
 ## Why can't I change the data format in My Dataset?
 
+We provide an R package, [`ScPCAr`](https://alexslemonade.github.io/ScPCAr/), to facilitate programmatic access to the ScPCA Portal.
+This package allows you to search for and download data from the ScPCA Portal directly within R.
+Please see the [package documentation](https://alexslemonade.github.io/ScPCAr/) for more details about installation and usage.
+Source code for the package can be found on [GitHub](https://github.com/AlexsLemonade/ScPCAr).
+
 When creating a {ref}`custom dataset for download<download_files:Custom datasets>` (`My Dataset`), all single-cell sample or project data included must be of the same {ref}`data format<download_options:Data format>`, either `SingleCellExperiment` for use in R or `AnnData` for use in Python.
 We currently do not support including both data formats at once in `My Dataset`.
 Once a sample or project of a given data format has been added to `My Dataset`, all subsequent single-cell or single-nuclei data added will automatically be in that same format.

docs/getting_started.md

@@ -408,7 +408,7 @@ subsetted_adata_merged_object = adata_merged_object[adata_merged_object.obs["lib
 ```
 
 The merged object additionally contains metadata such as information about sample diagnosis, subdiagnosis, or tissue location that may be useful for subsetting.
-A full set of merged object contents which can support subsetting {ref}`is available here<merged_objects:SingleCellExperiment cell metrics>`.
+A full set of merged object contents which can support subsetting is available in {ref}`this documentation<merged_objects:merged objects>`.
 
 As one example, to subset a `SingleCellExperiment` merged object to a given diagnosis, use the following R code:
 
@@ -489,7 +489,7 @@ Be aware that the processed objects have been filtered to remove low-quality cel
 ### Filtering cells based on ADT quality control
 
 The `adt_scpca_filter` column indicates which cells should be removed before proceeding with downstream analyses of the ADT data, as determined by [`DropletUtils::CleanTagCounts()`](https://rdrr.io/github/MarioniLab/DropletUtils/man/cleanTagCounts.html).
-This process identified cells with high levels of ambient contamination and/or high levels of negative control ADTs (if available).
+This process identified low-quality cells as those with either very low or high levels of ambient contamination and/or negative control ADTs (if available).
 Cells are labeled either as `"Keep"` (cells to retain) or `"Remove"` (cells to filter out).
 
 To filter cells based on this column in the `SingleCellExperiment` objects, use the following command:
@@ -520,6 +520,23 @@ Here are some additional resources that can be used for working with ADT counts
 - [Integrating with Protein Abundance, Orchestrating Single Cell Analysis](http://bioconductor.org/books/3.15/OSCA.advanced/integrating-with-protein-abundance.html)
 - [Seurat vignette on using with multimodal data](https://satijalab.org/seurat/articles/multimodal_vignette.html)
 
+### Using MuData objects for multimodal analysis
+
+It is also possible to combine the given RNA and ADT `AnnData` objects into a single [`MuData` object](https://mudata.readthedocs.io/en/latest/index.html) for multimodal analysis, as shown below.
+
+_Note: You will need to install the [`MuData` package](https://mudata.readthedocs.io/en/latest/index.html) to generate and work with `MuData` objects._
+
+```python
+import anndata
+import mudata
+
+# Read individual AnnData files
+rna_object = anndata.read_h5ad(file = "SCPCL000000_processed_rna.h5ad")
+adt_object = anndata.read_h5ad(file = "SCPCL000000_processed_adt.h5ad")
+
+# Combine into a MuData object, using keys "RNA" and "ADT" to distinguish modalities
+mdata_object = mudata.MuData({"RNA": rna_object, "ADT": adt_object})
+```
 
 ## Special considerations for multiplexed samples