Skip to contents

scSimEval: Unified Benchmarking and Accuracy Evaluation for Single-Cell Multiomics Simulation Methods

License: MIT Bioconductor submission

scSimEval is a unified evaluation and benchmarking toolkit designed to assess how realistically synthetic data generated by single-cell simulation methods match empirical experiments.

The package is primarily optimized for:

  • scRNA-seq (Single-Cell RNA Sequencing count matrices)
  • scATAC-seq (Single-Cell ATAC Sequencing peak accessibility matrices)
  • Paired Multiomics (Simultaneous scRNA-seq + scATAC-seq across identical cells, e.g. 10x Multiome, SHARE-seq)
  • Unpaired Multiomics (Independent scRNA-seq and scATAC-seq profiled from separate cells of the same tissue)
  • Mosaic Multiomics (Partially overlapping cellular cohorts or modality subsets)

Data Modalities & Extensibility: While scSimEval is primarily optimized and demonstrated for scRNA-seq, scATAC-seq, and multiomics, all evaluation measures operate on standard numeric count or continuous matrices. Consequently, the benchmarking framework can be readily extended to other single-cell data modalities, including CITE-seq surface protein counts, single-cell DNA methylation levels, and spatial transcriptomics.

Key Highlights

  • Zero External Ground Truth Required: Operates directly on the empirical reference matrices and simulated matrices you already have.
  • 62 Evaluation Measures Across 8 Canonical Categories: Comprehensively evaluates distributions, zero counts, cell clustering, batch mixing, marker genes, developmental lineages, cross-modal coupling, and runtime/RAM scalability.
  • Support for Paired, Unpaired & Mosaic Multiomics: Seamlessly switches between full cell-level coupling metrics (FOSCTTM, ) for paired co-assays and population-level concordance metrics (label transfer accuracy/F1, module co-expression) for unpaired or mosaic profiles.
  • Low-Dimensional Cell Embeddings (UMAP, t-SNE, PCA): Multi-dataset non-linear coordinate projections comparing reference and simulated cells, accompanied by an interactive table of quantitative quality metrics (Mean Silhouette width, ARI cluster fidelity, mean library size, mean detected features, and library size discrepancy %).
  • Automated Two-Step Score Normalization: Automatically inverts lower-is-better metrics (error, distance, runtime) and scales all criteria from 0.000.00 to 1.001.00 for fair comparison.
  • High-Resolution Visualizations (600 DPI): Multi-dimensional bubble matrices, executive summary horizontal bar charts, 14-panel distribution QC plots, raw score heatmaps, PCA biplots, MDS ordinations, and cell embedding layouts.
  • Embedded Interactive Shiny Studio GUI: Standalone interactive graphical interface to explore benchmark results, run new evaluations, and export 600 DPI figures and multi-sheet Excel workbooks without coding.

Installation

You can install scSimEval directly from GitHub:

# Install devtools if not already installed
if (!requireNamespace("devtools", quietly = TRUE)) {
  install.packages("devtools")
}

# Install scSimEval from GitHub
devtools::install_github("kabilanbio/scSimEval")

To view the included vignette during installation:

devtools::install_github("kabilanbio/scSimEval", build_vignettes = TRUE)

Workflow Architecture

scSimEval Workflow Architecture

The evaluation process follows four simple steps:

Step 1: Input Real and Simulated Data

Provide your real experimental reference and simulated count matrices (ref_rna, sim_rna, ref_atac, sim_atac), along with cell type labels (cell_types), batch labels (batch_info), and computer resource records (runtime in seconds and peak RAM in MiB).

Step 2: Calculate Evaluation Measures

scSimEval calculates 62 evaluation measures across 8 core categories without needing any artificial ground truth.

Step 3: Run the Complete Benchmark

Execute the complete evaluation in a single command using evaluate_simulation_accuracy(), evaluate_multiomics_accuracy(), or evaluate_multiple_datasets().

Step 4: Review Scores, Rankings, and Figures

Review clean summary tables, standardized scores (0.000.00 to 1.001.00), simulator ranking leaderboards, and high-resolution figures.

The 8 Evaluation Categories

scSimEval organizes 62 evaluation measures into eight foundational categories:

  1. Category 1: Distribution Properties (14 metrics) — Compares single-cell count distributions (such as library size, average expression, gene variance, and zero fraction) and data manifolds between real and simulated data using statistical distance measures (Kolmogorov-Smirnov, Wasserstein, Median Absolute Deviation, RMSE, Density Overlap, Bhattacharyya distance, ECDF area, Runs test, Maximum Mean Discrepancy, and Fréchet distance).

  2. Category 2: Correlations & Zero-Inflation (6 metrics) — Checks biological variation (BCV), logistic dropout curves (midpoint x0x_0, slope kk, R2R^2), cell-to-cell correlations, and gene co-expression networks.

  3. Category 3: Cellular Structure & Concordance (10 metrics) — Evaluates whether simulated data preserves distinct cell types in unsupervised clustering (Silhouette width, Dunn index, Davies-Bouldin, Calinski-Harabasz) and supervised group matching (Adjusted Rand Index, Normalized Mutual Information, Adjusted Mutual Information, VV-measure, Neighborhood Purity, and Hungarian optimal matching precision).

  4. Category 4: Batch Effects & Technical Confounders (7 metrics) — Tests whether technical batch differences are well-mixed without erasing true biological differences (Batch ASW, batch Shannon entropy, Principal Component Regression Rbatch2R^2_{\text{batch}}, CellMixS CMS, LISI, Seurat mixing metric, and local density factor discrepancy ldfDiff).

  5. Category 5: Marker Genes and Biological Signals (15 metrics) — Assesses differentially expressed gene (DEG) fidelity across three established benchmarking frameworks: Simpipe, SimBench, and Shaky Foundations (effect size errors, fold-change correlations, top-marker overlap, and cell classification accuracy).

  6. Category 6: Trajectory and Lineage Dynamics (2 metrics) — Evaluates developmental timelines (pseudotime rank correlation) and cell lineage tree branching (branch height RMSE), inferred directly from scRNA-seq counts.

  7. Category 7: Cross-Modal Coupling and Modularity (6 metrics) — Evaluates the connection between chromatin accessibility and gene expression in multiomics data, including cell-to-cell matching (FOSCTTM, ), cross-modal cell type transfer, and peak-to-gene regulatory links (RVR_V coefficient).

  8. Category 8: Computer Resources and Scalability (2 metrics) — Measures total execution time in seconds and peak computer memory usage in MiB.

Metric Measurement Capacity by Data & Metadata Mode

scSimEval dynamically scales its evaluation depth based on the experimental data layers and metadata files you provide:

Experiment Mode Uploaded Inputs Metrics Measured Active Evaluation Categories & Scope
Single-Cell (scRNA / scATAC) Count Matrix alone (no metadata) ~23 Metrics Cat I (Distributional Properties, 14 metrics), Cat II (Correlations & Zeroes, 6 metrics), Cat VI (Trajectory, 2 metrics), Cat VIII (Scalability, 2 metrics). Evaluates basic statistical distributions, library sizes, and sparsity curves.
Single-Cell (scRNA / scATAC) Matrix + Cell Types ~35–38 Metrics Adds +12–15 metrics: Unlocks Cat III (Cellular Structure & Clustering / ARI, NMI, AMI, Silhouette) and Cat V (Biological Signal / SimBench DE, DP, DV, top DEG Jaccard overlap).
Single-Cell (scRNA / scATAC) Matrix + Batch Info ~30 Metrics Adds +7 metrics: Unlocks Cat IV (Batch Effects & Confounder Mixing / Batch Silhouette, Shannon Entropy, PCR R2R^2, CMS, ISI, Seurat mixing).
Single-Cell (scRNA / scATAC) Full Metadata (Matrix + Cell Types + Batch) Up to ~55 Metrics Unlocks the complete unimodal suite across Categories I, II, III, IV, V, VI, and VIII.
Paired Multiomics (scRNA + scATAC) RNA + ATAC Matrices alone ~48 Metrics Unimodal distributions for both modalities (~40 metrics) PLUS Cat VII (Cross-Modal Coupling: FOSCTTM cell alignment, , cross-modal generation, RVR_V coefficient, peak-to-gene linkage).
Paired Multiomics (scRNA + scATAC) Full Metadata (RNA + ATAC + Cell Types + Batch) All 62 Metrics Complete multiomics benchmark suite across all 8 canonical categories (Categories I–VIII).
Unpaired Multiomics (Independent cells) RNA + ATAC Matrices alone ~38–42 Metrics Both unimodal distributions plus population-level coupling (co-expression module fidelity, Network Jaccard, cross-modality correlation).
Unpaired Multiomics (Independent cells) RNA + ATAC + Cell Types (+ Batch) Up to ~58 Metrics Essential for unpaired designs: Unlocks cluster-matched cross-modal label transfer concordance, cell-type regulatory linkage, and technical donor integration.

Quick Start Examples

1. Unimodal Evaluation (evaluate_simulation_accuracy) 

library(scSimEval)

# Load packaged example scRNA-seq dataset
data(example_scrna)

# Evaluate unimodal accuracy on scRNA-seq
unimodal_res <- evaluate_simulation_accuracy(
  ref_data = example_scrna$ref,
  sim_data = example_scrna$sim,
  compute_bivariate = FALSE,
  verbose = FALSE
)

# Inspect library size statistical distance metrics
head(subset(unimodal_res$metrics_summary_table, Property == "library_size"))

2. Multiomics Benchmarking Pipeline (evaluate_multiomics_accuracy) 

# Load packaged multiomics dataset (paired RNA + ATAC)
data(example_multiomics)

# Execute master benchmark across all categories
master_eval <- evaluate_multiomics_accuracy(
  ref_multi = example_multiomics$ref_multi,
  sim_multi = example_multiomics$sim_multi,
  cell_types = example_multiomics$cell_types,
  batch_info = example_multiomics$batch_info,
  cpu_time = example_multiomics$resource_stats$cpu_time,
  memory_mb = example_multiomics$resource_stats$memory_mb,
  verbose = FALSE
)

# View tidy master summary table
head(master_eval$benchmark_summary_table)

3. Multi-Dataset / Multi-Method Consolidated Benchmarking (evaluate_multiple_datasets) 

# Benchmark an arbitrary collection of datasets simultaneously
dataset_collection <- list(
  "Method 1-scRNA-seq" = list(ref = example_scrna$ref, sim = example_scrna$sim),
  "Method 1-scATAC-seq" = list(ref = example_scatac$ref, sim = example_scatac$sim),
  "Method 2-scRNA-seq" = list(ref = example_scrna$ref, sim = example_scrna$sim),
  "Method 2-scATAC-seq" = list(ref = example_scatac$ref, sim = example_scatac$sim)
)

consolidated <- evaluate_multiple_datasets(
  datasets = dataset_collection,
  pair_by_prefix = TRUE,
  compute_bivariate = FALSE,
  verbose = FALSE
)

# View dataset overview table
print(consolidated$dataset_overview)

4. Large-Scale Multi-Dataset Benchmark on HPC (benchmark_simulation_suite)

For benchmarking studies evaluating multiple simulators across multiple independent single-cell datasets (e.g. 16 diverse cell types or tissues × 6 simulators = 96 evaluation runs):

# 1. Define named list of reference datasets (supports matrices or RDS file paths)
references <- list(
  "PBMC_10k"       = pbmc_ref,
  "Pancreas_Baron" = pancreas_ref,
  "Brain_Cortex"   = cortex_ref
  # ... up to 16 datasets
)

# 2. Define named list of simulators and their simulated matrices per dataset
simulators <- list(
  "Splatter"  = list("PBMC_10k" = splat_pbmc, "Pancreas_Baron" = splat_panc, "Brain_Cortex" = splat_cortex),
  "scDesign3" = list("PBMC_10k" = scd3_pbmc,  "Pancreas_Baron" = scd3_panc,  "Brain_Cortex" = scd3_cortex),
  "SCRIP"     = list("PBMC_10k" = scrip_pbmc, "Pancreas_Baron" = scrip_panc, "Brain_Cortex" = scrip_cortex),
  "SymSim"    = list("PBMC_10k" = sym_pbmc,   "Pancreas_Baron" = sym_panc,   "Brain_Cortex" = sym_cortex),
  "dyngen"    = list("PBMC_10k" = dyn_pbmc,   "Pancreas_Baron" = dyn_panc,   "Brain_Cortex" = dyn_cortex),
  "simATAC"   = list("PBMC_10k" = sima_pbmc,  "Pancreas_Baron" = sima_panc,  "Brain_Cortex" = sima_cortex)
)

# 3. Run high-performance evaluation with automatic sparse matrix coercion (dgCMatrix),
# multi-threading, and proactive memory management for HPC clusters
suite_res <- benchmark_simulation_suite(
  datasets    = references,
  simulators  = simulators,
  threads     = 8,        # multi-core HPC parallelism
  auto_sparse = TRUE,     # dgCMatrix minimizes RAM consumption
  run_gc      = TRUE,     # gc() after each run prevents memory bloat
  save_path   = "multi_dataset_benchmark.rds" # persist for Shiny Studio
)

# 4. Plot comparative bubble matrix:
# Automatically averages metrics across all 16 datasets under each simulator!
# Only simulator names appear on the axis; no cluttered dataset axes.
plot_benchmark_bubble_matrix(suite_res)

In the interactive scSimEval Studio (launch_scSimEval_app()), users can navigate to Data Hub, choose Mode 1 (Pre-Computed Benchmark) or Mode 4 (Upload Saved Results) to instantly load multi-dataset studies or upload HPC-generated RDS files.

Score Normalization in Benchmark Bubble Plots

Score Normalization and Visual Mapping Workflow

To ensure fair comparison across metrics with different units and directions:

  1. Direction Inversion (Lower-is-Better Metrics): Metrics where smaller numbers denote superior fidelity (such as KS distance, Wasserstein distance, RMSE, and runtime) are inverted: Inverted Value=Maximum−Value\text{Inverted Value} = \text{Maximum} - \text{Value} This ensures that larger values always denote superior quality across every measure.

  2. Min-Max Scaling (0.000.00 to 1.001.00): All metrics are rescaled to a standard range: Standardized Score=Value−MinimumMaximum−Minimum\text{Standardized Score} = \frac{\text{Value} - \text{Minimum}}{\text{Maximum} - \text{Minimum}}

  3. Visual Mapping Rules:

    • Bubble Size: Directly proportional to the standardized score (0.000.00 to 1.001.00). Larger bubbles indicate higher simulation quality.
    • Glyph Shape: Top-performing tools (≥0.96\ge 0.96) are highlighted with bold squares, while standard scores are shown as circles.
    • Category Colors: Distinct color for each evaluation category.
    • Leaderboard Ranking: Simulation methods are ordered top-to-bottom on the vertical axis by composite average score.

Visualization Suite (600 DPI)

scSimEval provides built-in plotting functions rendering high-resolution figures at 600 DPI:

  • Flagship Bubble Matrix (plot_benchmark_bubble_matrix): Multi-tier bubble matrix across the 8 evaluation categories.
  • Evaluation Summary Bars (plot_evaluation_summary): Horizontal bar matrix ranking simulators with multi-panel category scores.
  • Comparative Distribution QC (plot_distribution_qc): 14-panel density and bivariate distribution comparison layout.
  • Cell Embeddings Suite (compute_dataset_embeddings, plot_dataset_embeddings, compute_embedding_quality_metrics): Publication-ready UMAP, t-SNE, and PCA projections comparing reference and simulated single cells across multi-simulator grid or 1-to-1 layouts, accompanied by quantitative Silhouette, ARI, and library size quality metrics.
  • Metric Comparison Boxplots (plot_metric_boxplots): Standardized or raw score boxplots with jittered data points.
  • Computational Scalability Suite (plot_scalability_benchmark): 6-panel scalability dashboard (runtime, RAM, Pareto frontier, throughput).
  • Metric Heatmap (plot_metric_heatmap): Method-by-metric grid displaying exact unnormalized raw scores in bold text.
  • PCA Biplot & MDS Ordination (plot_metric_pca, plot_metric_mds): Dimension reduction of simulator metric profiles.
# 1. Master benchmark bubble matrix
plot_benchmark_bubble_matrix(
  data = consolidated,
  method_classes = list("Multiomics" = c("Method 1", "Method 2")),
  title = "Benchmark Overview: Single-Cell Simulation Fidelity"
)

# 2. Evaluation summary horizontal bar matrix
plot_evaluation_summary(data = consolidated)

# 3. 14-panel distribution QC plot
plot_distribution_qc(ref_data = example_scrna$ref, sim_data = example_scrna$sim)

# 4. Multi-dataset cell embeddings (UMAP / PCA) & quality metrics
emb <- compute_dataset_embeddings(
  reference = example_scrna$ref,
  simulated = list("Splatter" = example_scrna$sim),
  reduction = "umap",
  cell_types = example_scrna$cell_types
)
plot_dataset_embeddings(emb, reduction = "umap", layout = "facet", color_by = "cell_type")
qm <- compute_embedding_quality_metrics(emb)
print(qm)

# 5. Computational scalability dashboard
plot_scalability_benchmark(scalability_data = consolidated, mode = "composite", panels = "6panel")

Interactive Graphical User Interface (Shiny App Studio)

For researchers who prefer an interactive visual interface rather than writing code, scSimEval includes an embedded Interactive Shiny App Studio (scSimEvalApp). This standalone web application gives researchers direct, point-and-click access to all 62 evaluation measures, multi-dataset benchmark studies, live dataset structural summaries, diagnostic visualization suites, and reproducible standalone R script generators:

library(scSimEval)

# Launch the interactive Shiny app in your default web browser
launch_scSimEval_app()

The 7 Modular Application Tabs:

  • Tab 1: Home — Executive overview of the 62 evaluation measures, biological KPI dimensions, and an interactive operational tour across the four ingestion modes (with all-time pageview tracking in the cloud app).
  • Tab 2: Data Hub — Comprehensive data ingestion centre featuring 4 operational modes:
    1. Mode 1: Pre-Computed Benchmark (Demo Data): Instant 1-click loading of a multi-dataset benchmark study across 6 simulators (744 metric observations) for zero-wait exploration (contrasting with the lightweight data(example_*) matrices provided for quick console testing).
    2. Mode 2: Single-Cell RNA/ATAC Evaluation: Evaluates empirical reference against simulated count matrices with optional cell-type and batch metadata.
    3. Mode 3: Multi-Omics Co-Assays: Evaluates paired or unpaired multiomics simulations assessing cross-modal coupling and feature alignment.
    4. Mode 4: Upload Pre-Computed Benchmark Table: Instant loading of previously calculated benchmark CSV or RDS files. Includes an interactive matrix inspector (Subtab 2), a live Dataset Properties Summary QC table (cells, features, sparsity %, library sizes), and an automated HPC Batch Script Generator (SLURM / PBS / Bash) for cluster execution.
  • Tab 3: Comparative Bubble Matrix & Leaderboard — Flagship 62-metric comparative matrix with a multi-dataset selector bar (All Datasets (Average) vs. individual datasets Dataset 1, Dataset 2, …), Minimalist vs. Publication layout toggles, confirmation plot buttons, 600 DPI figure downloads, and Download Standalone R Script for reproducible analysis workflows outside Shiny. Subtab 2 provides visual simulator ranking podiums, composite fidelity rankings, and custom category weight sliders.
  • Tab 4: Visualizations — 8 dedicated diagnostic panels equipped with global multi-dataset selectors, active dataset badges, confirmation controls, and workflow R script downloads:
    1. Evaluation Summary: Category horizontal bar ranking and composite metrics.
    2. Distribution QC: Comparative ECDF curves, KS distances, and gene/cell mean-variance trends.
    3. Scalability Benchmark: Runtime (seconds) vs. Peak RAM (MiB), resource cost index, and cell throughput rate.
    4. Metric Plots: Category and metric score boxplots/barplots with Raw vs. Normalized view toggles.
    5. Metric Heatmap: Unsupervised hierarchical clustering of simulators and metrics with exact numbers in bold.
    6. PCA Ordination: Principal component biplots projecting simulators in metric space.
    7. MDS Metric Space: Multidimensional scaling projection of simulator algorithmic proximity.
    8. Cell Embeddings: Dimension reduction projections (UMAP, t-SNE, PCA) comparing reference cells to simulated cells, with quantitative cluster fidelity tables.
  • Tab 5: Benchmark Results & Export — Master Benchmark Scores Table featuring per-column dropdown filtering, explicit dual-value reporting (Score_Raw in native statistical units, runtime in seconds, peak RAM in MiB, and normalized Score in [0, 1]), alongside exports for Excel (.xlsx), CSV, TSV, RDS, and full ZIP archives.
  • Tab 6: Help & Getting Started — Comprehensive user manual, formula glossary, metric measurement capacity reference table, hardware sizing guidelines, and input file templates.
  • Tab 7: Contact — ICAR-IASRI institutional details, author profiles, and citation information.

Documentation

Complete documentation of scSimEval R package is available at https://kabilanbio.github.io/scSimEval/.

Authors & Maintainers

scSimEval is developed and maintained by:

  • Kabilan Sakthivel (Ph.D Student and Maintainer, ICAR-IASRI) — kabilan151414@gmail.com

  • Dr. Dwijesh Chandra Mishra (Principal Scientist, Author & Thesis Guide, ICAR-IASRI) — dwij.mishra@gmail.com

  • Dr. Shashi Bhushan Lal (Author, ICAR-IASRI) — sblall16@gmail.com

  • Dr. Sudhir Srivastava (Author, ICAR-IASRI) — sudhir0401bm@gmail.com

  • Dr. Krishna Kumar Chaturvedi (Author, ICAR-IASRI) — kkcchaturvedi@gmail.com

  • Dr. Sharanbasappa (Contributor, ICAR-IASRI) — smadival509@gmail.com

  • GitHub Repository: https://github.com/kabilanbio/scSimEval

  • Bug Reports & Issues: https://github.com/kabilanbio/scSimEval/issues

License

MIT License. Copyright (c) 2026 Kabilan Sakthivel and scSimEval Authors.