Comprehensive Catalog of the 62 Evaluation Metrics

Overview of the Evaluation Architecture

scSimEval evaluates single-cell simulation methods using an objective, ground-truth-free framework. It includes 62 distinct evaluation measures organized into 8 core biological and computational categories.

Every metric operates directly on the real biological reference data and the synthetic data generated by single-cell and multiomics simulators (scRNA-seq, scATAC-seq, and paired/unpaired co-assays). All 62 metrics in this catalog match identically with the metric identifiers and labels evaluated within the package functions, exported tables, and the interactive scSimEval Studio Shiny application.

---

Summary Table of the 8 Categories

Category	Domain	Number of Metrics	Primary Focus
I	Distributional Properties	14	Statistical distance between real and simulated single-cell count distributions and manifolds
II	Correlations & Zero-Inflation	6	Biological overdispersion (BCV), dropout curves, cell correlations, and gene co-expression
III	Cellular Structure & Concordance	10	Unsupervised cluster separation (ASW, Dunn, DB, CH) and supervised cell type preservation (ARI, NMI, AMI, V-measure, purity, precision)
IV	Batch Effects & Confounder Mixing	7	Technical batch mixing (CellMixS, LISI, Shannon entropy, PCR $R^2$, Seurat mixing, batch ASW, and ldfDiff)
V	Biological Signal & Downstream Fidelity	15	Differentially expressed gene (DEG) preservation, fold-change correlation, null uniformity, classifier accuracy, and Shaky Foundations variance
VI	Trajectory & Lineage Dynamics	2	Inferred developmental pseudotime rank correlation and cell lineage branching tree topology
VII	Cross-Modal Coupling & Modularity	6	Paired and joint ATAC-RNA regulatory links, FOSCTTM, Match@1, label transfer, and $R_V$ coefficient
VIII	Computational Scalability	2	Execution runtime in seconds and peak RAM memory allocation in MiB
Total		62	Complete single-cell and multiomics simulation evaluation suite

---

Complete Master Reference Table (62 Metrics)

Below is the definitive cross-reference table mapping each metric’s internal identifier, user-facing display label in the Shiny app and figures, category, evaluation direction, and scope:

#	Metric ID	Shiny App Display Label	Category	Direction	Description
1	KS	KS Distance	(I) Distributional Properties	Lower is better	Kolmogorov-Smirnov supremum distance between cumulative distributions
2	Wasserstein	Wasserstein Dist.	(I) Distributional Properties	Lower is better	1D Earth Mover’s Distance measuring work required to transform distributions
3	MAD	MAD	(I) Distributional Properties	Lower is better	Median Absolute Deviation discrepancy across cell and gene properties
4	MAE	MAE	(I) Distributional Properties	Lower is better	Mean Absolute Error across summary statistics
5	RMSE	RMSE	(I) Distributional Properties	Lower is better	Root Mean Squared Error penalizing large deviations in moment statistics
6	OV	Overlap Coeff.	(I) Distributional Properties	Higher is better	Kernel density curve area intersection between reference and simulated distributions
7	Bhattacharyya	Bhattacharyya Dist.	(I) Distributional Properties	Lower is better	Statistical divergence between discrete count probability distributions
8	ECDF_DiffArea	ECDF Diff. Area	(I) Distributional Properties	Lower is better	Integrated area difference between empirical cumulative distribution functions
9	Runs_Statistic	Runs Statistic	(I) Distributional Properties	Lower is better	Wald-Wolfowitz runs test absolute $z$-score testing random interleaving
10	Fasano_Franceschini_2D_KS	Fasano-Franceschini 2D	(I) Distributional Properties	Lower is better	2D extension of the KS statistic for bivariate feature distributions
11	Peacock_2D_KS	Peacock 2D	(I) Distributional Properties	Lower is better	2D Peacock bivariate distribution test statistic
12	KDE_Bivariate_zstat	2D KDE z-stat	(I) Distributional Properties	Lower is better	2D Kernel Density Estimation divergence $z$-statistic between joint distributions
13	MMD	MMD	(I) Distributional Properties	Lower is better	Maximum Mean Discrepancy testing multivariate single-cell manifolds in RKHS
14	Frechet_SingleCell_Distance	Frechet SC Dist.	(I) Distributional Properties	Lower is better	Fréchet single-cell distance between multivariate Gaussian manifolds in PC space
15	bcv_discrepancy	Biological CV (BCV)	(II) Correlations & Zero-Inflation	Lower is better	Biological Coefficient of Variation discrepancy after Poisson noise subtraction
16	dropout_curve_midpoint_x0	Dropout Midpoint (x0)	(II) Correlations & Zero-Inflation	Lower is better	Logistic dropout curve midpoint ($x_0$) estimation error
17	dropout_curve_slope_k	Dropout Slope (k)	(II) Correlations & Zero-Inflation	Lower is better	Logistic dropout curve slope ($k$) steepness error
18	mean_variance_r2	Mean-Variance R2	(II) Correlations & Zero-Inflation	Higher is better	Mean-variance trend fit ($R^2$) and dropout model goodness-of-fit
19	cell_pearson_cor	Cell Pearson Corr.	(II) Correlations & Zero-Inflation	Higher is better	Cell-to-cell Pearson correlation matrix concordance
20	feature_pearson_cor	Feature Pearson Corr.	(II) Correlations & Zero-Inflation	Higher is better	Gene-to-gene Pearson co-expression network correlation concordance
21	silhouette_sim	Silhouette ASW	(III) Cellular Structure & Concordance	Higher is better	Average Silhouette Width (ASW) cluster separation fidelity
22	dunn_sim	Dunn Index	(III) Cellular Structure & Concordance	Higher is better	Dunn index measuring ratio of inter-cluster separation to cluster diameter
23	davies_bouldin_sim	Davies-Bouldin	(III) Cellular Structure & Concordance	Lower is better	Davies-Bouldin compactness and separation discrepancy
24	calinski_harabasz_sim	Calinski-Harabasz	(III) Cellular Structure & Concordance	Higher is better	Calinski-Harabasz variance ratio criterion between and within clusters
25	ari	ARI	(III) Cellular Structure & Concordance	Higher is better	Adjusted Rand Index comparing unsupervised clusters against cell type labels
26	nmi	NMI	(III) Cellular Structure & Concordance	Higher is better	Normalized Mutual Information between unsupervised clusters and known cell types
27	ami	AMI	(III) Cellular Structure & Concordance	Higher is better	Adjusted Mutual Information adjusted for chance cluster overlap
28	v_measure	V-Measure	(III) Cellular Structure & Concordance	Higher is better	Harmonic mean of cluster homogeneity and completeness
29	neighborhood_purity	Neighborhood Purity	(III) Cellular Structure & Concordance	Higher is better	Proportion of $k$-nearest neighbors sharing identical annotated cell type labels
30	Generative_Precision	Generative Precision	(III) Cellular Structure & Concordance	Higher is better	Hungarian optimal matching accuracy between simulated and real cell partitions
31	batch_silhouette	Batch Silhouette	(IV) Batch Effects & Confounder Mixing	Higher is better	Average Silhouette Width for batch confounders (lower batch clustering is better)
32	shannon_entropy	Shannon Entropy	(IV) Batch Effects & Confounder Mixing	Higher is better	Batch Shannon diversity entropy across local cell neighborhoods
33	pcr_r2	PC Regression R2	(IV) Batch Effects & Confounder Mixing	Lower is better	Principal Component Regression variance explained by technical batch labels ($R^2_{\text{batch}}$)
34	cms	CMS Score	(IV) Batch Effects & Confounder Mixing	Higher is better	CellMixS cell-specific mixing score testing random batch distributions
35	isi	LISI / ISI	(IV) Batch Effects & Confounder Mixing	Higher is better	Local Inverse Simpson Index measuring effective batches per neighborhood
36	seurat_mixing_metric	Seurat Mixing	(IV) Batch Effects & Confounder Mixing	Lower is better	Seurat batch mixing metric measuring average rank of same-batch neighbors
37	ldf_diff	ldfDiff	(IV) Batch Effects & Confounder Mixing	Lower is better	Local Density Factor discrepancy measuring manifold density distortion
38	SimBench_SMAPE	SimBench SMAPE	(V) Biological Signal & Downstream Fidelity	Lower is better	Symmetric Mean Absolute Percentage Error in differentially expressed log2FC
39	SimBench_DE_Fidelity_Score	DE Fidelity (1-SMAPE)	(V) Biological Signal & Downstream Fidelity	Higher is better	SimBench composite differential expression fidelity score ($1 - \text{SMAPE}$)
40	Log2FC_Pearson_Corr	Log2FC Pearson Corr	(V) Biological Signal & Downstream Fidelity	Higher is better	Pearson correlation of estimated log2 fold-changes across marker genes
41	Log2FC_Spearman_Corr	Log2FC Spearman Corr	(V) Biological Signal & Downstream Fidelity	Higher is better	Spearman rank correlation of estimated log2 fold-changes
42	Top_DEG_Jaccard_Overlap	Top DEG Jaccard	(V) Biological Signal & Downstream Fidelity	Higher is better	Jaccard index overlap of top differentially expressed genes between real and sim
43	DEG_Ratio	DEG Ratio (Sim/Real)	(V) Biological Signal & Downstream Fidelity	Matches 1.0	Ratio of identified DEG count in simulated data vs real reference data
44	PValue_Uniformity_Chisq	P-Val Unif. (Chisq)	(V) Biological Signal & Downstream Fidelity	Lower is better	$\chi^2$ divergence test verifying uniform p-value distribution under the null hypothesis
45	Distribution_Score	Distribution Score	(V) Biological Signal & Downstream Fidelity	Higher is better	Composite null p-value distribution preservation score without false positives
46	Classifier_Accuracy	Classifier Accuracy	(V) Biological Signal & Downstream Fidelity	Higher is better	Random forest / SVM cell type classification accuracy on marker transcripts
47	Classifier_Macro_F1	Classifier Macro-F1	(V) Biological Signal & Downstream Fidelity	Higher is better	Macro-averaged F1 classification score across rare and common cell populations
48	Classifier_Macro_Recall	Classifier Macro-Recall	(V) Biological Signal & Downstream Fidelity	Higher is better	Macro-averaged recall score across biological cell identities
49	Silhouette_Sim	Group Silhouette	(V) Biological Signal & Downstream Fidelity	Higher is better	Shaky Foundations cell-type silhouette similarity in DEG marker space
50	Silhouette_Discrepancy	Silhouette Discrepancy	(V) Biological Signal & Downstream Fidelity	Lower is better	Shaky Foundations absolute silhouette width discrepancy relative to reference
51	PVE_Group_Sim	Group PVE	(V) Biological Signal & Downstream Fidelity	Higher is better	Shaky Foundations Proportion of Variance Explained (PVE) by cell groups
52	PVE_Discrepancy	Group PVE Discrepancy	(V) Biological Signal & Downstream Fidelity	Lower is better	Discrepancy in proportion of variance explained by biological condition
53	pseudotime_correlation	Pseudotime Corr. (rho)	(VI) Trajectory & Lineage Dynamics	Higher is better	Spearman rank correlation of inferred developmental pseudotime ordering
54	tree_height_rmse	Lineage Tree RMSE	(VI) Trajectory & Lineage Dynamics	Lower is better	Root Mean Squared Error of lineage branching hierarchy tree heights
55	cross_modal_accuracy	Cross-Modal Accuracy	(VII) Cross-Modal Coupling & Modularity	Higher is better	Cross-modal cell type label transfer accuracy (RNA $\rightarrow$ ATAC)
56	cross_modal_F1	Cross-Modal F1	(VII) Cross-Modal Coupling & Modularity	Higher is better	Macro-averaged F1 score of cross-modal cell type label transfer
57	FOSCTTM	FOSCTTM	(VII) Cross-Modal Coupling & Modularity	Lower is better	Fraction of Samples Closer Than The True Match in joint multi-omics space
58	match_at_1	Match@1	(VII) Cross-Modal Coupling & Modularity	Higher is better	Top-1 cross-modal nearest neighbor exact cell pairing rate
59	rv_coefficient	RV Coefficient	(VII) Cross-Modal Coupling & Modularity	Higher is better	Vector correlation coefficient measuring chromatin co-accessibility and expression coupling
60	module_correlation_r	Module Corr. (r)	(VII) Cross-Modal Coupling & Modularity	Higher is better	Peak-to-gene linkage and co-regulatory module correlation concordance
61	elapsed_time_seconds	Elapsed Time (s)	(VIII) Computational Scalability	Lower is better	Wall-clock execution runtime in seconds required for model training and synthesis
62	peak_memory_mb	Peak RAM (MiB)	(VIII) Computational Scalability	Lower is better	Maximum resident set size / peak memory allocated during execution

---

Category 1: Distribution Properties (14 Metrics)

Evaluates whether single-cell count distributions (library size, mean expression, gene variance, and zero fraction) and higher-dimensional data manifolds in simulated data match real experimental data.

1. Kolmogorov-Smirnov (KS) Distance (KS)

• Direction: Lower is better (inverted during score normalization).
• Formula: $D_{KS} = \sup_x |F_{ref}(x) - F_{sim}(x)|$
• Biological Meaning: Measures the maximum vertical distance between the empirical cumulative distribution functions of real reference and simulated properties. A value of 0 indicates identical distributions.

2. Wasserstein Metric (Earth Mover’s Distance 1D) (Wasserstein)

• Direction: Lower is better.
• Formula: $W_1(P, Q) = \int_{-\infty}^{\infty} |F_{ref}(x) - F_{sim}(x)| \, dx$
• Biological Meaning: Represents the minimal cost required to transform the simulated distribution into the reference distribution, integrating differences across the full range of counts.

3. Median Absolute Deviation (MAD)

• Direction: Lower is better ($|\text{MAD}_{ref} - \text{MAD}_{sim}|$).
• Formula: $\text{MAD} = \text{median}(|x - \text{median}(x)|)$
• Biological Meaning: Robust non-parametric estimate of count spread that resists distortion by extreme sequencing outliers.

4. Mean Absolute Error (MAE)

• Direction: Lower is better.
• Formula: $\text{MAE} = \frac{1}{n} \sum_{i=1}^n |x_i - y_i|$
• Biological Meaning: Average absolute deviation between cell-level and gene-level summary statistics.

5. Root Mean Squared Error (RMSE)

• Direction: Lower is better.
• Formula: $\text{RMSE} = \sqrt{\frac{1}{n} \sum_{i=1}^n (x_i - y_i)^2}$
• Biological Meaning: Standard quadratic scoring rule penalizing large discrepancies between real and simulated statistical moments.

6. Density Curve Overlap (OV)

• Direction: Higher is better (range: $[0, 1]$).
• Formula: $\text{Overlap} = \int \min(f_{ref}(x), f_{sim}(x)) \, dx$
• Biological Meaning: Shared area under the kernel density estimates of real and simulated data. A value of $1.0$ represents identical density profiles.

7. Bhattacharyya Distance (Bhattacharyya)

• Direction: Lower is better.
• Formula: $D_B(P, Q) = -\ln(BC(P, Q))$, where $BC(P, Q) = \sum \sqrt{p(x) q(x)}$
• Biological Meaning: Measures statistical overlap and divergence between discrete single-cell count probability distributions.

8. Empirical Cumulative Distribution Area Difference (ECDF_DiffArea)

• Direction: Lower is better.
• Formula: $\Delta_{\text{ECDF}} = \int |F_{ref}(x) - F_{sim}(x)| \, dx$
• Biological Meaning: Directly integrates the area enclosed between real and simulated empirical cumulative distribution functions.

9. Wald-Wolfowitz Runs Test Statistic (Runs_Statistic)

• Direction: Lower is better ($|Z_{\text{runs}}|$, where values near 0 indicate identical orderings).
• Biological Meaning: Non-parametric test checking whether interleaved reference and simulated observations mix randomly, confirming they originate from the same continuous distribution.

10. 2D Fasano-Franceschini Test Statistic (Fasano_Franceschini_2D_KS)

• Direction: Lower is better.
• Biological Meaning: Computationally efficient bivariate generalization of the Kolmogorov-Smirnov test over quadrants, evaluating paired properties (e.g. library size vs. cell zero fraction).

11. 2D Peacock Bivariate Test Statistic (Peacock_2D_KS)

• Direction: Lower is better.
• Biological Meaning: Rigorous two-dimensional Kolmogorov-Smirnov test measuring supremum divergence across all four quadrants in bivariate planes (e.g. mean vs. variance).

12. 2D Bivariate Kernel Density Estimation z-Statistic (KDE_Bivariate_zstat)

• Direction: Lower is better.
• Biological Meaning: Non-parametric two-dimensional kernel density divergence statistic assessing the global shape and contours of paired feature distributions.

13. Maximum Mean Discrepancy (MMD)

• Direction: Lower is better.
• Formula: $\text{MMD}^2(P, Q) = \mathbb{E}[k(x, x')] - 2\mathbb{E}[k(x, y)] + \mathbb{E}[k(y, y')]$
• Biological Meaning: Kernel-based two-sample test comparing multivariate single-cell distributions in reproducing kernel Hilbert space without projecting down to 1D summaries.

14. Fréchet Single-Cell Distance (Frechet_SingleCell_Distance)

• Direction: Lower is better.
• Formula: $d_F^2 = \|\mu_{ref} - \mu_{sim}\|_2^2 + \text{Tr}(\Sigma_{ref} + \Sigma_{sim} - 2(\Sigma_{ref}\Sigma_{sim})^{1/2})$
• Biological Meaning: Multivariate Gaussian manifold divergence in principal component space between empirical and synthesized cell populations.

---

Category 2: Correlations and Zero Patterns (6 Metrics)

Evaluates whether simulators accurately capture biological overdispersion, zero-inflation dynamics, cell-to-cell dependencies, and gene co-expression networks.

15. Biological Coefficient of Variation Distance (bcv_discrepancy)

• Direction: Lower is better ($|\text{BCV}_{ref} - \text{BCV}_{sim}|$).
• Formula: $\text{BCV} = \sqrt{\frac{\sigma^2}{\mu^2} - \frac{1}{\mu}}$
• Biological Meaning: Isolates true biological variability across cells after subtracting Poisson technical sampling noise.

16. Logistic Dropout Curve Midpoint Error (dropout_curve_midpoint_x0)

• Direction: Lower is better ($|x_{0,\text{ref}} - x_{0,\text{sim}}|$).
• Formula: $P(\text{zero}) = \frac{1}{1 + \exp(-k(x - x_0))}$
• Biological Meaning: Evaluates the expression threshold ($x_0$) at which a gene has a 50% probability of exhibiting technical dropout (zero count).

17. Logistic Dropout Curve Slope Error (dropout_curve_slope_k)

• Direction: Lower is better ($|k_{\text{ref}} - k_{\text{sim}}|$).
• Biological Meaning: Evaluates the rate of transition ($k$) from unobserved dropouts to consistently detected counts as expression levels increase.

18. Mean-Variance Fit / Dropout Goodness-of-Fit (mean_variance_r2)

• Direction: Higher is better (range: $[0, 1]$).
• Biological Meaning: Quantifies how accurately the non-linear relationship between mean expression and zero frequency / variance is preserved in simulated counts.

19. Cell-Cell Pearson Correlation Distance (cell_pearson_cor)

• Direction: Higher is better (or Frobenius discrepancy, inverted).
• Biological Meaning: Confirms that pairwise cell similarity matrices and cellular manifold neighborhoods are accurately replicated.

20. Feature/Gene-Gene Pearson Correlation Distance (feature_pearson_cor)

• Direction: Higher is better.
• Biological Meaning: Assesses whether gene co-expression modules and transcriptional regulatory dependencies maintain their pairwise correlation strength.

---

Category 3: Cell Structure and Clustering (10 Metrics)

Evaluates whether simulated datasets preserve distinct cell types, cluster separation, and biological partitions without artificial blurring.

21. Average Silhouette Width Discrepancy (silhouette_sim)

• Direction: Higher is better (similarity) / Lower is better (discrepancy).
• Formula: $s(i) = \frac{b(i) - a(i)}{\max(a(i), b(i))}$
• Biological Meaning: Measures how well-separated distinct cell types are relative to within-cluster compactness.

22. Dunn Index Discrepancy (dunn_sim)

• Direction: Higher is better / Lower discrepancy.
• Formula: $D = \frac{\min_{i \neq j} d(C_i, C_j)}{\max_k \text{diam}(C_k)}$
• Biological Meaning: Ratio between the smallest inter-cluster distance and largest cluster diameter, evaluating worst-case cluster separation.

23. Davies-Bouldin Index Discrepancy (davies_bouldin_sim)

• Direction: Lower is better.
• Formula: $DB = \frac{1}{k} \sum_{i=1}^k \max_{j \neq i} \left( \frac{\sigma_i + \sigma_j}{d(c_i, c_j)} \right)$
• Biological Meaning: Measures similarity between clusters; lower scores reflect compact clusters that are widely separated.

24. Calinski-Harabasz Index Discrepancy (calinski_harabasz_sim)

• Direction: Higher is better / Lower discrepancy.
• Formula: $CH = \frac{\text{SS}_B / (k - 1)}{\text{SS}_W / (N - k)}$
• Biological Meaning: Variance ratio criterion measuring overall between-cluster dispersion relative to within-cluster dispersion.

25. Adjusted Rand Index Concordance (ari)

• Direction: Higher is better (range: $[-1, 1]$, where $1.0$ is perfect match).
• Biological Meaning: Evaluates agreement between unsupervised clusters and known biological cell type labels, corrected for chance agreement.

26. Normalized Mutual Information (nmi)

• Direction: Higher is better (range: $[0, 1]$).
• Formula: $\text{NMI}(U, V) = \frac{2 I(U; V)}{H(U) + H(V)}$
• Biological Meaning: Measures shared mutual information between unsupervised cluster assignments and biological cell types.

27. Adjusted Mutual Information (ami)

• Direction: Higher is better (range: $[0, 1]$).
• Biological Meaning: Information-theoretic concordance between cell partitions adjusted for expected random overlap across unbalanced clusters.

28. $V$-Measure (v_measure)

• Direction: Higher is better (range: $[0, 1]$).
• Formula: $V = \frac{2 \cdot h \cdot c}{h + c}$
• Biological Meaning: Harmonic mean of cluster homogeneity (clusters contain single cell types) and completeness (all cells of a given type are assigned together).

29. Neighborhood Purity (neighborhood_purity)

• Direction: Higher is better (range: $[0, 1]$).
• Biological Meaning: Fraction of $k$-nearest neighbors in embedding space that share the same biological cell identity.

30. Generative Precision / Hungarian Optimal Matching (Generative_Precision)

• Direction: Higher is better (range: $[0, 1]$).
• Biological Meaning: Solves the linear sum assignment problem (Hungarian algorithm) to find optimal one-to-one pairings between simulated clusters and real biological cell types.

---

Category 4: Batch Effects and Technical Confounders (7 Metrics)

Evaluates whether technical batch variation is realistically synthesized without washing out biological signals.

31. Batch Silhouette ASW (batch_silhouette)

• Direction: Higher is better for mixed batches (or discrepancy relative to reference batch ASW).
• Biological Meaning: Average Silhouette Width calculated using batch labels; near-zero values indicate well-mixed batches where cells do not cluster solely by technical batch.

32. Batch Shannon Entropy (shannon_entropy)

• Direction: Higher is better (range: $[0, \ln(B)]$).
• Formula: $H = -\sum_{b=1}^B p_b \ln(p_b)$
• Biological Meaning: Measures batch composition diversity in local single-cell neighborhoods; high entropy indicates thorough mixing across batches.

33. Principal Component Regression Batch Variance (pcr_r2)

• Direction: Lower is better (or low discrepancy relative to reference).
• Biological Meaning: Quantifies the proportion of global gene expression variance linearly explained by technical batch covariates ($R^2_{\text{batch}}$).

34. CellMixS Mixing Score (cms)

• Direction: Higher is better (range: $[0, 1]$).
• Biological Meaning: Tests whether the distribution of batch labels in local neighborhoods follows random expectations derived from overall batch proportions.

35. Local Inverse Simpson Index (isi)

• Direction: Higher is better (range: $[1, B]$).
• Formula: $\text{LISI} = \frac{1}{\sum_{b=1}^B p_b^2}$
• Biological Meaning: Effective number of distinct batches represented in each cell’s nearest-neighbor neighborhood.

36. Seurat Mixing Metric (seurat_mixing_metric)

• Direction: Lower is better.
• Biological Meaning: Computes the average rank of neighbors belonging to the same batch; lower ranks reflect thorough mixing without batch clustering.

37. Local Density Factor Discrepancy (ldf_diff)

• Direction: Lower is better.
• Biological Meaning: Measures whether local density factor differences between real and simulated data remain small, ensuring local manifold geometries are not distorted by batch artifacts.

---

Category 5: Marker Genes and Biological Signals (15 Metrics)

Assesses whether differentially expressed genes (DEGs), fold-changes, null p-value uniformity, and downstream classification models are faithfully preserved across established benchmarking frameworks (SimBench, Simpipe, and Shaky Foundations).

38. Symmetric Mean Absolute Percentage Error (SimBench_SMAPE)

• Direction: Lower is better (range: $[0, 200\%]$).
• Formula: $\text{SMAPE} = \frac{100\%}{n} \sum_{i=1}^n \frac{|y_i - \hat{y}_i|}{(|y_i| + |\hat{y}_i|)/2}$
• Biological Meaning: Relative percentage error in log2 fold-changes of differentially expressed genes between reference and simulated datasets.

39. SimBench Composite DE Fidelity Score (SimBench_DE_Fidelity_Score)

• Direction: Higher is better (range: $[0, 1]$).
• Formula: $\text{Score} = 1 - \frac{\text{SMAPE}}{200}$
• Biological Meaning: Unified SimBench score capturing overall differential expression effect size retention.

40. Log2 Fold-Change Pearson Correlation (Log2FC_Pearson_Corr)

• Direction: Higher is better (range: $[-1, 1]$).
• Biological Meaning: Measures linear proportionality of estimated DEG effect sizes between real and simulated experiments.

41. Log2 Fold-Change Spearman Correlation (Log2FC_Spearman_Corr)

• Direction: Higher is better (range: $[-1, 1]$).
• Biological Meaning: Measures whether the relative ranking of marker genes by effect size is preserved.

42. Top DEG Jaccard Overlap (Top_DEG_Jaccard_Overlap)

• Direction: Higher is better (range: $[0, 1]$).
• Formula: $J(A, B) = \frac{|A \cap B|}{|A \cup B|}$
• Biological Meaning: Set intersection overlap of the top differentially expressed marker genes between real and simulated data.

43. DEG Detection Ratio (DEG_Ratio)

• Direction: Matches 1.0 (range: $[0, \infty)$).
• Formula: $\text{Ratio} = \frac{N_{\text{DEG, sim}}}{N_{\text{DEG, ref}}}$
• Biological Meaning: Evaluates sensitivity and false discovery balance by checking whether simulated data generates the expected quantity of significant marker genes.

44. Uniform P-Value Distribution Discrepancy (PValue_Uniformity_Chisq)

• Direction: Lower is better.
• Biological Meaning: $\chi^2$ goodness-of-fit test assessing whether non-differentially expressed null genes yield uniform p-values without anti-conservative false positive spikes.

45. Null P-Value Distribution Preservation Score (Distribution_Score)

• Direction: Higher is better (range: $[0, 1]$).
• Biological Meaning: Composite score verifying that null p-value uniformity is preserved without systematic type I error inflation.

46. Machine Learning Cell Type Classification Accuracy (Classifier_Accuracy)

• Direction: Higher is better (range: $[0, 1]$).
• Biological Meaning: Accuracy of a machine learning classifier trained on marker genes in identifying cell types.

47. Macro-Averaged F1 Score (Classifier_Macro_F1)

• Direction: Higher is better (range: $[0, 1]$).
• Formula: $F_1 = 2 \cdot \frac{\text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}$
• Biological Meaning: Harmonic balance between precision and recall across all cell types, preventing common populations from masking rare cell types.

48. Macro-Averaged Recall Score (Classifier_Macro_Recall)

• Direction: Higher is better (range: $[0, 1]$).
• Biological Meaning: Unweighted average sensitivity across biological cell types in simulated datasets.

49. Shaky Foundations Group Silhouette Similarity (Silhouette_Sim)

• Direction: Higher is better (range: $[0, 1]$).
• Biological Meaning: Silhouette width concordance in differential marker space according to the Shaky Foundations methodology.

50. Shaky Foundations Silhouette Discrepancy (Silhouette_Discrepancy)

• Direction: Lower is better.
• Biological Meaning: Absolute difference in cell-group silhouette separation between real and simulated datasets.

51. Shaky Foundations Group PVE Similarity (PVE_Group_Sim)

• Direction: Higher is better (range: $[0, 1]$).
• Biological Meaning: Preservation of the Proportion of Variance Explained (PVE) by biological condition/cell type.

52. Shaky Foundations PVE Discrepancy (PVE_Discrepancy)

• Direction: Lower is better.
• Biological Meaning: Absolute discrepancy in the proportion of total variance explained by biological group structure.

---

Category 6: Trajectory & Lineage Dynamics (2 Metrics)

Evaluates whether continuous developmental progressions and branching lineage hierarchies are accurately reconstructed.

53. Pseudotime Rank Correlation (pseudotime_correlation)

• Direction: Higher is better (range: $[-1, 1]$).
• Biological Meaning: Spearman rank correlation of cell progression ordering along developmental trajectories (e.g. stem $\rightarrow$ progenitor $\rightarrow$ differentiated), inferred directly from expression counts.

54. Lineage Tree Height Discrepancy (tree_height_rmse)

• Direction: Lower is better.
• Biological Meaning: Root Mean Squared Error of lineage tree branch lengths and branching points between real and simulated developmental trees.

---

Category 7: Cross-Modal Coupling & Modularity (6 Metrics)

Evaluates the joint regulatory linkage between chromatin accessibility (scATAC-seq) and gene expression (scRNA-seq) in multiomics datasets.

55. Cross-Modal Cell Type Transfer Accuracy (cross_modal_accuracy)

• Direction: Higher is better (range: $[0, 1]$).
• Biological Meaning: Accuracy of predicting cell types in Modality 2 (scATAC-seq) using a classifier trained strictly on Modality 1 (scRNA-seq).

56. Cross-Modal Cell Type Transfer Macro-F1 Score (cross_modal_F1)

• Direction: Higher is better (range: $[0, 1]$).
• Biological Meaning: Macro-averaged F1 score evaluating cross-modality cell identity transfer across rare and common cell states.

57. Fraction of Samples Closer Than The True Match (FOSCTTM)

• Direction: Lower is better (range: $[0, 0.5]$, where $0.0$ indicates perfect alignment and $0.5$ represents random chance).
• Formula: $\text{FOSCTTM} = \frac{1}{2N} \left( \sum_{i=1}^N \frac{|\{j : d(x_i, y_j) < d(x_i, y_i)\}|}{N} + \sum_{j=1}^N \frac{|\{i : d(y_j, x_i) < d(y_j, x_j)\}|}{N} \right)$
• Biological Meaning: Measures single-cell alignment accuracy in joint multi-omics latent space for co-assayed cells.

58. Top-1 Cross-Modal Match Rate (match_at_1)

• Direction: Higher is better (range: $[0, 1]$).
• Biological Meaning: Proportion of single cells whose true co-assayed profile is its nearest neighbor (rank 1) in joint embedding space.

59. Regulatory Coupling RV Coefficient (rv_coefficient)

• Direction: Higher is better (range: $[0, 1]$).
• Formula: $R_V(X, Y) = \frac{\text{Tr}(X X^T Y Y^T)}{\sqrt{\text{Tr}((X X^T)^2) \text{Tr}((Y Y^T)^2)}}$
• Biological Meaning: Vector matrix correlation measuring global co-inertia and shared regulatory geometry between chromatin accessibility and transcriptional output.

60. Cross-Modal Module Correlation Fidelity (module_correlation_r)

• Direction: Higher is better (range: $[-1, 1]$).
• Biological Meaning: Evaluates whether peak-to-gene correlations and cis-regulatory co-accessibility modules maintain biological concordance.

---

Category 8: Computational Scalability (2 Metrics)

Evaluates computational runtime and memory resource consumption across simulators.

61. Wall-Clock Execution Time (elapsed_time_seconds)

• Direction: Lower is better.
• Biological Meaning: Total elapsed runtime in seconds required to train the generative model and synthesize simulated single-cell profiles.

62. Peak Memory Allocation (peak_memory_mb)

• Direction: Lower is better.
• Biological Meaning: Maximum resident memory (RAM in MiB) consumed during simulation execution.

---

Automated Score Normalization Pipeline

To compare metrics with completely different units (seconds, megabytes, p-values, correlation coefficients) fairly on the same scale, scSimEval executes an automated two-step normalization:

Raw Metric Value
      │
      ▼
Step 1: Direction Inversion (for lower-is-better metrics)
      Inverted Value = Maximum - Value
      │
      ▼
Step 2: Min-Max Normalization
      Standardized Score = (Value - Minimum) / (Maximum - Minimum)
      │
      ▼
Standardized Score in [0.00, 1.00]
(Higher is always better; 1.00 = Top Performer)

In visual representations (such as plot_benchmark_bubble_matrix()), standardized scores are mapped directly to bubble sizes, and top performers ($\ge 0.96$) receive bold highlighted square glyphs.

sessionInfo()
#> R version 4.6.1 (2026-06-24 ucrt)
#> Platform: x86_64-w64-mingw32/x64
#> Running under: Windows 11 x64 (build 26200)
#> 
#> Matrix products: default
#>   LAPACK version 3.12.1
#> 
#> locale:
#> [1] LC_COLLATE=English_India.utf8  LC_CTYPE=English_India.utf8   
#> [3] LC_MONETARY=English_India.utf8 LC_NUMERIC=C                  
#> [5] LC_TIME=English_India.utf8    
#> 
#> time zone: Asia/Calcutta
#> tzcode source: internal
#> 
#> attached base packages:
#> [1] stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> loaded via a namespace (and not attached):
#>  [1] digest_0.6.39     desc_1.4.3        R6_2.6.1          fastmap_1.2.0    
#>  [5] xfun_0.60         cachem_1.1.0      knitr_1.52        htmltools_0.5.9  
#>  [9] rmarkdown_2.32    lifecycle_1.0.5   cli_3.6.6         sass_0.4.10      
#> [13] pkgdown_2.2.1     textshaping_1.0.5 jquerylib_0.1.4   systemfonts_1.3.2
#> [17] compiler_4.6.1    tools_4.6.1       ragg_1.5.2        bslib_0.12.0     
#> [21] evaluate_1.0.5    yaml_2.3.12       otel_0.2.0        jsonlite_2.0.0   
#> [25] rlang_1.3.0       fs_2.1.0          htmlwidgets_1.6.4