Best holdout assessment is sufficient for cancer transcriptomic model selection

Article type: Research Article

Keywords: machine learning, transcriptomics, classifier, gene signature, occam’s razor

Authors: Jake Crawford, Maria Chikina, Casey S. Greene

Affiliations: Genomics and Computational Biology Graduate Group, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA; Department of Computational and Systems Biology, School of Medicine, University of Pittsburgh, Pittsburgh, PA, USA; Department of Biomedical Informatics, University of Colorado School of Medicine, Aurora, CO, USA; Center for Health AI, University of Colorado School of Medicine, Aurora, CO, USA

License: © 2024 The Authors CC BY 4.0 This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).

Article links: DOI: 10.1016/j.patter.2024.101115  |  PubMed: 39776849  |  PMC: PMC11701843

Relevance: Moderate: mentioned 3+ times in text

Full text: PDF (6.1 MB)

Introduction

Gene expression datasets are typically “wide,” with many gene features and relatively few samples. These feature-rich datasets present obstacles in many aspects of machine learning, including overfitting and multicollinearity, and challenges in interpretation. To facilitate the use of feature-rich gene expression data in machine learning models, feature selection and/or dimension reduction are commonly used to distill a more condensed data representation from the input space of all genes.ref. bib1^,ref. bib2 The intuition is that many gene expression features are likely irrelevant to the prediction problem, redundant, or contain no meaningful variation across samples, so transforming them or selecting a subset can generate a more reliable predictor.

In cancer transcriptomics, this preference for small, parsimonious sets of genes can be seen in the popularity of “gene signatures.” These are groups of genes whose expression levels are used to define cancer subtypes or predict prognosis or therapeutic response.ref. bib3^,ref. bib4 Many studies specify the size of the signature in the paper’s title or abstract, suggesting that the fewer genes in a gene signature, the better, e.g., Chen et al.,ref. bib5 Landemaine et al.,ref. bib6 and Cardoso et al.ref. bib7 Clinically, there are many reasons why a smaller gene signature may be preferable, including cost (fewer genes may be less expensive to profile or validate, whereas a large signature likely requires a targeted array or next-generation sequencing [NGS] analysisref. bib8) and interpretability (it is easier to reason about the function and biological role of a smaller gene set than a large one since even disjoint gene signatures tend to converge on common biological pathwaysref. bib9^,ref. bib10).

Behind much of this work, there is an underlying assumption that smaller gene signatures tend to be more robust: that for a new patient or in a new biological context, a smaller gene set or more parsimonious model will be more likely to maintain its predictive performance than a larger one. Similar ideas are described in the statistics literature, suggesting that simpler models with performances that are comparable to the best model are more likely to perform robustly across datasets or resist overfitting.ref. bib11^,ref. bib12 Although these assumptions have rarely been formally stated or systematically tested in genomics applications, they are often included in guidelines or rules of thumb for applied statistical modeling or machine learning in biology, e.g., Altman and Royston,ref. bib13 Boulesteix et al.,ref. bib14 and Kass et al.ref. bib15

In this study, we sought to test the robustness assumption directly by evaluating model generalization across biological contexts, inspired by previous work on domain adaptation and transfer learning in cancer transcriptomics.ref. bib16^,ref. bib17^,ref. bib18 We used two large, heterogeneous public cancer datasets: The Cancer Genome Atlas (TCGA) for human tumor sample dataref. bib19 and the Cancer Cell Line Encyclopedia (CCLE) for human cell line data.ref. bib20 These datasets contain overlapping -omics data types derived from distinct data sources, allowing us to quantify model generalization across data sources. In addition, each dataset contains samples from a wide range of different cancer types/tissues of origin, allowing us to quantify model generalization across cancer types. We trained both linear and non-linear models to predict mutation status (presence or absence) from RNA sequencing (RNA-seq) gene expression for approximately 70 cancer driver genes across varying levels of model simplicity and degrees of regularization, resulting in a variety of gene signature sizes. We compared two simple procedures for model selection, one that combines cross-validation performance with model parsimony and one that only relies on cross-validation performance, for each classifier in each context.

Our results suggest that, in general, mutation status classification models that perform well in cross-validation within a biological context also generalize well across biological contexts. There are some individual genes and some individual cancer types where more regularized, well-performing models outperform the best-performing model. However, we do not observe a systematic generalization advantage for smaller/more regularized models across all genes and cancer types. These results provide evidence that good cross-validation performance within a biological context (data source or cancer type) is a sufficient proxy for robust performance across contexts.

Results

Evaluating model generalization using public cancer data

We collected data from TCGA Pan-Cancer Atlas and CCLE to predict the presence or absence of mutations in cancer genes as a benchmark of cancer-related information content across cancer types and contexts. We trained mutation status classifiers across approximately 70 genes involved in cancer development and progression from Vogelstein et al.ref. bib21 using LASSO logistic regression with gene expression (RNA-seq) values as predictive features and integrating point mutation and copy-number data to label each sample as mutated or not mutated in the target gene (Note S1). We fit each classifier across a variety of regularization parameters, resulting in models with a variety of different sparsity levels between the extremes of 0 nonzero features and all features included (Figure S2). Inspired by the generalization experiments across tissues and model systems in Ma et al.,ref. bib16 we designed experiments to evaluate the generalization of mutation status classifiers across datasets (TCGA to CCLE and CCLE to TCGA) and across biological contexts (cancer types) within TCGA, relative to a within-dataset baseline (Figure 1).

Generalization from human tumor samples to cell lines is more effective than the reverse

To evaluate “cross-dataset” generalization, we trained mutation status classifiers on human tumor data from TCGA and evaluated them on cell line data from CCLE, as well as the reverse, from the CCLE to TCGA. As an example, we examined EGFR, an oncogenic tyrosine kinase that is commonly mutated in diverse cancer types and cancer cell lines, including lung cancer, colorectal cancer, and glioblastoma.ref. bib22^,ref. bib23 For EGFR mutation status classifiers trained on TCGA and evaluated on CCLE, we saw that the area under the precision-recall curve (AUPR) on cell lines was slightly worse than on held-out tumor samples but comparable across regularization levels/LASSO parameters (Figure 2A). On the other hand, EGFR classifiers trained on CCLE and evaluated on TCGA performed considerably worse on human tumor samples as compared to held-out cell lines (Figure 2B). When we compared performance with norms of model coefficient vectors, including the $L_1$ norm that LASSO models explicitly optimize, as opposed to the LASSO parameter values, the observed performance trends were similar (Figure S3).

To explore these tendencies more generally, we compared performance across all genes in the Vogelstein et al. dataset for both TCGA-to-CCLE and CCLE-to-TCGA generalization. We measured the difference between performance on the holdout data within the training dataset and performance across datasets after correcting for the baseline frequency of mutation occurrence in the relevant dataset (i.e., the expected AUPR value for a random classifier). A positive difference indicates poor generalization (better holdout performance than test performance), and a 0 or negative difference indicates good generalization (comparable test performance to holdout performance). For generalization from TCGA to CCLE, we observed that median AUPR differences were mostly centered around 0 for most genes, with some exceptions at the extremes (Figure 2C; performance differences on the y axis). An example of a gene exhibiting poor generalization was IDH1, shown toward the left of Figure 2C as having good performance on held-out TCGA data and poor performance on CCLE data. IDH-mutant glioma cell lines are poorly represented compared to IDH-mutant patient tumors, which may explain the difficulty of generalization to cell lines for IDH1 mutation classifiers.ref. bib24 For generalization from CCLE to TCGA, we observed a more pronounced upward shift toward better performance on CCLE and worse on TCGA, with most genes performing better on the CCLE holdout data and very few genes generalizing comparably to TCGA samples (Figure 2D).

“Best” and “smallest good” model selection strategies perform comparably

To address the question of whether sparser or more parsimonious models tend to generalize better or not, we implemented two model selection schemes and compared them for TCGA-to-CCLE and CCLE-to-TCGA mutation prediction problems (Figure 3A). The best model selection scheme chooses the top-performing model (LASSO parameter) on the holdout dataset from the same source as the training data and applies it to the test data from the other data source. The intention of the smallest good model selection scheme is to balance parsimony with reasonable performance on the holdout data since simply selecting the smallest possible model (generally, the dummy regressor/mean predictor) is not likely to generalize well.

To accomplish this, we rely on the “lambda.1se” heuristic used in the glmnet R package for generalized linear models, which is one of the default methods for parameter choice and model selection.ref. bib25 We first identify models with performance within one standard error of the top-performing model on the holdout dataset. Then, from this subset of relatively well-performing models, we choose the smallest (i.e., strongest LASSO penalty) to apply to the test data. In both cases, we exclusively use the holdout data to select a model and only apply the model to out-of-dataset samples to evaluate generalization performance after model selection. Applying these criteria to both TCGA-to-CCLE and CCLE-to-TCGA prediction problems, we saw that model sizes (number of nonzero gene expression features) tended to differ by approximately an order of magnitude between model selection approaches, with medians on the order of 100 nonzero features for the best models and on the order of 10 nonzero features for the smallest good models (Figure S4). Still, there was considerable variation between target genes, and some best-performing models included substantially more features than the median, including classifiers we have previously observed to perform well such as TP53, PTEN, and SETD2.

For TCGA-to-CCLE generalization, 37/71 genes (52.1%) had better performance for the best model, and 24/71 genes (33.8%) had better generalization performance with the smallest good model. The other 10 genes had the same best and smallest good model performances: in other words, the smallest good model was also the best performing overall, so the performance difference between the two was exactly 0 (Figure 3B). For CCLE-to-TCGA generalization, 30/66 genes (45.5%) had better performance for the best model and 25/66 (37.9%) for the smallest good, with the other 11 having the same model fulfill both criteria (Figure 3C). Overall, these results do not support the hypothesis that the most parsimonious model generalizes the best: for both generalization problems, there are slightly more genes where the best-performing model on the holdout dataset is also the best-performing on the test set, although there are some genes where the smallest good approach works well (CCLE-to-TCGA Wilcoxon signed-rank p = 0.721, TCGA-to-CCLE Wilcoxon signed-rank p = 0.963).

We examined genes that fell into either category for TCGA-to-CCLE generalization (dotted lines on Figure 3B). For NF1, the best model outperforms the smallest good model (Figure 3D). Comparing holdout (orange) and cross-dataset (green) performance, both generally follow a similar trend, with the cross-dataset performance near its peak when the holdout performance peaks at a regularization parameter of $\alpha =0.01$. PIK3CA is an example of the opposite, a gene where the smallest good model tends to outperform the best model (Figure 3E). In this case, better cross-dataset performance occurs at a higher level of regularization (further left on the x axis), $\alpha =0.019$, than the peak for the holdout performance, $\alpha =0.027$. This suggests that a PIK3CA mutation status classifier that is more parsimonious but has slightly worse performance does tend to generalize more effectively across datasets from TCGA to CCLE.

Generalization across cancer types yields similar results to generalization across datasets

To evaluate generalization across biological contexts within a dataset, we trained mutation prediction classifiers on all but one cancer type in TCGA, performed model selection on a holdout set stratified by cancer type, and held out the remaining cancer type as a test set. We performed the same best vs. smallest good analysis that was previously described across 291 gene/holdout cancer type combinations (Figure 4A). We observed 135/291 gene/cancer type combinations (46.4%) that had better generalization performance with the best model compared to 130/291 (44.7%) for the smallest good model. The other 26 gene/cancer type combinations had the same best and smallest good model and thus no difference in performance. This is consistent with our cross-dataset experiments, with slightly more instances where the best model on the stratified holdout data also generalizes the best but no pronounced distributional shift in either direction (Wilcoxon signed-rank p = 0.599).

We looked in more detail at two examples of gene/cancer type combinations, one on either side of the 0 point for cross-cancer type generalization. For prediction of PIK3CA mutation status in rectal adenocarcinoma (READ), we observed the best cross-cancer type performance for relatively low levels of regularization/high x axis values at $\alpha =0.027$ (Figure 4B). For prediction of NF1 mutation status in uterine corpus endometrial carcinoma (UCEC), on the other hand, we observed the best cross-cancer generalization for a high level of regularization ($\alpha =0.0072$), and generalization capability for the best parameter on the stratified holdout set ($\alpha =0.01$) was lower (Figure 4C). It is also interesting to note that in the previous experiments generalizing from TCGA to CCLE, we used PIK3CA as an example of a gene where the smallest good model performs best and NF1 as an example where the best model was selected, and this tendency was reversed for these two cancer types. This highlights the importance of considering generalization to the cancer type or sample cohort of interest independently of general trends for a particular classifier whenever possible.

We aggregated results across genes for each cancer type, looking at performance in the held-out cancer type compared to performance on the stratified holdout set (Figure 4D). Cancer types that were particularly difficult to generalize to (better performance on stratified data than cancer type holdout or positive y axis values) include testicular cancer (TGCT) and soft tissue sarcoma (SARC), which are notable because they are not carcinomas like the majority of cancer types included in TCGA, potentially making generalization harder. We also aggregated results across cancer types for each gene, identifying a distinct set of genes where classifiers tend to generalize poorly no matter what cancer type is held out (Figure S5). Included in this set of genes with poor generalization performance are HRAS, NRAS, and BRAF, suggesting that a classifier that combines mutations in Ras pathway genes into a single “pathway mutation status” label (as described in Way et al.ref. bib26 or using more general computational approaches such as those in Haan et al.ref. bib27 and Bakhtiar et al.ref. bib28) could be a better approach than separate classifiers for each gene.

In the cancer type aggregation plot (Figure 4D), thyroid carcinoma (THCA) stood out as a carcinoma that had poor performance when held out. In our experiments, the only genes in which THCA is included as a held-out cancer type are BRAF and NRAS; generalization performance for both genes is below cross-validation performance but slightly worse for NRAS than BRAF (Figure S6). Previous work suggests that the BRAF mutation tends to have a different functional signature in THCA than other cancer types, and withholding THCA from the training set improved classifier performance, which could at least in part explain the difficulty of generalizing to THCA we observe.ref. bib26

Restricting neural network hidden layer size does not improve generalization

To test whether or not findings generalize to non-linear models, we trained a 3-layer neural network to predict mutation status from gene expression for generalization from TCGA to CCLE, and we varied the size of the first hidden layer to control regularization/model complexity. We fixed the size of the second hidden layer to be half the size of the first layer, rounded up to the nearest integer; further details are provided in methods. For EGFR mutation status prediction, we saw that performance for small hidden layer sizes was noisy but generally lower than for higher hidden layer sizes on train, holdout, and test sets, reflecting “underfitting” or high bias (Figure 5A). On average, over all 71 genes from Vogelstein et al., performance on both held-out TCGA data and CCLE data tends to increase until a hidden layer size of 10–50, then flattens (Figure 5B). To explore additional approaches to neural network regularization, we also tried varying the dropout and weight decay for EGFR and KRAS mutation status classification while holding the hidden layer size constant. The results followed a similar trend, with generalization performance generally tracking performance on holdout data (Figure S7). We also preprocessed the input gene expression features using principal-component analysis (PCA) and varied the number of PCA features retained as input to the neural network; for EGFR, the best generalization performance and holdout performance both occurred at 1,000 PCs, but for KRAS, the model generalized better to cell line data for fewer PCs than its peak holdout performance (Figure S8).

It can be challenging to measure which hidden layer sizes tended to perform relatively well or poorly across classifiers, as different genes may have different baseline performance AUPR values and overall classifier effect sizes. In order to summarize across genes, for each gene, we ranked the range of hidden layer sizes by the corresponding models’ generalization performance on CCLE (Figure 5C). Concretely, for a particular hidden layer size, low ranks represent good performance, and high ranks represent poor performance. We then visualized the distribution of ranks above and below the median rank of 5.5/10 for each hidden layer size across all genes. In summary, for a given hidden layer size, a high proportion of ranks above the median (true, or blue bar, Figure 5C) signifies poor overall performance for that hidden layer size, and a high proportion of ranks below the median (false, or orange bar, Figure 5C) signifies good performance. We saw that small hidden layer sizes tended to generalize poorly (<5 but most pronounced for 1 and 2), and intermediate hidden layer sizes tended to generalize well (10–100 and sometimes 500/1,000). This suggests that some degree of parsimony or simplicity could be useful but that very simple models do not tend to generalize well.

We also performed the same best/smallest good analysis as with the linear models, using hidden layer size as the regularization axis instead of LASSO regularization strength. We observed a distribution centered around 0, suggesting that the best and smallest good models tend to generalize similarly (Figure 5D). 28/71 genes (45.2%) had better generalization performance with the best model compared to 21/71 (28.6%) for the smallest good model and 22 with the same best and smallest good model. We extended our analyses to two additional non-linear model classes as well, for TCGA-to-CCLE generalization: XGBoost gradient boosting classification and a deeper neural network with 5 hidden layers. For XGBoost, using the n_estimators (number of tree estimators to combine) and max_depth (maximum depth of each tree) parameters to control model complexity, we saw a similar relationship between holdout performance on TCGA and generalization performance on CCLE to that of the LASSO experiments, although model performance was generally more stable across parameter settings (Figure S9). For the 5-layer neural networks, the generalization results were similar to the 3-layer neural networks, although underfitting/high bias was more obvious for very small hidden layer sizes, and there was a slightly more pronounced preference for larger hidden layer sizes overall (Figure S10).

Discussion

Using public cancer genomics and transcriptomics data from TCGA and CCLE, we studied the generalization of mutation status classifiers for a wide variety of cancer driver genes. We designed experiments to evaluate generalization across biological contexts by holding out cancer types in TCGA and across datasets by training models on TCGA and evaluating them on CCLE, and vice versa. We found that, in general, smaller or more parsimonious models do not tend to generalize more effectively across cancer types or across datasets, and in the absence of prior knowledge about a prediction problem, simply choosing the model that performs the best on a holdout dataset is at least as effective for selecting models that generalize. Given that similar smallest good heuristics are used broadly across genomics studies (see, e.g., Wang et al.,ref. bib29 Shao et al.,ref. bib30 and Li et al.ref. bib31), we expect these results to have implications on current practices.

Our results were similar in both linear models (LASSO logistic regression) and non-linear deep neural networks when using hidden layer size as the regularization parameter of interest. In our non-linear model experiments, we did not observe better generalization across datasets for fully connected neural networks with fewer hidden layer nodes, and our preliminary results indicated a similar trend for dropout and weight decay. Compared to linear models, it is less clear how to define a small or parsimonious neural network model since there are many regularization techniques that one may use to control complexity. Rather than simply removing nodes and keeping the network fully connected, another approach to parsimony could be to select an inductive bias to guide the size reduction of the network. Existing examples include network structures guided by protein-protein interaction networks or function/pathway ontologies.ref. bib32^,ref. bib33^,ref. bib34^,ref. bib35 It is possible that a smaller neural network with a structure that corresponds more appropriately to the prediction problem would achieve better generalization results, although choosing an apt network structure or data source can be a challenging aspect of such efforts.

For generalization from CCLE to TCGA, we observed that performance was generally worse on human tumor samples from TCGA than for held-out cell lines. This could, at least in part, be a function of sample size: the number of cell lines in CCLE is approximately an order of magnitude smaller than the number of tumor samples in TCGA (∼10,000 samples in TCGA vs. ∼1,500 cell lines in CCLE, although the exact number of samples used to train and evaluate our classifiers varies by gene; see methods for further details). There are also plausible biological and technical explanations for the difficulty of generalizing to human tumor samples. This result could reflect the imperfect and limited nature of cancer cell lines as a model system for human tumors, which previous studies have pointed out.ref. bib36^,ref. bib37^,ref. bib38 In addition, the CCLE data are collected and processed uniformly, as described in Ghandi et al.,ref. bib20 while TCGA data are processed by a uniform pipeline but collected from a wide variety of different cancer centers around the US.ref. bib19

When we ranked cancer types in order of their generalization difficulty aggregated across genes, we noticed a slight tendency toward non-carcinoma cancer types (testicular cancer [TGCT], soft tissue sarcoma [SARC], skin cutaneous melanoma [SKCM]) being difficult to generalize to. It has been pointed out in other biological data types that holding out entire contexts or domains is necessary for a full picture of generalization performance,ref. bib39^,ref. bib40 which our results corroborate. This highlights a potential weakness of using TCGA’s carcinoma-dominant pan-cancer data as a training set for a broad range of tasks, for instance in foundation models, which are becoming feasible for some genomics applications.ref. bib41^,ref. bib42^,ref. bib43 One caveat of our analysis is that each cancer type is included in the training data or held out for a different subset of genes, so it is difficult to detangle gene-specific effects (some mutations have less distinguishable functional effects on gene expression than others) from cancer-type-specific effects (some cancer types are less similar to each other than others) on prediction performance using our experimental design.

Other aspects of TCGA that may make it less representative for certain prediction problems is that it is composed of primary tumor samples from adult patients with relatively high quality (fresh frozen, generally high purity, although this varies by tissueref. bib44), so it is possible that generalization to metastatic samples, pediatric patients, or lower-quality (e.g., formalin-fixed paraffin-embedded [FFPE]) clinical samples would present different properties. Similarly, mutation calling in CCLE cell lines is limited by the lack of a matched normal reference, although we generally observed reasonable generalization to cell lines, suggesting that the quality of mutation calls is likely adequate in the genes we considered. Overall, however, we believe the size and tissue representations of TCGA and CCLE make them apt benchmarks for model performance in cancer -omics.

Methods

Mutation data download and preprocessing

To generate binary mutated/non-mutated gene labels for our machine learning model, we used mutation calls for TCGA samples from MC3ref. bib45 and copy-number threshold calls from GISTIC2.0.ref. bib46 MC3 mutation calls were downloaded from the Genomic Data Commons (GDC) of the National Cancer Institute at https://gdc.cancer.gov/about-data/publications/pancanatlas. Thresholded copy-number calls are from an older version of the GDC data and are available here: https://figshare.com/articles/dataset/TCGA_PanCanAtlas_Copy_Number_Data/6144122. We removed hypermutated samples, defined as two or more standard deviations above the mean non-silent somatic mutation count, from our dataset to reduce the number of false positives (i.e., non-driver mutations). Any sample with either a non-silent somatic variant or a copy-number variation (CNV; copy-number gain in the target gene for oncogenes and copy-number loss in the target gene for tumor-suppressor genes) was included in the positive set; all remaining samples were considered negative for mutation in the target gene.

We followed a similar procedure to generate binary labels for cell lines from CCLE using the data available on the DepMap download portal at https://depmap.org/portal/download/all/. Mutation information was retrieved from the OmicsSomaticMutations.csv data file, and copy-number information was retrieved from the OmicsCNGene.csv data file, both from the 22Q2 public release. We thresholded the CNV log ratios provided by CCLE into binary gain/loss calls using a lower threshold of log₂(3/2) (i.e., cell lines with a log ratio below this threshold were considered to have a full copy loss in the corresponding gene) and an upper threshold of log₂(5/2) (i.e., cell lines with a log ratio above this threshold were considered to have a full copy gain in the corresponding gene). After applying the same hypermutation criteria that we used for TCGA, no cell lines in CCLE were identified as hypermutated. After preprocessing, 1,402 cell lines with mutation and copy-number data remained. We then combined non-silent point mutations and copy-number gain/loss information into binary labels using the same criteria as for TCGA.

Gene expression data download and preprocessing

RNA-seq data for TCGA were downloaded from GDC at the same link provided above for the Pan-Cancer Atlas. We discarded non-protein-coding genes and genes that failed to map and removed tumors that were measured from multiple sites. After filtering to remove hypermutated samples and taking the intersection of samples with both mutation and gene expression data, 9,074 TCGA samples remained.

RNA-seq data for CCLE were downloaded from the DepMap download portal in the CCLE_expression.csv data file from the 22Q2 public release. After taking the intersection of CCLE cell lines with both mutation and gene expression data, 1,402 cell lines remained. For experiments making predictions across datasets (i.e., training models on TCGA and evaluating performance on CCLE, or vice versa), we took the intersection of genes in both datasets, resulting in 16,041 gene features. For experiments where only TCGA data were used (i.e., evaluating models on held-out cancer types), we used all 16,148 gene features present in TCGA after the filtering described above.

Cancer gene set construction

In order to study mutation status classification for a diverse set of cancer driver genes, we started with the set of 125 frequently altered genes from Vogelstein et al.ref. bib21 (all genes from Table S2A in that study). For each target gene, to ensure that the training dataset was reasonably balanced (i.e., that there would be enough mutated samples to train an effective classifier), we included only cancer types with at least 15 mutated samples and at least 5% mutated samples, which we refer to here as valid cancer types. In some cases, this resulted in genes with no valid cancer types, which we dropped from the analysis. Out of the 125 genes originally listed in the Vogelstein et al. cancer gene set, we retained 71 target genes for TCGA-to-CCLE analysis and 66 genes for CCLE-to-TCGA analyses. For these analyses, each gene needed at least one valid cancer type in TCGA and one valid cancer type in CCLE to construct the train and test sets. For the cancer type holdout analysis, we retained 56 target genes: in this case, each gene needed at least two valid cancer types in TCGA to be retained, one to train on and one to hold out.

Classifier setup and cross-validation design

We trained logistic regression classifiers to predict whether or not a given sample had a mutational event in a given target gene using gene expression features as explanatory variables. Our model was trained on gene expression data (X) to predict somatic mutation presence or absence (y) in a target gene. To control for varying mutation burden per sample, we included log₁₀(sample mutation count) in our models as a covariate. Since gene expression datasets tend to have many dimensions and comparatively few samples, we used a LASSO penalty to perform feature selection.ref. bib47 LASSO logistic regression has the ability to generate sparse models (some or most coefficients are 0), as well as having a single tunable hyperparameter which can be easily interpreted as an indicator of regularization strength/model simplicity.

LASSO ($L_1$-penalized) logistic regression finds the feature weights $\hat{w}\in {\mathbb{R}}^p$, solving the following optimization problem:

where $i\in \left\{1,\dots ,n\right\}$ denotes a sample in the dataset, $X_i\in {\mathbb{R}}^p$ denotes features (gene expression measurements) from the given sample, $y_i\in \left\{0,1\right\}$ denotes the label (mutation presence/absence) for the given sample, and $l(\cdot )$ denotes the negative log likelihood of the observed data given a particular choice of feature weights, i.e.,

Given weight values $\hat{w}$, it is straightforward to predict the probability of a positive label (mutation in the target gene) $P\left(y^{\ast }=1\mid X^{\ast };\hat{w}\right)$ for a test sample $X^{\ast }$:

and the probability of no mutation in the target gene, $P\left(y^{\ast }=0\mid X^{\ast };\hat{w}\right)$, is given by (1 − the above quantity).

This optimization problem leaves one hyperparameter to select: $C$, which controls the inverse of the strength of the L₁ penalty on the weight values (i.e., regularization strength scales with $\frac{1}{C}$). Although the LASSO optimization problem does not have a closed form solution, the loss function is convex, and iterative optimization algorithms are commonly used for finding reasonable solutions. For fixed values of $C$, we solved for $\hat{w}$ using scikit-learn’s LogisticRegression method,ref. bib48 which uses the coordinate descent optimization method implemented in liblinear.ref. bib49 We selected this implementation rather than the SGDClassifier stochastic gradient descent implementation because coordinate descent/liblinear tends to generate sparser models and does not depend on a learning rate parameter, although after hyperparameter tuning, performance is generally comparable between the implementations.ref. bib50

To assess model selection across contexts (datasets and cancer types), we trained models using a variety of LASSO parameters on 75% of the training dataset, holding out 25% of the training dataset as the cross-validation set and also evaluating across contexts as the test set. We trained models using $C$ values evenly spaced on a dense logarithmic scale between (10⁻³ and 10³), which was where we generally observed that performance varied the most, and a sparser logarithmic scale between (10³ and 10⁷) in order to capture models with very little regularization that included all features. In other words, the exact range we used is the output of the command: numpy.concatenate(numpy.logspace(−3, 3, 43), numpy.logspace(3, 7, 21)).

This range of regularization strength/sparsity levels was intended to give evenly distributed coverage across genes and cancer types that included underfit models (predicting only the mean or using very few features, poor performance on all datasets), overfit models (performing perfectly on training data but comparatively poorly on cross-validation and test data), and a wide variety of models in between that typically included the best fits to the cross-validation and test data. To assess variability between train/CV splits, we used all 4 splits (25% holdout sets) × 2 random seeds for a total of 8 different training sets for each gene, using the same test set (i.e., all of the held-out context, either one cancer type or one dataset) in each case.

Best model vs. smallest good model analysis

For the best vs. smallest good model selection comparison, we started with 8 performance measurements (4 cross-validation folds × 2 random seeds) for each LASSO parameter. We took the mean over these 8 measurements to get a single performance measurement for each model (LASSO parameter) on the holdout dataset, which has the same composition as the training set. We used these per-parameter mean performance measurements to select the best model (LASSO parameter with the best mean performance on the holdout dataset) and the smallest good model (strongest LASSO parameter with mean performance within 1 standard error of the best mean performance value on the holdout dataset, as implemented in the glmnet R package’s lambda.1se model selection methodref. bib25). For the distributions of differences shown in the results, we took the difference in mean performance for the best and smallest good models for each gene, with positive differences indicating better performance for the best model and negative differences better performance for the smallest good model. Note that in each case, we are comparing model selection procedures for models trained on the same data (same training set/cross-validation split) and measuring the difference in model performance between procedures, so correcting for the baseline AUPR is unnecessary here.

Neural network setup and parameter selection

As a trade-off between computational cost and ability to represent non-linear decision boundaries, inspired by the architecture of the intermediate-complexity model described in Heil et al.,ref. bib51 we trained a three-layer fully connected neural network with rectified linear unit (ReLU) non-linearitiesref. bib52 to predict mutation status. For the experiments described in the results and discussions sections, we varied the size of the first hidden layer in the range {1, 2, 3, 4, 5, 10, 50, 100, 500, 1,000}. We fixed the size of the second hidden layer to be half of the size of the first hidden layer, rounded up to the nearest integer, and the size of the third hidden layer was the number of classes, 2 in our case. Our models were trained for 100 epochs of mini-batch stochastic gradient descent in PyTorchref. bib53 using the Adam optimizerref. bib54 and a fixed batch size of 50. To select the remaining hyperparameters for each hidden layer size, we performed a random search over 10 combinations, with a single train/test split stratified by cancer type, using the following hyperparameter ranges: learning rate {0.1, 0.01, 0.001, 5e−4, 1e−4}, dropout proportion {0.1, 0.5, 0.75}, and weight decay (L₂ penalty) {0, 0.1, 1, 10, 100}. We used the same train/cross-validation split strategy described above for one random seed and 4 cross-validation splits, generating 4 different performance measurements for each gene and hidden layer size.

Although L₁ regularization can be used to more directly induce model sparsity in convex settings, we note that using L₁ regularization to control model complexity in neural networks is considerably more complex. Simply adding an additional loss term is not enough to achieve convergence to a sparse solution; the problem requires special optimizers and is the subject of ongoing research (see, e.g., Yun et al.ref. bib55). For this reason, we focused on controlling neural network model complexity via the size and number of hidden layers, as well as the other approaches described above.

For the EGFR gene, we also ran experiments where we varied the dropout proportion and the weight decay hyperparameter as the regularization axis and selected the remaining hyperparameters (including the hidden layer size) using a random search. In these cases, we used a fixed range for dropout of {0.0, 0.05, 0.125, 0.25, 0.375, 0.5, 0.625, 0.75, 0.875, 0.95} and a fixed range for weight decay of {0.0, 0.001, 0.005, 0.01, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5, 0.75, 1.0, 10.0}. All neural network analyses were performed on a Ubuntu 18.04 machine with an NVIDIA RTX 2060 GPU.

Resource availability

Lead contact

Requests for information on or further resources from this study should be directed to Casey S. Greene (casey.s.greene@cuanschutz.edu).

Materials availability

This study did not generate new materials or reagents.

Data and code availability

The data from TCGA analyzed during this study were previously published as part of TCGA Pan-Cancer Atlas projectref. bib19 and are available from the NIH NCI GDC. The data from CCLE analyzed during this study were previously publishedref. bib20 and are available from the Broad Institute’s DepMap Portal. Raw classification results, performance figures for all genes in the Vogelstein et al. dataset, and parameter selection results and performance comparisons for each individual gene in the best vs. smallest good analyses are available on Figshare under a CC0 license.ref. bib56

Software developed and used in this manuscript is available on GitHub at https://github.com/greenelab/pancancer-evaluation/ and on Zenodo.ref. bib57 The scripts used to download and preprocess the datasets for this study are available at https://github.com/greenelab/pancancer-evaluation/tree/master/00_process_data. Scripts for TCGA-to-CCLE and CCLE-to-TCGA CCLE comparisons (Figures 2 and 3) and neural network experiments (Figure 5) are available in the https://github.com/greenelab/pancancer-evaluation/tree/master/08_cell_line_prediction directory. Scripts for TCGA cancer type comparisons (Figure 4) are available in the https://github.com/greenelab/pancancer-evaluation/tree/master/02_cancer_type_classification directory. All scripts are available under the open-source BSD 3-clause license.

This manuscript was written using Manubotref. bib58 and is available on GitHub at https://github.com/greenelab/generalization-manuscript under the CC0-1.0 license.

Acknowledgments

This research was supported in part by grants from the 10.13039/100000002National Institutes of Health (R01 CA237170, R01 HD109765, and R01 HG010067). This research was supported in part by the 10.13039/100016300University of Pittsburgh Center for Research Computing through the resources provided. Specifically, this work used the HTC cluster, which is supported by 10.13039/100000002NIH award number S10OD028483.

Author contributions

Conceptualization, J.C. and C.S.G.; methodology, J.C., M.C., and C.S.G.; software, J.C.; visualization, J.C.; writing – original draft, J.C.; writing – review & editing, J.C., M.C., and C.S.G.; funding acquisition, C.S.G.; supervision, C.S.G. All authors read and approved the final manuscript.

Declaration of interests

During the manuscript revision, J.C. was employed at Repare Therapeutics.

References

1. A. Antoniadis, S. Lambert-Lacroix, F. Leblanc. Effective dimension reduction methods for tumor classification using gene expression data. Bioinformatics, 2003. [DOI | PubMed]
2. F.W. Townes, S.C. Hicks, M.J. Aryee, R.A. Irizarry. Feature selection and dimension reduction for single-cell RNA-Seq based on a multinomial model. Genome Biol., 2019. [DOI]
3. M. Cieślik, A.M. Chinnaiyan. Cancer transcriptome profiling at the juncture of clinical translation. Nat. Rev. Genet., 2017. [DOI | PubMed]
4. F. Chibon. Cancer gene expression signatures – The rise and fall?. Eur. J. Cancer, 2013. [DOI | PubMed]
5. H.-Y. Chen, S.-L. Yu, C.-H. Chen, G.-C. Chang, C.-Y. Chen, A. Yuan, C.-L. Cheng, C.-H. Wang, H.-J. Terng, S.-F. Kao. A Five-Gene Signature and Clinical Outcome in Non–Small-Cell Lung Cancer. N. Engl. J. Med., 2007. [DOI | PubMed]
6. T. Landemaine, A. Jackson, A. Bellahcène, N. Rucci, S. Sin, B.M. Abad, A. Sierra, A. Boudinet, J.M. Guinebretière, E. Ricevuto. A Six-Gene Signature Predicting Breast Cancer Lung Metastasis. Cancer Res., 2008. [DOI | PubMed]
7. F. Cardoso, L.J. van’t Veer, J. Bogaerts, L. Slaets, G. Viale, S. Delaloge, J.-Y. Pierga, E. Brain, S. Causeret, M. DeLorenzi. 70-Gene Signature as an Aid to Treatment Decisions in Early-Stage Breast Cancer. N. Engl. J. Med., 2016. [DOI | PubMed]
8. E.A. Slodkowska, J.S. Ross. MammaPrint™ 70-gene signature: another milestone in personalized medical care for breast cancer patients. Expert Rev. Mol. Diagn., 2009. [DOI | PubMed]
9. J. Massagué. Sorting Out Breast-Cancer Gene Signatures. N. Engl. J. Med., 2007. [DOI | PubMed]
10. B. Weigelt, L. Pusztai, A. Ashworth, J.S. Reis-Filho. Challenges translating breast cancer gene signatures into the clinic. Nat. Rev. Clin. Oncol., 2011. [DOI | PubMed]
11. J. Friedman, T. Hastie, R. Tibshirani. Regularization Paths for Generalized Linear Models via Coordinate Descent. J. Stat. Softw., 2010. [PubMed]
12. D. Krstajic, L.J. Buturovic, D.E. Leahy, S. Thomas. Cross-validation pitfalls when selecting and assessing regression and classification models. J. Cheminform., 2014. [DOI | PubMed]
13. D.G. Altman, P. Royston. What do we mean by validating a prognostic model?. Stat. Med., 2000. [DOI | PubMed]
14. A.-L. Boulesteix, C. Strobl, T. Augustin, M. Daumer. Evaluating Microarray-based Classifiers: An Overview. Cancer Inform., 2008. [DOI | PubMed]
15. R.E. Kass, B.S. Caffo, M. Davidian, X.-L. Meng, B. Yu, N. Reid. Ten Simple Rules for Effective Statistical Practice. PLoS Comput. Biol., 2016. [DOI]
16. J. Ma, S.H. Fong, Y. Luo, C.J. Bakkenist, J.P. Shen, S. Mourragui, L.F.A. Wessels, M. Hafner, R. Sharan, J. Peng, T. Ideker. Few-shot learning creates predictive models of drug response that translate from high-throughput screens to individual patients. Nat. Cancer, 2021. [DOI | PubMed]
17. H. Sharifi-Noghabi, P.A. Harjandi, O. Zolotareva, C.C. Collins, M. Ester. Out-of-distribution generalization from labelled and unlabelled gene expression data for drug response prediction. Nat. Mach. Intell., 2021. [DOI]
18. S.M.C. Mourragui, M. Loog, D.J. Vis, K. Moore, A.G. Manjon, M.A. van de Wiel, M.J.T. Reinders, L.F.A. Wessels. Predicting patient response with models trained on cell lines and patient-derived xenografts by nonlinear transfer learning. Proc. Natl. Acad. Sci. USA, 2021. [DOI]
19. J.N. Weinstein, E.A. Collisson, G.B. Mills, K.R.M. Shaw, B.A. Ozenberger, K. Ellrott, I. Shmulevich, C. Sander, J.M. Stuart. The Cancer Genome Atlas Pan-Cancer analysis project. Nat. Genet., 2013. [DOI | PubMed]
20. M. Ghandi, F.W. Huang, J. Jané-Valbuena, G.V. Kryukov, C.C. Lo, E.R. McDonald, J. Barretina, E.T. Gelfand, C.M. Bielski, H. Li. Next-generation characterization of the Cancer Cell Line Encyclopedia. Nature, 2019. [DOI | PubMed]
21. B. Vogelstein, N. Papadopoulos, V.E. Velculescu, S. Zhou, L.A. Diaz, K.W. Kinzler. Cancer Genome Landscapes. Science, 2013. [DOI | PubMed]
22. G. da Cunha Santos, F.A. Shepherd, M.S. Tsao. EGFR Mutations and Lung Cancer. Annu. Rev. Pathol., 2011. [DOI | PubMed]
23. H. Liu, B. Zhang, Z. Sun. Spectrum of EGFR aberrations and potential clinical implications: insights from integrative pan-cancer analysis. Cancer Commun., 2020. [DOI]
24. L.E. Jones, S. Hilz, M.R. Grimmer, T. Mazor, C. Najac, J. Mukherjee, A. McKinney, T. Chow, R.O. Pieper, S.M. Ronen. Patient-derived cells from recurrent tumors that model the evolution of IDH-mutant glioma. Neurooncol. Adv., 2020. [DOI]
25. J. Friedman, T. Hastie, R. Tibshirani. Regularization Paths for Generalized Linear Models via Coordinate Descent. J. Stat. Soft., 2010. [DOI]
26. G.P. Way, F. Sanchez-Vega, K. La, J. Armenia, W.K. Chatila, A. Luna, C. Sander, A.D. Cherniack, M. Mina, G. Ciriello. Machine Learning Detects Pan-cancer Ras Pathway Activation in The Cancer Genome Atlas. Cell Rep., 2018. [DOI | PubMed]
27. D. Haan, R. Tao, V. Friedl, I.N. Anastopoulos, C.K. Wong, A.S. Weinstein, J.M. Stuart. Pacific Symposium on Biocomputing 2020 (WORLD SCIENTIFIC), 2019. [DOI]
28. H. Bakhtiar, K.T. Helzer, Y. Park, Y. Chen, N.R. Rydzewski, M.L. Bootsma, Y. Shi, P.M. Harari, M. Sharifi, M. Sjöström. Identification of phenocopies improves prediction of targeted therapy response over DNA mutations alone. npj Genom. Med., 2022. [DOI]
29. L. Wang, Y. Yang, L. Feng, C. Tan, H. Ma, S. He, M. Lian, R. Wang, J. Fang. A novel seven-gene panel predicts the sensitivity and prognosis of head and neck squamous cell carcinoma treated with platinum-based radio(chemo)therapy. Eur. Arch. Oto-Rhino-Laryngol., 2021. [DOI]
30. F. Shao, Z. Wang, S. Wang. Identification of MYCN-Related Gene as a Potential Biomarker for Neuroblastoma Prognostic Model by Integrated Analysis and Quantitative Real-Time PCR. DNA Cell Biol., 2021. [DOI | PubMed]
31. X. Li, H. Zhang, J. Liu, P. Li, Y. Sun. Five crucial prognostic-related autophagy genes stratified female breast cancer patients aged 40–60 years. BMC Bioinf., 2021. [DOI]
32. B.M. Kuenzi, J. Park, S.H. Fong, K.S. Sanchez, J. Lee, J.F. Kreisberg, J. Ma, T. Ideker. Predicting Drug Response and Synergy Using a Deep Learning Model of Human Cancer Cells. Cancer Cell, 2020. [DOI | PubMed]
33. M. Kulmanov, M.A. Khan, R. Hoehndorf, J. Wren. DeepGO: predicting protein functions from sequence and interactions using a deep ontology-aware classifier. Bioinformatics, 2018. [DOI | PubMed]
34. N. Fortelny, C. Bock. Knowledge-primed neural networks enable biologically interpretable deep learning on single-cell sequencing data. Genome Biol., 2020. [DOI]
35. Y. Hao, J.D. Romano, J.H. Moore. Knowledge-guided deep learning models of drug toxicity improve interpretation. Patterns, 2022. [DOI]
36. J.-P. Gillet, S. Varma, M.M. Gottesman. The Clinical Relevance of Cancer Cell Lines. J. Natl. Cancer Inst., 2013. [DOI | PubMed]
37. J.L. Wilding, W.F. Bodmer. Cancer Cell Lines for Drug Discovery and Development. Cancer Res., 2014. [DOI | PubMed]
38. F. Iorio, T.A. Knijnenburg, D.J. Vis, G.R. Bignell, M.P. Menden, M. Schubert, N. Aben, E. Gonçalves, S. Barthorpe, H. Lightfoot. A Landscape of Pharmacogenomic Interactions in Cancer. Cell, 2016. [DOI | PubMed]
39. J. Schreiber, R. Singh, J. Bilmes, W.S. Noble. A pitfall for machine learning methods aiming to predict across cell types. Genome Biol., 2020. [DOI]
40. S. Whalen, J. Schreiber, W.S. Noble, K.S. Pollard. Navigating the pitfalls of applying machine learning in genomics. Nat. Rev. Genet., 2022. [DOI | PubMed]
41. E. Nguyen, M. Poli, M. Faizi, A. Thomas, C. Birch-Sykes, M. Wornow, A. Patel, C. Rabideau, S. Massaroli, Y. Bengio. HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution. arXiv, 2023. [DOI]
42. H. Cui, C. Wang, H. Maan, K. Pang, F. Luo, B. Wang. scGPT: Towards Building a Foundation Model for Single-Cell Multi-omics Using Generative AI. Nature Methods., 2024. [DOI]
43. M. Hao, J. Gong, X. Zeng, C. Liu, Y. Guo, X. Cheng, T. Wang, J. Ma, L. Song, X. Zhang. Large Scale Foundation Model on Single-cell Transcriptomics. Nature Methods, 2024. [DOI]
44. D. Aran, M. Sirota, A.J. Butte. Systematic pan-cancer analysis of tumour purity. Nat. Commun., 2015. [DOI]
45. K. Ellrott, M.H. Bailey, G. Saksena, K.R. Covington, C. Kandoth, C. Stewart, J. Hess, S. Ma, K.E. Chiotti, M. McLellan. Scalable Open Science Approach for Mutation Calling of Tumor Exomes Using Multiple Genomic Pipelines. Cell Syst., 2018. [DOI | PubMed]
46. C.H. Mermel, S.E. Schumacher, B. Hill, M.L. Meyerson, R. Beroukhim, G. Getz. GISTIC2.0 facilitates sensitive and confident localization of the targets of focal somatic copy-number alteration in human cancers. Genome Biol., 2011. [DOI | PubMed]
47. R. Tibshirani. Regression Shrinkage and Selection Via the Lasso. J. Roy. Stat. Soc. B Stat. Methodol., 1996. [DOI]
48. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg. Scikit-learn: Machine Learning in Python. J. Mach. Learn. Res., 2011
49. R.-E. Fan, K.-W. Chang, C.-J. Hsieh, X.-R. Wang, C.-J. Lin. LIBLINEAR: A Library for Large Linear Classification. J. Mach. Learn. Res., 2008
50. J. Crawford, M. Chikina, C.S. Greene. Optimizer’s dilemma: optimization strongly influences model selection in transcriptomic prediction. Bioinformatics Advances, 2024. [DOI]
51. B.J. Heil, J. Crawford, C.S. Greene. The effect of non-linear signal in classification problems using gene expression. PLoS Comput. Biol., 2023. [DOI]
52. V. Nair, G.E. Hinton. Proceedings of the 27th International Conference on Machine Learning ICML’10, 2010
53. A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga. PyTorch: An Imperative Style, High-Performance Deep Learning Library. arXiv, 2019. [DOI]
54. D.P. Kingma, J. Ba. Adam: A Method for Stochastic Optimization. arXiv, 2017. [DOI]
55. J. Yun, A.C. Lozano, E. Yang. Proceedings of the 35th International Conference on Neural Information Processing Systems 1865, 2021
56. J. Crawford. 2023. [DOI]
57. J. Crawford. greenelab/pancancer-evaluation: Publication release. Zenodo, 2024. [DOI]
58. D.S. Himmelstein, V. Rubinetti, D.R. Slochower, D. Hu, V.S. Malladi, C.S. Greene, A. Gitter. Open collaborative writing with Manubot. PLoS Comput. Biol., 2019. [DOI]