Defining hypoxia in cancer: A landmark evaluation of hypoxia gene expression signatures

Article type: Research Article

Keywords: hypoxia, tumorigenesis, biomarkers, transcriptomics, gene signature, signature scores, hypoxia-targeting therapies, radiotherapy, patient stratification, single cell

Authors: Matteo Di Giovannantonio, Fiona Hartley, Badran Elshenawy, Alessandro Barberis, Dan Hudson, Hana S. Shafique, Vincent E.S. Allott, David A. Harris, Simon R. Lord, Syed Haider, Adrian L. Harris, Francesca M. Buffa, Benjamin H.L. Harris

Affiliations: Computational Biology and Integrative Genomics Lab, Department of Oncology, University of Oxford, Oxford, UK; Chinese Academy of Medical Sciences Oxford Institute, University of Oxford, Oxford, UK; The Rosalind Franklin Institute, Didcot, UK; Duke University School of Medicine, Durham, NC, USA; St. Catherine’s College, University of Oxford, Oxford, UK; Merton College, University of Oxford, Oxford, UK; Breast Cancer Now Toby Robins Breast Cancer Research Centre, The Institute of Cancer Research, London, UK; CompBio Lab, Department of Computing Sciences, Bocconi University, Milan, Italy; AI and Systems Biology Lab, IFOM – Istituto Fondazione di Oncologia Molecolare ETS, Milan, Italy; Cutrale Perioperative and Ageing Group, Imperial College London, London, UK

License: © 2025 The Authors CC BY 4.0 This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).

Article links: DOI: 10.1016/j.xgen.2025.100764  |  PubMed: 39892389  |  PMC: PMC11872601

Relevance: Moderate: mentioned 3+ times in text

Full text: PDF (6.3 MB)

Introduction

Hypoxia is a decrease in the normal level of oxygen in tissues, commonly observed in vascular and pulmonary diseases, as well as cancer.ref. bib1 Hypoxia emerges in the tumor microenvironment as cancer cells proliferate and aberrant angiogenesis fails to keep pace with increased oxygen demand.ref. bib2 This environment exerts a strong selective pressure, favoring adaptations associated with resistance to chemotherapy and radiotherapy,ref. bib3^,ref. bib4 increased genomic instability,ref. bib5 protection from antitumor immune responses,ref. bib6 creation of protective stem cell niches,ref. bib7 and enhanced metastatic potential.ref. bib8 Tumor hypoxia is therefore associated with poor prognosis in tumors across tissues, including breast,ref. bib9 bladder,ref. bib10 brain,ref. bib11 gastric,ref. bib12 head and neck,ref. bib13 liver,ref. bib14 lung,ref. bib15 esophageal,ref. bib16 and prostate.ref. bib17

Over several decades, cancer hypoxia has been the focus of research aiming at therapy. However, hypoxia-targeted therapies have had limited integration into medical practice. A rare exception is the hypoxic radiosensitizer, nimorazole, which has been made standard of care in Denmark following the seminal the Danish Head and Neck Cancer Group (DAHANCA) study.ref. bib18 However, other countries have not followed suit, awaiting results of further studies.

Overall, hypoxia-targeted therapies have produced inconsistent results in clinical trials.ref. bib19 For instance, the promising hypoxia-activated prodrug tirapazamine, although successful in multiple phase I and II trials,ref. bib20^,ref. bib21^,ref. bib22^,ref. bib23^,ref. bib24^,ref. bib25^,ref. bib26^,ref. bib27^,ref. bib28^,ref. bib29 failed to improve overall survival or progression-free survival in phase III in cervical and head and neck cancers.ref. bib30^,ref. bib31 Mixed results were seen in non-small cell lung cancer.ref. bib32^,ref. bib33

The lack of stratification has been cited to be sufficient to account for the failure of phase III trials for hypoxia-activated prodrugs.ref. bib34 This effect is not likely just limited to these prodrugs. Indeed, the failure to accurately identify patients with hypoxic tumors, and the lack of integration of validated hypoxia biomarkers into clinical trials, has contributed to disappointing clinical trial results across the field.ref. bib35^,ref. bib36^,ref. bib37^,ref. bib38^,ref. bib39

Reliably and accurately identifying hypoxia in tumors at scale may hold the key to unlocking the potential of hypoxia-targeting therapies. The development of predictive tools for patient stratification has been described as the most crucial step to the successful integration of these treatments.ref. bib40 One promising approach is the use of hypoxia gene expression signatures (hypoxia signatures): sets of genes whose expression is altered in the context of hypoxic environments and that can be applied to quantify the response to hypoxia in tumor tissue.ref. bib41 As well as being used prospectively to select patients for hypoxia-modifying/targeting agents, hypoxia gene expression signatures can be used in retrospective datasets, opening up other data sources not specifically aimed at studying hypoxia.ref. bib42

Hypoxia signatures have been developed using a variety of approaches.ref. bib41 At present, there is no consensus in the field on how hypoxia signatures should be applied (in vitro or in vivo). Previous studies have found heterogeneity both in gene content and signature performance.ref. bib41^,ref. bib43 A central problem that limits the applicability of gene signatures to newly generated independent datasets is the difficulty of summarizing the expression of a disparate set of genes as a robust and transferable score.ref. bib44 Thus, two principal challenges exist: selecting the correct signature and finding the most effective method to represent the gene expression within it.

Currently, there is no agreement on which signature to use in which context, nor a systematic evaluation of differential signature performance with different summarization methods (scores). This work intends to bridge this lacuna in the literature and address the pressing questions tied to the hypoxia signature application: (1) which are the most appropriate signatures and most effective scoring methods (e.g., median, mean, gene set variation analysis [GSVA]) for measurement of hypoxia in cell lines; (2) which are the most promising signatures for stratifying patients for treatment with hypoxia-targeting agents in clinical practice? If a single signature/score combination proves promising in one or both areas, it would provide substantial benefits to both laboratory and clinical researchers by improving consistency and streamlining research efforts.

Here, we systematically assess published hypoxia signatures and common scoring methods in the largest analysis to date. We investigate the performance of 70 hypoxia signatures in hypoxia vs. normoxia experiments across 104 cell lines, as well as in over 5,000 clinical samples from 10 solid tumor types. Since no consensus exists about how hypoxia signatures should be applied to the burgeoning field of single-cell data, we investigate signature effectiveness in this high-resolution technology. Furthermore, we present a novel approach to solving one of the key controversies emerging in the field of gene signatures as a whole, testing whether signatures truly differ in performance to random gene sets/signatures (RGSs).ref. bib45^,ref. bib46^,ref. bib47 This work provides much-needed clarity to the field and helps establish a new foundation in how to apply hypoxia signatures so we can enhance our understanding of tumor microenvironmental biology, elucidate new pathways and biomarkers, and, ultimately, drive effective patient stratification for hypoxia modifiers and other treatment solutions.

Results

Systematic review reveals 70 published hypoxia signatures

A systematic search of the three major databases for academic publishing yielded 70 publications on hypoxia gene expression signatures (Figure 1A), 38 more than were identified in a 2015 study.ref. bib41 The size of these signatures ranged from 759 genes (Starmansref. bib48) to three genes (Sun,ref. bib49 Xu,ref. bib50 Zhang 2020ref. bib51), and their makeup and annotations are given in Table S1 (S1a, symbol annotation; S1b, Entrez annotation; S1c, Ensembl annotation). Their mean and median signature sizes were 55 and 24 genes respectively. 35 signatures were derived using clinical samples, leaving 35 derived from in vitro approaches alone. Signatures derived using clinical samples ranged from three to 158 in length with a median number of 14 genes. In vitro signatures tended to be longer, with a median number of genes of 42. Surprisingly, no individual gene was found in all 70 signatures. This could reflect their origin in terms of the derived tissues’ response to hypoxia (different cell lines/tumor types), or this might reflect differences in the experimental conditions used (percentage of oxygen, length of time under hypoxia, etc.; Table 1). The number of overlapping genes between signatures is shown in (Figure S1).

Nonetheless, several genes occurred frequently across signatures (top occurring genes highlighted in Figure 1B). The most frequently occurring genes across signatures were NDRG1 and VEGFA (both present in 28 out of the 70 signatures) and PGK1 (27 out of 70). Frequencies for all genes are found in Table S2.

Pathway enrichment analysis of all genes occurring in any of the 70 signatures confirmed several enriched pathways known to be activated in response to hypoxia (Figure 1C; Table S3). Reassuringly, the most enriched pathways were HIF-1 signaling (4.5-fold enrichment, p = 3.18E−17) and renal cell carcinoma (4-fold enrichment, p = 5.65E−09), the latter a disease where the HIF transcriptional response is active due to VHL inactivation by mutation.ref. bib52 Protein processing in the endoplasmic reticulum (ER) was also enriched (3.7-fold enrichment, p = 3.18E−17) driven by several genes known to be induced in response to ER stress (including ATF4 and ATF6), three members of the ER degradation-enhancing alpha-mannosidase-like protein family (EDEM1, EDEM2, and EDEM3) and genes coding for components of coat protein-complex II (SEC23Aand SEC24A). This might be linked to enrichment of autophagy (2.9-fold enrichment, p = 1.56E−08) as both ER stress and autophagy are known to be linked.ref. bib53 Several diseases previously associated with the expression of genes regulated in the hypoxic response were also found to be enriched, including diabetes, non-alcoholic liver disease, Parkinson’s disease, and prion disease.ref. bib54^,ref. bib55^,ref. bib56^,ref. bib57

Signature and summary score choices strongly influence predictive value for hypoxic exposure

Gene signatures can be informative of the status of a biological sample, from cell cultures to tissue. For example, if all the genes associated with the hypoxia response are highly expressed in one sample and not expressed in another sample, we would conclude that the first sample is experiencing a hypoxia response. However, this clear dichotomy is rarely observed, as transcriptional readouts are intrinsically noisy and tissue cultures and tissues are not uniform. Nevertheless, we could expect that, overall, the expression of genes that are canonically regulated by hypoxia will tend to be higher in a sample cultured in hypoxia, with respect to a sample cultured in normoxia. Thus, hypoxia gene expression signatures need to be summarized optimally to infer a “hypoxic phenotype” in a biological system/clinical setting of interest. Similarly, for other phenotypes, gene signatures can be used as a proxy.

Most authors selected one such scoring method when applying and assessing their derived hypoxia signature(s), and at present there is no consensus on which score is most appropriate to use for a given context. Thus, we assessed 14 summary scoring methods across the 70 hypoxia signatures through an analytical framework that can be applied to any collection of gene signatures (Table S4).

To illustrate our analytical approach and the impact of the different scoring methods on signature performance we show the performance of one signature in human breast cancer cell lines (the Toustrup signature; Figures 2A–2C). The distribution derived from genes in the Toustrup signature show marked differences when comparing hypoxic and normoxic samples from GEO: GSE29406 (three normoxic and three hypoxic MCF7 cell-line samples, hypoxic = 1% oxygen for 24 h; Figure 2A). This distinction is missing when using RGSs of the same length. Such differences can be summarized using a p value, derived from the null distribution estimated using RGS (Figure 2B). These p values can be summarized across multiple cell lines and datasets as an “accuracy index” in an approach fundamental to our in vitro signature evaluation. This index is defined as the number of pairwise combinations with a significant difference over the total number of pairs analyzed (see STAR Methods). This measure can be used to determine the best score and signature combination. For instance, with the Toustrup signature, interquartile mean (IQM), which excludes outlier influences, reaches the highest accuracy index in breast cancer cell-line experiments (Figure 2C).

To find the most effective signature/scoring combination for measurement of hypoxia across cell lines, we carried out a pan-cancer analysis. Here, the validity of all hypoxia signature/score combinations was investigated across 28 sequencing platforms, 104 cell lines, and 1,198 pairwise comparisons of hypoxic and normoxic samples (Table S5). Results were compared to 54 million RGSs for each of the 14 scores.

As in breast cancer cell lines, the choice of scoring method markedly affected the ability of the signature to reflect hypoxic status. The best-performing signature/score combination was the Tardon signature when using the IQM score (Tardon/IQM). Although derived from glioblastoma multiforme cell lines exposed to 1% oxygen for 48 h, this 19-gene signature was widely effective across multiple cell lines, oxygen tensions, and hypoxic exposures (Figure 3), achieving an accuracy of 94.0%. If not the top-ranking signature/score combination in the individual tumor types tested, Tardon/IQM was always within the top 10 of the 1,050 signature/score combinations tests in each cancer type and within the top three in eight of the 12 tissue types tested. Where not the most accurate, Tardon/IQM was within 0.28%–5% of the most accurate combination (Table S6). Of note, samples that were misclassified by the Tardon signature (blue values for −log₁₀(p) in Figure 3) tended to be non-significant across most/all signatures (horizontal blue lines). These samples typically originated from experiments where colorectal cancer cell lines were exposed to hypoxia for a short time (i.e., between 1 and 2 h in HCT116, HCT-15, LoVo, WiDr, and COLO-125), which might be too early to observe a marked transcriptional response to hypoxia, and experiments when VHL was reintroduced in hypoxia vs. VHL mutated in normoxia in 786-O cells, which could be expected as reintroduction of VHL might not completely recapitulate the hypoxia response. Some breast cancer samples also were universally misclassified across signatures. This could be due to experimental artifacts, such as residual oxygen being left in plastic tissue culture platesref. bib124 or oxygen being present via other channels. Full details of those misclassified by Tardon/IQM are found in Table S7.

A second key observation was that several scoring methods appeared consistently ineffective at denoting hypoxia in this bulk RNA sequencing (RNA-seq) data, irrespective of the experiment used (for instance, midrange; further examples in Figures S2–S14). Even the best-performing signature, Tardon, dropped to 0.6% accuracy with single-sample gene set enrichment analysis (ssGSEA) or 33.1% if GSVA was used, illustrating the importance of summary score choice. IQM had the highest number of signatures (28) with over 85% accuracy with 22 of these derived solely from in vitro cell culture. Other measures of central tendency also performed well, showing over 85% accuracy in a number of signatures: Tukey’s trimean (24 signatures), midhinge (21 signatures), and median (16 signatures). Tardon was also the top performer using each of these metrics with 93.7%, 93.2%, and 92.6%, respectively.

The best-performing signature derived using clinical samples with IQM was Yang 2018 sarcoma, achieving 92.8% accuracy in our cell-line analysis (Figure 3). This 24-gene signature was derived from seven soft-tissue sarcoma cell lines and refined in clinical cohorts. Some signatures, for instance the 16-gene Sun 2020 signature, six-gene Liu signature, and seven-gene Shou 2020 signature, appeared generally ineffective at identifying hypoxia irrespective of score choice. The highest accuracies achieved with Sun 2020, Liu and Shou 2020 were 3.1% (normalized comulative score, NCS), 2.7% (mean), and 1.4% (NCS), respectively. Interestingly, these signatures were derived using clinical samples only.

We also investigated the performance of the signatures in GEO: GSE30979, which consisted of normoxic/hypoxic comparison of ex vivo tumor fragments from non-small cell lung cancer, containing mixed cell types.ref. bib125 This analysis showed only 61.1% accuracy with Tardon/IQM, with other signatures performing better (e.g., Starmans obtaining 88.3% using the mean score). How cell-line and ex vivo results are transferable to the clinic needs to be elucidated and will be explored later.

To conclude, this analysis of bulk RNA datasets of 104 cell lines demonstrates variability in the performance of both scores and signatures. Thus, it is essential to carefully consider the choice of hypoxia gene expression signature and score used to identify an in vitro active hypoxia response.

Hypoxia signatures with efficacy in bulk RNA from cell lines show validity in single-cell data

Single-cell RNA-seq (scRNA-seq) data from hypoxia and normoxia experiments are rare, and so very little is known on how to apply hypoxia signatures to single-cell data. To address this, we interrogated two in-house-generated scRNA-seq datasets from cell lines to investigate whether the performance of hypoxia signatures in scRNA-seq was similar to that in the bulk RNA-seq data. The best-performing signature/score combination in bulk RNA-seq, Tardon/IQM, clearly divides normoxic and hypoxic cells in both cell lines (MCF7 and HCC1806), and, overall, results in bulk RNA-seq platforms appear similar to those in these single-cell datasets (Figure 4). This clear division is also seen in normoxic/hypoxic normal adjacent tissue (NAT) from human kidneys, generated in a previous studyref. bib123 (GEO: GSE200207; Figure S15).

One particular signature appeared to have superior performance when applied to our single-cell dataset compared to bulk RNA data (van Malenstein). The van Malenstein signature is a concise four-gene signature derived from HepG2 cells (CCNG2, EGLN3, ERO1A, and WDR45B). In the single-cell datasets, this signature is dominated by the expression of ERO1A and WDR45B (Figure S16). ERO1A appears upregulated in hypoxia and downregulated in normoxia, whereas WDR45B is upregulated in normoxia and downregulated in hypoxia. As these two genes have opposite patterns of expression, the intra-signature correlation is low (Figure S17). The unusual performance of this signature can be traced back to its small set of two dominant features (ERO1A and WDR45B), resulting from the inherent sparsity of single-cell data versus bulk. In single-cell data, not all signature genes contribute equally to the score. For small signatures, a few features can dominate the overall score, skewing analysis results and reducing robustness against technical variation. Indeed, normalized rank of gene expression values for the van Malenstein signature in both scRNA-seq and bulk data revealed higher expression of both ERO1A and WDR45B in single-cell data compared to bulk, particularly in hypoxia. This explains this difference in signature performance between technologies. The impact of single-cell data sparsity on analytical methods is well described.ref. bib126 However, despite observable differences between bulk and single-cell data, with van Malenstein serving as a notable example, most signatures exhibit impressive consistency in performance between these two sequencing platforms. The Tardon signature continues to stand out as a leading performer distinctly separating normoxic from hypoxic cells.

Two signatures stand out as the most promising for clinical use across tumor types

Because of the major microenvironmental differences between in vitro cellular models and human tumors, it is essential to assess whether hypoxia signatures and scores that work well in cell lines also are the most appropriate in human tumors. There are debates surrounding the most effective type of hypoxia measurements with the advances in modern technology, as perhaps the “gold standard” oxygen electrode measurements have been surpassed. There are no large cohort studies across tumor types to establish the most reliable and relevant indicator of hypoxia in the clinical setting. Further, at present, there are no large datasets that have gene expression data, prognostic information, and hypoxia measurements from a variety of technologies. Thus, one commonly used method of assessing the clinical relevance of a hypoxia signature is by examining prognosis in patient cohorts (i.e., if those individuals with a higher score have a worse prognosis). This assumes that hypoxia is linked with prognosis, which has support both from biological and previous clinical studies (exemplar publicationsref. bib1^,ref. bib127^,ref. bib128). However, other factors aside from hypoxia clearly can influence prognosis. The current study is limited to the validation of hypoxia signatures in datasets that have prognostic information. However, we take a novel approach, utilizing tumor and adjacent normal-tissue samples from one of largest cancer datasets available: The Cancer Genome Atlas (TCGA).

To find the most promising signatures for clinical use, we identified signatures that fulfill three criteria: (1) NAT should have a lower hypoxia score than tumor samples (criterion 1), (2) effective hypoxia signatures should be significantly different from random gene signatures of the same length when comparing NAT and tumor tissues (criterion 2), and (3) efficacious hypoxia signatures should confer prognostic information (criterion 3).

Two signatures, Buffa and Ragnum, showed higher scores in tumors compared to NAT across all 10 cancer types (Figure S18). Both signatures were developed using clinical samples: the 51-gene Buffa signature was derived using human head and neck and breast cancer samples,ref. bib83 whereas the 32-gene Ragnum signature was developed from four prostate cancer cell lines (22Rv1, LNCaP, PC-3, and DU 145) and honed in clinical samples from prostate patients.ref. bib93 Pathway analysis revealed both signatures were significantly enriched for glycolysis/gluconeogenesis, central carbon metabolism in cancer, as well as more generally carbon metabolism (Figure S19). While there is a notable overlap in terms of significantly enriched pathways, with four of the six pathways in the Ragnum signature also enriched in Buffa, the two signatures modestly intersect at the level of individual genes. The Buffa and Ragnum signatures share just a quartet of genes: ADM, DDIT4, P4HA1, and HILPDA.

Buffa and Ragnum achieved higher scores in tumors compared to NAT across studied tumor types using two and six scores respectively (Buffa: mean and NCS. Ragnum : mean, IQM, median, midhinge, trimean, and NCS). When comparing these signature score combinations, they were consistently very different from RGS over 1,000 simulations (illustrated in Figure S20 and shown Table S8). All eight signature/score combinations achieved over 99% signature performance index (SPI), with the exception of Buffa/NCS, which achieved 97.6%. It is noteworthy that Tardon/IQM, the best-performing signature and score combination in cell lines, had a higher average score in tumor compared to normal tissues in six of the 10 cancer types (HNSC, LUSC, COAD, LUAD, THCA, and UCEC) and had an SPI of only 55.9%. Further, in cell lines, Buffa signature demonstrated reasonable effectiveness in cell lines, achieving its peak accuracy of 87.3% with the IQM score. In contrast, the Ragnum signature’s highest accuracy was significantly lower, at only 44.3%, also recorded with the IQM score.

Prognostic ability was investigated across 5,401 solid tumor samples in TCGA (for further details, see STAR Methods). The best-performing signature/score combination was Buffa/mean, which was significantly prognostic in seven out of 10 individual tumor types (HNSC, BRCA, LUAD, LIHC, UCEC, THCA, and PRAD; Figure S21). Although Ragnum/IQM was significantly prognostic in six out of 10 tumor types (BRCA, LUAD, PRAD, THCA, LIHC, and UCEC; Figure S22). Neither combination was significantly prognostic on COAD, LUSC, or STAD. Using the median as the cutoff point for high and low hypoxia showed both Buffa/mean with Ragnum/IQM were significantly prognostic in four tumor types, with both being prognostic in BRCA and LUAD (Buffa/mean: HNSC, BRCA, LUAD, and LIHC. Ragnum/IQM: BRCA, LUAD, LIHC, and UCEC). Revisiting the results from the 104 in vitro cell lines previously interrogated, Buffa/mean had a superior accuracy compared to Ragnum/IQM (78.5% vs. 44.3%). In colorectal cancer cell lines, Buffa/mean was 56.3% accurate, whereas Ragnum/IQM was only 17.5% accurate. It is hard to comment on the in vitro performance of these signature/score combinations in gastric adenocarcinoma and lung squamous cell carcinoma cell lines, as data from hypoxia experiments using these cancer types are very limited (Table S5).

In order to leverage hypoxia signatures to stratify patients to therapies, a threshold to define low and high hypoxia and thus at which to administer the treatment is helpful. Previous studies have employed the median hypoxia score within a distribution to determine eligibility for hypoxia-modifying therapies.ref. bib91^,ref. bib129 However, little is known as to whether the choice of median is optimal in individual or across tumor types. This is a complex question to answer. However, using TCGA data, Kaplan-Meier analyses suggest that perhaps a promising and practicable cut point for Buffa/mean at which to stratify patients with LIHC, LUAD, or HNSC is at the top 20th percentile (Figure 5). Multivariate Cox proportional hazard models underscore this (Table S9).

Discussion

Transcriptomic signatures have been transformative in the oncology clinic, for example, in treatment selection for patients with estrogen receptor-positive early breast cancer.ref. bib130^,ref. bib131 Hypoxia-targeted therapies have utility across tumor types; however, the inability to select accurately patients for clinical trials has proved a critical impediment. Hypoxia signatures may hold the key, and this work is the largest and most comprehensive analysis and validation of hypoxia signatures to date. This study includes (1) a systematic review of published hypoxia signatures, (2) an unbiased analysis of hypoxia signature performance in all open-access hypoxia cell-line data from GEO spanning 104 different cell lines, and finally (3) a data-driven approach to find the most promising signature for use in clinical samples using the landmark cancer genomics study, TCGA. We also present a new method for signature evaluation using RGSs, which addresses a major need to develop methodology to evaluate gene signatures.

Looking at signatures as a whole, measures of central tendency appear to work better than those using ranking (e.g., GSVA). This indicates that the magnitude of gene expression, rather than just the relative ranking, might be important. The most effective signature and score combination on any cell line tested was Tardon/IQM. Across 1,198 pairwise combinations, Tardon/IQM yields an impressive 94.0% accuracy in identifying hypoxic samples in experiments (i.e., cells in hypoxic chambers vs. normoxic conditions). There were also some samples where most signatures failed, suggesting an experimental issue, either by design or otherwise (e.g., too short a time under hypoxia). However, it must be noted that this analysis is biased with respect to cell type and tissues commonly used in published models.

Although this is the most comprehensive evaluation of hypoxia signatures and scores in cell lines to date, it does not cover all cell lines, oxygen tensions, or durations under hypoxia. Therefore, when applying the results from this study, it is important to consider the experimental data upon which our conclusions are currently based (Table S5). Another consideration is that it is hard to quantify (1) how much weight to place on small percentage increases in accuracy, which may relate to only a handful of samples, and (2) the extent to which overfitting plays a role in tissue types where limited samples exist. Therefore, researchers may choose to use Tardon/IQM alongside the highest-ranking signature/score combination(s) for their cancer type of interest (Table S6) and compare the results. Undoubtedly, as more hypoxia experiments are added to the literature, this work should be revisited. In particular, further experiments on colorectal and kidney cell lines at a range of oxygen tensions/hypoxic exposures would be beneficial.

Tardon/IQM continued to perform well in single-cell data, which is relatively underexplored in this field, achieving excellent separation between hypoxic and normoxic breast cancer cells across two cell lines (MCF7 and HCC1806). Further single-cell validation is needed across datasets from different tissue types, as such data are currently rare.

For clinical samples, one major limitation of this study, and of others in the literature, was the lack of hypoxia measurements by methods alternative to gene expression that could provide a comparative assessment. Thus, we suggested a novel three-stage approach to identify the most promising signature and score for use in clinical samples (see STAR Methods). Throughout these three stages, Buffa and Ragnum performed extremely well. Prognostic efficacy across tumor types was best achieved by Buffa/mean and Ragnum/IQM. It should be noted that prognostic analyses were carried out in TCGA, which, like all resources, has limitations, including incomplete annotation of patient outcome and treatment data and its relatively short-term clinical follow-up interval.ref. bib132^,ref. bib133^,ref. bib134^,ref. bib135 Previous work has already shown that the Buffa signature is prognostic in other cohorts in breast cancerref. bib83^,ref. bib136 (including METABRICref. bib137^,ref. bib138^,ref. bib139), head and neck cancer,ref. bib83 and lung cancer.ref. bib83^,ref. bib140 Similarly, the Ragnum signature has demonstrated prognostic value in prostateref. bib93 and pancreatic cancers.ref. bib141 While these findings highlight the potential of these signatures in predicting patient outcomes, it is unknown how much weight should be given to prognostic ability when assessing the accuracy of these signatures in measuring hypoxia. Large-scale prospective studies are needed to validate, ideally with multiple potential hypoxic measures, whether these signatures do truly reflect hypoxia (e.g., with [18F]-fluoromisonidazole PET-CT).ref. bib142 However, such studies have both methodological and logistical complications. Even if carried out, their results may be challenging to interpret where lack of concordance exists. Perhaps a more practical approach is to test promising signature/score combinations prospectively.

Following our comprehensive analysis, we recommend using Tardon/IQM to confirm hypoxic status in laboratory experiments, perhaps alongside the highest-ranking signature/score combination(s) for the cancer type of interest. For prospective evaluation in clinical trials, Buffa/mean and Ragnum/IQM pass our rigorous three-step evaluation. This work gives much-needed clarity to the field and provides an important reference to laboratory and clinician scientists who seek validation of hypoxic status and/or are considering orchestration of prospective trials.

Limitations of the study

This study represents a significant step forward in understanding the performance of hypoxia signatures across bulk and scRNA-seq data; however, there are some salient limitations. This study’s significant reliance on publicly available datasets introduces variability arising from differences in experimental design, oxygen tensions, and durations of hypoxic exposure. This variability impacts the comparability and generalizability of the findings, underscoring the need for more standardized experimental protocols in future studies. While the analysis leverages a diverse range of datasets, the focus remains predominantly on commonly used cell lines publicly available in the GEO. These cell lines may not fully capture the heterogeneity of hypoxic responses observed across diverse tissue types and cancer subtypes. Expanding validations to underrepresented cell lines, particularly those reflective of less-studied malignancies such as colorectal or kidney cancers, would enhance the robustness of these findings. Further, the limited availability of hypoxia versus normoxia experiments with scRNA-seq data is important to note. A concerted effort by the scientific community to contribute more such scRNA-seq data will be valuable.

Resource availability

Lead contact

Requests for further information should be directed to Dr. Benjamin Harris (benjamin.harris@oncology.ox.ac.uk).

Materials availability

This study did not generate new unique reagents.

Data and code availability

Dataset availability is detailed in the table in STAR★Methods. The SigScores package is found on GitHub (https://alebarberis.github.io/sigscores/index.html).

Acknowledgments

This work was supported by Cancer Research UK Programme grant 23969 and European Research Council Programme grant 772970 to F.M.B. B.H.L.H. was supported by 10.13039/501100000289Cancer Research UK, Oxfordshire Health Service Research Committee, and the Thouron Award. M.D.G. received support from Urology Cancer Research and Education (UCARE). We would like to thank Professor Michael B. Fertleman, Dr. Louis J. Koizia, and Dr. Jason L. Walsh for their support during this work.

Author contributions

Conceptualization, M.D.G., F.M.B., and B.H.L.H.; methodology, M.D.G., F.M.B., and B.H.L.H.; data curation, M.D.G., B.E., and A.B.; formal analysis, M.D.G., B.E., F.H., and B.H.L.H.; software, A.B. and B.E.; visualization, M.D.G., B.E., and B.H.L.H.; writing—review & editing, all authors; supervision, F.M.B. and B.H.L.H.; investigation, M.D.G., F.M.B., S.H., A.L.H., and B.H.L.H.; funding acquisition, M.D.G., F.M.B., and B.H.L.H.

Declaration of interests

The authors declare no competing interests.

STAR★Methods

Key resources table

Method details

Identification of hypoxia gene expression signatures

A gene expression signature consists of a collection of one or more genes whose expression levels reflect a specific biological status or phenotype of the sample under examination. Gene expression signatures can be derived in a number of different ways, from the comparison of two samples (with or without relevant condition) with a simple two-samples statistical test, to machine learning techniques such as Generalized Linear Models (GLM),ref. bib143 Random Forestref. bib144 or Support Vector Machines.ref. bib145 Published hypoxia gene expression signatures were identified from two main sources.

• A previously published extensive literature review from Harris et al.ref. bib41 containing 32 signatures
• A systematic review interrogating Web of Science, Scopus and Pubmed databases using the query: (“hypoxia signature”) OR (“hypoxia” AND “signature”)

Articles resulting from the initial query were further filtered to exclude those not directly relevant to hypoxia signatures. Due to the methodology employed in this study, only genes identified as upregulated were incorporated into the final signatures. An exception was made for two specific signatures: Boidot 2014 Continuous Hypoxia and Boidot 2014 Cyclic Hypoxia, for which upregulation information was not specified.

Reannotation of hypoxia signatures

Gene expression signatures have been developed using a variety of sequencing methodologies, from microarrays to Next Generation Sequencing (NGS). Over the last decade, microarrays were still widely used to derive gene expression signatures. However, the most recent signatures tend to rely on RNA sequencing data, thanks to the decreasing costs of this technology. Often, the authors of the signatures mainly report the genes in their publications as Gene Name (or HGNC symbol) and rarely include more specific and stable identifiers, such as microarray Probe ID or the Ensembl/Entrez Gene ID. This provides a challenge as biological understanding and gene nomenclature and definition evolve over time. A commonly used way to standardise gene annotations across signatures involves converting gene symbols from the articles into stable identifiers using the biomaRt software.ref. bib146 biomaRt is a federated database system (FDBS) providing unified access to disparate data sources. The European Bioinformatics Institute provides access to Ensembl genomes through a biomaRt implementation, accessible via browser and several software packages.

However, this ID conversion might still be challenging for two reasons:

• Gene names change over time: for example, the Vascular Endothelial Growth Factor A (VEGFA) gene, which plays a key role in hypoxia, was previously reported as VEGF. Thus any signatures that contain VEGF, will lose this gene during the conversion process using the most recent Ensembl BiomaRt versions.
• Ensembl annotation database change over time: a new release of the Ensembl dataset could result in loss of information since Ensembl does not include outdated annotations in its relational database. For example, the above-mentioned VEGF gene name is not available on the recent Ensembl release.

Therefore, in order to convert gene symbols from papers into Ensembl gene IDs, it is critical to reannotate them into their most updated version first. For this task, the multi-symbol checker dataset from HGNC was used.ref. bib147 Each gene expression signature was reannotated using the multi-symbol checker to convert each gene symbol to the latest “approved symbol”. If more alias symbols were available, the relevance score from GeneCardsref. bib148 was used to select the most relevant annotation. Gene names annotated as protein coding were prioritised.

Once updated to the most recent and relevant gene symbol, genes in the signature were converted into both Ensembl and Entrez Gene IDs using biomaRtref. bib146 (Ensembl 107: Jul 2022), an R interface to the BioMart software suite. Finally, a conversion dictionary was created to match each Ensembl Gene ID with any other external annotation such as Probe IDs or Ensembl transcript IDs, useful to select the correct gene from the gene expression datasets included in this study. A list of all gene symbols and Ensembl Gene IDs is available in Appendix 1.

Pathway analysis

Over-representation pathway analysis was performed using ShinyGOref. bib149 v0.77 to identify gene pathways that were predominantly enriched in the signatures. A pathway was deemed significantly enriched if a False Discovery Rate (FDR) of less than 0.05 was observed.

Cell line data

Publicly available gene expression datasets consisting of at least one hypoxic and normoxic control were retrieved from The Gene Expression Omnibus (GEO) archive.ref. bib150 GEO data is organised in Series, Samples, and Platforms.

• Series: a Series record provides a general description of the whole study, and links together a group of related Samples. Each Series is identified by a unique accession number having “GSE” as prefix (i.e., GSExxx).
• Platforms: a Platform record is composed of a summary description of the array or sequencer. A Platform ID is reported with the letters “GPL” and the platform number. For example, GPL570 is the corresponding ID for the Affymetrix Human Genome U133 Plus 2.0 Array. One platform can be assigned to multiple Series. Each microarray Platform record includes a manifest file, a dataset reporting the microarray probe IDs annotated with the corresponding gene information such as gene symbol(s) or transcript(s). Bulk RNAseq platforms do not contain a manifest file and gene expression IDs are reported in the gene expression dataset for each Series.
• Samples: a Sample record describes all the conditions under which an individual sample was handled. Each Sample is identified with a unique ID, the letters “GSM” followed by the sample number (e.g., GSM71498). A Sample can be part of multiple Series, but must reference only one Platform.

Batch effects can occur due to differences in the sequencing technology used, but also different technical and experimental conditions, such as the lab users and the consumables utilised. Combining all samples from different Series together in a unique dataset could mask underlying batch effects and correcting them could be challenging, leading to inaccurate conclusions. To obviate this problem, the performance of hypoxia signatures was assessed by evaluating the difference between paired hypoxic/normoxic samples (defined as pairwise combinations) according to the following rules.

• Both samples must be part of the same Series and sequenced using the same technology
• Both samples are from the same cell line (for example, MDA-MB-231 samples cannot be combined with MCF-7 samples)

Unfortunately, the GEO archive collects most of the sample-related information as unstructured data (e.g., free text). Thus, all the information such as oxygen concentration, time under hypoxia, cell type or additional treatments had to be extracted through manual curation.

In the current study, the GEO database was queried to identify cell line experiments testing hypoxic conditions at different time points and oxygen concentrations. Other exogenous conditions in addition to hypoxia, such as exposure to a low glucose environment or experiments involving hypoxia mimicking agents were also identified. The following criteria were included in the query.

• Series must contain the word “Hypoxia” and in any of the fields in the Series description
• Experiments must have been performed using human-derived biological samples
• Gene expression profiles were obtained using high-throughput sequencing or arrays

The query identified 204 Series and a total number of 2,134 Samples. Three sequential filtering rules were then applied.

• Rule #1: The first filter was performed on each Series by looking at its experimental description. Series containing at least one in vitro hypoxic sample and one normoxic control were selected leaving 103 Series and 1511 samples after the filtering
• Rule #2: The second filter consisted of removing all Series that had not been published or referenced in any peer-reviewed article, leaving 97 Series and 1423 samples after this step.
• Rule #3: The third filter was performed by manual curation of all the Series and their corresponding papers, retrieving experimental information such as cell lines, oxygen tensions used, etc. Series were excluded if any samples in the Series reported different annotation between the Series information in GEO and the information in the scientific article.

A total number of 73 Series and 767 samples were finally selected. Series used are found in Table S5. Both RNA-seq and microarray gene expression data were used in their post-processed form, as used in their published reference papers. A further normalisation step was included for RNA-seq data to allow comparison of gene expression data across different samples, converting each gene expression value into Transcript Per Million (TPM) using GeoTcgaData.

For each microarray dataset, the corresponding manifest included in the GEO Series Platform was used to intersect the gene IDs from the signatures with their respective probeset IDs available in the platform manifest using a conversion dictionary. For bulk RNAseq data, the gene or transcript ID in the gene expression dataset was used to match the corresponding gene ID from the hypoxia signatures.

After selecting only up-regulated genes from hypoxia gene expression signatures, a final quality check was performed on all the GEO Series. For each dataset, the overlap between genes in the signature and available genes in each series was evaluated. All the platforms and RNAseq data where >20% of the signatures had >20% of genes missing from the dataset were excluded. This was done in order to keep the number of genes as consistent as possible across all the platforms and resulted in the exclusion of three platforms for microarray data and four datasets for bulk RNAseq (Figures S23 and S24).

Single-cell data

MCF7 and HCC1806 cells were maintained as previously described.ref. bib151 Hypoxia was defined as 0.1% oxygen concentration. The choice of hypoxia duration was guided by CA9 expression levels which were measured by FACS and western blotting after 24 h and 72 h of hypoxia. Maximal induction was reached at 72 h for MCF7 and 24 h for HCC1806. Thus, the MCF7 and HCC1806 cells were cultured in hypoxia for these respective durations.

SmartSeq single-cell sequencing was carried out as in previous studies.ref. bib152 Pre-processing was carried out using the following: FastQC for quality control,ref. bib153 Cutadapt for read trimming,ref. bib154 STAR for read alignment,ref. bib155 and featureCounts feature counting.ref. bib156 MultiQC was used for aggregation of all quality control metrics associated with every single step in the pipeline.ref. bib157 Snakemake was the pipeline language used to chain and execute the entire pre-processing pipeline.ref. bib158

The primary analysis pipeline was executed in the R programming language and the analysis was done in the Seurat package for scRNA-Seq analysis.ref. bib159 Dimensionality reduction was applied to facilitate the visualisation of the data and the detection of underlying clusters. The first dimensionality reduction step was principal component analysis (PCA), and the second dimensionality reduction step was Uniform Manifold Approximation and Projection for Dimension Reduction (UMAP).ref. bib160

Clustering analysis was done using the louvain algorithm of community detection. The choice of the number of principal components and clustering resolution parameters was based on elbow plots and clustering trees respectively (generated from the clustree packageref. bib161).

Batch effects are a common issue in scRNA-Seq datasets, and integration methods are commonly used to correct for them.ref. bib162 In the present study, integration methods were avoided due to their tendency to eliminate all biological variability in the data, a well-known limitation of such approaches. Signature quality control metrics, e.g., intra signature correlation, feature expression heatmaps, radar plots, etc., were generated using SigQC.ref. bib44

Clinical data

The TCGA gene expression data was retrieved from the Broad GDAC Firehose. TCGA is a “landmark cancer genomics program”, coordinated by the National Cancer Institute and National Human Genome Research (USA).ref. bib163 The project includes profiling of 20,000 primary cancer samples from 33 cancer types with, in some cases, matched normal samples. The TCGA dataset has been made publicly available in an anonymised fashion for the scientific community. In this work, mRNA expression in TCGA samples across 10 key solid tumors is examined, with a focus on those where at least 30 normal adjacent tissue (NAT) samples were available. The performance of hypoxia signatures was evaluated against these samples. Standard TCGA annotation is used throughout this manuscript, but for clarity here are the acronyms used for tumor types investigated: Breast invasive carcinoma: BRCA, Colon adenocarcinoma: COAD, Head and Neck squamous cell carcinoma: HNSC, Liver hepatocellular carcinoma: LIHC, Lung adenocarcinoma: LUAD, Lung squamous cell carcinoma: LUSC, Prostate adenocarcinoma: PRAD, Stomach adenocarcinoma: STAD, Thyroid carcinoma: THCA, Uterine Corpus Endometrial Carcinoma: UCEC.

Calculation of hypoxia signature summary scores

Gene expression signatures can be reduced to a single measure, which is referred to as a score. This score reflects the status or strength of the phenotype they represent and can be determined through various methods.ref. bib44 With regards to hypoxia signatures, hypoxia scores are single values indicating the level of hypoxia in biological samples. Fourteen scores have been evaluated in this study, including common measures of central tendency such as mean or median, and composite measures for differential gene expression analysis, such as Gene Set Variation Analysis (GSVA) or single-sample Gene Set Enrichment Analysis (ssGSEA). Further information about these scores can be found in Table S4. Scores included in the study are.

• Mean
• Median
• Mode
• Midrange
• Midhinge
• Trimean
• Interquartile Range
• Interquartile Mean
• Mean Absolute Deviation
• Average Absolute Deviation
• Z score
• GSVA
• ssGSEA
• Normalised Cumulative Score

Since the proposed methodology consists of measuring and comparing differences across hypoxic and normoxic samples based on their hypoxia score, only up-regulated genes from gene expression signatures were selected.

The calculation of hypoxia scores was performed using SigScores, an extended version of SigQC,ref. bib44 a previously developed tool by our lab for evaluating the quality of gene expression signatures and calculating gene signature scores. SigScores is an R package that is designed to offer a simple function to compute all available scores. The intention behind its creation is to offer different summary measures in a single software, facilitating effortless computation and comparison of the metrics. Specifically for this study, SigScores was developed and is now accessible via GitHub (https://alebarberis.github.io/sigscores/index.html) and Zenodo (https://doi.org/10.5281/zenodo.14608695). Using the SigScores package, hypoxia scores in both bulk and single-cell data were calculated.

Comparison with random gene signatures (RGS) in bulk gene expression data

The performance of hypoxia signatures was assessed by comparing them to Random Gene Signatures (RGS) of the same length. This evaluation aimed to assess whether the hypoxia signatures significantly outperformed random sets of genes. Similar approaches have been used elsewhere in other contexts.ref. bib164^,ref. bib165 It is particularly important to take such an approach, as previous work has highlighted that unrelated random gene signatures may have been wrongly associated with clinical outcomes.ref. bib166 To avoid this, in this study, we used a permutation-based analysis approach. Each Series was tested separately to avoid experimental biases and batch effects, i.e., samples from GSE15530 were not compared to samples from GSE3188, even if belonging to the same cell line.

The first step in our study was to define how the hypoxia scores can be evaluated in controlled cell line experiments. These scores have been assessed by looking at their value according to the oxygen status of the samples (hypoxic or normoxic). For example, if the scores truly reflect hypoxia, a hypoxic sample would be expected to have a higher hypoxia score than a normoxic sample since downregulated genes have been excluded from each gene expression signature.

Each GEO Series contains multiple hypoxic and normoxic samples, often including replicates. However, most of the samples do not include information on how the replicates were processed (e.g., in parallel, paired or all together). Therefore, to obtain comprehensive insights, we compared all possible pairwise combinations of hypoxic and normoxic samples within the same GEO Series. This strategy enabled the evaluation of every hypoxic/normoxic sample pairing without a priori selection. Thus, each hypoxic sample was matched against every normoxic sample. For instance, a Series comprising two hypoxic and two normoxic samples resulted 4 pairwise combinations (calculated as $\#hypoxicsamples\times \#normoxicsamples$).

One way of measuring how a signature differentiates between pairwise hypoxia/normoxia combinations consists of calculating the absolute value between their two hypoxia scores. Measuring this distance, and evaluating it against RGS, is the core element of the method presented in this study. Given a hypoxia signature and a score method, the Euclidean score distance $d_{SIG}\left(hyp,norm\right)$ is calculated as follows, with $h_{score}$ and $n_{score}$ representing the score of the hypoxic and normoxic samples respectively:

If a hypoxia signature and score combination is predictive of an active hypoxia response, one would expect a large distance $d_{SIG}\left(hyp,norm\right)$. The same approach also is carried out using an RGS and this too gives a score distance, defined as $d_{RGS}\left(hyp,norm\right)$, which is expected to be distributed around zero. Thus, if the hypoxia signature is separating the samples correctly, the $d_{SIG}\left(hyp,norm\right)$ will be greater than $d_{RGS}\left(hyp,norm\right)$ (Figure S25). To further enhance statistical robustness, 1000 different RGS have been used to calculate multiple $d_{RGS}\left(hyp,norm\right)$. These distances from multiple RGS obtained with a permutation test can then be used to derive a null distribution and can be compared to the hypoxia signature $d_{SIG}\left(hyp,norm\right)$ on each individual pair of normoxic and hypoxic samples. This is demonstrated in Figure 2B on one pair of samples: $d_{SIG}$ was compared to the $d_{RGS}$ from 1000 different RGS with the red bar indicating where the the $d_{SIG}$ falls in the distribution. When $d_{SIG}$ registers the highest value within the distribution, it indicates superior performance of the corresponding hypoxia signature and score compared to all evaluated RGS. This can be measured by calculating a significance value ($p$) as below (with $\epsilon =0.0009$ representing the correction factorref. bib167) and this was repeated for all pairwise combinations of hypoxic/normoxic samples in each GEO Series:

The percentile of score is the number of distances (expressed as percentage) in the distribution that are lower than $d_{SIG}$. For example, a percentile of score of 100% means that all the 1000 $d_{RGS}$ are lower than the $d_{SIG}$. However, if a score in the normoxic sample is greater than that in the hypoxic sample using the hypoxia gene expression signature, the significance between the two samples is automatically made non-significant ($p=1$). The correction factor $\epsilon$ is used in the permutation test to prevent a zero p-value, occurring when no RGS outperforms the original signature, reflecting the uncertainty inherent in any statistical test. This correction factor is calculated as the lowest possible $p$ obtainable from the permutation test when only one random signature performs better than the original signature ($1/1001=0.0009$).

The last step of the analysis consists of calculating an accuracy index for each signature as the ratio between the number of significant pairwise combinations (threshold $p\le 0.005$) over the total number of combinations for each cancer type or experimental condition. The p-value corresponds to the probability of rejecting the null hypothesis that RGS perform equally to the hypoxia signature. Therefore, a stringent threshold of $p=0.005$ was chosen in order to allow a maximum of five RGS to outperform the original signature and allow the original signature to still remain significant ($5/1001=0.0049$). The accuracy index is reported as the percentage of correct classifications. For example, if 80 hypoxic/normoxic pairwise combinations out of 100 tested on all the breast cancer cell lines are significant compared to their respective RGS (as described above), the accuracy is stated as 80% in breast cancer. This approach was repeated for each of the fourteen scoring metrics and seventy signatures.

Finally, other common gene expression scores such as the first Principal Component (PC1) from Principal Component Analysis (PCA) or Pathway Level Gene Expression Analysis (PLAGE) were not considered in the analysis. These scores cannot be assessed using the presented method, as there would be no way to determine if the direction of the implied variance using the hypoxia signature is the same as in any RGS.

Comparison with random gene signatures (RGS) in single-cell gene expression data

To assess the efficacy of hypoxia gene expression signatures in single-cell data, an approach analogous to that applied to bulk RNAseq data was adapted. Single-cell data necessitate a modification in the statistical procedure due to the provision of a score for each individual cell as opposed to one for an entire sample. In particular, the following aspects need to be considered.

• Pairwise combinations and P-value calculation: using a pairwise combination approach in single-cell data would be inappropriate, given that each hypoxic cell would be compared with all normoxic cells. This is not only computationally intensive but also potentially misleading, as each cell is likely to manifest a unique gene expression profile. For example, a pair of hypoxic and normoxic samples, each comprising 100 cells, would result in 10,000 pairwise combinations, necessitating approximately 10 million permutations (1000 RGS for each combination) to derive p-values for a single signature and score. Therefore, quantifying the hypoxic response of a single pairwise combination of cells in comparison to all other pairs would be challenging and difficult to interpret.
• Calculation of signatures accuracy: In the context of single-cell data, the determination of accuracy must be performed at the individual cell level, making traditional sample-wide metrics less suitable.

Given these complexities, an alternative approach was employed compared to bulk data analysis.

For each signature and gene expression score, hypoxia scores were calculated for each cell within normoxic and hypoxic samples. These scores were utilised to form two distinct distributions: one for hypoxic cells and another for normoxic cells. The aim is to evaluate whether the score distribution originating from hypoxic cells is statistically different from that derived from normoxic cells. To statistically compare these distributions, the Wasserstein metric was employed as a robust measure of distance between the distributions. Following calculation of the Wasserstein metric for the original hypoxia signature, the same permutation-based analysis was applied to generate a null distribution from 1000 random gene signatures, facilitating statistical comparisons between the original hypoxia signature and RGS. Significance was determined at a p-value <0.005, consistent with the approach for bulk data. The Wasserstein distance metric was computed using the R function wasserstein.test() from the package waddR.ref. bib168

Notably, in contrast to bulk data, where the accuracy of each signature could be calculated, only the p-value was used here to identify signatures with statistically significant differences. This statistical framework offers a solid methodology for assessing hypoxia gene signature performance in single-cell data, taking into account its unique challenges while preserving the essential attributes of single-cell analysis.

Statistical evaluation of tumor VS NAT

It is generally expected that tumor tissue is, on average, more hypoxic than normal tissue due to the imbalance between metabolic demand and the insufficient supply from aberrant blood vessels. Normoxic tumors, in fact, should have hypoxia scores comparable to NAT. In this analysis, we examined differences across solid tumors in TCGA for each signature. The distributions of hypoxia signature scores in cancer tissues were compared to those in NAT using the nonparametric Mann-Whitney U test. An alternative hypothesis was defined based on the assumption that the average hypoxia score in cancer samples exceeds that in NAT (but not vice versa). Specifically, let T(u) and N(u) represent the cumulative distribution functions for the distributions underlying x and y, respectively. The hypothesis asserts that the distribution underlying x (tumor hypoxia scores) is stochastically greater than that underlying y (NAT hypoxia scores), i.e., T(u) > N(u) for all u.

For instance, if the hypoxia signature scores in the cancer group are on average lower than those in NAT, the assumption is violated, and the difference is considered non-significant. A significance threshold of p < 7.14E−05 was applied to identify signatures that met the criteria, corrected using the Bonferroni method, accounting for the number of signatures (70) and cancer types (10) evaluated.

Comparison with RGS in clinical samples

A similar approach used for comparing hypoxia signatures in in-vitro samples can be applied to clinical samples. However, instead of evaluating pairs of samples, this analysis focuses on determining whether significant differences exist between the distributions of hypoxia scores in tumors compared to NAT. The analysis involves two main steps, performed for each signature, score, and cancer type.

• The non-parametric Mann-Whitney U test is performed to derive a p-value, assessing whether a statistically significant difference exists in hypoxia signature scores between tumor and NAT samples. This statistical evaluation is repeated on 1000 RGS, and for each, a p-value is calculated as described previously.
• A Signature Performance Index (SPI), expressed as a percentage, is then derived. This is done by calculating how often the p-value from the original hypoxia signature is lower than any of the 1000 RGS p-values. For example, if 990 out of 1000 RGS have a higher p-value than the original hypoxia signature, the SPI will be 99%.

It is noteworthy to mention that the sample size of both tumor and adjacent normal tissues will be the same in each of the 1000 RGS. This approach enables the comparison of 1000 p-values from RGS with the p-value from the original signature to derive the SPI and determine if the hypoxia signature is statistically significant compared to the RGS.

Survival analysis

Pan-cancer prognostic performance was investigated by using the Kaplan-Meier (KM) estimator at five years in TCGA data followed by a multivariate analysis using the Cox Proportional Hazard (CPH) model. The KM estimator is a non-parametric statistic used to derive the survival function from lifetime data. A total of 5221 out of 5407 samples were included in the prognostic analysis using Disease-Specific Survival (DSS), with 186 samples being excluded due to missing DSS information (Table S10). A point of contention in the field is what type of thresholds should be used across tumor types to denote low and high hypoxic samples. Previous works have used above and below the median, exemplar ref. ref. bib66, however this is not necessarily the most comprehensive approach. Thus, to be more thorough we investigated the prognostic ability of the signature/score combinations comparing every fifth percentile using DSS, as different tumor types have different degrees of hypoxia. The log-rank test was used to compare the survival distribution between samples and the KM estimator was calculated using the Python module lifelines v.0.26.4. A signature/score at a percentile was defined as significantly prognostic if the log-rank p ≤ 0.005, threshold Bonferroni-corrected by the 10 incremental thresholds tested starting from the 50th percentile, with an incremental step of 5 (e.g., 50th, 55th, 60th, until 95th).

Confirmatory CPH analyses were conducted using R (version 4.3.0), with the proportional hazards assumption tested via Schoenfeld residuals. Observation time was measured until the date of death or the end of the five-year monitoring period. Hypoxia signature/score combinations of interest were analyzed by dividing them into five quantiles in multivariable-adjusted models. These models were adjusted for: age (in 10-year age groups), stage, grade, gender (sex), ethnicity, and smoking history for all cancers. HPV status and alcohol consumption was included for HNSC, eGFR/ALK mutation status for LUAD, and Hepatitis B, Hepatitis C, non-alcoholic fatty liver disease, haemochromatosis, alpha-1-antitrypsin deficiency, alcohol consumption and vascular invasion for LIHC.

References

1. A.L. Harris. Hypoxia — a key regulatory factor in tumour growth. Nat. Rev. Cancer, 2002. [PubMed]
2. V. Petrova, M. Annicchiarico-Petruzzelli, G. Melino, I. Amelio. The hypoxic tumour microenvironment. Oncogenesis, 2018. [PubMed]
3. D.M. Brizel, R.K. Dodge, R.W. Clough, M.W. Dewhirst. Oxygenation of head and neck cancer: changes during radiotherapy and impact on treatment outcome. Radiother. Oncol., 1999. [PubMed]
4. M. Nordsmark, J. Overgaard. Tumor hypoxia is independent of hemoglobin and prognostic for loco-regional tumor control after primary radiotherapy in advanced head and neck cancer. Acta Oncol., 2004. [PubMed]
5. K.R. Luoto, R. Kumareswaran, R.G. Bristow. Tumor hypoxia as a driving force in genetic instability. Genome Integr., 2013. [PubMed]
6. M.Z. Noman, M. Hasmim, Y. Messai, S. Terry, C. Kieda, B. Janji, S. Chouaib. Hypoxia: a key player in antitumor immune response. A Review in the Theme: Cellular Responses to Hypoxia. Am. J. Physiol. Cell Physiol., 2015. [PubMed]
7. A. Mohyeldin, T. Garzón-Muvdi, A. Quiñones-Hinojosa. Oxygen in stem cell biology: a critical component of the stem cell niche. Cell Stem Cell, 2010. [PubMed]
8. E.B. Rankin, A.J. Giaccia. Hypoxic control of metastasis. Science, 2016. [PubMed]
9. J. Tutzauer, M. Sjöström, E. Holmberg, P. Karlsson, F. Killander, L.M.F. Leeb-Lundberg, P. Malmström, E. Niméus, M. Fernö, A. Jögi. Breast cancer hypoxia in relation to prognosis and benefit from radiotherapy after breast-conserving surgery in a large, randomised trial with long-term follow-up. Br. J. Cancer, 2022. [PubMed]
10. V.E. Theodoropoulos, A.C. Lazaris, F. Sofras, I. Gerzelis, V. Tsoukala, I. Ghikonti, K. Manikas, I. Kastriotis. Hypoxia-Inducible Factor 1a Expression Correlates with Angiogenesis and Unfavorable Prognosis in Bladder Cancer. Eur. Urol., 2004. [PubMed]
11. M. Domènech, A. Hernández, A. Plaja, E. Martínez-Balibrea, C. Balañà. Hypoxia: The Cornerstone of Glioblastoma. Int. J. Mol. Sci., 2021. [PubMed]
12. E.A. Griffiths, S.A. Pritchard, I.M. Welch, P.M. Price, C.M. West. Is the hypoxia-inducible factor pathway important in gastric cancer?. Eur. J. Cancer, 2005. [PubMed]
13. D.M. Brizel, G.S. Sibley, L.R. Prosnitz, R.L. Scher, M.W. Dewhirst. Tumor hypoxia adversely affects the prognosis of carcinoma of the head and neck. Int. J. Radiat. Oncol. Biol. Phys., 1997. [PubMed]
14. Z.-L. Xiang, Z.C. Zeng, J. Fan, Z.Y. Tang, H.Y. Zeng, D.M. Gao. Gene expression profiling of fixed tissues identified hypoxia-inducible factor-1α, VEGF, and matrix metalloproteinase-2 as biomarkers of lymph node metastasis in hepatocellular carcinoma. Clin. Cancer Res., 2011. [PubMed]
15. J.-J. Hung, M.H. Yang, H.S. Hsu, W.H. Hsu, J.S. Liu, K.J. Wu. Prognostic significance of hypoxia-inducible factor-1 , TWIST1 and Snail expression in resectable non-small cell lung cancer. Thorax, 2009. [PubMed]
16. N. Tanaka, H. Kato, T. Inose, H. Kimura, A. Faried, M. Sohda, M. Nakajima, Y. Fukai, T. Miyazaki, N. Masuda. Expression of carbonic anhydrase 9, a potential intrinsic marker of hypoxia, is associated with poor prognosis in oesophageal squamous cell carcinoma. Br. J. Cancer, 2008. [PubMed]
17. M. Milosevic, P. Warde, C. Ménard, P. Chung, A. Toi, A. Ishkanian, M. McLean, M. Pintilie, J. Sykes, M. Gospodarowicz. Tumor hypoxia predicts biochemical failure following radiotherapy for clinically localized prostate cancer. Clin. Cancer Res., 2012. [PubMed]
18. J. Overgaard, H.S. Hansen, M. Overgaard, L. Bastholt, A. Berthelsen, L. Specht, B. Lindeløv, K. Jørgensen. A randomized double-blind phase III study of nimorazole as a hypoxic radiosensitizer of primary radiotherapy in supraglottic larynx and pharynx carcinoma. Results of the Danish Head and Neck Cancer Study (DAHANCA) Protocol 5-85. Radiother. Oncol., 1998. [PubMed]
19. N. Vilaplana-Lopera, M. Besh, E.J. Moon. Targeting Hypoxia: Revival of Old Remedies. Biomolecules, 2021
20. D. Rischin, L. Peters, R. Hicks, P. Hughes, R. Fisher, R. Hart, M. Sexton, I. D’Costa, R. von Roemeling. Phase I trial of concurrent tirapazamine, cisplatin, and radiotherapy in patients with advanced head and neck cancer. J. Clin. Oncol., 2001. [PubMed]
21. C. Aghajanian, C. Brown, C. O’flaherty, A. Fleischauer, J. Curtin, R. Roemeling, D.R. Spriggs. Phase I study of tirapazamine and cisplatin in patients with recurrent cervical cancer. Gynecol. Oncol., 1997. [PubMed]
22. P.S. Craighead, R. Pearcey, G. Stuart. A phase I/II evaluation of tirapazamine administered intravenously concurrent with cisplatin and radiotherapy in women with locally advanced cervical cancer. Int. J. Radiat. Oncol. Biol. Phys., 2000. [PubMed]
23. Q.T. Le, J. McCoy, S. Williamson, J. Ryu, L.E. Gaspar, M.J. Edelman, S.R. Dakhil, S.D. Sides, J.J. Crowley, D.R. Gandara. Phase I study of tirapazamine plus cisplatin/etoposide and concurrent thoracic radiotherapy in limited-stage small cell lung cancer (S0004): a Southwest Oncology Group study. Clin. Cancer Res., 2004. [PubMed]
24. C.A. Johnson, D. Kilpatrick, R. von Roemeling, C. Langer, M.A. Graham, D. Greenslade, G. Kennedy, E. Keenan, P.J. O’Dwyer. Phase I trial of tirapazamine in combination with cisplatin in a single dose every 3 weeks in patients with solid tumors. J. Clin. Oncol., 1997. [PubMed]
25. S. Senan, R. Rampling, M.A. Graham, P. Wilson, H. Robin, N. Eckardt, N. Lawson, A. McDonald, R. von Roemeling, P. Workman, S.B. Kaye. Phase I and pharmacokinetic study of tirapazamine (SR 4233) administered every three weeks. Clin. Cancer Res., 1997. [PubMed]
26. L.N. Shulman, L. Buswell, N. Riese, N. Doherty, J.S. Loeffler, R.W. von Roemeling, C.N. Coleman. Phase I trial of the hypoxic cell cytotoxin tirapazamine with concurrent radiation therapy in the treatment of refractory solid tumors. Int. J. Radiat. Oncol. Biol. Phys., 1999. [PubMed]
27. A.Y. Bedikian, S.S. Legha, O. Eton, A.C. Buzaid, N. Papadopoulos, S. Coates, T. Simmons, J. Neefe, R. von Roemeling. Phase II trial of tirapazamine combined with cisplatin in chemotherapy of advanced malignant melanoma. Ann. Oncol., 1997. [PubMed]
28. D. Rischin, L. Peters, R. Fisher, A. Macann, J. Denham, M. Poulsen, M. Jackson, L. Kenny, M. Penniment, J. Corry. Tirapazamine, Cisplatin, and Radiation versus Fluorouracil, Cisplatin, and Radiation in patients with locally advanced head and neck cancer: a randomized phase II trial of the Trans-Tasman Radiation Oncology Group (TROG 98.02). J. Clin. Oncol., 2005. [PubMed]
29. D.J. Lee, A. Trotti, S. Spencer, R. Rostock, C. Fisher, R. von Roemeling, E. Harvey, E. Groves. Concurrent tirapazamine and radiotherapy for advanced head and neck carcinomas: a Phase II study. Int. J. Radiat. Oncol. Biol. Phys., 1998. [PubMed]
30. J. Treat, E. Johnson, C. Langer, C. Belani, B. Haynes, R. Greenberg, R. Rodriquez, P. Drobins, W. Miller, L. Meehan. Tirapazamine with cisplatin in patients with advanced non-small-cell lung cancer: a phase II study. J. Clin. Oncol., 1998. [PubMed]
31. M. Reck, J. von Pawel, C. Nimmermann, G. Groth, U. Gatzemeier. Phase II-trial of tirapazamine in combination with cisplatin and gemcitabine in patients with advanced non-small-cell-lung-cancer (NSCLC). Pneumologie, 2004. [PubMed]
32. J. von Pawel, R. von Roemeling, U. Gatzemeier, M. Boyer, L.O. Elisson, P. Clark, D. Talbot, A. Rey, T.W. Butler, V. Hirsh. Tirapazamine plus cisplatin versus cisplatin in advanced non-small-cell lung cancer: A report of the international CATAPULT I study group. Cisplatin and Tirapazamine in Subjects with Advanced Previously Untreated Non-Small-Cell Lung Tumors. J. Clin. Oncol., 2000. [PubMed]
33. F. Shepherd, G. Koschel, J. Von Pawel, U. Gatzmeier, N. Van Zandwiyk, P. Woll, R. Van Klavren, P. Krasko, P. Desimone, M. Nicolson. Comparison of Tirazone (Tirapazamine) and cisplatin vs. etoposide and cisplatin in advanced non-small cell lung cancer (NSCLC): Final results of the international Phase III CATAPULT II Trial. Lung Cancer, 2000
34. L. Spiegelberg, R. Houben, R. Niemans, D. de Ruysscher, A. Yaromina, J. Theys, C.P. Guise, J.B. Smaill, A.V. Patterson, P. Lambin, L.J. Dubois. Hypoxia-activated prodrugs and (lack of) clinical progress: The need for hypoxia-based biomarker patient selection in phase III clinical trials. Clin. Transl. Radiat. Oncol., 2019. [PubMed]
35. B. Harris, S. Saleem, N. Cook, E. Searle. Targeting hypoxia in solid and haematological malignancies. J. Exp. Clin. Cancer Res., 2022
36. E. Anduran, L.J. Dubois, P. Lambin, J.-Y. Winum. Hypoxia-activated prodrug derivatives of anti-cancer drugs: a patent review 2006 – 2021. Expert Opin. Ther. Pat., 2022. [PubMed]
37. C. Wigerup, S. Påhlman, D. Bexell. Therapeutic targeting of hypoxia and hypoxia-inducible factors in cancer. Pharmacol. Ther., 2016. [PubMed]
38. I.N. Misry, M. Thomas, E.D. Calder, S.J. Conway, E.M. Hammond. Clinical Advances of Hypoxia-Activated Prodrugs in Combination With Radiation Therapy. Int. J. Radiat. Oncol. Biol. Phys., 2017. [PubMed]
39. C. Bernauer, Y.K.S. Man, J.C. Chisholm, E.Y. Lepicard, S.P. Robinson, J.M. Shipley. Hypoxia and its therapeutic possibilities in paediatric cancers. Br. J. Cancer, 2021. [PubMed]
40. W.R. Wilson, M.P. Hay. Targeting hypoxia in cancer therapy. Nat. Rev. Cancer, 2011. [PubMed]
41. B.H.L. Harris, A. Barberis, C.M.L. West, F.M. Buffa. Gene Expression Signatures as Biomarkers of Tumour Hypoxia. Clin. Oncol., 2015
42. V. Bhandari, C. Hoey, L.Y. Liu, E. Lalonde, J. Ray, J. Livingstone, R. Lesurf, Y.J. Shiah, T. Vujcic, X. Huang. Molecular landmarks of tumor hypoxia across cancer types. Nat. Genet., 2019. [PubMed]
43. N.S. Fox, M.H.W. Starmans, S. Haider, P. Lambin, P.C. Boutros. Ensemble analyses improve signatures of tumour hypoxia and reveal inter-platform differences. BMC Bioinf., 2014
44. A. Dhawan, A. Barberis, W.C. Cheng, E. Domingo, C. West, T. Maughan, J.G. Scott, A.L. Harris, F.M. Buffa. Guidelines for using sigQC for systematic evaluation of gene signatures. Nat. Protoc., 2019. [PubMed]
45. D. Venet, J.E. Dumont, V. Detours. Most Random Gene Expression Signatures Are Significantly Associated with Breast Cancer Outcome. PLoS Comput. Biol., 2011
46. Y. Shimoni. Association between expression of random gene sets and survival is evident in multiple cancer types and may be explained by sub-classification. PLoS Comput. Biol., 2018
47. K. Manjang, S. Tripathi, O. Yli-Harja, M. Dehmer, G. Glazko, F. Emmert-Streib. Prognostic gene expression signatures of breast cancer are lacking a sensible biological meaning. Sci. Rep., 2021. [PubMed]
48. M.H.W. Starmans, K.C. Chu, S. Haider, F. Nguyen, R. Seigneuric, M.G. Magagnin, M. Koritzinsky, A. Kasprzyk, P.C. Boutros, B.G. Wouters, P. Lambin. The prognostic value of temporal in vitro and in vivo derived hypoxia gene-expression signatures in breast cancer. Radiother. Oncol., 2012. [PubMed]
49. X. Sun, H. Luo, C. Han, Y. Zhang, C. Yan. Identification of a Hypoxia-Related Molecular Classification and Hypoxic Tumor Microenvironment Signature for Predicting the Prognosis of Patients with Triple-Negative Breast Cancer. Front. Oncol., 2021
50. Y. Xu, C. Cao, Z. Zhu, Y. Wang, Y. Tan, X. Xu. Novel Hypoxia-Associated Gene Signature Depicts Tumor Immune Microenvironment and Predicts Prognosis of Colon Cancer Patients. Front. Genet., 2022
51. B. Zhang, B. Tang, J. Gao, J. Li, L. Kong, L. Qin. A hypoxia-related signature for clinically predicting diagnosis, prognosis and immune microenvironment of hepatocellular carcinoma patients. J. Transl. Med., 2020
52. A.C. Koong, N.C. Denko, K.M. Hudson, C. Schindler, L. Swiersz, C. Koch, S. Evans, H. Ibrahim, Q.T. Le, D.J. Terris, A.J. Giaccia. Candidate genes for the hypoxic tumor phenotype. Cancer Res., 2000. [PubMed]
53. A.B. Scandurro, C.W. Weldon, Y.G. Figueroa, J. Alam, B.S. Beckman. Gene microarray analysis reveals a novel hypoxia signal transduction pathway in human hepatocellular carcinoma cells. Int. J. Oncol., 2001. [PubMed]
54. C.C. Wykoff, C.W. Pugh, P.H. Maxwell, A.L. Harris, P.J. Ratcliffe. Identification of novel hypoxia dependent and independent target genes of the von Hippel-Lindau (VHL) tumour suppressor by mRNA differential expression profiling. Oncogene, 2000. [PubMed]
55. N. Denko, C. Schindler, A. Koong, K. Laderoute, C. Green, A. Giaccia. Epigenetic Regulation of Gene Expression in Cervical Cancer Cells by the Tumor Microenvironment. Clin. Cancer Res., 2000. [PubMed]
56. N.C. Denko, L.A. Fontana, K.M. Hudson, P.D. Sutphin, S. Raychaudhuri, R. Altman, A.J. Giaccia. Investigating hypoxic tumor physiology through gene expression patterns. Oncogene, 2003. [PubMed]
57. A. Jögi, J. Vallon-Christersson, L. Holmquist, H. Axelson, A. Borg, S. Påhlman. Human neuroblastoma cells exposed to hypoxia: induction of genes associated with growth, survival, and aggressive behavior. Exp. Cell Res., 2004. [PubMed]
58. W. Ning, T.J. Chu, C.J. Li, A.M.K. Choi, D.G. Peters. Genome-wide analysis of the endothelial transcriptome under short-term chronic hypoxia. Physiol. Genomics, 2004. [PubMed]
59. D.J. Manalo, A. Rowan, T. Lavoie, L. Natarajan, B.D. Kelly, S.Q. Ye, J.G.N. Garcia, G.L. Semenza. Transcriptional regulation of vascular endothelial cell responses to hypoxia by HIF-1. Blood, 2005. [PubMed]
60. V. Wang, D.A. Davis, M. Haque, L.E. Huang, R. Yarchoan. Differential gene up-regulation by hypoxia-inducible factor-1alpha and hypoxia-inducible factor-2alpha in HEK293T cells. Cancer Res., 2005. [PubMed]
61. K.Y. Detwiller, N.T. Fernando, N.H. Segal, S.W. Ryeom, P.A. D’Amore, S.S. Yoon. Analysis of hypoxia-related gene expression in sarcomas and effect of hypoxia on RNA interference of vascular endothelial cell growth factor A. Cancer Res., 2005. [PubMed]
62. J.-T. Chi, Z. Wang, D.S.A. Nuyten, E.H. Rodriguez, M.E. Schaner, A. Salim, Y. Wang, G.B. Kristensen, A. Helland, A.L. Børresen-Dale. Gene expression programs in response to hypoxia: cell type specificity and prognostic significance in human cancers. PLoS Med., 2006
63. S.M. Mense, A. Sengupta, M. Zhou, C. Lan, G. Bentsman, D.J. Volsky, L. Zhang. Gene expression profiling reveals the profound upregulation of hypoxia-responsive genes in primary human astrocytes. Physiol. Genomics, 2006. [PubMed]
64. G.P. Elvidge, L. Glenny, R.J. Appelhoff, P.J. Ratcliffe, J. Ragoussis, J.M. Gleadle. Concordant regulation of gene expression by hypoxia and 2-oxoglutarate-dependent dioxygenase inhibition: the role of HIF-1alpha, HIF-2alpha, and other pathways. J. Biol. Chem., 2006. [PubMed]
65. D.G. Peters, W. Ning, T.J. Chu, C.J. Li, A.M.K. Choi. Comparative SAGE analysis of the response to hypoxia in human pulmonary and aortic endothelial cells. Physiol. Genomics, 2006. [PubMed]
66. O. Aprelikova, M. Wood, S. Tackett, G.V.R. Chandramouli, J.C. Barrett. Role of ETS transcription factors in the hypoxia-inducible factor-2 target gene selection. Cancer Res., 2006. [PubMed]
67. M.C. Bosco, M. Puppo, C. Santangelo, L. Anfosso, U. Pfeffer, P. Fardin, F. Battaglia, L. Varesio. Hypoxia modifies the transcriptome of primary human monocytes: modulation of novel immune-related genes and identification of CC-chemokine ligand 20 as a new hypoxia-inducible gene. J. Immunol., 2006. [PubMed]
68. Y.-F. Shi, C.C. Fong, Q. Zhang, P.Y. Cheung, C.H. Tzang, R.S.S. Wu, M. Yang. Hypoxia induces the activation of human hepatic stellate cells LX-2 through TGF-beta signaling pathway. FEBS Lett., 2007. [PubMed]
69. F.L. Sung, E.P. Hui, Q. Tao, H. Li, N.B.Y. Tsui, Y.M.D. Lo, B.B.Y. Ma, K.F. To, A.L. Harris, A.T.C. Chan. Genome-wide expression analysis using microarray identified complex signaling pathways modulated by hypoxia in nasopharyngeal carcinoma. Cancer Lett., 2007. [PubMed]
70. S.C. Winter, F.M. Buffa, P. Silva, C. Miller, H.R. Valentine, H. Turley, K.A. Shah, G.J. Cox, R.J. Corbridge, J.J. Homer. Relation of a hypoxia metagene derived from head and neck cancer to prognosis of multiple cancers. Cancer Res., 2007. [PubMed]
71. R. Seigneuric, M.H.W. Starmans, G. Fung, B. Krishnapuram, D.S.A. Nuyten, A. van Erk, M.G. Magagnin, K.M. Rouschop, S. Krishnan, R.B. Rao. Impact of supervised gene signatures of early hypoxia on patient survival. Radiother. Oncol., 2007. [PubMed]
72. S. Beyer, M.M. Kristensen, K.S. Jensen, J.V. Johansen, P. Staller. The histone demethylases JMJD1A and JMJD2B are transcriptional targets of hypoxia-inducible factor HIF. J. Biol. Chem., 2008. [PubMed]
73. Z. Hu, C. Fan, C. Livasy, X. He, D.S. Oh, M.G. Ewend, L.A. Carey, S. Subramanian, R. West, F. Ikpatt. A compact VEGF signature associated with distant metastases and poor outcomes. BMC Med., 2009. [PubMed]
74. Y. Benita, H. Kikuchi, A.D. Smith, M.Q. Zhang, D.C. Chung, R.J. Xavier. An integrative genomics approach identifies Hypoxia Inducible Factor-1 (HIF-1)-target genes that form the core response to hypoxia. Nucleic Acids Res., 2009. [PubMed]
75. P. Fardin, A. Barla, S. Mosci, L. Rosasco, A. Verri, L. Varesio. The l1-l2 regularization framework unmasks the hypoxia signature hidden in the transcriptome of a set of heterogeneous neuroblastoma cell lines. BMC Genom., 2009
76. U. Lendahl, K.L. Lee, H. Yang, L. Poellinger. Generating specificity and diversity in the transcriptional response to hypoxia. Nat. Rev. Genet., 2009. [PubMed]
77. F.M. Buffa, A.L. Harris, C.M. West, C.J. Miller. Large meta-analysis of multiple cancers reveals a common, compact and highly prognostic hypoxia metagene. Br. J. Cancer, 2010. [PubMed]
78. M.T. Ghorbel, M. Cherif, E. Jenkins, A. Mokhtari, D. Kenny, G.D. Angelini, M. Caputo. Transcriptomic analysis of patients with tetralogy of Fallot reveals the effect of chronic hypoxia on myocardial gene expression. J. Thorac. Cardiovasc. Surg., 2010. [PubMed]
79. B.S. Sørensen, K. Toustrup, M.R. Horsman, J. Overgaard, J. Alsner. Identifying pH independent hypoxia induced genes in human squamous cell carcinomas in vitro. Acta Oncol., 2010. [PubMed]
80. H. van Malenstein, O. Gevaert, L. Libbrecht, A. Daemen, J. Allemeersch, F. Nevens, E. Van Cutsem, D. Cassiman, B. De Moor, C. Verslype, J. van Pelt. A seven-gene set associated with chronic hypoxia of prognostic importance in hepatocellular carcinoma. Clin. Cancer Res., 2010. [PubMed]
81. P. Fardin, A. Barla, S. Mosci, L. Rosasco, A. Verri, R. Versteeg, H.N. Caron, J.J. Molenaar, I. Ora, A. Eva. A biology-driven approach identifies the hypoxia gene signature as a predictor of the outcome of neuroblastoma patients. Mol. Cancer, 2010. [PubMed]
82. Z. Ghazoui, F.M. Buffa, A.K. Dunbier, H. Anderson, T. Dexter, S. Detre, J. Salter, I.E. Smith, A.L. Harris, M. Dowsett. Close and Stable Relationship between Proliferation and a Hypoxia Metagene in Aromatase Inhibitor–Treated ER-Positive Breast Cancer. Clin. Cancer Res., 2011. [PubMed]
83. K. Toustrup, B.S. Sørensen, M. Nordsmark, M. Busk, C. Wiuf, J. Alsner, J. Overgaard. Development of a hypoxia gene expression classifier with predictive impact for hypoxic modification of radiotherapy in head and neck cancer. Cancer Res., 2011. [PubMed]
84. C. Halle, E. Andersen, M. Lando, E.K. Aarnes, G. Hasvold, M. Holden, R.G. Syljuåsen, K. Sundfør, G.B. Kristensen, R. Holm. Hypoxia-induced gene expression in chemoradioresistant cervical cancer revealed by dynamic contrast-enhanced MRI. Cancer Res., 2012. [PubMed]
85. A. Eustace, N. Mani, P.N. Span, J.J. Irlam, J. Taylor, G.N.J. Betts, H. Denley, C.J. Miller, J.J. Homer, A.M. Rojas. A 26-gene hypoxia signature predicts benefit from hypoxia-modifying therapy in laryngeal cancer but not bladder cancer. Clin. Cancer Res., 2013. [PubMed]
86. R. Boidot, S. Branders, T. Helleputte, L.I. Rubio, P. Dupont, O. Feron. A generic cycling hypoxia-derived prognostic gene signature: application to breast cancer profiling. Oncotarget, 2014. [PubMed]
87. H.B. Ragnum, L. Vlatkovic, A.K. Lie, K. Axcrona, C.H. Julin, K.M. Frikstad, K.H. Hole, T. Seierstad, H. Lyng. The tumour hypoxia marker pimonidazole reflects a transcriptional programme associated with aggressive prostate cancer. Br. J. Cancer, 2015. [PubMed]
88. C.S. Fjeldbo, C.H. Julin, M. Lando, M.F. Forsberg, E.K. Aarnes, J. Alsner, G.B. Kristensen, E. Malinen, H. Lyng. Integrative Analysis of DCE-MRI and Gene Expression Profiles in Construction of a Gene Classifier for Assessment of Hypoxia-Related Risk of Chemoradiotherapy Failure in Cervical Cancer. Clin. Cancer Res., 2016. [PubMed]
89. Y.-E. Suh, K. Lawler, R. Henley-Smith, L. Pike, R. Leek, S. Barrington, E.W. Odell, T. Ng, F. Pezzella, T. Guerrero-Urbano, M. Tavassoli. Association between hypoxic volume and underlying hypoxia-induced gene expression in oropharyngeal squamous cell carcinoma. Br. J. Cancer, 2017. [PubMed]
90. L. Yang, J. Taylor, A. Eustace, J.J. Irlam, H. Denley, P.J. Hoskin, J. Alsner, F.M. Buffa, A.L. Harris, A. Choudhury, C.M.L. West. A Gene Signature for Selecting Benefit from Hypoxia Modification of Radiotherapy for High-Risk Bladder Cancer Patients. Clin. Cancer Res., 2017. [PubMed]
91. I.C. Ye, E.J. Fertig, J.W. DiGiacomo, M. Considine, I. Godet, D.M. Gilkes. Molecular Portrait of Hypoxia in Breast Cancer: A Prognostic Signature and Novel HIF-Regulated Genes. Mol. Cancer Res., 2018. [PubMed]
92. L. Yang, D. Roberts, M. Takhar, N. Erho, B.A. Bibby, N. Thiruthaneeswaran, V. Bhandari, W.C. Cheng, S. Haider, A.M. McCorry. Development and Validation of a 28-gene Hypoxia-related Prognostic Signature for Localized. Prostate Cancer. EBioMedicine, 2018. [PubMed]
93. L. Yang, L. Forker, J.J. Irlam, N. Pillay, A. Choudhury, C.M.L. West. Validation of a hypoxia related gene signature in multiple soft tissue sarcoma cohorts. Oncotarget, 2018. [PubMed]
94. T.P. Dao, S. Rösch, H. Mairbäurl, S. Pusch, A. Unterberg, C. Herold-Mende, R. Warta. Identification of a Prognostic Hypoxia-Associated Gene Set in IDH-Mutant Glioma. Int. J. Mol. Sci., 2018. [PubMed]
95. Y.L. Chen, Y. Zhang, J. Wang, N. Chen, W. Fang, J. Zhong, Y. Liu, R. Qin, X. Yu, Z. Sun, F. Gao. A 17 gene panel for non-small-cell lung cancer prognosis identified through integrative epigenomic-transcriptomic analyses of hypoxia-induced epithelial-mesenchymal transition. Mol. Oncol., 2019. [PubMed]
96. Y.F. Zou, Y.M. Rong, Y.X. Tan, J. Xiao, Z.L. Yu, Y.F. Chen, J. Ke, C.H. Li, X. Chen, X.J. Wu. A signature of hypoxia-related factors reveals functional dysregulation and robustly predicts clinical outcomes in stage I/II colorectal cancer patients. Cancer Cell Int., 2019
97. J. Wang, Y. Wang, P. Xing, Q. Liu, C. Zhang, Y. Sui, C. Wu. Development and validation of a hypoxia-related prognostic signature for breast cancer. Oncol. Lett., 2020. [PubMed]
98. Y. Shou, L. Yang, Y. Yang, X. Zhu, F. Li, J. Xu. Identification of Signatures of Prognosis Prediction for Melanoma Using a Hypoxia Score. Front. Genet., 2020
99. W. Lin, S. Wu, X. Chen, Y. Ye, Y. Weng, Y. Pan, Z. Chen, L. Chen, X. Qiu, S. Qiu. Characterization of Hypoxia Signature to Evaluate the Tumor Immune Microenvironment and Predict Prognosis in Glioma Groups. Front. Oncol., 2020
100. Z. Mo, L. Yu, Z. Cao, H. Hu, S. Luo, S. Zhang. Identification of a Hypoxia-Associated Signature for Lung Adenocarcinoma. Front. Genet., 2020
101. J. Sun, T. Zhao, D. Zhao, X. Qi, X. Bao, R. Shi, C. Su. Development and validation of a hypoxia-related gene signature to predict overall survival in early-stage lung adenocarcinoma patients. Ther. Adv. Med. Oncol., 2020
102. T.M. Calvo, E. Marinari, D. Migliorini, V. Bes, S. Tankov, E. Charrier, T.A. McKee, V. Dutoit, P.Y. Dietrich, E. Cosset, P.R. Walker. An Experimentally Defined Hypoxia Gene Signature in Glioblastoma and Its Modulation by Metformin. Biology, 2020. [PubMed]
103. L. Puente-Santamaría, L. Sanchez-Gonzalez, R. Ramos-Ruiz, L. Del Peso. Hypoxia classifier for transcriptome datasets. BMC Bioinf., 2022
104. K. Wang, Y. Lu, Z. Liu, M. Diao, L. Yang. Establishment and External Validation of a Hypoxia-Derived Gene Signature for Robustly Predicting Prognosis and Therapeutic Responses in Glioblastoma Multiforme. BioMed Res. Int., 2022
105. B. Lane, M.T. Khan, A. Choudhury, A. Salem, C.M.L. West. Development and validation of a hypoxia-associated signature for lung adenocarcinoma. Sci. Rep., 2022. [PubMed]
106. F. Gao, Z. Wang, J. Gu, X. Zhang, H. Wang. A Hypoxia-Associated Prognostic Gene Signature Risk Model and Prognosis Predictors in Gliomas. Front. Oncol., 2021
107. R. Abou Khouzam, S.P. Rao, G.H. Venkatesh, N.A. Zeinelabdin, S. Buart, M. Meylan, M. Nimmakayalu, S. Terry, S. Chouaib. An Eight-Gene Hypoxia Signature Predicts Survival in Pancreatic Cancer and Is Associated With an Immunosuppressed Tumor Microenvironment. Front. Immunol., 2021
108. Y. Shou, L. Yang, Y. Yang, X. Zhu, F. Li, J. Xu. Determination of hypoxia signature to predict prognosis and the tumor immune microenvironment in melanoma. Mol. Omics, 2021. [PubMed]
109. Q. Zhang, L. Qiao, J. Liao, Q. Liu, P. Liu, L. Liu. A novel hypoxia gene signature indicates prognosis and immune microenvironments characters in patients with hepatocellular carcinoma. J. Cell Mol. Med., 2021. [PubMed]
110. R. Shi, X. Bao, K. Unger, J. Sun, S. Lu, F. Manapov, X. Wang, C. Belka, M. Li. Identification and validation of hypoxia-derived gene signatures to predict clinical outcomes and therapeutic responses in stage I lung adenocarcinoma patients. Theranostics, 2021. [PubMed]
111. L. Liu, H. Zhu, P. Wang, S. Wu. Construction of a Six-Gene Prognostic Risk Model Related to Hypoxia and Angiogenesis for Cervical Cancer. Front. Genet., 2022
112. C. Wei, X. Liu, Q. Wang, Q. Li, M. Xie. Identification of Hypoxia Signature to Assess the Tumor Immune Microenvironment and Predict Prognosis in Patients with Ovarian Cancer. Int. J. Endocrinol., 2021
113. X. He, J. Ding, X. Cheng, M. Xiong. Hypoxia-Related Gene-Based Signature Can Evaluate the Tumor Immune Microenvironment and Predict the Prognosis of Colon Adenocarcinoma Patients. Int. J. Gen. Med., 2021. [PubMed]
114. H. Xia, J. Wang, X. Guo, Z. Lv, J. Liu, Q. Yan, M. Liu, J. Wang. Identification of a Hypoxia-Related Gene Signature for Predicting Systemic Metastasis in Prostate Cancer. Front. Cell Dev. Biol., 2021
115. Q. Li, L. Jin, M. Jin. Novel Hypoxia-Related Gene Signature for Risk Stratification and Prognosis in Hepatocellular Carcinoma. Front. Genet., 2021
116. C. Zhao, Y. Zhou, H. Ma, J. Wang, H. Guo, H. Liu. A four-hypoxia-genes-based prognostic signature for oral squamous cell carcinoma. BMC Oral Health, 2021. [PubMed]
117. O. Lombardi, R. Li, S. Halim, H. Choudhry, P.J. Ratcliffe, D.R. Mole. Pan-cancer analysis of tissue and single-cell HIF-pathway activation using a conserved gene signature. Cell Rep., 2022
118. W.G. Kaelin. The von Hippel-Lindau tumor suppressor protein and clear cell renal carcinoma. Clin. Cancer Res., 2007. [PubMed]
119. T. Yorimitsu, U. Nair, Z. Yang, D.J. Klionsky. Endoplasmic reticulum stress triggers autophagy. J. Biol. Chem., 2006. [PubMed]
120. S.B. Catrina, X. Zheng. Hypoxia and hypoxia-inducible factors in diabetes and its complications. Diabetologia, 2021. [PubMed]
121. E.L. LaGory, A.J. Giaccia. Long-range hypoxia signaling in NAFLD. Nat. Med., 2017. [PubMed]
122. L.L. Pinilla, A. Ugun-Klusek, S. Rutella, L.A. De Girolamo. Hypoxia Signaling in Parkinson’s Disease: There Is Use in Asking ‘What HIF?. Biology, 2021
123. S. Ramljak, H. Herlyn, I. Zerr. Cellular Prion Protein (PrPc) and Hypoxia: True to Each Other in Good Times and in Bad, in Sickness, and in Health. Front. Cell. Neurosci., 2016
124. T. Rzymski, M. Milani, L. Pike, F. Buffa, H.R. Mellor, L. Winchester, I. Pires, E. Hammond, I. Ragoussis, A.L. Harris. Regulation of autophagy by ATF4 in response to severe hypoxia. Oncogene, 2010. [PubMed]
125. K. Leithner, C. Wohlkoenig, E. Stacher, J. Lindenmann, N.A. Hofmann, B. Gallé, C. Guelly, F. Quehenberger, P. Stiegler, F.M. Smolle-Jüttner. Hypoxia increases membrane metallo-endopeptidase expression in a novel lung cancer ex vivo model – role of tumor stroma cells. BMC Cancer, 2014. [PubMed]
126. D. Lähnemann, J. Köster, E. Szczurek, D.J. McCarthy, S.C. Hicks, M.D. Robinson, C.A. Vallejos, K.R. Campbell, N. Beerenwinkel, A. Mahfouz. Eleven grand challenges in single-cell data science. Genome Biol., 2020
127. G.L. Semenza. Defining the role of hypoxia-inducible factor 1 in cancer biology and therapeutics. Oncogene, 2010. [PubMed]
128. X. Jing, F. Yang, C. Shao, K. Wei, M. Xie, H. Shen, Y. Shu. Role of hypoxia in cancer therapy by regulating the tumor microenvironment. Mol. Cancer, 2019
129. D.J. Thomson, N.J. Slevin, H. Baines, G. Betts, S. Bolton, M. Evans, K. Garcez, J. Irlam, L. Lee, N. Melillo. Randomized Phase 3 Trial of the Hypoxia Modifier Nimorazole Added to Radiation Therapy With Benefit Assessed in Hypoxic Head and Neck Cancers Determined Using a Gene Signature (NIMRAD). Int. J. Radiat. Oncol. Biol. Phys., 2024. [PubMed]
130. J.A. Sparano, R.J. Gray, P.M. Ravdin, D.F. Makower, K.I. Pritchard, K.S. Albain, D.F. Hayes, C.E. Geyer, E.C. Dees, M.P. Goetz. Clinical and Genomic Risk to Guide the Use of Adjuvant Therapy for Breast Cancer. N. Engl. J. Med., 2019. [PubMed]
131. M. Piccart, L.J. van ‘t Veer, C. Poncet, J.M.N. Lopes Cardozo, S. Delaloge, J.Y. Pierga, P. Vuylsteke, E. Brain, S. Vrijaldenhoven, P.A. Neijenhuis. 70-gene signature as an aid for treatment decisions in early breast cancer: updated results of the phase 3 randomised MINDACT trial with an exploratory analysis by age. Lancet Oncol., 2021. [PubMed]
132. J. Liu, T. Lichtenberg, K.A. Hoadley, L.M. Poisson, A.J. Lazar, A.D. Cherniack, A.J. Kovatich, C.C. Benz, D.A. Levine, A.V. Lee. An Integrated TCGA Pan-Cancer Clinical Data Resource to Drive High-Quality Survival Outcome Analytics. Cell, 2018. [PubMed]
133. K.A. Hoadley, C. Yau, D.M. Wolf, A.D. Cherniack, D. Tamborero, S. Ng, M.D.M. Leiserson, B. Niu, M.D. McLellan, V. Uzunangelov. Multiplatform analysis of 12 cancer types reveals molecular classification within and across tissues of origin. Cell, 2014. [PubMed]
134. D. Huo, H. Hu, S.K. Rhie, E.R. Gamazon, A.D. Cherniack, J. Liu, T.F. Yoshimatsu, J.J. Pitt, K.A. Hoadley, M. Troester. Comparison of Breast Cancer Molecular Features and Survival by African and European Ancestry in The Cancer Genome Atlas. JAMA Oncol., 2017. [PubMed]
135. L.D. Miller, J. Smeds, J. George, V.B. Vega, L. Vergara, A. Ploner, Y. Pawitan, P. Hall, S. Klaar, E.T. Liu, J. Bergh. An expression signature for p53 status in human breast cancer predicts mutation status, transcriptional effects, and patient survival. Proc. Natl. Acad. Sci. USA, 2005. [PubMed]
136. Y. Wang, J.G.M. Klijn, Y. Zhang, A.M. Sieuwerts, M.P. Look, F. Yang, D. Talantov, M. Timmermans, M.E. Meijer-van Gelder, J. Yu. Gene-expression profiles to predict distant metastasis of lymph-node-negative primary breast cancer. Lancet, 2005. [PubMed]
137. M. Kawashima, K. Bensaad, C.E. Zois, A. Barberis, E. Bridges, S. Wigfield, C. Lagerholm, R.I. Dmitriev, M. Tokiwa, M. Toi. Disruption of hypoxia-inducible fatty acid binding protein 7 induces beige fat-like differentiation and thermogenesis in breast cancer cells. Cancer Metab., 2020. [PubMed]
138. S.M. Kyerewah-Kersi, E. Khan, G.H. Venkatesh, R. Abou Khouzam. Hypoxia is Associated with Increased Expression of APOBEC Genes in Breast Cancer. Hamdan Medical Journal, 2023
139. Z. Xiong, W. Huang, W. Zhong, J. Fu, J. Feng, X. Wang, F. Ling. Breast Cancer Subtypes Based on Hypoxia-Related Gene Sets Identify Potential Therapeutic Agents. Front. Mol. Biosci., 2022
140. D.G. Beer, S.L.R. Kardia, C.C. Huang, T.J. Giordano, A.M. Levin, D.E. Misek, L. Lin, G. Chen, T.G. Gharib, D.G. Thomas. Gene-expression profiles predict survival of patients with lung adenocarcinoma. Nat. Med., 2002. [PubMed]
141. H. Zhuang, S. Wang, B. Chen, Z. Zhang, Z. Ma, Z. Li, C. Liu, Z. Zhou, Y. Gong, S. Huang. Prognostic Stratification Based on HIF-1 Signaling for Evaluating Hypoxic Status and Immune Infiltration in Pancreatic Ductal Adenocarcinomas. Front. Immunol., 2021
142. Z. Xu, X.-F. Li, H. Zou, X. Sun, B. Shen. 18F-Fluoromisonidazole in tumor hypoxia imaging. Oncotarget, 2017. [PubMed]
143. F. Mo, D. Lin, M. Takhar, V.R. Ramnarine, X. Dong, R.H. Bell, S.V. Volik, K. Wang, H. Xue, Y. Wang. Stromal Gene Expression is Predictive for Metastatic Primary Prostate Cancer. Eur. Urol., 2018. [PubMed]
144. A. Zare, L.-M. Postovit, J.M. Githaka. Robust inflammatory breast cancer gene signature using nonparametric random forest analysis. Breast Cancer Res., 2021. [PubMed]
145. N. Dawany, L.C. Showe, A.V. Kossenkov, C. Chang, P. Ive, F. Conradie, W. Stevens, I. Sanne, L. Azzoni, L.J. Montaner. Identification of a 251 gene expression signature that can accurately detect M. tuberculosis in patients with and without HIV co-infection. PLoS One, 2014
146. S. Durinck, P.T. Spellman, E. Birney, W. Huber. Mapping identifiers for the integration of genomic datasets with the R/Bioconductor package biomaRt. Nat. Protoc., 2009. [PubMed]
147. 147Multi-symbol checker. https://www.genenames.org/tools/multi-symbol-checker/.
148. G. Stelzer, R. Rosen, I. Plaschkes, S. Zimmerman, M. Twik, S. Fishilevich, T. Iny Stein, R. Nudel, R. Lieder, Y. Mazor. GeneCards – the human gene database (www.genecards.org). Curr. Protoc. Bioinformatics, 2016
149. S.X. Ge, D. Jung, R. Yao. ShinyGO: a graphical gene-set enrichment tool for animals and plants. Bioinformatics, 2020. [PubMed]
150. E. Clough, T. Barrett. The Gene Expression Omnibus Database. Methods Mol. Biol., 2016. [PubMed]
151. I. Ledaki, A. McIntyre, S. Wigfield, F. Buffa, S. McGowan, D. Baban, J.L. Li, A.L. Harris. Carbonic anhydrase IX induction defines a heterogeneous cancer cell response to hypoxia and mediates stem cell-like properties and sensitivity to HDAC inhibition. Oncotarget, 2015. [PubMed]
152. S. Picelli, Å.K. Björklund, O.R. Faridani, S. Sagasser, G. Winberg, R. Sandberg. Smart-seq2 for sensitive full-length transcriptome profiling in single cells. Nat. Methods, 2013. [PubMed]
153. G. De Sena Brandine, A.D. Smith. Falco: high-speed FastQC emulation for quality control of sequencing data. F1000Res., 2019. [PubMed]
154. M. Martin. Cutadapt removes adapter sequences from high-throughput sequencing reads. EMBnet. j., 2011
155. A. Dobin, C.A. Davis, F. Schlesinger, J. Drenkow, C. Zaleski, S. Jha, P. Batut, M. Chaisson, T.R. Gingeras. STAR: ultrafast universal RNA-seq aligner. Bioinformatics, 2013. [PubMed]
156. Y. Liao, G.K. Smyth, W. Shi. featureCounts: an efficient general purpose program for assigning sequence reads to genomic features. Bioinformatics, 2014. [PubMed]
157. P. Ewels, M. Magnusson, S. Lundin, M. Käller. MultiQC: summarize analysis results for multiple tools and samples in a single report. Bioinformatics, 2016. [PubMed]
158. J. Köster, S. Rahmann. Snakemake–a scalable bioinformatics workflow engine. Bioinformatics, 2012
159. Y. Hao, S. Hao, E. Andersen-Nissen, W.M. Mauck, S. Zheng, A. Butler, M.J. Lee, A.J. Wilk, C. Darby, M. Zager. Integrated analysis of multimodal single-cell data. Cell, 2021. [PubMed]
160. L. McInnes, J. Healy, N. Saul, L. Großberger. UMAP: Uniform Manifold Approximation and Projection. J. Open Source Softw., 2018
161. L. Zappia, A. Oshlack. Clustering trees: a visualization for evaluating clusterings at multiple resolutions. GigaScience, 2018
162. H.T.N. Tran, K.S. Ang, M. Chevrier, X. Zhang, N.Y.S. Lee, M. Goh, J. Chen. A benchmark of batch-effect correction methods for single-cell RNA sequencing data. Genome Biol., 2020
163. J.N. Weinstein, E.A. Collisson, G.B. Mills, K.R.M. Shaw, B.A. Ozenberger, K. Ellrott, I. Shmulevich, C. Sander, J.M. Stuart. The Cancer Genome Atlas Pan-Cancer analysis project. Nat. Genet., 2013. [PubMed]
164. P.C. Boutros, S.K. Lau, M. Pintilie, N. Liu, F.A. Shepherd, S.D. Der, M.S. Tsao, L.Z. Penn, I. Jurisica. Prognostic gene signatures for non-small-cell lung cancer. Proc. Natl. Acad. Sci. USA, 2009. [PubMed]
165. M.H.W. Starmans, G. Fung, H. Steck, B.G. Wouters, P. Lambin. A simple but highly effective approach to evaluate the prognostic performance of gene expression signatures. PLoS One, 2011
166. D. Venet, J.E. Dumont, V. Detours. Most random gene expression signatures are significantly associated with breast cancer outcome. PLoS Comput. Biol., 2011
167. T.A. Knijnenburg, L.F.A. Wessels, M.J.T. Reinders, I. Shmulevich. Fewer permutations, more accurate P-values. Bioinformatics, 2009. [PubMed]
168. R. Schefzik, J. Flesch, A. Goncalves. Fast identification of differential distributions in single-cell RNA-sequencing data with waddR. Bioinformatics, 2021. [PubMed]