Are the values shown in the heatmap really the values used for DEG analysis?
In papers reporting RNA-Seq data analysis, genes identified by DEG (Differentially Expressed Genes) analysis are often visualized in a heatmap. For readers of such papers, it is natural to assume that the heatmap simply presents, in an easier-to-read form, the values that were used to determine differential expression.
However, this is not always the case.
Gene Counts may be used for DEG analysis, while the heatmap may show values converted to gene-wise Z-scores. PCA may be performed using values transformed by VST or rlog, while TPM or FPKM may be used to display expression levels.
In other words, even when everything is presented as part of a single RNA-Seq analysis, the meaning and properties of the values being used may change during the workflow. In such a situation, how can the validity of the analysis be judged from the presented data and figures?
Why does this happen?
Analysts are not intentionally confusing different types of values. In most cases, they are trying to perform each analysis as correctly as possible.
When they look up DEG analysis, they are told to use Gene Counts with edgeR or DESeq2. When they look up PCA, they are advised not to use Gene Counts directly, but instead to transform the data using VST or rlog. When they look up heatmaps, they find methods that convert values to gene-wise Z-scores to make expression patterns easier to see. When comparing expression levels, they may also be told to use TPM or FPKM.
Analysts consult manuals, papers, official documentation, and tutorials, and select methods considered appropriate for each type of analysis. As a result, they may construct a workflow such as the following:
Gene Counts
↓
DEG analysis using edgeR or DESeq2
↓
Transformation using VST or rlog
↓
PCA
↓
Gene-wise Z-score transformation
↓
Heatmap
When each step is considered individually, all of these are commonly recommended methods. However, these methods were developed for different purposes and are explained separately in different manuals and publications. Those individual sources do not necessarily explain how the resulting outputs should ultimately be connected and interpreted as one coherent analysis.
Analysts collect the “correct methods” presented in different places and assemble them into a single workflow. As a result, the values used along the way may change from Gene Counts to VST, rlog, TPM, and Z-scores. Even when analysts believe they are analyzing the same RNA-Seq data consistently, they may in fact be looking at values with different properties at each stage.
This problem does not arise because analysts have failed to make a serious effort to perform the analysis correctly. It can arise precisely because they try to perform each analysis correctly and combine the recommended procedures for each step.
For example, edgeR takes Gene Counts as input and incorporates normalization factors estimated using TMM (trimmed mean of M-values) into the statistical model used for DEG analysis. In contrast, clustering and heatmaps may use values produced through different normalization or data transformation procedures. Therefore, even when all analyses begin with the same Gene Counts, they are not necessarily examining values with the same underlying properties.
What is missing is not only knowledge of how to use each individual tool correctly. What is also needed is a perspective on how those individual forms of correctness should be connected across the analysis as a whole.
The analyst’s job is to connect individual results
RNA-Seq analysis uses many tools and methods, including edgeR, DESeq2, PCA, clustering, heatmaps, GO analysis, and pathway analysis.
However, the analyst’s job is not merely to run them in sequence.
The analyst’s job is to connect the results produced by individual tools
and organize them into one biological interpretation.
To do this, the analyst must understand
which values were used to calculate the PCA,
how the heatmap represents the DEG analysis results,
and what changed as a result of each data transformation.
How should different analysis results be connected to one another?
Judging that relationship is the responsibility of the analyst.
Another dimension of correctness is needed beyond the correctness of individual tools
Analysts have responsibilities beyond running individual tools with the correct inputs and settings. They must ensure that the analysis as a whole has the following properties:
- Validity
- Consistency
- Verifiability
- Reproducibility
Reproducibility means that another analyst can obtain the same result by using the same data and procedures. However, if values with different properties are connected without being examined within the reproduced workflow, the interpretation may still be invalid, even if the same result can be reproduced. For that reason, reproducibility alone is not enough. It also requires the underlying validity, consistency, and verifiability.
Validity means that the selected analysis method is appropriate for the actual characteristics of the data and the purpose of the analysis, and that the interpretation drawn from the result is reasonable.
Consistency means that each analysis result is logically connected to the results that come before and after it.
Verifiability means that it is possible to trace a result or interpretation back to the values and processing steps from which it was produced.
Even when the same procedure can be reproduced, the analysis is not necessarily valid if the method is unsuitable for the data being examined. Likewise, even when each individual processing step is generally valid, the analysis as a whole may lack consistency and verifiability if results derived from different values are connected without being examined.
Only when validity, consistency, verifiability, and reproducibility are all present can confidence in the analysis results be increased.
The analyst’s role is to examine the workflow and reorganize it when necessary
When an analysis is constructed using R scripts, the analyst has the flexibility to extract intermediate data, compare different data representations, and perform the necessary checks.
Of course, using R does not automatically solve the problem. If an analyst merely arranges recommended procedures from manuals in sequence, they may still fail to notice that the data representation has changed or that the relationship between results has become weaker. Nevertheless, the analyst at least has the freedom to inspect intermediate data, compare different data representations, and reorganize the workflow when necessary.
At the same time, the analyst is responsible for using that freedom to maintain the validity, consistency, and verifiability of the analysis as a whole. Simply following the recommended procedure for each individual tool is not enough to fulfill that responsibility. If changes in the values being used make the relationship with earlier results difficult to understand, the transformation or visualization method should be reconsidered, and the workflow should be modified or adjusted when necessary. At the same time, the analyst must ensure that those changes remain within a range that preserves the assumptions and validity of each analytical method.
The data in front of you should take priority over the manual
The highest priority in analysis is not to reproduce the method described in a manual exactly. It is to select and, when necessary, adjust the analysis method according to the characteristics of the actual data.
The recommended procedures presented in manuals and tutorials often assume relatively clean data that satisfy the conditions expected by the method. Actual RNA-Seq data, however, may contain instability in the low-count range, differences in measurement depth or dynamic range between samples, outliers, missing values, and genes that are highly expressed only in particular samples. Applying the recommended procedure without modification does not necessarily produce a valid result for such data.
When the assumptions of the manual differ from the actual data, priority should be given not to the manual, but to the data in front of you. If the properties of the data differ from those assumed by the manual, the analysis method must be modified so that those properties can be handled appropriately. The analysis method should be adapted to the data; the data should not be forced to fit the manual.
For example, in this article on performing RNA-Seq deconvolution analysis with the help of AI, we did not use TPM directly as the input for xCell, even though TPM is generally recommended for RNA-Seq input. This decision was based on an earlier analysis using the same RNA-Seq dataset, which showed that using values in the low-count range without adjustment could cause differences in dynamic range between samples to be detected as if they were biological expression differences. This is an example of maintaining consistency across an analysis by not treating each analysis as an independent task, but instead applying knowledge about the data obtained from an earlier analysis to the choice of method used in the next one.
Differences in values become even harder to see in integrated packages
Many RNA-Seq analysis packages combine multiple tools and algorithms into a single workflow. Analysts can perform DEG analysis, PCA, clustering, and heatmap generation continuously within one interface. Operationally, this can make it appear as though the same data are simply being passed from one analysis to the next.
In reality, however, Gene Counts may be internally transformed into VST or rlog values, and then further converted into Z-scores. Continuity of operation is not the same as consistency of data.
With an R script, it is at least possible to inspect intermediate objects and export values for comparison. In an integrated package, however, the values used internally or the transformed data may not be accessible. In addition, users who do not read the documentation carefully may be led to believe that the analysis is internally consistent.
However, this is not simply a matter of the analyst failing to check carefully. Some packages place strong emphasis on the validity and reproducibility of individual analyses, while giving insufficient consideration to the consistency of values used across analyses or to whether results can be traced back to the original values for verification.
The respective responsibilities of the package and the analyst
Using a package is not itself the problem.
What should be expected of a package is that it clearly shows, without leaving room for misunderstanding, which values were used at each stage of the analysis and how each result was produced from the preceding processing steps.
However, that alone does not guarantee consistency across the entire analysis. It remains the analyst’s responsibility to determine how the results from individual tools should be connected and to verify whether those connections are valid.
Analysts also have a responsibility to determine whether a tool allows them to inspect the intermediate values and processing details they need, and whether that tool should be used for their analysis. If the values and processing steps used internally cannot be examined, the analyst cannot fully verify the relationship between the results. In such cases, it is necessary to reconsider whether the package should be adopted merely because it is easy to operate.
Comparison between a typical integrated package and Subio Platform
The differences between typical integrated packages and Subio Platform can be summarized as follows from the perspective of whether the analyst can maintain consistency and verifiability.
| Comparison item | Typical package integrating multiple tools | Subio Platform |
|---|---|---|
| Values used for analysis | Values may be transformed internally into different representations for different analyses, making it difficult to confirm which values were actually used and when the data representation changed | Visualization and comparative analyses within Subio Platform are generally performed using the same preprocessed and normalized Processed Signal. The display can be switched to the original Ch1 Raw Signal to examine the relationship between values before and after transformation. When testing different preprocessing or normalization methods, the Series can be duplicated and managed as a separate analysis |
| Verification of filtering | It may be difficult to confirm in detail which genes were excluded and under which conditions | Genes matching the filtering conditions can be saved as a Measurement List. Measurement Lists can be organized in a folder structure, allowing the analyst to manage clearly which results were derived from which list |
| Relationship between DEG analysis and visualization | The values used for DEG analysis may differ from those used for visualizations such as heatmaps, without that relationship being made explicit | The DEG analysis classifications can remain displayed while the expression pattern of each gene is examined in Subio Platform. Even when DEG analysis is performed using an external tool, the analyst can visually examine the relationship between the input values used for the analysis and the values used for visualization |
| Comparison and verification against the original values | Only figures or gene lists may be shown, making it difficult to return to the original Gene Counts or other input values and the pre-transformation values to confirm which values produced the result | While keeping the analysis result displayed, the analyst can switch between Processed Signal and the original Ch1 Raw Signal and visually examine the relationship between the classification result and the expression pattern before and after transformation. When preprocessing or normalization is changed, the results can be saved as independent Series and compared before and after the change |
| Integration with R and external tools | Even when R packages are used internally, the input values and intermediate output values may not be available to the user | Raw data such as Gene Counts, or preprocessed and normalized Processed Signal, can be exported for analysis in R. The resulting output can then be displayed in Subio Platform and examined together with the original expression patterns |
| Recording decisions and modifications | Even when analysis procedures and settings are saved automatically, the reason why a particular method was selected is often not recorded | Analytical decisions and processing details can be written as notes, and related files can be saved together. The data, analysis results, notes, and other information can also be saved in an SSA file for sharing with collaborators or for backup. However, the analyst remains responsible for recording the necessary information |
| Responsibility of the analyst | The analyst must not simply accept the presented results, but must examine the values and processing steps used and determine whether the relationship between the results is valid. However, sufficient verification may be difficult when the necessary information is not accessible | Using an environment in which intermediate values and processing results can be examined, the analyst must decide which values and methods to adopt and record the changes and the reasons for them. The ability to examine the analysis does not replace the analyst’s responsibility for judgment |
Some analysis software automatically records who performed which operation and when, in the form of an immutable audit trail. Such systems are important in regulated environments, but because they involve access control, electronic signatures, and system validation, they are generally expensive and complex.
Subio Platform is not a system that automatically generates this kind of audit trail. It allows analysts to inspect intermediate values, compare the results of different processing methods, and save decisions and processing details as notes together with the data. However, the analyst remains responsible for deciding what to record, which methods to select, and how to connect and interpret the results. What Subio Platform provides is an environment in which analysts can inspect, compare, adjust, and document their analysis.
This Is Different from the Consistency Discussed in Workflow Management
In RNA-Seq data analysis, the term “consistency” is also used in the context of workflow management with tools such as Nextflow and Snakemake, as well as the use of Docker, Singularity, and similar technologies to fix the analysis environment. These are important efforts to ensure that an analysis can be performed with the same tools, versions, and parameters, allowing the same results to be obtained regardless of who runs it or when it is run. However, the consistency discussed in this context is, in essence, an extension of reproducibility, because it concerns whether the same analytical procedures can be repeated under the same conditions.
What this article addresses is whether, throughout the entire analysis workflow, it is possible to explain consistently which values each result was derived from and how the different results are related to one another. This kind of consistency across the entire analysis is conceptually different from the consistency traditionally discussed in workflow management. In this article, we refer to this idea as analysis-wide consistency and propose adding it as a new criterion for evaluating RNA-Seq data analysis.
AI-Driven Automated Analysis May Make This Problem Even Harder to See
The automation of RNA-Seq data analysis through AI is likely to accelerate further. As this happens, maintaining awareness of analysis-wide consistency and checking it throughout the entire analysis will become critically important.
This is because the more smoothly AI presents a series of analytical results, the more internally consistent those results may appear. Unless analysts consciously verify which values each result was derived from, examine how the results are related to one another, return to the original data when necessary, and explicitly instruct AI to do the same, they may become overwhelmed by an even greater volume of analytical results than before.
