Can RNA-Seq Analysis Tools Help Identify Problems in the Data?
RNA-Seq analysis tools increasingly make it easy to generate a wide range of results, including PCA plots, heatmaps, differential expression analyses, and enrichment analyses. However, determining whether those results can be trusted requires examining the underlying data.
In an earlier article, Identifying Problems Through Data Visualization, we reanalyzed an RNA-Seq dataset (GSE173789) comparing patients with multiple sclerosis (MS) and healthy controls (HC). We found distortions in the gene count distributions of some samples. These distortions caused many genes to appear relatively upregulated after normalization, and we confirmed that they also affected the results of differential expression analysis. We also identified a subgroup of HC samples with distinct expression patterns.
Can commonly used RNA-Seq analysis tools help researchers identify these underlying patterns and investigate their impact on analysis results? In this series, we use the same dataset to examine whether different tools allow researchers to discover potential problems, investigate their causes, and assess their effects on downstream analyses. Our goal is not to rank analysis tools, but to explore what kinds of analysis environments make it easier for researchers to recognize problems in their data and return to the underlying data to investigate them.
Before continuing, we recommend reviewing the earlier case study or watching the following video from Short Instruction 1 to familiarize yourself with the characteristics of this dataset.
Short Instruction 1: Detecting Potential Outliers with PCA and Confirming Them with Multiple Visualizations
In this article, we use Galaxy, a web-based analysis platform, to examine how a standard RNA-Seq workflow supports quality control and the investigation of potential problems in the data.
We reanalyzed the data from FASTQ files to examine what QC information was available at each step and how easily we could connect it to expression profiles and comparison results. We referred to the Galaxy Training Network (GTN) tutorials RNA-Seq reads to counts and RNA-seq counts to genes.
In Galaxy, attention can shift toward getting the workflow to finish
One thing that stood out while using Galaxy was the amount of work involved in connecting tools. We passed FASTQ files to FastQC, summarized the results with MultiQC, aligned paired-end reads with HISAT2, passed the resulting BAM files to featureCounts, and organized the gene counts into a format suitable for limma.
Small problems arose at several steps. In this analysis, difficulties began with downloading the FASTQ files. When we tried retrieving them from SRA accessions in Galaxy, fasterq-dump failed, and identifying the cause was not straightforward. It was difficult to distinguish a problem with the data from an issue involving NCBI, Galaxy, the network, or a temporary service disruption. We worked around the problem by switching to fastq-dump.
Other tasks included reorganizing collections so that FastQC results could be passed to MultiQC. During alignment, HISAT2 terminated with signal 9 for two samples. We could not resolve the issue and omitted these two samples from subsequent analysis. They were omitted because processing could not be completed, rather than as a result of a sample quality assessment.
Each problem had to be addressed before we could proceed. Our attention therefore turned toward questions such as “Why did this job fail?”, “What format does the next tool require?”, and “How can we get the workflow to finish?” After resolving errors and reaching differential expression analysis, it was easy to feel that the analysis had been successfully completed.
AI assistance made it easier to work through settings and errors. Even so, our attention tended to remain on getting the analysis to run to completion, while the difficulty of interpreting the results remained.
FastQC and MultiQC provide extensive information, but do not establish its impact on the comparison
We first ran FastQC on all FASTQ files and summarized the results with MultiQC. The report provides an overview of read counts, sequence quality, duplication levels, adapter content, and other QC metrics. In this dataset, read counts varied considerably among samples. Some samples also showed duplication levels and adapter content that differed from those of other samples.
We examined these differences carefully because our previous analysis had already revealed associations between QC metrics and expression profiles. Without that prior knowledge, however, it would be difficult to tell from this report how much the differences might affect the final group comparison. High duplication levels do not necessarily indicate a quality problem in RNA-Seq data, and read counts can also vary among samples.
An experienced analyst might identify samples that warrant further investigation and track them through subsequent steps. Without knowing which findings to follow up or how to do so, however, a user might simply note the differences and proceed.
Sample IDs alone make reports harder to interpret
Another observation was how strongly sample naming affected our ability to understand the reports. When we retrieved FASTQ files from SRA in Galaxy, the resulting datasets were named using their SRR accessions. In MultiQC, featureCounts, and the subsequent multidimensional scaling (MDS) plots, identifiers such as SRR14409217 and SRR14409222 did not immediately tell us whether a sample belonged to the multiple sclerosis (MS) or healthy control (HC) group.
A reference table provides that information, but repeatedly consulting it while comparing reports adds to the effort required. This was particularly relevant when examining relationships among read counts, gene count distributions, gene assignment rates, MDS positions, and disease status.
Where possible, it is helpful to retain the original ID while incorporating key experimental conditions into the sample label early in the analysis. For example:
SRR14409217_MS_sample01
Labels like this make it easier to connect samples with their experimental conditions across reports.
However, after retrieving many SRA runs together using tools such as fastq-dump in Galaxy, renaming the resulting datasets individually can be tedious. This experience highlighted the value of organizing sample metadata before analysis and assigning informative labels early.
The featureCounts report also reveals differences among samples
After running featureCounts, we examined gene assignment rates in MultiQC. Most samples had assignment rates of approximately 55–64%, while some had substantially lower rates. However, the relationship between this metric, gene count distributions, and expression profiles still needed to be examined. A low assignment rate alone does not establish how strongly a sample affects the group comparison.
The report therefore contained information worth investigating. Unless that information was connected to the expression data, however, we could simply note the differences in assignment rates and proceed to statistical analysis.
We reached limma-voom before connecting the QC findings to the sample structure
We combined the gene counts obtained with featureCounts and performed differential expression analysis using limma-voom, comparing MS with HC. We filtered low-count genes using CPM values, applied TMM normalization, and used voom to model the relationship between the mean and variance.
The limma report includes an MDS plot, mean–variance trend, model fit diagnostics, MD plots, and a volcano plot. Processing completed, the mean–variance trend showed no obvious irregularities, and the volcano plot displayed many DEG candidates. We could then select DEGs using criteria such as adjusted P-values and proceed to GO or pathway analysis.
The MDS plot appeared to show some separation between MS and HC. Coloring samples only by DiseaseState drew attention to differences between the two groups. At that point, however, we had not established which sample groups contributed most strongly to the separation or how their positions related to the QC differences observed earlier.
Completing the workflow, checking diagnostic plots, and obtaining DEG candidates can create a sense that the results have been adequately assessed. In this analysis, we found that we could reach differential expression analysis without fully understanding the relationship between QC findings and expression profiles.
Viewing the results alongside the original gene counts helps connect the findings
We brought the gene counts obtained with featureCounts and the analysis results together in Subio Platform. Figure 4 shows the same genes and samples before normalization on the left and after normalization on the right. The line graphs above show individual gene expression profiles, while the histograms below show the distribution of values within each sample.
The SampleGroup labels refer to sample groups identified in the earlier case study. The label doubt identifies samples with distributional distortion, while closeToDoubt identifies samples that clustered with them in hierarchical clustering.
The histograms below show that, before normalization, the gene count distributions of these samples are shifted to the left relative to those of other samples. Normalization raises their values, bringing the right-hand peaks into approximate alignment with those of other samples. However, this shift also leaves a gap with no displayed values in the low-value region on the left. Although the peaks align, differences in distribution shape remain, particularly in the low-value region.
This relative lack of low values was also observed in the publicly available CPM data used in the earlier case study. After generating gene counts again from FASTQ files, we observed a similar distributional feature in the corresponding samples after normalization.
The line graphs above show how this distributional change relates to individual genes. The black lines represent a subset of the genes identified as more highly expressed in MS than in HC. For many of these genes, the original counts in the corresponding samples are lower than those in other samples, whereas their normalized values are relatively higher. The upward shift and the gap in the low-value region seen in the histograms correspond to individual genes appearing more highly expressed in these samples after normalization.
Comparing values before and after normalization allows us to examine which measurements from which sample groups contribute to the finding of higher expression in MS. Examining FASTQ read counts, featureCounts assignment rates, original gene count distributions, changes after normalization, and MDS and DEG results together for the same samples is important for correctly interpreting the DEG results.
Galaxy’s QC report also showed that some samples had gene count distributions shifted toward lower counts. However, I think it would be quite difficult to infer from that plot alone that some genes would show relatively higher values in these samples after normalization, contributing to their identification as more highly expressed in MS.
What limma-voom evaluates, and what the analyst must judge
The limma-voom workflow accounts for library sizes and the mean–variance relationship, then evaluates group differences according to the specified experimental design. Interpreting those differences as changes associated with disease also requires considering sample backgrounds, experimental conditions, and the observed expression profiles.
Obtaining an expression matrix through steps such as FastQC, HISAT2, and featureCounts is one achievement. Establishing what the resulting group differences mean for the research question requires further assessment. Completing the statistical analysis and obtaining significant differences do not, by themselves, establish what those differences reflect.
Checking each report separately can make relationships harder to recognize
FastQC provides information about read quality and related metrics. featureCounts assigns reads or fragments to genes, generates counts, and reports assignment statistics. limma performs statistical analysis using an expression matrix and experimental design, and produces diagnostic plots and results. Each step provides information relevant to the analysis.
The GTN tutorials also describe QC using MDS and distribution plots, as well as analyses that account for additional experimental factors. Tools and guidance for further investigation are available. In our analysis, however, connecting the reports still required tracking which samples had low read counts, what gene counts they produced, where they appeared in MDS, and which disease group they belonged to.
Differences whose implications were difficult to assess in individual reports became easier to recognize as a pattern when we examined the same samples across steps. In addition to checking individual QC metrics, we need to connect them with expression data and experimental conditions. This also explains why SRR-only sample names made the reports harder to interpret.
The analyst still has to make decisions
Even when information is brought together, it may not uniquely determine whether a sample should be included or excluded. The significance of read counts, duplication levels, assignment rates, and MDS positions depends on the experimental context and analysis purpose. These features alone do not determine how samples should be handled.
For example, if technical factors are confounded with disease status, expression data alone may not allow their respective contributions to be distinguished. Assessing the cause requires information such as experimental records and sample backgrounds. Nevertheless, even when the cause remains unresolved, we can examine how strongly the comparison results depend on including particular sample groups.
The analyst must use this evidence to judge what each comparison supports and how confidently its results can be interpreted. This is why we need ways to examine read counts, assignment rates, gene count distributions, changes after normalization, MDS and DEG results, sample IDs, and experimental conditions together.
Running a pipeline and analyzing data are different tasks
In Can we use different values for heatmaps and DEG analysis in RNA-Seq?, we discussed the risk of interpreting outputs from different tools as interchangeable views of the same phenomenon without adequately considering the values each tool uses. When the underlying values or transformations differ, we need to understand the distinction between the relationships shown by a visualization and the hypotheses tested by a statistical analysis.
Our experience with Galaxy highlighted the difficulty of connecting information across analysis steps. Reports from FastQC, featureCounts, and limma contained important information. Checking them separately, however, did not make it easy to recognize how the observed sample differences related to subsequent expression profiles and group comparisons.
A shared challenge in these two cases is interpreting information from multiple tools together. Which values were used? Which samples were analyzed? What comparison produced each result? We need to preserve these connections while considering the analysis as a whole.
The same challenge can arise when combining tools and packages in Bioconductor, Python, or other environments. Alongside running a pipeline and resolving settings or errors, analysts need to connect and examine findings across its steps.
Completing a pipeline built from reliable tools does not guarantee that the results have been interpreted appropriately. Bringing together information from multiple steps and interpreting it in the context of the whole analysis is a central responsibility of the analyst.
Series: Can RNA-Seq Analysis Tools Help Identify Problems in the Data?
In this series, we use the same RNA-Seq dataset (GSE173789) to investigate whether different analysis tools can help researchers identify hidden problems, explore their causes, and assess their impact on analysis results.
- Identifying Problems Through Data Visualization — How distorted expression distributions affect analysis results
- RNA-Seq Analysis Using a Standard Galaxy Workflow — Exploring data quality through a standard analysis workflow
- Trying RaNA-Seq — Automated analysis from FASTQ files and evaluation of the results
- Trying RNAseqChef — Extensive analysis features and their limitations for exploratory QC
- Trying iDEP — Can its QC visualizations help identify problems in the data?
- A Third Perspective on Bioinformatics Education — The importance of critically evaluating analysis results in the AI era