How thoroughly can we evaluate RNA-Seq results?
In an earlier case study, we reanalyzed an RNA-Seq dataset from patients with multiple sclerosis (MS) and identified substantial differences in gene expression distributions across samples. In a subset of MS samples, many genes showed unusually high expression values, influencing which genes were identified as upregulated differentially expressed genes (DEGs). The healthy control samples also appeared to fall into two groups with distinct expression patterns. With data like these, it is important not to accept DEG results at face value. We need to investigate the differences among samples that may be driving those results.
We previously examined Galaxy and RaNA-Seq using the same dataset, evaluating how well each tool allowed us to identify and investigate these issues. This time, we tested RNAseqChef.
RNAseqChef offers a wide range of analysis features
RNAseqChef is described as offering a wide range of visualization and analysis capabilities. We therefore expected it might allow us to examine results from different perspectives and investigate potential problems in the data. In practice, however, its basic workflow and presentation of results were quite similar to those of RaNA-Seq. After uploading the data, a series of analyses runs automatically, and the results are presented in a report.
RNAseqChef does provide a broad selection of analytical features. However, we did not find an interface that would allow us to move easily between plots and trace unusual results back to particular samples or genes. Having a wide range of analysis and visualization features does not necessarily mean that users can freely explore and validate their results.
Uploading Gene Counts automatically generates analysis results
RNAseqChef requires two input files: a gene count matrix and a text file containing sample metadata. Gene identifiers must be Ensembl IDs or gene symbols; NCBI Gene IDs are not supported. Unlike Galaxy and RaNA-Seq, RNAseqChef does not process FASTQ files. To make the comparison as consistent as possible, we uploaded the Gene Counts generated using Galaxy.
For DEG analysis, RNAseqChef offers DESeq2, edgeR, and EBSeq. Users can also select an FDR adjustment method and set thresholds for fold change, FDR, and baseMean. We ran the analysis using the default settings. RNAseqChef automatically generated PCA plots, clustering results, MA plots, heatmaps, DEG results, and other outputs.
One important difference from our previous evaluations of Galaxy and RaNA-Seq is the starting point of the analysis. Because RNAseqChef begins with Gene Counts, it does not provide upstream QC information, such as total read counts or the number of reads assigned to genes. This is a consequence of the different input requirements.
However, the uploaded Gene Counts should still make it possible to compare gene expression distributions across samples. Unfortunately, we did not find a plot for this purpose in the Pair-wise DEG mode. This leaves out an important source of information needed to investigate the distorted expression distributions observed in this dataset.
PCA and heatmaps reveal the underlying problems in the data
The Result overview in RNAseqChef includes PCA, MDS, hierarchical clustering, MA plots, and heatmaps. These visualizations revealed the same characteristic patterns discussed in our original case study.
In the PCA plot, healthy controls and MS samples did not separate clearly, and several samples lay far from the others. The heatmap also showed relatively high expression values for many genes in a subset of MS samples, while the healthy control samples appeared to form two groups with distinct expression patterns. This combination of patterns was also visible in Figure 4 of the original research paper. In other words, RNAseqChef does reveal clues suggesting that the dataset requires closer examination.
How should we interpret samples that appear as outliers in PCA?
When a sample lies far from the others in a PCA plot, its position alone is not sufficient justification for excluding it from the analysis. PCA cannot tell us whether the difference reflects a technical problem or genuine biological variation.
In this dataset, some MS samples showed distinctive expression patterns. One might interpret these patterns as evidence of different MS subtypes. Since MS is a biologically heterogeneous disease, considering this possibility is not unreasonable.
However, as shown in our original case study, differences in Gene Counts distributions contribute to the observed sample structure. Examining these distributions would help us determine how much of the observed structure may be attributable to distributional distortions and provide a basis for assessing whether a biological subtype interpretation is justified. RNAseqChef, however, does not provide the plots or interactive features needed to carry out this investigation.
The apparent separation of healthy control samples into two groups also deserves attention when evaluating the validity of the results. Whether this pattern raises concerns depends on how the analyst interprets the heatmap, but RNAseqChef does not offer a straightforward way to investigate these samples further.
The plots are largely independent of one another. We cannot select a sample of interest in a PCA plot and directly examine the corresponding data in another visualization. For example, after identifying a sample that lies far from the others, we could not directly examine how its overall gene expression distribution differed from those of the other samples.
Z-score heatmaps obscure differences in the magnitude of variation across genes
RNAseqChef displays its heatmaps using Z-scores. Centering each gene by subtracting its mean expression value is useful because it removes differences in overall expression levels among genes, making it easier to compare expression patterns across samples. Z-score transformation, however, goes further by dividing the centered values by each gene's standard deviation, thereby standardizing the magnitude of variation across genes. (For a more detailed discussion, see Why Z-Score Normalization Should Not Be Used Uncritically in RNA-Seq Heatmaps.)
This is particularly important when distorted expression distributions produce large variation in a subset of samples, as in the present dataset. Such distortions can substantially affect how the results are interpreted. Unfortunately, RNAseqChef uses a fixed Z-score heatmap display, with no option to change the scaling method.
GOI profiling allows us to examine individual gene expression patterns
RNAseqChef includes a feature called GOI profiling, which allows users to examine genes of interest. After entering gene names, users can view their positions on a volcano plot, a heatmap, and box plots showing normalized counts. This is a useful feature. MA plots and volcano plots do not show the variation in expression values among individual samples, whereas box plots allow us to examine this variation directly.
However, when an unusually high or low value appears in a box plot, we cannot identify which sample it represents. Nor could we determine whether that same sample showed similarly unusual values for other genes or whether its overall gene expression distribution differed from those of other samples. Thus, although GOI profiling provides useful information about individual genes, it remains difficult to trace these observations back to potential sample-level problems.
DEG results are automatically used for enrichment analysis
In RNAseqChef, genes identified as DEGs are automatically passed on to enrichment analysis, much as in RaNA-Seq. When DEG identification may be influenced by differences in gene expression distributions across samples, as in this dataset, the workflow proceeds to enrichment analysis before the reliability of the DEG results has been adequately examined. Automatically performing enrichment analysis is convenient, but researchers still need to assess whether the input gene list is appropriate.
On the positive side, RNAseqChef allows users to see which genes contribute to each enriched gene set. Below the enrichment plots, users can scroll horizontally through a results table to reveal the corresponding gene names. This makes it possible to use the browser's Find function to identify genes shared among different gene sets. Being able to go beyond the plots and examine the specific genes contributing to the results is a valuable feature.
Choosing between RaNA-Seq and RNAseqChef
After using RNAseqChef, we found that its basic workflow and presentation of results were very similar to those of RaNA-Seq. Both tools automatically perform standard analyses after data are uploaded, generating PCA, clustering, DEG, enrichment, and other results. Of course, they differ in their individual features and capabilities.
However, for the type of automated analysis examined in this series, the choice may largely depend on the type of input data available. If you have FASTQ files, RaNA-Seq provides a convenient starting point. If you already have Gene Counts, RNAseqChef is another option. For comparisons involving three or more experimental groups, one possible workflow would be to download Gene Counts from RaNA-Seq and import them into RNAseqChef.
Generating results and assessing their reliability are different tasks
In this evaluation, RNAseqChef made it easy to perform DEG analysis and proceed to enrichment analysis. Its PCA plots and heatmaps displayed the same problematic data structure identified in our original case study. GOI profiling also allowed us to examine expression distributions for selected genes. However, we could not easily investigate the questions raised by those observations.
Generating analysis results is not the same as understanding which characteristics of the data produced them or determining whether those results are reliable.
This issue is not unique to RNAseqChef. Galaxy, RaNA-Seq, and RNAseqChef differ in their flexibility and workflows, but in our evaluations, none made it easy to investigate the underlying causes of the observed problems and assess the validity of the results. One possible contributing factor is that bioinformatics education has often placed more emphasis on learning how to run analysis tools than on learning how to critically evaluate the results they produce.
The review "Designing RNA sequencing experiments: A practical guide to reproducible gene expression analysis" discusses challenges related to reproducibility and data interpretation in RNA-Seq analysis. It emphasizes the importance of experimental design, quality control, appropriate analytical methods, and biological interpretation in obtaining reproducible and biologically meaningful results.
What matters is not simply generating PCA plots or clustering diagrams, but examining what those visualizations reveal and determining whether the conclusions drawn from them are reliable.
As analysis tools make it increasingly easy to generate large numbers of results automatically, researchers' ability to critically examine and validate those results becomes more important than ever.
