Bad Request (HTTP 400)

[irvinng98]

Hi Chen,

I am running into a similar as #148 but I have verified that my features are in the format of K00001 etc.

daa_results_df <- pathway_daa(abundance = kegg_abundance, metadata = metadata, group = "Flareup", daa_method = "DESeq2", p.adjust = "BH")

daa_sub_method_results_df <- daa_results_df[daa_results_df$method == "DESeq2", ]

daa_annotated_sub_method_results_df <- pathway_annotation(pathway = "KO", daa_results_df = daa_sub_method_results_df, ko_to_kegg = TRUE)

The output of pathway_annotation is:

KO to KEGG is set to TRUE. Proceeding with KEGG pathway annotations...
No organism specified. Using generic KO information across all species.
We are connecting to the KEGG database to get the latest results, please wait patiently.
Processing pathways in chunks...
[2025-06-11 12:05:42] INFO: Starting KEGG annotation process
[2025-06-11 12:05:42] WARN: Attempt 1 failed: Bad Request (HTTP 400)., retrying...
Warning: restarting interrupted promise evaluation[2025-06-11 12:05:45] WARN: Attempt 2 failed: Bad Request (HTTP 400)., retrying...
Warning: restarting interrupted promise evaluation[2025-06-11 12:05:49] ERROR: Error querying KEGG for NA: Bad Request (HTTP 400).
  [--------]   0% | Elapsed:  8s | ETA:  3h
  2/2540 (0 B/s/sec) | Processing NA[2025-06-11 12:05:50] WARN: Attempt 1 failed: Bad Request (HTTP 400)., retrying...
Warning: restarting interrupted promise evaluation[2025-06-11 12:05:52] WARN: Attempt 2 failed: Bad Request (HTTP 400)., retrying...
Warning: restarting interrupted promise evaluation[2025-06-11 12:05:57] ERROR: Error querying KEGG for NA: Bad Request (HTTP 400).
  [--------]   0% | Elapsed: 15s | ETA:  4h
  3/2540 (0 B/s/sec) | Processing NA[2025-06-11 12:05:58] WARN: Attempt 1 failed: Bad Request (HTTP 400)., retrying...

My daa_sub_method_results_df dataframe looks like this:

Thank you so much for you help!

[irvinng98]

I also ran

> any(is.na(daa_sub_method_results_df$feature))
[1] FALSE

[cafferychen777]

Hi @irvinng98,

Problem Description

When running pathway_annotation with ko_to_kegg = TRUE, users encounter HTTP 400 errors:

[2025-06-11 12:05:42] WARN: Attempt 1 failed: Bad Request (HTTP 400)., retrying...
[2025-06-11 12:05:45] WARN: Attempt 2 failed: Bad Request (HTTP 400)., retrying...
[2025-06-11 12:05:49] ERROR: Error querying KEGG for NA: Bad Request (HTTP 400).

The key indicator is "Processing NA" in the progress bar, suggesting that NA values are being passed to the KEGG API.

Root Cause Analysis

After examining the code, I identified the most likely cause: NA values in the feature column of your daa_sub_method_results_df dataframe are being passed to the KEGG REST API, which returns HTTP 400 (Bad Request) because "NA" is not a valid KO identifier.

The issue occurs in the process_kegg_annotations function where:

1. The function iterates through each feature: ko_id <- filtered_df$feature[i] (line 334)
2. If ko_id is NA, it gets passed to get_kegg_with_cache(ko_id, organism) (line 341)
3. The KEGG API receives "NA" as a query parameter and returns HTTP 400

Diagnostic Steps

First, let's verify if this is the issue:

# Check for NA values in your feature column
print("Checking for NA values in feature column:")
print(any(is.na(daa_sub_method_results_df$feature)))

# Count NA values
na_count <- sum(is.na(daa_sub_method_results_df$feature))
print(paste("Number of NA features:", na_count))

# Show rows with NA features
if (na_count > 0) {
  print("Rows with NA features:")
  print(daa_sub_method_results_df[is.na(daa_sub_method_results_df$feature), ])
}

# Check the structure of your feature column
print("Feature column summary:")
print(summary(daa_sub_method_results_df$feature))
print("First few features:")
print(head(daa_sub_method_results_df$feature, 10))

Solutions

Solution 1: Remove NA Features (Recommended)

# Remove rows with NA features before annotation
daa_sub_method_results_df_clean <- daa_sub_method_results_df[!is.na(daa_sub_method_results_df$feature), ]

# Verify no NA values remain
print(paste("Rows before cleaning:", nrow(daa_sub_method_results_df)))
print(paste("Rows after cleaning:", nrow(daa_sub_method_results_df_clean)))
print(paste("NA features removed:", sum(is.na(daa_sub_method_results_df$feature))))

# Now run pathway annotation with cleaned data
daa_annotated_sub_method_results_df <- pathway_annotation(
  pathway = "KO", 
  daa_results_df = daa_sub_method_results_df_clean, 
  ko_to_kegg = TRUE
)

Solution 2: Check Your Original Data

The NA values might come from your original abundance data or the DESeq2 analysis. Check:

# Check your original kegg_abundance data
print("Checking original abundance data:")
print("Row names (KO IDs):")
print(head(rownames(kegg_abundance), 10))
print("Any NA in row names:")
print(any(is.na(rownames(kegg_abundance))))

# Check if DESeq2 introduced NA values
print("Checking DESeq2 results:")
print("Any NA in feature column from DESeq2:")
print(any(is.na(daa_results_df$feature)))

Solution 3: Complete Workflow with Validation

library(ggpicrust2)
library(dplyr)

# Your original code with added validation
daa_results_df <- pathway_daa(
  abundance = kegg_abundance, 
  metadata = metadata, 
  group = "Flareup", 
  daa_method = "DESeq2", 
  p.adjust = "BH"
)

# Filter to DESeq2 method
daa_sub_method_results_df <- daa_results_df[daa_results_df$method == "DESeq2", ]

# VALIDATION STEP: Check for and remove NA features
print("=== Data Validation ===")
print(paste("Total features before cleaning:", nrow(daa_sub_method_results_df)))
print(paste("Features with NA:", sum(is.na(daa_sub_method_results_df$feature))))

# Remove NA features
daa_sub_method_results_df <- daa_sub_method_results_df[!is.na(daa_sub_method_results_df$feature), ]

print(paste("Total features after cleaning:", nrow(daa_sub_method_results_df)))

# Additional validation: Check KO format
ko_pattern <- "^K\\d{5}$"
invalid_kos <- daa_sub_method_results_df$feature[!grepl(ko_pattern, daa_sub_method_results_df$feature)]
if (length(invalid_kos) > 0) {
  print(paste("Warning: Found", length(invalid_kos), "features that don't match KO format"))
  print("First few invalid features:")
  print(head(invalid_kos, 5))
}

# Now run pathway annotation
daa_annotated_sub_method_results_df <- pathway_annotation(
  pathway = "KO", 
  daa_results_df = daa_sub_method_results_df, 
  ko_to_kegg = TRUE
)

Alternative Causes and Solutions

If the NA removal doesn't solve the issue, consider these other potential causes:

1. Network/Firewall Issues

# Test KEGG connectivity
library(KEGGREST)

# Test a simple KEGG query
test_result <- tryCatch({
  KEGGREST::keggGet("K00001")
}, error = function(e) {
  print(paste("KEGG connection test failed:", e$message))
  NULL
})

if (!is.null(test_result)) {
  print("KEGG connection test successful")
} else {
  print("KEGG connection failed - check network/firewall")
}

2. Restart R Session

As mentioned in the documentation, sometimes restarting the R session helps:

# Restart R session and reload libraries
.rs.restartR()  # In RStudio
# Or restart R manually

# Then reload libraries and try again
library(ggpicrust2)
library(KEGGREST)

3. Check KEGG Server Status

# Try with a smaller subset first
small_subset <- daa_sub_method_results_df[1:5, ]
test_annotation <- pathway_annotation(
  pathway = "KO", 
  daa_results_df = small_subset, 
  ko_to_kegg = TRUE
)

4. Use Different Network/Device

If the issue persists:

• Try from a different network connection
• Use a VPN if you're in a region with restricted access
• Try from a different computer/device

Prevention

To prevent this issue in the future:

# Add this validation function to your workflow
validate_daa_results <- function(daa_df) {
  # Check for NA features
  na_features <- sum(is.na(daa_df$feature))
  if (na_features > 0) {
    warning(paste("Found", na_features, "NA features that will be removed"))
    daa_df <- daa_df[!is.na(daa_df$feature), ]
  }
  
  # Check KO format
  ko_pattern <- "^K\\d{5}$"
  invalid_kos <- sum(!grepl(ko_pattern, daa_df$feature))
  if (invalid_kos > 0) {
    warning(paste("Found", invalid_kos, "features with invalid KO format"))
  }
  
  return(daa_df)
}

# Use in your workflow
daa_sub_method_results_df <- validate_daa_results(daa_sub_method_results_df)

Summary

The most likely cause is NA values in your feature column. The solution is to:

1. Check for NA values in daa_sub_method_results_df$feature
2. Remove rows with NA features before running pathway_annotation
3. Validate your data at each step of the pipeline

If the issue persists after removing NA values, it's likely a network connectivity issue, and you should try:

• Restarting R session
• Using a different network/device
• Checking KEGG server accessibility

The code validation shows that the ggpicrust2 package has robust error handling, but it doesn't pre-filter NA values before sending them to the KEGG API, which causes the HTTP 400 errors you're experiencing.

Best,
Chen