[Bug]: JSONCSSSelector Fails to Handle Sibling CSS Selectors

[itsskofficial]

crawl4ai version

0.6.4

Expected Behavior

The JSONCSSSelector should correctly process sibling CSS selectors like "+ tr span.age" to extract data from elements such as located within the immediately following sibling of a base element (e.g., on Hacker News). Specifically, when using a selector like "+ tr span.age", it should select the element from the next sibling and extract the desired attribute or text, such as the title attribute containing timestamp data.

Current Behavior

When applying a sibling selector such as "+ tr span.age" to a base element like , the JSONCSSSelector fails to select any elements. As a result, no data is extracted, and the expected fields (e.g., timestamps in the time field) are missing or empty in the output.

Is this reproducible?

Yes

Inputs Causing the Bug

- URL(s): 
https://news.ycombinator.com/ (Hacker News homepage)

- Settings used:
Crawler: AsyncWebCrawler
Extraction Strategy: JsonCssExtractionStrategy with a schema containing sibling selectors.

- Input data:
Base selector: "tr.athing"
Field selector: "+ tr span.age" (for extracting the title attribute of <span class="age">)

Steps to Reproduce

1. Set up a Crawl4AI crawler with AsyncWebCrawler and JsonCssExtractionStrategy.

2. Define a schema with:
Base selector: "tr.athing"
Field:
{
    "name": "time",
    "selector": "+ tr span.age",
    "type": "attribute",
    "attribute": "title"
}

3. Run the crawler on https://news.ycombinator.com/.

4. Observe that the extracted data does not include the time field or contains empty values for it, despite the presence of <span class="age"> elements in the sibling <tr>.

Code snippets

from crawl4ai import AsyncWebCrawler, JsonCssExtractionStrategy

# Define the schema with a sibling selector
schema = {
    "name": "HackerNewsScraper",
    "baseSelector": "tr.athing",
    "fields": [
        {
            "name": "time",
            "selector": "+ tr span.age",
            "type": "attribute",
            "attribute": "title",
            "default": ""
        }
    ]
}

# Initialize the crawler with the extraction strategy
async with AsyncWebCrawler() as crawler:
    strategy = JsonCssExtractionStrategy(schema=schema)
    results = await crawler.arun(url="https://news.ycombinator.com/", config={"extraction_strategy": strategy})
    
    # Print the extracted content (expected to have empty 'time' fields)
    print(results[0].extracted_content)

OS

Windows

Python version

3.12.9

Browser

Chrome

Browser version

126.0.6478.127

Error logs & Screenshots (if applicable)

No explicit errors are raised; however, the extracted data for the time field is consistently empty or missing.

[Ahmed-Tawfik94]

Root Cause Analysis

• Primary cause: Both JsonCssExtractionStrategy and
JsonLxmlExtractionStrategy reject CSS selectors that start with combinators
(+, ~, >, etc.).
• JsonCssExtractionStrategy: Calls element.select(selector) directly; BeautifulSoup/soupsieve rejects leading combinators.
• JsonLxmlExtractionStrategy: Uses lxml's CSS selector parser which also rejects leading combinators.
• Contributing factors:
• Relative selectors with leading combinators are not valid at top level in CSS selector parsers.
• The extraction strategies do not rewrite or normalize selectors for relative sibling selection.
• This affects real-world usage (e.g., Amazon product extraction examples use ~ span span).
• Evidence:
• crawl4ai/extraction_strategy.py JsonCssExtractionStrategy _get_elements method (lines 1469-1472):

      def _get_elements(self, element, selector: str):                      
          return element.select(selector)                        

• JsonLxmlExtractionStrategy has similar limitations despite more sophisticated selector handling.
• Real-world examples: Amazon extraction uses [data-csa-c-func-deps='aui-da-a-popover'] ~ span span.
• Soup Sieve and lxml CSS parsers both reject leading combinators at the top level.

Resolution Check (Current Version)

• Status: not resolved
• Evidence:
• Reproduced on 0.8.0 with JsonCssExtractionStrategy using + tr span.age.
• Also reproduced with JsonLxmlExtractionStrategy using same selector.
• PR #1255 implements selector normalization that successfully handles +
tr span.age selectors.
• Tested PR implementation manually - it correctly extracts data from
sibling elements.
• PR #1255 only fixes JsonCssExtractionStrategy;
JsonLxmlExtractionStrategy still needs similar fix.

Suggested Fixes (Backward-Compatible)

CRITICAL: Solution must be extended to BOTH extraction strategies

1. Implemented in PR Fix JSONCSSSelector to Handle Sibling CSS Selectors #1255 for JsonCssExtractionStrategy: Add selector
   normalization to handle leading combinators by using sibling traversal:

    def _get_elements(self, element, selector: str):                          
        if selector.startswith('+ '):                                         
            # Remove '+ ' prefix and strip whitespace                         
            rest = selector[2:].strip()                                       
            # Split on the first space to separate sibling selector from the  rest                                                                        
            space_index = rest.find(' ')                                      
            if space_index != -1:                                             
                sibling_selector = rest[:space_index]                         
                rest_selector = rest[space_index+1:].strip()                  
            else:                                                             
                sibling_selector = rest                                       
                rest_selector = None                                          
                                                                              
            # Find the next sibling matching the sibling_selector             
            sibling = element.find_next_sibling(sibling_selector)             
                                                                              
            if sibling and rest_selector:                                     
                # Apply the remaining selector to the sibling                 
                return sibling.select(rest_selector)                          
            elif sibling:                                                     
                # If no further selector, return the sibling itself           
                return [sibling]                                              
            else:                                                             
                # No sibling found                                            
                return []                                                     
        else:                                                                 
            # Default behavior for descendant selectors                       
            return element.select(selector)                                   
                                                                              

2. URGENTLY NEEDED for JsonLxmlExtractionStrategy: Implement similar
   normalization in JsonLxmlExtractionStrategy's selector handling. The
   sophisticated selector caching and fallback mechanisms in
   JsonLxmlExtractionStrategy should be extended to handle leading combinators
   before passing to lxml's CSS parser.
3. Comprehensive solution: Extend normalization to handle all leading
   combinators (+, ~, >, +, ||) in both strategies.