CharacterTextSplitter seems to always include embedded column as meta_data

[miaomiaocui]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

# Columns to be embedded
embed_columns = ['text'] #text describes the property like 'luxury condo with fitness center'

# Columns that should be numeric
numeric_columns = ['sqft', 'year_built', 'floor'] 
    
def normalize_text(text):
    return ' '.join(str(text).lower().strip().split())

def process_dataframe(df, batch_size=1000):
    documents = []
    for i in range(0, len(df), batch_size):
        batch_df = df.iloc[i:i+batch_size]
        batch_documents = process_batch(batch_df)
        documents.extend(batch_documents)
    return documents

def process_batch(batch_df):
    batch_documents = []
    for _, row in batch_df.iterrows():
        try:
            # Combine and normalize the specified columns into a single text
            content = " ".join(normalize_text(row[col]) for col in embed_columns)
            
            # Create metadata dictionary, filtering out null values
            metadata = {col: row[col] for col in batch_df.columns if col not in embed_columns and pd.notna(row[col])}
            
            # Ensure numeric columns remain numeric
            for col in numeric_columns:
                if col in metadata:
                    metadata[col] = float(metadata[col])
            
            # Create Document
            doc = Document(page_content=content, metadata=metadata)
            batch_documents.append(doc)
        except Exception as e:
            print(f"Error creating document for row {_}: {e}")
    
    return batch_documents

# Create documents with metadata and page_content
if __name__ == "__main__":
    # Assuming df is your DataFrame
    # df = pd.read_csv('your_data.csv')  # Uncomment and use your data source
    
    umentsdoc = process_dataframe(df)
    
    print(f"Created {len(documents)} documents")

# Text splitting
text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
docs = text_splitter.split_documents(documents)

# Print metadata before and after text splitting
print("Before splitting:")
print("Metadata keys:", documents[0].metadata.keys())
print("\nAfter splitting:")
print("Metadata keys:", docs[0].metadata.keys())

Output:

Before splitting:
Metadata keys: dict_keys(['sqft', 'year_built', 'floor'])

After splitting:
Metadata keys: dict_keys(['sqft', 'year_built', 'floor', 'text'])

Description

CharacterTextSplitter seems to automatically include texts (page_content) in metadata. Is this expected or a bug?

System Info

langchain==0.2.11
langchain-community==0.2.10
langchain-core==0.2.24
langchain-openai==0.1.19
langchain-pinecone==0.1.3
langchain-text-splitters==0.2.2

Python version: 3.11.9

Platform is mac

[dosubot[bot]]

It is expected behavior for the CharacterTextSplitter in LangChain to include the text (page_content) in the metadata. This is evident from the code where each chunk of text is appended to lines_with_metadata with its associated metadata:

lines_with_metadata.append(
    {
        "content": "\n".join(current_content),
        "metadata": current_metadata.copy(),
    }
)

This behavior ensures that each piece of text is paired with its corresponding metadata ^[1].

To continue talking to Dosu, mention @dosu.