Skip to content

project_summary.md

Latest commit

 

History

History
54 lines (32 loc) · 3.35 KB

project_summary.md

File metadata and controls

54 lines (32 loc) · 3.35 KB

Project Summary: Linguistic Alpha Pipeline

This document provides a comprehensive overview of the work done to build and refine a linguistic analysis pipeline for financial documents.

1. Initial State & Goal

The project started with a set of Python scripts designed to perform psycholinguistic feature engineering on corporate communications. The initial structure was functional but fragmented, with separate mock data files and a monolithic feature engineering script that mixed different analysis types (core linguistic features, catalyst event scores).

The primary goal was to integrate a new data source: the "Management’s Discussion and Analysis" (MD&A) section from SEC filings, and refactor the project into a clean, scalable pipeline.

2. The Refactoring Plan

After an initial exploration, we decided on a complete refactoring to streamline the workflow. The approved plan was as follows:

Plan: Consolidate Data and Refactor Pipeline

This plan streamlines the entire project by adopting a new, unified data structure. It involves cleaning up obsolete files, creating a single source of mock data, and refactoring the feature engineering and backtesting scripts to work with the new format.

  1. Clean Up Project Structure:

    • Delete obsolete files and directories (output/, old mock data, old feature scripts, run_pipeline.py, scraper/).

  2. Create New Consolidated Mock Data:

    • Create a single new mock data file: data/mock_sec_data.json.
    • Populate this file with data following a specific structure, with distinct entries for "MD&A" and "Risk_Factors" sections.

  3. Create a New Unified Feature Engineering Script:

    • Create a new, single analysis/feature_engineering.py script.
    • This script contains one primary function, calculate_features_by_section, to apply logic based on the "section" field in the data.

  4. Simplify and Update the Backtesting Script:

    • Update analysis/backtesting.py to directly load the consolidated JSON, call the new feature engineering function, and run the correlation analysis.

3. Implementation and Final State

The plan was implemented successfully. The project is now organized around a clean, simple, and powerful pipeline:

  • Data Source: A single source of truth for mock data exists at data/mock_sec_data.json. All new data, including MD&A and Risk Factors, should be added here in the specified JSON format.

  • Feature Engineering: The analysis/feature_engineering.py script contains a single, unified function that:

    1. Reads the consolidated data.
    2. Applies the correct feature logic based on the section field ("MD&A" or "Risk_Factors").
    3. Pivots the data to create a clean, wide-format DataFrame where each row represents a single filing (ticker + date) and the columns represent the linguistic features from each section.

  • Backtesting: The analysis/backtesting.py script:

    1. Loads the JSON data and calls the feature engineering function.
    2. Fetches historical stock price data from Yahoo Finance.
    3. Calculates the future stock price volatility for the 90-day period following each filing.
    4. Runs a correlation analysis to measure the relationship between the linguistic features and the future volatility.

This refactored structure is minimal, clean, and easily extensible for future work, such as integrating audio transcript data.