Analyst Report Document Processing Toolkit

Standalone pipeline for inventorying downloaded analyst report PDFs, parsing their file-level metadata, matching them back to calendar metadata, and detecting duplicate reports.

Overview

This repository takes a local folder of downloaded analyst report PDFs and turns it into a structured report-processing inventory. It is designed as a reproducible four-stage pipeline:

1. Inventory downloaded PDF files and build lightweight file-level metadata.
2. Parse report filenames into structured report fields.
3. Match parsed reports back to a calendar metadata extract.
4. Detect exact and metadata-level duplicates and summarize processing coverage.

The repo is meant to be a standalone toolkit rather than a loose set of notebooks. Each stage writes explicit CSV outputs and a Markdown run log so intermediate artifacts can be inspected directly.

Purpose

The repository does four things:

1. Scans a downloaded analyst report folder and records file names, relative paths, file sizes, approximate page counts, hashes, and lightweight text previews.
2. Parses the downloaded filenames into structured report variables such as date, ticker, contributor, partial title, and document identifier.
3. Links those parsed report rows to an analyst-report calendar metadata extract using date, page count, ticker, contributor, and title similarity.
4. Flags exact duplicate files and metadata-signature duplicates, then exports compact summary tables for review.

The main outputs are stage-level CSVs describing the raw PDF inventory, the cleaned parsed report inventory, the matched report inventory, and the final duplicate-review tables.

Key Definitions

• PDF Inventory: a file-level table describing the downloaded report set, including paths, hashes, sizes, and lightweight previews.
• Calendar Metadata: a structured report-level metadata table used to describe reports by date, contributor, title, ticker, and related attributes.
• Document Identifier: a report-level identifier parsed from a file name or metadata record.
• Deduplication: the process of identifying repeated documents using exact hashes or metadata-based duplicate signatures.
• Text Preview: a lightweight snippet extracted from a PDF to help with inspection and matching, rather than a full document parse.

Data Access Notes

The raw input data is not tracked in this repository. If you'd like to discuss the sample data structure, expected schema, or reproduction details, feel free to contact me.

Pipeline Stages

Stage 1: Extract PDF Inventory

This stage recursively scans the local PDF folder, computes file-level metadata, estimates page counts from PDF markers, hashes each file, and extracts a lightweight text preview using the system strings utility with a byte-level fallback.

Primary outputs:

• Output file: output/exports/001_extract_pdf_inventory/pdf_inventory_raw.csv

Stage 2: Clean and Parse Downloaded Reports

This stage parses the downloaded PDF filenames into structured report fields, standardizes contributor and title values, converts date and document identifiers into typed fields, and produces a clean report-level inventory.

Primary outputs:

• Output file: output/exports/002_clean_and_parse_downloaded_reports/cleaned_downloaded_reports.csv

Stage 3: Match Downloaded Reports to Metadata

This stage links the cleaned report inventory to the calendar metadata sample using date and page-count blocking, then scores candidates using ticker, contributor, and title agreement.

Primary outputs:

• Output file: output/exports/003_match_downloaded_reports_to_metadata/matched_reports_to_metadata.csv
• Output file: output/exports/003_match_downloaded_reports_to_metadata/unmatched_reports.csv

Stage 4: Deduplicate and Summarize

This stage assigns duplicate-group identifiers based on exact file hashes and metadata signatures, then builds summary tables for duplicate groups, overall processing metrics, and contributor-level match coverage.

Primary outputs:

• Output file: output/exports/004_deduplicate_and_summarize/deduplicated_report_inventory.csv
• Output file: output/exports/004_deduplicate_and_summarize/duplicate_report_groups.csv
• Output file: output/exports/004_deduplicate_and_summarize/processing_summary_metrics.csv
• Output file: output/exports/004_deduplicate_and_summarize/contributor_coverage_summary.csv

Repository Structure

analyst-report-document-processing-toolkit/
├── requirements.txt
├── .env.example
├── .gitignore
├── README.md
├── pyproject.toml
├── scripts/
│   ├── 001_extract_pdf_inventory.py
│   ├── 002_clean_and_parse_downloaded_reports.py
│   ├── 003_match_downloaded_reports_to_metadata.py
│   ├── 004_deduplicate_and_summarize.py
│   ├── 00A_run_all.py
│   ├── check_env.py
│   ├── run_pipeline.py
│   └── setup_env.py
├── src/
│   └── project/
│       ├── __init__.py
│       ├── config.py
│       ├── env.py
│       ├── io.py
│       ├── logger.py
│       ├── paths.py
│       ├── settings.py
│       ├── utils.py
│       ├── validation.py
│       ├── document_processing/
│       │   ├── __init__.py
│       │   ├── deduplication.py
│       │   └── pdf_inventory.py
│       ├── matching/
│       │   ├── __init__.py
│       │   └── report_metadata.py
│       └── pipelines/
│           ├── __init__.py
│           ├── clean_parse_downloaded_reports.py
│           ├── deduplicate_and_summarize.py
│           ├── extract_pdf_inventory.py
│           └── match_downloaded_reports_to_metadata.py
├── tests/
│   ├── conftest.py
│   ├── test_clean_parse_downloaded_reports.py
│   ├── test_deduplication.py
│   ├── test_pdf_inventory.py
│   ├── test_report_metadata_matching.py
│   └── test_validation.py
├── input/
│   ├── downloaded_reports_sample/
│   │   └── pdfs/
│   └── reference/
│       └── lseg_calendar_metadata_sample.csv
└── output/
    ├── exports/
    ├── figures/
    ├── logs/
    └── tables/

Required Inputs

This repository expects two local inputs:

• A folder of downloaded analyst report PDFs at DOWNLOADED_REPORTS_DIR
• A calendar metadata CSV at CALENDAR_METADATA_PATH

These inputs are local-only. Everything under input/ and output/ is gitignored, so if the repo is moved or cloned elsewhere you will need to place the inputs back into the expected local paths or update .env.

The expected local raw layout is:

input/
├── downloaded_reports_sample/
│   └── pdfs/
│       ├── 2018-09-12-EXMP.N-Example Research-Consumer Platform Initiation-83105769.pdf
│       ├── 2021-11-02-DEMO.OQ-Sample Securities-Digital Commerce Update-94382701.pdf
│       └── ...
└── reference/
    └── lseg_calendar_metadata_sample.csv

Input Data Examples

Example Downloaded PDF Inputs

The raw PDF input is a directory of report files rather than a CSV. Stage 1 scans that folder and converts it into a tabular inventory. The filenames are expected to resemble the downloaded analyst-report naming pattern shown above.

Example Calendar Metadata Rows

Below is a fake example using the full input schema for the calendar metadata CSV. The values are shortened and anonymized, but the column names match the real input file used by the repo.

DocumentID	Date	Available	Company Name	Ticker	Title	Contributor	Top Analyst	Analysts	Pages	Retail Value	Subjects	Categories	Count of Additional Companies	Count of Additional Tickers	Count of Additional Analysts	PrimarySymbols	SecondarySymbols
90000001	2021-11-02	2021-11-05	Example Commerce Group	EXMP.N	China Technology: Initiating Coverage	Example Research	Analyst A	Analyst A, Analyst B	169	1828.5	Equity, Initiation	Industry Report	7	7	2	EXMP.N	OTHR.N, DEMO.OQ
90000002	2020-08-30	2021-06-01	Sample Marketplace Inc.	SAMP.OQ	Consumer Technology Update	Sample Securities	Analyst C	Analyst C, Analyst D	243	2794.5	Equity	Company Report	1	1	2	SAMP.OQ	ALT.N

Setup

setup_env.py creates the expected local directories, creates a repo-local .env from .env.example when one does not already exist, and installs the packages listed in requirements.txt into the current interpreter or the local .venv if one exists.

python3 scripts/setup_env.py
python3 scripts/check_env.py

This repository expects a repo-local .env file at the project root. The tracked .env.example contains the sample configuration below, and python3 scripts/setup_env.py will copy it to .env on first run without overwriting an existing local file.

DOWNLOADED_REPORTS_DIR=input/downloaded_reports_sample/pdfs
CALENDAR_METADATA_PATH=input/reference/lseg_calendar_metadata_sample.csv
TEXT_PREVIEW_LIMIT=250
TITLE_SIMILARITY_THRESHOLD=0.72
STRINGS_MIN_LENGTH=8
MAX_REPORTS=25
RUNTIME_MODE=local

check_env.py also validates that the system strings binary is available, because Stage 1 uses it for lightweight text-preview extraction.

Run

Run all stages:

python3 scripts/run_pipeline.py --all

Run one stage:

python3 scripts/run_pipeline.py --stage 003_match_downloaded_reports_to_metadata

Each stage is independently executable as an entry script, but later stages depend on earlier-stage outputs already existing.

Outputs

The repository writes two kinds of outputs:

• Stage logs in output/logs/
• Stage exports in output/exports/

The expected output structure is:

output/
├── exports/
│   ├── 001_extract_pdf_inventory/
│   │   └── pdf_inventory_raw.csv
│   ├── 002_clean_and_parse_downloaded_reports/
│   │   └── cleaned_downloaded_reports.csv
│   ├── 003_match_downloaded_reports_to_metadata/
│   │   ├── matched_reports_to_metadata.csv
│   │   └── unmatched_reports.csv
│   └── 004_deduplicate_and_summarize/
│       ├── contributor_coverage_summary.csv
│       ├── deduplicated_report_inventory.csv
│       ├── duplicate_report_groups.csv
│       └── processing_summary_metrics.csv
└── logs/
    ├── 001_extract_pdf_inventory.md
    ├── 002_clean_and_parse_downloaded_reports.md
    ├── 003_match_downloaded_reports_to_metadata.md
    └── 004_deduplicate_and_summarize.md

Example Output Rows

The examples below are fake but aligned to the actual output schemas written by the pipeline.

Stage 1 PDF inventory output should look like this:

PDF Relative Path	PDF File Name	File Size (Bytes)	Estimated Pages	SHA256	Extracted Text Preview	Extracted Text Character Count
2021-11-02-EXMP.N-Example Research-China Technology Update-90000001.pdf	2021-11-02-EXMP.N-Example Research-China Technology Update-90000001.pdf	2456789	169	abc123examplehash	%PDF-1.4 Example text preview ...	278434
2020-08-30-Sample Securities-Consumer Technology Update-90000002.pdf	2020-08-30-Sample Securities-Consumer Technology Update-90000002.pdf	1987654	243	def456examplehash	%PDF-1.4 Another preview ...	270426

Stage 2 cleaned report inventory should look like this:

DocumentID	Date	Year	Contributor	Contributor Normalized	Report File Ticker	Partial Title	Partial Title Normalized	Pages	PDF Relative Path	PDF File Name	File Size (Bytes)	SHA256	Extracted Text Preview	Extracted Text Character Count
90000001	2021-11-02	2021	Example Research	example research	EXMP.N	China Technology Update	china technology update	169	2021-11-02-EXMP.N-Example Research-China Technology Update-90000001.pdf	2021-11-02-EXMP.N-Example Research-China Technology Update-90000001.pdf	2456789	abc123examplehash	%PDF-1.4 Example text preview ...	278434
90000002	2020-08-30	2020	Sample Securities	sample securities	``	Consumer Technology Update	consumer technology update	243	2020-08-30-Sample Securities-Consumer Technology Update-90000002.pdf	2020-08-30-Sample Securities-Consumer Technology Update-90000002.pdf	1987654	def456examplehash	%PDF-1.4 Another preview ...	270426

Stage 3 matched report inventory should look like this:

DocumentID	Year	Date	Contributor	Contributor Normalized	Report File Ticker	Partial Title	Partial Title Normalized	Pages	PDF Relative Path	PDF File Name	File Size (Bytes)	SHA256	Extracted Text Preview	Extracted Text Character Count	Metadata Available	Metadata Company Name	Metadata Ticker	Metadata Title	Metadata Contributor	Metadata Top Analyst	Metadata Analysts	Metadata Retail Value	Match Status	Match Type	Match Score
90000001	2021	2021-11-02	Example Research	example research	EXMP.N	China Technology Update	china technology update	169	2021-11-02-EXMP.N-Example Research-China Technology Update-90000001.pdf	2021-11-02-EXMP.N-Example Research-China Technology Update-90000001.pdf	2456789	abc123examplehash	%PDF-1.4 Example text preview ...	278434	2021-11-05	Example Commerce Group	EXMP.N	China Technology: Initiating Coverage	Example Research	Analyst A	Analyst A, Analyst B	1828.5	matched	ticker_exact+contributor_exact+title_fuzzy	9.25
90000002	2020	2020-08-30	Sample Securities	sample securities	``	Consumer Technology Update	consumer technology update	243	2020-08-30-Sample Securities-Consumer Technology Update-90000002.pdf	2020-08-30-Sample Securities-Consumer Technology Update-90000002.pdf	1987654	def456examplehash	%PDF-1.4 Another preview ...	270426	2021-06-01	Sample Marketplace Inc.	SAMP.OQ	Consumer Technology Update	Sample Securities	Analyst C	Analyst C, Analyst D	2794.5	matched	contributor_exact+title_exact	6.0

Stage 3 unmatched report inventory uses the same full schema as the matched file but only contains rows with Match Status = unmatched.

Stage 4 deduplicated report inventory should look like this:

DocumentID	Year	Date	Contributor	Report File Ticker	Partial Title	Pages	PDF Relative Path	PDF File Name	File Size (Bytes)	SHA256	Match Status	Match Type	Match Score	Exact Duplicate Group ID	Metadata Duplicate Group ID	Keep Report
90000001	2021	2021-11-02	Example Research	EXMP.N	China Technology Update	169	2021-11-02-EXMP.N-Example Research-China Technology Update-90000001.pdf	2021-11-02-EXMP.N-Example Research-China Technology Update-90000001.pdf	2456789	abc123examplehash	matched	ticker_exact+contributor_exact+title_fuzzy	9.25	EXACT-001	META-001	yes
90000001	2021	2021-11-02	Example Research	EXMP.N	China Technology Update	169	2021-11-02-EXMP.N-Example Research-China Technology Update-90000001(1).pdf	2021-11-02-EXMP.N-Example Research-China Technology Update-90000001(1).pdf	2456789	abc123examplehash	matched	ticker_exact+contributor_exact+title_fuzzy	9.25	EXACT-001	META-001	no

Stage 4 duplicate group summary should look like this:

Duplicate Group ID	Duplicate Type	Representative PDF File Name	Duplicate Count	Duplicate File Names
EXACT-001	exact_hash	2021-11-02-EXMP.N-Example Research-China Technology Update-90000001(1).pdf	2	2021-11-02-EXMP.N-Example Research-China Technology Update-90000001(1).pdf | 2021-11-02-EXMP.N-Example Research-China Technology Update-90000001.pdf
META-001	metadata_signature	2021-11-02-EXMP.N-Example Research-China Technology Update-90000001(1).pdf	2	2021-11-02-EXMP.N-Example Research-China Technology Update-90000001(1).pdf | 2021-11-02-EXMP.N-Example Research-China Technology Update-90000001.pdf

Stage 4 processing summary metrics should look like this:

section	metric	value	notes
inventory	row_count	16	Total cleaned report rows carried into the deduplication stage.
matching	matched_rows	16	Rows successfully matched to calendar metadata.
deduplication	exact_duplicate_groups	1	Duplicate groups detected from identical SHA256 hashes.
deduplication	metadata_duplicate_groups	1	Duplicate groups detected from date-contributor-title-page signatures.
deduplication	kept_reports	15	Reports retained after exact duplicate filtering.

Stage 4 contributor coverage summary should look like this:

Contributor	Report Count	Matched Report Count
Example Research	2	2
Sample Securities	1	1

Tests

The included tests cover core inventory extraction, filename parsing, metadata matching, deduplication logic, and validation helpers.

These tests are unit-level only. They do not yet act as a full integration test of the complete staged pipeline or every written CSV artifact.

Limitations

• Stage 1 uses a lightweight strings-based text preview rather than a full PDF text parser. The preview field is therefore useful for rough inspection, not high-quality text extraction.
• The page count is an estimate based on PDF byte markers, not a verified parser count.
• Matching is intentionally lightweight and sample-friendly; it uses date and page-count blocking plus ticker, contributor, and title similarity scoring.
• Blank duplicate-group cells in the Stage 4 CSV may appear as NaN if the file is later reloaded with pandas’ default missing-value parsing. In the raw CSV they are written as empty fields.