Context Information Model (CIM) Framework Construction

This repository contains the pipeline for constructing Context Information Models (CIM) from instructional video datasets. The system processes video transcripts to extract information units, build facet candidates, and generate context schemas that organize knowledge in a structured, fine-grained manner.

Repository Structure

• main.py: Main entry point for constructing CIM frameworks
• src: Core framework construction logic
  • framework_split.py: Main pipeline for processing videos and building CIM schemas
• helpers: Helper scripts and utilities
  • cim_scripts.py: CIM framework construction functions
  • dataset.py: Dataset loading and task definitions
  • nlp.py: Natural language processing utilities
  • pydantic_models: Pydantic models for data validation
  • framework.py: Framework-related models
  • evaluation.py: Evaluation models
  • rag.py: RAG-related models
  • summarization.py: Summarization models
• prompts: Prompt templates for LLM interactions
  • framework.py: Framework construction prompts
• static: Storage of datasets and processing results
  • datasets: Input video datasets in JSON format
  • results: Processing results and generated CIM schemas
• requirements.txt: Python package dependencies

Development environment

• Ubuntu 22.04, CUDA 12.8

Installation

1. Create a new conda environment (Python 3.10)

conda create -n cim python=3.10
conda activate cim

2. Install required dependencies

pip install -r requirements.txt

3. Set up environment variables for LMs (OpenAI)

Get your OpenAI API key from OpenAI Platform. You'll need to create an account and generate an API key.

Option A: Temporary (current terminal session only)

export OPENAI_API_KEY="your-api-key-here"

Option B: Conda environment variable

If you're using conda, you can set it as an environment variable:

# Make sure conda environment is activate (i.e., `cim`)
conda env config vars set OPENAI_API_KEY="your-api-key-here"
conda activate cim  # Reactivate the environment

Verify the environment variable is set:

echo $OPENAI_API_KEY

This should output your API key. If it's empty, the environment variable is not set correctly.

Available Tasks

The system supports multiple instructional video tasks. Available tasks include:

Cross-Task Tasks:

• Change a Tire
• Build Simple Floating Shelves
• Make French Toast
• Make Irish Coffee

Custom Tasks:

• How to Make a Sushi Roll
• How to Make Caramel Apples
• How to Make a Milkshake Without Ice Cream
• How to Grill Steak
• How to Make Scrambled Eggs in a Microwave
• How to Grow Hydrangea from Cuttings
• How to Grow a Pumpkin
• How to Clean Bathroom Tile
• How to Polish Stainless Steel
• How to Clean a Glass Top Stove
• How to Get Rid of a Wasp's Nest
• How to Plant a Living Christmas Tree
• How to Wrap Your Hands for Boxing
• How to Catch Trout
• How to Make a Paper Hat

Run

Run the pipeline to construct a CIM framework for a specific task:

# -t TASK, --task TASK
#                       The task to construct the CIM for (required)
# -e EMBEDDING_METHOD, --embedding_method EMBEDDING_METHOD
#                       Embedding method: openai (default) | bert
# -x EXTRACTION_MODEL, --extraction_model EXTRACTION_MODEL
#                       LLM used for information unit extraction (default: gpt-4.1-mini-2025-04-14)
# -g GENERATION_MODEL, --generation_model GENERATION_MODEL
#                       LLM used for facet mining and labeling (default: gpt-4.1-mini-2025-04-14)
# -v VERSION, --version VERSION
#                       Optionally specify the version of the CIM (default: full_run_1)
python main.py -t <task> [-e <embedding_method>] [-x <extraction_model>] [-g <generation_model>] [-v <version>]

Examples:

# Minimal (uses defaults: openai embeddings, gpt-4.1-mini-2025-04-14 models)
python main.py -t "How to Make a Paper Hat"

# Specify version label for results directory
python main.py -t "How to Make a Paper Hat" -v "full_run_1"

# Use local BERT embeddings instead of OpenAI embeddings
python main.py -t "How to Make a Paper Hat" -e bert

# Override LLMs used for extraction/generation
python main.py -t "How to Make a Paper Hat" -x gpt-4.1-mini-2025-04-14 -g gpt-4.1-mini-2025-04-14

The pipeline will:

1. Load the dataset for the specified task from static/datasets/
2. Build information units from video transcripts
3. Mine facet candidates iteratively
4. Compute the Pareto frontier using knapsack optimization
5. Generate the final context schema

Output

Processing results are stored in static/results/{task-name}/split_results_{version}.json.

The output format includes:

{
    "context_schema": [
        {
            "id": str,
            "type": str,  # e.g., "what", "why", "when", "where"
            "title": str,
            "title_plural": str,
            "definition": str,
            "guidelines": [str, ...],
            "vocabulary": [
                {
                    "label": str,
                    "definition": str,
                },
                ...
            ]
        },
        ...
    ],
    "facet_candidates": [...],  # All discovered facet candidates
    "labeled_dataset": [...]     # Processed dataset with labels
}

The context schema organizes knowledge into facets (dimensions of variation) with associated vocabularies (labels) that can be used to partition and structure instructional content.

LLM batching results

• Framework prompts are dispatched in batches (see prompts/framework_batch.py), which writes JSONL request logs to static/results/lm-batches/{batch_id}_llm_requests.jsonl.
• Responses are streamed to static/results/lm-batches/{batch_id}_llm_responses.jsonl by helpers.process_api_requests, which rate-limits and retries automatically.
• Every run prints the batch_id; use it to inspect or rerun process_api_requests(batch_id) if a batch was interrupted.

What is CIM?

We introduce Context-Information Maps (CIM) to automatically organize scattered task knowledge across a large corpus of tutorial videos about the same task. By transforming static tutorials into a unified bipartite graph that links specific "contexts" (applicability conditions such as tools, ingredients, or goals) with atomic "information units" (specific instructions, tips, or explanations), the system explicitly aligns diverse knowledge pieces from different sources according to when/where they are relevant (i.e., task contexts). CIM enables applications such as the automatic augmentation of tutorials with missing information, navigation between alternative methods across different sources, and the analytical detection of knowledge gaps or saturation points within a corpus of tutorials for the task.

Notes

• The system uses OpenAI embeddings (text-embedding-3-large) and language models (gpt-4.1-mini-2025-04-14) for extraction and processing
• Processing is iterative and may take time depending on dataset size
• Intermediate results are saved automatically, allowing for resumption of interrupted runs
• The system targets a discriminativeness score below 0.8 to ensure fine-grained knowledge organization

For any questions please contact: Bekzat Tilekbay