Experiments

[kamil271e]

• Synthetic neural nets generator with analysis
• Detailed experiments with non-singular values
• Started experiments with sparsity maximization in ReLU networks
• Refactor

[Copilot]

Pull Request Overview

This PR introduces several experimental components for synthetic neural net generation and analysis, alongside a major refactor of the autoencoder and extraction framework. Key changes include:

• New implementations for SoftmaxBottleneckModel and an updated Autoencoder with integrated weight initialization and activation handling.
• Refactoring of the extraction framework (SVDClassifier, ExtractionFramework) to use the updated configuration (AUTOENCODER_LAYER_SIZES) and more robust Q-matrix update logic.
• Updates to experiment utilities, end-to-end pipeline (experiments/run.py), dependencies (pyproject.toml), and multiple notebooks for SVD and sparsity experiments.

Reviewed Changes

Copilot reviewed 32 out of 33 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
src/models/softmax_bottleneck_model.py	Introduces a new model with flexible weight initialization and activation management.
src/models/autoencoder.py	Implements a refactored autoencoder with improved activation handling and weight initialization.
src/extraction_framework/svd_classifier.py	Replaces usage of a legacy layers variable with AUTOENCODER_LAYER_SIZES, aligning classification with new config.
src/extraction_framework/main.py	Updates import paths to match the restructured package organization.
src/extraction_framework/extraction_framework.py	Refactors the extraction logic to include iterative Q-matrix updates, visualization, and error recovery.
src/experiments/utils.py & src/experiments/run.py	Add end-to-end experiment pipelines and additional utilities for analyzing neural net behavior.
src/config.py	Updates configuration parameters, directory creation, and dependency definitions.
pyproject.toml	Adjusts dependencies and linter ignore rules for improved code quality.
notebooks/*	Adds and revises multiple notebooks to demonstrate SVD, sparsity optimization, and full pipeline experiments.

Files not reviewed (1)

• Makefile: Language not supported

Comments suppressed due to low confidence (2)

src/extraction_framework/extraction_framework.py:47

• The iterative update of 'self.h' within the extraction loop is ambiguous and may lead to unexpected behavior during layer dimension tracking. Consider revisiting this update logic and adding clear documentation to explain the intended invariant.

self.h = h if h != 0 else self.h

src/extraction_framework/extraction_framework.py:42

• [nitpick] Consider defining '_clamp_eps' and '_max_abs' as constants or initializing them in the constructor rather than resetting them inside the inverse_activation method. This change would improve clarity and ensure consistency across method calls.

self._clamp_eps = 1e-7