Development

This guide is for developers who want to contribute to the Substack API client library or understand its internal architecture.

Development Setup

Prerequisites

Node.js 16 or higher
npm or yarn package manager
Git
A Substack account for testing

Getting Started

Option 1: Dev Container (Recommended)

For the most consistent development experience, use the provided dev container configuration:

GitHub Codespaces:

Click the "Code" button in the GitHub repository
Select "Codespaces" tab
Click "Create codespace on main"
Dependencies will be installed automatically

VS Code Remote - Containers:

Install the "Remote - Containers" extension in VS Code
Clone the repository locally
Open in VS Code and click "Reopen in Container" when prompted
Dependencies will be installed automatically

Option 2: Local Development

Clone the repository:

git clone https://github.com/jakub-k-slys/substack-api.git
cd substack-api

Install dependencies:
```
npm install
```
Build the project:
```
npm run build
```

Environment Setup

Create a .env file for testing:

# .env
SUBSTACK_API_KEY=your-connect-sid-cookie-value
SUBSTACK_HOSTNAME=example.substack.com

To get your substack.sid cookie value:

Login to Substack in your browser
Open Developer Tools (F12)
Go to Application/Storage → Cookies → https://substack.com
Copy the substack.sid value

Project Structure

substack-api/
├── src/
│   ├── substack-client.ts    # Main SubstackClient implementation  
│   ├── http-client.ts        # HTTP client with authentication
│   ├── entities/             # Entity classes (Profile, Post, Note, Comment)
│   │   ├── profile.ts
│   │   ├── own-profile.ts  
│   │   ├── post.ts
│   │   ├── note.ts
│   │   └── comment.ts
│   ├── types/               # TypeScript type definitions
│   │   ├── api-types.ts
│   │   ├── entity-types.ts
│   │   └── config-types.ts
│   ├── note-builder.ts      # Helper for building formatted notes
│   └── index.ts             # Public API exports
├── tests/
│   ├── unit/                # Unit tests for individual components
│   ├── integration/         # Integration tests for entity interactions
│   └── e2e/                 # End-to-end tests with real API
├── docs/                    # Comprehensive documentation
│   ├── api-reference.md     # Complete API documentation
│   ├── entity-model.md      # Entity model guide
│   ├── examples.md          # Real-world usage examples
│   ├── quickstart.md        # Getting started guide
│   └── ...
├── samples/                 # Sample applications and scripts
├── dist/                    # Compiled JavaScript files
├── .env.example             # Environment variables template
├── jest.config.js           # Jest configuration for unit tests
├── jest.integration.config.js # Jest configuration for integration tests
├── jest.e2e.config.js       # Jest configuration for E2E tests
├── package.json             # Project configuration
├── tsconfig.json            # TypeScript configuration
└── README.md                # Project overview

Development Workflow

Building

To build the project:

npm run build

This will:

Clean the dist directory
Compile TypeScript files
Generate type definitions

Testing Strategy

The project uses a comprehensive 3-tier testing strategy:

1. Unit Tests (`npm test`)

Purpose: Test individual components in isolation
Location: tests/unit/
Speed: Very fast (< 1 second)
Scope: Functions, classes, utilities
Mocking: Heavy use of mocks for external dependencies

npm test              # Run all unit tests
npm run test:watch    # Run in watch mode for development

2. Integration Tests (`npm run test:integration`)

Purpose: Test component interactions and entity relationships
Location: tests/integration/
Speed: Fast (few seconds)
Scope: Entity navigation, async iteration, error handling
Mocking: Mock HTTP layer, real entity logic

npm run test:integration          # Run integration tests
npm run test:integration:watch    # Watch mode

3. End-to-End Tests (`npm run test:e2e`)

Purpose: Validate against real Substack API
Location: tests/e2e/
Speed: Slower (network dependent)
Scope: Full workflow validation, API compatibility
Mocking: No mocks - real API calls

npm run test:e2e     # Run E2E tests (requires credentials)

End-to-End Testing

The project includes end-to-end (E2E) tests that validate integration with the real Substack server. These tests are located in the tests/e2e/ directory.

Setting Up E2E Tests

Set up credentials: Create a .env file in the project root with your Substack credentials:
```
# Copy the example file
cp .env.example .env
```
Edit the .env file and add your substack.sid cookie:
```
SUBSTACK_API_KEY=your-connect-sid-cookie-value
SUBSTACK_HOSTNAME=yoursite.substack.com  # optional
```
Important: Never commit your .env file to version control. It's already included in .gitignore.
Obtain credentials: Get your substack.sid cookie value:
- Login to Substack in your browser
- Open Developer Tools (F12)
- Go to Application/Storage → Cookies → https://substack.com
- Copy the substack.sid value

Running E2E Tests

npm run test:e2e              # Run all E2E tests
npm run test:e2e -- --testNamePattern="Profile" # Run specific tests

E2E tests are designed to be:

Safe: Read-only operations where possible, minimal writes
Isolated: Each test cleans up after itself
Conditional: Skip gracefully when credentials are unavailable
Respectful: Include delays to avoid overwhelming the API

Run E2E tests in watch mode:

npm run test:e2e:watch

Run both unit and E2E tests:

npm run test:all

E2E Test Behavior

Without credentials: Tests will be automatically skipped with a warning message explaining how to set up credentials.
With credentials: Tests will run against the real Substack API using your provided credentials.
Test isolation: E2E tests are designed to be read-only and safe to run multiple times without creating unwanted content.
Timeout: E2E tests have a 30-second timeout to account for network latency.

E2E Test Coverage

The E2E test suite covers:

Authentication: Verifying API key authentication works
Publication operations: Getting publication details and metadata
Post operations: Fetching posts, pagination, searching, and individual post retrieval
Comment operations: Fetching comments for posts and individual comments
Notes operations: Fetching notes and pagination (note publishing tests are commented out to avoid creating content)
Profile operations: Getting user profiles and public profiles

Creating New E2E Tests

When adding new E2E tests:

Use the conditional test pattern with skipIfNoCredentials()
Handle API errors gracefully (some operations may not be available for all accounts)
Avoid tests that create persistent content unless absolutely necessary
Add logging for skipped operations to help with debugging
Follow the existing test structure and naming conventions

Code Style

The project follows TypeScript best practices:

Use explicit types where beneficial
Document public APIs with JSDoc comments
Follow consistent naming conventions
Write unit tests for new functionality

Documentation

Documentation is written in Markdown and built using Sphinx with MyST parser:

Install documentation dependencies:

pip install sphinx sphinx-rtd-theme myst-parser

Build the documentation:
```
cd docs
make html
```

The built documentation will be available in docs/build/html.

Contributing

Contribution Process

Fork the repository
Create a feature branch
Make your changes
Write or update tests
Update documentation
Submit a pull request

Pull Request Guidelines

Keep changes focused and atomic
Follow existing code style
Include tests for new functionality
Update documentation as needed
Describe your changes in the PR description

Running Checks

Before submitting a PR, ensure:

All tests pass:
```
npm test
```
TypeScript compiles without errors:
```
npm run build
```
Documentation builds successfully:
```
cd docs
make html
```

Release Process

Update version in package.json
Update CHANGELOG.md
Build the project:
```
npm run build
```
Run tests:
```
npm test
```

Commit changes:

git add .
git commit -m "Release v1.x.x"
git tag v1.x.x
git push origin main --tags

Publish to npm:
```
npm publish
```

Development Tips

TypeScript Configuration

The project uses a strict TypeScript configuration. Key settings in tsconfig.json:

{
  "compilerOptions": {
    "strict": true,
    "esModuleInterop": true,
    "declaration": true
  }
}

Testing Tips

Use Jest's mock capabilities for testing API calls
Test error conditions and edge cases
Use TypeScript in test files for better type checking

Example test structure:

import { Substack, SubstackError } from './client';

describe('Substack', () => {
  let client: Substack;

  beforeEach(() => {
    client = new Substack();
  });

  it('should handle successful requests', async () => {
    // Test implementation
  });

  it('should handle errors', async () => {
    // Test implementation
  });
});

Runtime Type Validation

The library uses io-ts for runtime type validation of API responses to ensure data safety beyond TypeScript's compile-time checks.

Why Runtime Validation?

While TypeScript provides excellent static typing, API responses are dynamic and can change without notice. Runtime validation with io-ts provides:

Type safety at runtime: Validates that API responses match expected shapes
Early error detection: Catches data inconsistencies before they reach domain models
Robust error handling: Provides detailed error messages for invalid data
Composable validation: Uses composable codecs for complex nested structures

Using io-ts Codecs

The library defines io-ts codecs for key internal types in src/internal/types/io-ts-codecs.ts:

// Raw Post codec
export const RawPostCodec = t.type({
  id: t.number,
  title: t.string,
  slug: t.string,
  post_date: t.string,
  canonical_url: t.string,
  type: t.union([t.literal('newsletter'), t.literal('podcast'), t.literal('thread')])
  // ... other fields
})

export type RawPost = t.TypeOf<typeof RawPostCodec>

Validation in Services

Services use validation utilities to validate API responses:

import { decodeOrThrow } from '../validation'
import { RawPostCodec } from '../types'

async getPostById(id: number): Promise<SubstackPost> {
  const rawResponse = await this.httpClient.get<unknown>(`/api/v1/posts/by-id/${id}`)
  
  // Validate the response with io-ts before returning
  return decodeOrThrow(RawPostCodec, rawResponse, 'Post response')
}

Validation Utilities

Two main utilities are provided in src/internal/validation.ts:

decodeOrThrow: Validates data and throws an error on failure (used in production code)
decodeEither: Returns an Either type for error handling (used in tests and error-safe contexts)

Adding New Codecs

When adding new API endpoints or modifying existing ones:

Define io-ts codecs for the expected response shapes
Use decodeOrThrow in service methods to validate responses
Add tests for both successful and failing validation scenarios

Example test:

it('should validate valid post data', () => {
  const validPost = { id: 123, title: 'Test', /* ... */ }
  const result = decodeEither(RawPostCodec, validPost)
  expect(isRight(result)).toBe(true)
})

it('should reject invalid post data', () => {
  const invalidPost = { id: 'not-a-number', /* ... */ }
  expect(() => decodeOrThrow(RawPostCodec, invalidPost, 'test')).toThrow()
})

Debugging

For debugging during development:

Use the debug npm package for logging

Add source maps in tsconfig.json:

{
  "compilerOptions": {
    "sourceMap": true
  }
}

Use the VS Code debugger with the following launch configuration:

{
  "type": "node",
  "request": "launch",
  "name": "Debug Tests",
  "program": "${workspaceFolder}/node_modules/.bin/jest",
  "args": ["--runInBand"],
  "console": "integratedTerminal",
  "internalConsoleOptions": "neverOpen"
}

Uh oh!

FilesExpand file tree

development.md

Latest commit

History