Skip to content

Commit 40b8626

Browse files
authored
docs: Add comprehensive contributor guide (#69)
- Create guides/contributor/ folder with detailed documentation - Add Getting Started guide for development environment setup - Add Code Style Guide with TypeScript conventions and best practices - Add Build and Test guide for testing, CI/CD, and troubleshooting - Add Submitting Changes guide for pull request workflow - Simplify main README with links to contributor documentation - Update guides README to organize user and contributor sections This helps new contributors quickly onboard with styling, build, test requirements and contribution guidelines.
1 parent 6936706 commit 40b8626

File tree

7 files changed

+1689
-138
lines changed

7 files changed

+1689
-138
lines changed

README.md

Lines changed: 31 additions & 134 deletions
Original file line numberDiff line numberDiff line change
@@ -316,90 +316,35 @@ Logging is configured in `log4js.json`. Logs are written to both console and `lo
316316

317317
## Development
318318

319-
### Prerequisites
319+
For contributors, comprehensive documentation is available in the [Contributor Guide](guides/contributor/):
320320

321-
- Node.js 20+ (CI uses Node 23)
322-
- Docker (for running Spark Connect server)
323-
- npm
321+
- **[Getting Started](guides/contributor/GETTING_STARTED.md)** - Set up your development environment
322+
- **[Code Style Guide](guides/contributor/CODE_STYLE.md)** - Coding conventions and best practices
323+
- **[Build and Test](guides/contributor/BUILD_AND_TEST.md)** - Building, testing, and running the project
324+
- **[IDE Setup](guides/contributor/IDE_SETUP.md)** - Recommended IDE configurations
325+
- **[Submitting Changes](guides/contributor/SUBMITTING_CHANGES.md)** - How to submit pull requests
324326

325-
### Setup
326-
327-
1. **Clone the repository**:
328-
```bash
329-
git clone https://github.com/yaooqinn/spark.js.git
330-
cd spark.js
331-
```
332-
333-
2. **Install dependencies**:
334-
```bash
335-
npm install
336-
```
337-
338-
3. **Start Spark Connect server** (required for tests):
339-
```bash
340-
# Build the Docker image (first time only, ~20-30 seconds)
341-
docker build -t scs .github/docker
342-
343-
# Start the server
344-
docker run --name sparkconnect -p 15002:15002 -d scs
345-
346-
# Wait 15-20 seconds for server to be ready
347-
# Check logs to confirm it's running:
348-
docker logs sparkconnect
349-
# Look for: "Spark Connect server started at: [::]:15002"
350-
```
351-
352-
4. **Stop the server** when done:
353-
```bash
354-
docker stop sparkconnect
355-
docker rm sparkconnect
356-
```
357-
358-
### Running Tests
327+
### Quick Start for Contributors
359328

360329
```bash
361-
# Make sure Spark Connect server is running first!
362-
npm test
363-
```
364-
365-
The test suite includes:
366-
- 28 test files
367-
- 179+ test cases
368-
- Tests for DataFrame operations, SQL, readers/writers, catalog, types, etc.
369-
370-
Tests require a running Spark Connect server on port 15002.
330+
# Clone and install
331+
git clone https://github.com/yaooqinn/spark.js.git
332+
cd spark.js
333+
npm install
371334

372-
### Linting
335+
# Start Spark Connect server for testing
336+
docker build -t scs .github/docker
337+
docker run --name sparkconnect -p 15002:15002 -d scs
373338

374-
```bash
339+
# Run tests and linting
340+
npm test
375341
npm run lint
376-
```
377-
378-
ESLint is configured with TypeScript support. Generated protobuf code in `src/gen/` is ignored.
379-
380-
### Building
381342

382-
Compile TypeScript to JavaScript:
383-
384-
```bash
385-
npx tsc --build
386-
```
387-
388-
Type check without emitting files:
389-
390-
```bash
391-
npx tsc --noEmit
392-
```
393-
394-
### Protobuf Code Generation
395-
396-
If you modify `.proto` files in `protobuf/`:
397-
398-
```bash
399-
npx buf generate
343+
# Clean up
344+
docker stop sparkconnect && docker rm sparkconnect
400345
```
401346

402-
Generated TypeScript code is placed in `src/gen/`.
347+
For detailed instructions, see the [Contributor Guide](guides/contributor/).
403348

404349
### Project Structure
405350

@@ -452,70 +397,22 @@ npx ts-node example/org/apache/spark/sql/example/Pi.ts
452397

453398
## Contributing
454399

455-
Contributions are welcome! Here's how to contribute:
456-
457-
1. **Fork the repository** and create a feature branch
458-
2. **Make your changes** following the existing code style
459-
3. **Add tests** for new functionality
460-
4. **Run linting**: `npm run lint`
461-
5. **Run tests**: `npm test` (requires Spark Connect server)
462-
6. **Submit a pull request** with a clear description
463-
464-
### Code Style
465-
466-
- Use 2 spaces for indentation
467-
- Follow TypeScript best practices
468-
- Add Apache License header to new files
469-
- Write descriptive commit messages
470-
471-
### Running CI Checks Locally
472-
473-
Before submitting a PR, ensure CI checks pass:
474-
475-
```bash
476-
# Linting
477-
npm run lint
400+
Contributions are welcome! Please read the [Contributor Guide](guides/contributor/) for detailed information on:
478401

479-
# Tests (requires Docker)
480-
docker build -t scs .github/docker
481-
docker run --name sparkconnect -p 15002:15002 -d scs
482-
npm test
483-
docker stop sparkconnect && docker rm sparkconnect
484-
```
402+
- Setting up your development environment
403+
- Code style and conventions
404+
- Building and testing
405+
- Submitting pull requests
485406

486-
### Publishing to npm
407+
### Quick Contributing Guide
487408

488-
To publish a new version to npm (maintainers only):
489-
490-
1. **Update the version** in `package.json`:
491-
```bash
492-
npm version patch # or minor, or major
493-
```
494-
495-
2. **Build the package**:
496-
```bash
497-
npm run build
498-
```
499-
500-
3. **Test the package locally** (optional):
501-
```bash
502-
npm pack
503-
# This creates a .tgz file you can test with: npm install spark.js-0.1.0.tgz
504-
```
505-
506-
4. **Publish to npm**:
507-
```bash
508-
npm login # Login to npm (first time only)
509-
npm publish
510-
```
511-
512-
5. **Tag the release** on GitHub:
513-
```bash
514-
git tag v0.1.0
515-
git push origin v0.1.0
516-
```
409+
1. **Fork** the repository and create a feature branch
410+
2. **Follow** the [Code Style Guide](guides/contributor/CODE_STYLE.md)
411+
3. **Add tests** for new functionality
412+
4. **Run checks**: `npm run lint` and `npm test`
413+
5. **Submit** a pull request with a clear description
517414

518-
**Note**: The `prepublishOnly` script automatically runs the build before publishing.
415+
See [Submitting Changes](guides/contributor/SUBMITTING_CHANGES.md) for detailed instructions.
519416

520417
## Roadmap
521418

guides/README.md

Lines changed: 14 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,29 @@
1-
# User Guides
1+
# Guides
22

3-
This directory contains hand-written guides and tutorials for spark.js.
3+
This directory contains documentation and guides for spark.js users and contributors.
44

5-
## Available Guides
5+
## User Guides
66

77
- **[DataFrameWriterV2.md](DataFrameWriterV2.md)** - Advanced write operations using the V2 Writer API
88
- **[STATISTICAL_FUNCTIONS.md](STATISTICAL_FUNCTIONS.md)** - Statistical functions and their usage
99

10+
## Contributor Guides
11+
12+
For contributors, see the [Contributor Guide](contributor/):
13+
14+
- **[Getting Started](contributor/GETTING_STARTED.md)** - Set up your development environment
15+
- **[Code Style Guide](contributor/CODE_STYLE.md)** - Coding conventions and best practices
16+
- **[Build and Test](contributor/BUILD_AND_TEST.md)** - Building, testing, and running the project
17+
- **[Submitting Changes](contributor/SUBMITTING_CHANGES.md)** - How to submit pull requests
18+
1019
## API Documentation
1120

1221
For comprehensive API reference documentation, see the [auto-generated API docs](https://yaooqinn.github.io/spark.js/).
1322

1423
## Contributing
1524

1625
When adding new guides:
17-
1. Create a new `.md` file in this directory
26+
1. Create a new `.md` file in the appropriate directory
1827
2. Add a link to it in this README
1928
3. Reference it from the main README if appropriate
29+

0 commit comments

Comments
 (0)