|
2 | 2 |
|
3 | 3 | # html2rss-web |
4 | 4 |
|
5 | | -This web application scrapes websites to build and deliver RSS 2.0 feeds with a modern, responsive interface. |
| 5 | +html2rss-web converts arbitrary websites into RSS 2.0 feeds with a slim Ruby backend and an Astro-powered frontend. |
6 | 6 |
|
7 | | -**Features:** |
| 7 | +## Highlights |
| 8 | +- Responsive Astro interface with gallery and custom feed creation. |
| 9 | +- Automatic source discovery with token-scoped permissions. |
| 10 | +- Signed public feed URLs that work in standard RSS readers. |
| 11 | +- Built-in SSRF defences, input validation, and HMAC-protected tokens. |
8 | 12 |
|
9 | | -- **Modern UI**: Clean, responsive Astro-based frontend with component architecture |
10 | | -- **Feed Gallery**: Browse and discover popular RSS feeds |
11 | | -- **Auto Source**: Generate feeds from any website automatically |
12 | | -- **Stable URLs**: Provides stable URLs for feeds generated by automatic sourcing |
13 | | -- **Public Feed Access**: Secure, token-based public access to RSS feeds without authentication headers |
14 | | -- **Custom Feeds**: [Create your custom feeds](https://html2rss.github.io/web-application/tutorials/building-feeds)! |
15 | | -- **Pre-built Configs**: Comes with plenty of [included configs](https://html2rss.github.io/web-application/how-to/use-included-configs) |
16 | | -- **Performance**: Handles request caching and sets caching-related HTTP headers |
17 | | -- **Progressive Enhancement**: Works without JavaScript, enhanced with modern features |
18 | | -- **Security**: URL restrictions, authentication, SSRF protection, and input validation |
| 13 | +## Architecture |
| 14 | +- **Backend:** Ruby + Roda, backed by the `html2rss` gem for extraction. |
| 15 | +- **Frontend:** Astro static site with progressive enhancement. |
| 16 | +- **Distribution:** Docker Compose by default; other deployments require manual wiring. |
19 | 17 |
|
20 | | -**Architecture:** |
21 | | - |
22 | | -- **Backend**: Ruby + Roda for API and RSS generation |
23 | | -- **Frontend**: Astro for modern, fast static site generation |
24 | | -- **Core Engine**: [`html2rss`](https://github.com/html2rss/html2rss) Ruby gem for feed generation |
25 | | - |
26 | | -## Configuration |
27 | | - |
28 | | -The application can be configured using environment variables. See the [configuration guide](CONFIGURATION.md) for details. |
29 | | - |
30 | | -### Security Features |
31 | | - |
32 | | -- **URL Restrictions**: Public instances can restrict auto source to specific URLs |
33 | | -- **Authentication**: Token-based authentication for auto source and health check endpoints |
34 | | -- **Public Feed Access**: Secure, stateless feed tokens for public RSS access |
35 | | -- **HMAC-SHA256 Signing**: Cryptographically signed tokens prevent tampering |
36 | | -- **URL Binding**: Feed tokens are bound to specific URLs for security |
37 | | -- **SSRF Protection**: Built-in protection against Server-Side Request Forgery |
38 | | -- **Input Validation**: Comprehensive validation of all inputs |
39 | | -- **XML Sanitization**: Prevents XML injection attacks in RSS output |
40 | | -- **CSP Headers**: Content Security Policy headers prevent XSS attacks |
41 | | - |
42 | | -## RESTful API |
43 | | - |
44 | | -The application now provides a modern RESTful API v1 that follows industry standards: |
45 | | - |
46 | | -- **Resource-based URLs**: `/api/v1/feeds`, `/api/v1/strategies` |
47 | | -- **HTTP methods**: GET, POST, PUT, DELETE |
48 | | -- **Proper status codes**: 200, 201, 400, 401, 403, 404, 500 |
49 | | -- **JSON responses**: Consistent response format |
50 | | -- **Content negotiation**: XML for RSS feeds, JSON for metadata |
51 | | -- **OpenAPI documentation**: Complete API specification |
52 | | - |
53 | | -### Quick Start |
| 18 | +## Documentation |
| 19 | +The in-repo docs live under `frontend/src/content/docs/` and are published by Astro. |
| 20 | +- [Configuration Guide](frontend/src/content/docs/configuration.md) |
| 21 | +- [Security Guide](frontend/src/content/docs/security.md) |
| 22 | +- [REST API v1](frontend/src/content/docs/api/v1.md) |
| 23 | +- [Testing Overview](frontend/src/content/docs/testing.md) |
54 | 24 |
|
| 25 | +## REST API Snapshot |
55 | 26 | ```bash |
56 | | -# List available feeds |
57 | | -curl "https://your-domain.com/api/v1/feeds" |
| 27 | +# List feeds available to the token |
| 28 | +curl -H "Authorization: Bearer <token>" \ |
| 29 | + "https://your-domain.com/api/v1/feeds" |
58 | 30 |
|
59 | | -# Create a new feed |
| 31 | +# Create a feed and capture the signed public URL |
60 | 32 | curl -X POST "https://your-domain.com/api/v1/feeds" \ |
61 | | - -H "Authorization: Bearer your-token" \ |
| 33 | + -H "Authorization: Bearer <token>" \ |
62 | 34 | -H "Content-Type: application/json" \ |
63 | | - -d '{"url": "https://example.com", "name": "Example Feed"}' |
64 | | - |
65 | | -# Get feed metadata |
66 | | -curl "https://your-domain.com/api/v1/feeds/example" |
67 | | - |
68 | | -# Get RSS content |
69 | | -curl -H "Accept: application/xml" "https://your-domain.com/api/v1/feeds/example" |
| 35 | + -d '{"url":"https://example.com","name":"Example Feed"}' |
70 | 36 | ``` |
71 | 37 |
|
72 | | -### Documentation |
| 38 | +## Deploy with Docker Compose |
| 39 | +The supported path is Docker Compose. |
73 | 40 |
|
74 | | -- [RESTful API v1 Documentation](docs/api/v1/README.md) |
75 | | -- [OpenAPI Specification](docs/api/v1/openapi.yaml) |
76 | | - |
77 | | -## Public Feed Access |
78 | | - |
79 | | -The application supports secure public access to RSS feeds without requiring authentication headers. This is perfect for sharing feeds with RSS readers and other applications. |
80 | | - |
81 | | -### How It Works |
| 41 | +### Prerequisites |
| 42 | +- Docker Engine and Docker Compose |
| 43 | +- Git for cloning the repository |
82 | 44 |
|
83 | | -1. **Create a Feed**: Use the RESTful API or auto source feature to generate a feed |
84 | | -2. **Get Public URL**: The system returns a public URL with an embedded token |
85 | | -3. **Share the URL**: Anyone can access the feed using this URL |
86 | | -4. **Secure Access**: The token is cryptographically signed and URL-bound |
| 45 | +### Steps |
| 46 | +1. Clone the repository and change into the directory. |
| 47 | +2. Generate a 64-character hexadecimal key: `openssl rand -hex 32`. |
| 48 | +3. Update `docker-compose.yml` with the key via the `HTML2RSS_SECRET_KEY` environment variable. |
| 49 | +4. Start the stack: `docker-compose up`. |
87 | 50 |
|
88 | | -### Example |
| 51 | +The application serves the UI and API at `http://localhost:3000`. It fails fast if the secret key is missing. |
89 | 52 |
|
90 | | -```bash |
91 | | -# Create a feed via RESTful API |
92 | | -curl -X POST "https://your-domain.com/api/v1/feeds" \ |
93 | | - -H "Authorization: Bearer your-token" \ |
94 | | - -H "Content-Type: application/json" \ |
95 | | - -d '{"url": "https://example.com", "name": "Example Feed"}' |
96 | | - |
97 | | -# Response includes public_url |
98 | | -{ |
99 | | - "success": true, |
100 | | - "data": { |
101 | | - "feed": { |
102 | | - "id": "abc123", |
103 | | - "name": "Example Feed", |
104 | | - "url": "https://example.com", |
105 | | - "public_url": "/api/v1/feeds/eyJwYXlsb2FkIjoi..." |
106 | | - } |
107 | | - } |
108 | | -} |
109 | | - |
110 | | -# Access the feed publicly using the signed token |
111 | | -curl "https://your-domain.com/api/v1/feeds/eyJwYXlsb2FkIjoi..." |
| 53 | +## Frontend Development |
112 | 54 | ``` |
113 | | - |
114 | | -### Security Features |
115 | | - |
116 | | -- **10-Year Expiry**: Tokens are valid for 10 years (perfect for RSS) |
117 | | -- **URL Binding**: Tokens only work for their specific URL |
118 | | -- **HMAC Signing**: Tokens cannot be tampered with |
119 | | -- **No Server Storage**: Stateless validation, no database required |
120 | | - |
121 | | -## Documentation |
122 | | - |
123 | | -For full documentation, please see the [html2rss-web documentation](https://html2rss.github.io/web-application/). |
124 | | - |
125 | | -### Security and Deployment |
126 | | - |
127 | | -- [Security Guide](SECURITY.md) - Comprehensive security documentation |
128 | | -- [Project Website](https://html2rss.github.io/html2rss-web/) - Deployment and usage instructions |
129 | | - |
130 | | -## Quick Start |
131 | | - |
132 | | -This application is designed to be used via Docker Compose only. |
133 | | - |
134 | | -### Prerequisites |
135 | | - |
136 | | -- Docker and Docker Compose installed |
137 | | -- Git (to clone the repository) |
138 | | - |
139 | | -### Setup and Run |
140 | | - |
141 | | -1. **Clone the repository:** |
142 | | - ```bash |
143 | | - git clone https://github.com/html2rss/html2rss-web.git |
144 | | - cd html2rss-web |
145 | | - ``` |
146 | | - |
147 | | -2. **Generate a secret key:** |
148 | | - ```bash |
149 | | - openssl rand -hex 32 |
150 | | - ``` |
151 | | - |
152 | | -3. **Configure docker-compose.yml:** |
153 | | - ```bash |
154 | | - # Edit the file and replace 'your-generated-secret-key-here' with your actual secret key |
155 | | - # The docker-compose.yml file is already included in the repository |
156 | | - ``` |
157 | | - |
158 | | -4. **Start the application:** |
159 | | - ```bash |
160 | | - docker-compose up |
161 | | - ``` |
162 | | - |
163 | | -The application will be available at `http://localhost:3000`. |
164 | | - |
165 | | -**⚠️ Important**: The `HTML2RSS_SECRET_KEY` environment variable is required. Without it, the application will not start and will display setup instructions. |
166 | | - |
167 | | -### Frontend Development |
168 | | - |
169 | | -The project includes a modern Astro frontend alongside the Ruby backend: |
170 | | - |
171 | | -1. **Frontend development:** |
172 | | - |
173 | | - ```bash |
174 | | - cd frontend |
175 | | - npm install |
176 | | - npm run dev |
177 | | - ``` |
178 | | - |
179 | | -2. **Build frontend:** |
180 | | - ```bash |
181 | | - cd frontend |
182 | | - npm run build |
183 | | - ``` |
184 | | - |
185 | | -**Development Workflow:** |
186 | | - |
187 | | -- **`make dev`** - Starts both Ruby server (port 3000) and Astro dev server (port 4321) with hot reload |
188 | | -- **Ruby server** - Serves the API and built frontend at `/app` and `/app/gallery` |
189 | | -- **Astro dev server** - Provides hot reload for frontend development at `http://localhost:4321` |
190 | | -- **Hot reload** - Changes to frontend files automatically rebuild and refresh |
191 | | - |
192 | | -### Development Commands |
193 | | - |
194 | | -| Command | Description | |
195 | | -| -------------------- | ------------------------------------- | |
196 | | -| `make help` | Show all available commands | |
197 | | -| `make setup` | Full development setup | |
198 | | -| `make dev` | Start both Ruby and Astro dev servers | |
199 | | -| `make dev-ruby` | Start Ruby server only | |
200 | | -| `make dev-frontend` | Start Astro dev server only | |
201 | | -| `make test` | Run all tests (Ruby + Frontend) | |
202 | | -| `make test-ruby` | Run Ruby tests only | |
203 | | -| `make test-frontend` | Run frontend unit + contract tests | |
204 | | -| `make test-frontend-unit` | Run frontend unit tests only | |
205 | | -| `make test-frontend-contract` | Run frontend contract tests (Vitest + MSW) | |
206 | | -| `make lint` | Run all linters (Ruby + Frontend) | |
207 | | -| `make lint-ruby` | Run Ruby linter only | |
208 | | -| `make lint-js` | Run frontend linter only | |
209 | | -| `make lintfix` | Auto-fix all linting issues | |
210 | | -| `make lintfix-ruby` | Auto-fix Ruby linting issues | |
211 | | -| `make lintfix-js` | Auto-fix frontend linting issues | |
212 | | -| `make clean` | Clean temporary files | |
213 | | - |
214 | | -### Frontend Commands |
215 | | - |
216 | | -| Command | Description | |
217 | | -| ------------------------------ | ----------------------------- | |
218 | | -| `cd frontend && npm install` | Install frontend dependencies | |
219 | | -| `cd frontend && npm run dev` | Start frontend dev server | |
220 | | -| `cd frontend && npm run build` | Build frontend for production | |
221 | | -| `cd frontend && npm run test:unit` | Run unit tests (Vitest) | |
222 | | -| `cd frontend && npm run test:contract` | Run contract tests with MSW | |
223 | | - |
224 | | -### Testing Strategy |
225 | | - |
226 | | -| Layer | Tooling | Notes | |
227 | | -| ---------------- | -------------------------- | ----- | |
228 | | -| Ruby API | RSpec + Rack::Test | Deterministic request specs for the Roda app | |
229 | | -| Frontend Unit | Vitest + Testing Library | Hook/component behaviour with mocked fetch | |
230 | | -| Frontend Contract| Vitest + MSW | Exercises real fetch flows against mocked API responses | |
231 | | -| Docker Smoke | RSpec (tagged `:docker`) | Net::HTTP checks against a bundled Puma container | |
| 55 | +cd frontend |
| 56 | +npm install |
| 57 | +npm run dev |
| 58 | +``` |
| 59 | +The Ruby server continues to serve the production build while Astro runs with hot reload on port 4321. |
| 60 | + |
| 61 | +## Make Targets |
| 62 | + |
| 63 | +| Command | Purpose | |
| 64 | +| --- | --- | |
| 65 | +| `make help` | List available shortcuts. | |
| 66 | +| `make setup` | Install Ruby and Node dependencies. | |
| 67 | +| `make dev` | Run Ruby (port 3000) and Astro (port 4321) dev servers. | |
| 68 | +| `make dev-ruby` | Start only the Ruby server. | |
| 69 | +| `make dev-frontend` | Start only the Astro dev server. | |
| 70 | +| `make test` | Run Ruby and frontend test suites. | |
| 71 | +| `make test-ruby` | Run Ruby specs. | |
| 72 | +| `make test-frontend` | Run frontend unit and contract tests. | |
| 73 | +| `make lint` | Run all linters. | |
| 74 | +| `make lintfix` | Auto-fix lint warnings where possible. | |
| 75 | +| `make clean` | Remove build artefacts. | |
| 76 | + |
| 77 | +## Frontend npm Scripts |
| 78 | + |
| 79 | +| Command | Purpose | |
| 80 | +| --- | --- | |
| 81 | +| `npm run dev` | Astro dev server with hot reload. | |
| 82 | +| `npm run build` | Production build. | |
| 83 | +| `npm run test:run` | Unit tests (Vitest). | |
| 84 | +| `npm run test:contract` | Contract tests with MSW. | |
| 85 | + |
| 86 | +## Testing Strategy |
| 87 | + |
| 88 | +| Layer | Tooling | Focus | |
| 89 | +| --- | --- | --- | |
| 90 | +| Ruby API | RSpec + Rack::Test | Feed creation, retrieval, auth paths. | |
| 91 | +| Frontend unit | Vitest + Testing Library | Component rendering and hooks with mocked fetch. | |
| 92 | +| Frontend contract | Vitest + MSW | End-to-end fetch flows against mocked API responses. | |
| 93 | +| Docker smoke | RSpec (`:docker`) | Net::HTTP probes against the containerised service. | |
232 | 94 |
|
233 | 95 | ## Contributing |
234 | 96 |
|
235 | | -Contributions are welcome! Please see the [contributing guide](https://html2rss.github.io/get-involved/contributing) for more information. |
| 97 | +Contributions are welcome. See the [html2rss project guidelines](https://html2rss.github.io/get-involved/contributing) before opening a pull request. |
236 | 98 |
|
237 | 99 | ## Sponsoring |
238 | 100 |
|
239 | | -If you find this project useful, please consider [sponsoring the project](https://github.com/sponsors/gildesmarais). |
| 101 | +Support ongoing development via [GitHub Sponsors](https://github.com/sponsors/gildesmarais). |
0 commit comments