Skip to content

Commit df99cb8

Browse files
elliotBraemelliotBraem
authored
Upgrade RSS plugin and object transform (#7)
* wip * update telegram to escape characters * update transform to transformer type * fix plugin type map * near-rewards * Adds near social plugin (#6) * init plugin * fixed frontend * working plugin * fix test * fmt * fmt * fix tests * wip * simpler plugin * wip * clean-up * move rss-service * example * updates and adds docker support * upgrade object-transform * fmt * bun lock * clean up rss according to comments * nit picks * nit pick * add template syncing * update readme * use actions/checkout@v4
1 parent f0e1be6 commit df99cb8

39 files changed

+2813
-180
lines changed

‎.github/workflows/README.md‎

Lines changed: 152 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,152 @@
1+
# 🔄 Service Sync Workflows
2+
3+
> Automate syncing service directories to deployment repositories
4+
5+
## 📋 Overview
6+
7+
This repository contains a reusable GitHub Actions workflow for syncing service directories to deployment repositories. The workflow is designed to be flexible and can be used with any service directory and target repository.
8+
9+
## 🚀 Getting Started
10+
11+
### Reusable Workflow Template
12+
13+
The `sync-service-template.yml` workflow handles:
14+
15+
- ✅ Validating source directory exists
16+
- 🔄 Cloning target repository
17+
- 🌱 Creating target branch if needed
18+
- 📂 Syncing directory contents
19+
- 🚢 Committing and pushing changes
20+
21+
### Usage
22+
23+
Create a new workflow file for your service (e.g. `sync-your-service.yml`):
24+
25+
```yaml
26+
name: Sync Your Service
27+
on:
28+
push:
29+
paths:
30+
- 'path/to/your/service/**'
31+
branches:
32+
- main
33+
34+
jobs:
35+
sync-service:
36+
uses: ./.github/workflows/sync-service-template.yml
37+
with:
38+
source_path: path/to/your/service
39+
target_repo: org/deployment-repo-name
40+
target_branch: main # optional, defaults to main
41+
secrets:
42+
deploy_token: ${{ secrets.YOUR_DEPLOY_TOKEN }}
43+
```
44+
45+
## 🔑 Setting Up Deploy Keys
46+
47+
For each target repository, set up a deploy key for secure repository access. Deploy keys are SSH keys that grant access to a single repository. See the [GitHub documentation on deploy keys](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/managing-deploy-keys#deploy-keys) for more information.
48+
49+
### 1. Generate an SSH Key Pair
50+
51+
Generate a new SSH key pair specifically for this deployment:
52+
53+
```bash
54+
ssh-keygen -t ed25519 -C "your_email@example.com" -f deploy_key_filename
55+
```
56+
57+
This creates:
58+
59+
- `deploy_key_filename` (private key)
60+
- `deploy_key_filename.pub` (public key)
61+
62+
When prompted for a passphrase, press Enter to create a key without a passphrase (since it will be used in automated workflows).
63+
64+
To display the public key so you can copy it:
65+
66+
```bash
67+
# On macOS/Linux
68+
cat deploy_key_filename.pub
69+
70+
# On Windows (PowerShell)
71+
Get-Content deploy_key_filename.pub
72+
```
73+
74+
For more detailed instructions on generating SSH keys, see [GitHub's documentation on generating a new SSH key](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent#generating-a-new-ssh-key).
75+
76+
> **Security Note**: Always create a unique key for each repository. Never reuse SSH keys across different repositories or purposes.
77+
78+
### 2. Add the Public Key to the Target Repository
79+
80+
- Go to the target repository on GitHub
81+
- Navigate to Settings > Deploy keys
82+
- Click "Add deploy key"
83+
- Title: Give it a descriptive name (e.g. "Service Sync Deploy Key")
84+
- Key: Paste the contents of `deploy_key_filename.pub`
85+
- ✅ Check "Allow write access" if the workflow needs to push changes
86+
- Click "Add key"
87+
88+
### 3. Add the Private Key as a Repository Secret
89+
90+
- Go to your source repository's Settings > Secrets and variables > Actions
91+
- Click "New repository secret"
92+
- Name: Use a descriptive name (e.g. `RSS_DEPLOY_TOKEN`)
93+
- Value: Paste the contents of the `deploy_key_filename` file
94+
- Click "Add secret"
95+
96+
### 4. Use the Deploy Key in Your Workflow
97+
98+
The workflow template in this repository is already configured to use SSH deploy keys. The `sync-service-template.yml` workflow:
99+
100+
1. Uses the [webfactory/ssh-agent](https://github.com/webfactory/ssh-agent) action to securely install the SSH key
101+
2. Automatically handles SSH authentication for git operations
102+
3. Uses SSH for git clone and push operations
103+
104+
No additional modifications are needed to use deploy keys with this workflow.
105+
106+
## 📊 Example: RSS Service
107+
108+
The RSS service workflow (`sync-rss-service.yml`) demonstrates how to use the template:
109+
110+
```yaml
111+
name: Sync RSS Service
112+
on:
113+
push:
114+
paths:
115+
- 'packages/rss/service/**'
116+
branches:
117+
- main
118+
119+
jobs:
120+
sync-rss:
121+
uses: ./.github/workflows/sync-service-template.yml
122+
with:
123+
source_path: packages/rss/service
124+
target_repo: curatedotfun/rss-service-template
125+
target_branch: main
126+
secrets:
127+
deploy_token: ${{ secrets.RSS_DEPLOY_TOKEN }}
128+
```
129+
130+
## 🔒 Security Best Practices
131+
132+
1. **Repository-Specific Keys**: Use a unique deploy key for each target repository.
133+
134+
2. **Minimal Permissions**: Configure deploy keys with read-only access when possible.
135+
136+
3. **Key Rotation**: Regularly rotate your deploy keys for better security.
137+
138+
4. **Branch Protection**: Enable branch protection rules on target repositories to ensure changes follow your workflow.
139+
140+
## ❓ Troubleshooting
141+
142+
If the workflow fails, check:
143+
144+
1. **Source Path**: Ensure the source directory exists and contains the files you want to sync.
145+
146+
2. **Deploy Key**: Verify the key has the correct permissions and is properly configured.
147+
148+
3. **Target Repository**: Confirm the repository exists and is accessible with the provided key.
149+
150+
4. **SSH Configuration**: Check if the SSH configuration in the workflow is correct.
151+
152+
5. **Branch Access**: Verify there are no branch protection rules preventing pushes.

‎.github/workflows/near-rewards.yml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -14,10 +14,10 @@ jobs:
1414
issues: read
1515
pull-requests: read
1616
steps:
17-
- uses: actions/checkout@v3
18-
- uses: actions/setup-node@v3
17+
- uses: actions/checkout@v4
18+
- uses: actions/setup-node@v4
1919
with:
20-
node-version: '18'
20+
node-version: '20'
2121

2222
- name: Calculate Rewards
2323
env:

‎.github/workflows/publish.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,12 +9,12 @@ jobs:
99
release:
1010
runs-on: ubuntu-latest
1111
steps:
12-
- uses: actions/checkout@v3
12+
- uses: actions/checkout@v4
1313
- uses: oven-sh/setup-bun@v1
1414
- run: bun install
1515
- run: bun run build
1616
- name: Setup Node.js
17-
uses: actions/setup-node@v3
17+
uses: actions/setup-node@v4
1818
with:
1919
node-version: "lts/*"
2020
registry-url: 'https://registry.npmjs.org'

‎.github/workflows/rss-service-template.yml‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
name: Sync Service Directory
2+
on:
3+
push:
4+
paths:
5+
- 'packages/rss/service/**'
6+
branches:
7+
- main
8+
9+
jobs:
10+
sync:
11+
runs-on: ubuntu-latest
12+
steps:
13+
- uses: actions/checkout@v4
14+
- name: Sync Service Directory
15+
run: |
16+
git config --global user.name "GitHub Action"
17+
git config --global user.email "action@github.com"
18+
19+
# Clone the deployment repo
20+
git clone https://x-access-token:${{ secrets.DEPLOY_TOKEN }}@github.com/yourusername/rss-service-deploy.git
21+
22+
# Copy service directory contents
23+
rm -rf rss-service-deploy/*
24+
cp -r packages/rss/service/* rss-service-deploy/
25+
26+
# Commit and push changes
27+
cd rss-service-deploy
28+
git add .
29+
git commit -m "Sync from main repo" || echo "No changes to commit"
30+
git push

‎.github/workflows/sync-rss-service.yml‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
name: Sync RSS Service
2+
on:
3+
push:
4+
paths:
5+
- 'packages/rss/service/**'
6+
branches:
7+
- main
8+
9+
jobs:
10+
sync-rss:
11+
uses: ./.github/workflows/sync-service-template.yml
12+
with:
13+
source_path: packages/rss/service
14+
target_repo: curatedotfun/rss-service-template
15+
target_branch: main
16+
secrets:
17+
deploy_token: ${{ secrets.RSS_DEPLOY_TOKEN }}

‎.github/workflows/sync-service-template.yml‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
name: Sync Service Directory
2+
on:
3+
workflow_call:
4+
inputs:
5+
source_path:
6+
description: 'Path to the source directory to sync'
7+
required: true
8+
type: string
9+
target_repo:
10+
description: 'Name of the target repository (e.g. org/repo)'
11+
required: true
12+
type: string
13+
target_branch:
14+
description: 'Branch to push to in the target repository'
15+
required: false
16+
type: string
17+
default: 'main'
18+
secrets:
19+
deploy_token:
20+
description: 'SSH deploy key for target repository'
21+
required: true
22+
23+
jobs:
24+
sync:
25+
runs-on: ubuntu-latest
26+
steps:
27+
- uses: actions/checkout@v4
28+
with:
29+
fetch-depth: 1
30+
31+
- name: Validate inputs
32+
run: |
33+
if [ ! -d "${{ inputs.source_path }}" ]; then
34+
echo "❌ Source path ${{ inputs.source_path }} does not exist"
35+
exit 1
36+
fi
37+
echo "✅ Source path validated"
38+
39+
- name: Configure Git
40+
run: |
41+
git config --global user.name "GitHub Action"
42+
git config --global user.email "action@github.com"
43+
echo "✅ Git configured"
44+
45+
- name: Install SSH key
46+
uses: webfactory/ssh-agent@v0.8.0
47+
with:
48+
ssh-private-key: ${{ secrets.deploy_token }}
49+
50+
- name: Clone target repository
51+
run: |
52+
# Clone the deployment repo using SSH with the installed key
53+
git clone git@github.com:${{ inputs.target_repo }}.git target_repo
54+
echo "✅ Repository cloned"
55+
56+
# Ensure target branch exists
57+
cd target_repo
58+
if ! git ls-remote --heads origin ${{ inputs.target_branch }} | grep ${{ inputs.target_branch }} > /dev/null; then
59+
echo "🌱 Creating target branch ${{ inputs.target_branch }}"
60+
git checkout --orphan ${{ inputs.target_branch }}
61+
git rm -rf .
62+
git commit --allow-empty -m "Initialize branch"
63+
git push origin ${{ inputs.target_branch }}
64+
fi
65+
git checkout ${{ inputs.target_branch }}
66+
echo "✅ Branch checked out"
67+
cd ..
68+
69+
- name: Sync directory contents
70+
run: |
71+
echo "🔄 Syncing directory contents"
72+
73+
# Remove existing contents but preserve .git
74+
rm -rf target_repo/*
75+
76+
# Copy new contents
77+
cp -r ${{ inputs.source_path }}/* target_repo/
78+
79+
# Commit and push changes
80+
cd target_repo
81+
git add .
82+
git commit -m "Sync from main repo" || echo "No changes to commit"
83+
git push origin ${{ inputs.target_branch }}

‎apps/example/.env.example‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# Environment
2+
NODE_ENV=development
3+
4+
# @curatedotfun/telegram
5+
TELEGRAM_BOT_TOKEN=your:telegram-bot-token
6+
7+
# @curatedotfun/ai-transform
8+
OPENROUTER_API_KEY=sk-or-v1-your-open-router-key
9+
10+
# @curatedotfun/notion
11+
NOTION_TOKEN=ntn_your-notion-key
12+
13+
# @curatedotfun/near-social
14+
NEAR_PRIVATE_KEY=ed25519:your-private-key
15+
16+
# @curatedotfun/rss
17+
API_SECRET=dev-secret

‎apps/example/src/frontend/frontend.js‎

Lines changed: 30 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -111,7 +111,8 @@ const PLUGIN_DEFAULTS = {
111111
channelId: "@your_channel",
112112
},
113113
"@curatedotfun/rss": {
114-
url: "https://example.com/feed.xml",
114+
serviceUrl: "http://localhost:4001",
115+
apiSecret: "{API_SECRET}",
115116
},
116117
"@curatedotfun/supabase": {
117118
url: "{SUPABASE_URL}",
@@ -130,11 +131,33 @@ const DEFAULT_CONFIG = {
130131
transform: [
131132
{
132133
plugin: "@curatedotfun/ai-transform",
133-
config: PLUGIN_DEFAULTS["@curatedotfun/ai-transform"],
134+
config: {
135+
prompt:
136+
"Transform this into an engaging news article with a title and content.",
137+
apiKey: "{OPENROUTER_API_KEY}",
138+
schema: {
139+
title: {
140+
type: "string",
141+
description: "Engaging title for the article",
142+
},
143+
content: {
144+
type: "string",
145+
description: "Article content in HTML format",
146+
},
147+
},
148+
},
134149
},
135150
{
136151
plugin: "@curatedotfun/object-transform",
137-
config: PLUGIN_DEFAULTS["@curatedotfun/object-transform"],
152+
config: {
153+
mappings: {
154+
title: "{{title}}",
155+
content: "{{content}}",
156+
link: "https://example.com/posts/{{id}}",
157+
publishedAt: "{{timestamp}}",
158+
guid: "post-{{id}}",
159+
},
160+
},
138161
},
139162
{
140163
plugin: "@curatedotfun/simple-transform",
@@ -150,6 +173,10 @@ const DEFAULT_CONFIG = {
150173
plugin: "@curatedotfun/near-social",
151174
config: PLUGIN_DEFAULTS["@curatedotfun/near-social"],
152175
},
176+
{
177+
plugin: "@curatedotfun/rss",
178+
config: PLUGIN_DEFAULTS["@curatedotfun/rss"],
179+
},
153180
],
154181
};
155182

‎bun.lockb‎

2.88 KB
Binary file not shown.

‎packages/ai-transform/src/index.ts‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -130,7 +130,6 @@ export default class AITransformer<T = string>
130130
const result = (await response.json()) as OpenRouterResponse;
131131

132132
if (!result.choices?.[0]?.message?.content) {
133-
console.log("result", result);
134133
throw new Error("Invalid response structure from OpenRouter API");
135134
}
136135

0 commit comments

Comments
 (0)