Skip to content

Commit d9fec11

Browse files
jfk9wjfk9w
committed
docs & fixes
1 parent 751070d commit d9fec11

File tree

10 files changed

+380
-154
lines changed

10 files changed

+380
-154
lines changed

‎README.md‎

Lines changed: 165 additions & 102 deletions
Original file line numberDiff line numberDiff line change
@@ -1,129 +1,192 @@
11
# hikkabot
22

3-
A Telegram subscription service for [2ch.hk](https://2ch.hk) and [reddit.com](https://reddit.com).
4-
Available at [@h1kkabot](https://t.me/h1kkabot).
3+
Telegram bot which allows relaying third-party feed updates to Telegram chats.
54

5+
## Usage
6+
7+
Hikkabot requires a configuration file. The path must be provided as the first command-line argument.
8+
9+
You can use [this template](https://github.com/jfk9w/hikkabot/raw/master/config_template.yml) to build the configuration upon.
10+
11+
### Installation and execution
12+
13+
Install using Go package manager:
14+
15+
```bash
16+
$ go install github.com/jfk9w/hikkabot
17+
$ hikkabot config.yml
18+
```
19+
20+
Logs are printed to stdout.
621

722
## Features
823

9-
* Subscribe to a 2ch thread/board or a subreddit.
10-
* Manage group and channel subscriptions.
11-
* Receive alerts to all group/channel administrators about subscription changes.
12-
* Automatic WebM-to-MP4 conversion to leverage Telegram built-in video player.
24+
* Aggregator relays updates from various pluggable content feed providers ("vendors").
25+
* Supports PostgreSQL and SQLite3 as aggregator backends (including in-memory with no strings attached).
26+
* Automatically extracts direct media links from reddit submissions.
27+
* Converts webm to mp4 in order to leverage Telegram built-in video player.
28+
* Supports navigation/filtration via hashtags (use Telegram X on mobiles for best experience).
29+
* Detects media duplicates and filters them out when applicable.
30+
31+
### Vendors
32+
33+
Vendor is a content feed provider. It is responsible for parsing subscription options
34+
and loading, parsing and formatting feed updates and media attachments.
35+
36+
A user interacts with a vendor via subscription command. It looks like this:
37+
`/sub SUB [CHAT_REF] [OPTIONS]`, where:
38+
* `SUB` is the desired subscription. It can be in the form of URL or ID, whatever is required by the vendor.
39+
* `[CHAT_REF]` is the reference to the chat you wish to add the subscription to.
40+
It should either be an alias, a channel username without the leading `@`, or `.` for the current chat.
41+
Defaults to `.`
42+
* `[OPTIONS]` is a string of subscription options. These are vendor-specific.
43+
44+
#### 2ch/catalog
45+
46+
###### Features
47+
48+
* Watches new threads on the specified board of [2ch.hk](https://2ch.hk).
49+
* Can filter threads based on a regular expression applied to the OP text.
50+
* Can render a "subscribe" button in order to quickly subscribe a given chat to new threads via `2ch/thread` vendor.
51+
52+
###### Options
53+
54+
A regular expression can be passed in order to filter new threads based on their contents.
55+
56+
`auto` option enables thread subscription button rendering.
57+
`auto` is followed by `[CHAT_REF] [OPTIONS]` which are passed directly to the subscription command when pressing the rendered button.
58+
59+
###### Examples
60+
61+
* `/sub /b .` will subscribe the current chat to all new thread updates in /b/.
62+
* `/sub /mobi channel_a (привет|пока)` will subscribe @channel_a to all new threads in /mobi/
63+
where a content substring matches `(привет|пока)` regular expression.
64+
* `/sub /pr channel_a привет auto channel_b !m` will subscribe @channel_a to all new threads in /pr/
65+
where a content substring matches `привет` regular expression
66+
with thread subscription button targeted at @channel_b with an `!m` option.
67+
68+
#### 2ch/thread
69+
70+
###### Features
71+
72+
* Watch for post updates in any given thread on [2ch.hk](https://2ch.hk).
73+
* Relay both text and media updates with preserved formatting.
74+
* Relay only images and videos from new posts with automatic media deduplication.
75+
* Reply and thread navigation based on hashtags.
76+
* Automatic webm to mp4 conversion.
77+
78+
###### Options
79+
80+
`m` option can be passed in order to relay only media updates.
81+
82+
`#hashtag_text` can be passed in order to insert
83+
`#hashtag_text` in every thread post instead of a hashtag inferred from
84+
thread title text. May be useful for thread grouping based on a common subject.
85+
86+
###### Examples
87+
88+
* `/sub https://2ch.hk/b/res/123456.html .` will subscribe the current chat to all post updates in https://2ch.hk/b/res/123456.html.
89+
* `/sub https://2ch.hk/b/res/123456.html channel_a m` will subscribe @channel_a to all media updates in https://2ch.hk/b/res/123456.html.
90+
91+
#### subreddit
92+
93+
###### Features
94+
95+
* Watch for new posts updates in any given subreddit on [reddit](https://reddit.com).
96+
* Filter out unpopular posts.
97+
* Relay both text and media updates with preserved formatting.
98+
* Relay only images and videos from new posts with automatic media deduplication.
99+
* Reply and thread navigation based on hashtags.
100+
* Automatic image & video direct link extraction and embedding.
101+
102+
###### Options
103+
104+
`!m` option can be passed in order to relay both text and media updates.
105+
By default, only media updates are relayed.
106+
107+
A floating number between `0` and `1` can be passed in order
108+
to specify the ratio of best posts which will be relayed.
109+
By default `0.3`, this means that only top 30% of all posts will make it into updates.
110+
111+
###### Examples
112+
113+
* `/sub /r/meirl .` will subscribe the current chat to media updates from `/r/meirl`.
114+
* `/sub /r/meirl channel_a !m 0.5` will relay to `@channel_a` top 50% of both text and media posts of all posts from `/r/meirl`.
115+
116+
###### Post samples
117+
118+
<img src="https://github.com/jfk9w/hikkabot/raw/master/doc/subreddit-image.png" height="400px"></img>
119+
<img src="https://github.com/jfk9w/hikkabot/raw/master/doc/subreddit-text.png" height="300px"></img>
120+
121+
### Subscription management
122+
123+
All notifications about subscription changes will be sent to `supervisor_id`.
124+
These will contain buttons to help you manage the subscription during its lifecycle.
125+
Note that some emoji-coding is used: fire emoji means "started" or "resumed",
126+
stop sign means "suspended", and wastebasket means "removed".
13127

14128
### Available commands
15129

16-
#### sub
130+
In addition to button control there are also commands which you can
131+
enter manually. Apart from `/sub` mentioned earlier there are also:
17132

18-
Usage: `/sub ITEM [CHAT_REF] [OPTIONS]`
133+
###### /status
19134

20-
| Parameter | Description |
21-
|-----------|-------------|
22-
| `ITEM` | See available services and their items description below. |
23-
| `CHAT_REF` | Chat to be subscribed to this item. Should be either empty or `.` for this chat and a username for any channel. |
24-
| `OPTIONS` | Each item has its own options, check below. |
135+
Returns `OK` for all users enriched with some debug information
136+
only for the supervisor.
25137

138+
###### /clear PATTERN [CHAT_REF]
26139

27-
#### suspend & resume
140+
Removes all subscriptions with errors like `PATTERN`.
28141

29-
Usage: `/suspend ITEM_ID` OR `/resume ITEM_ID`
142+
`PATTERN` is the value which will be passed to SQL "like" query. So
143+
something like `%404%` will match `Error code 404: page not found`.
30144

31-
Suspend an active subscription OR Resume an inactive subscription.
145+
`CHAT_REF` is optional and is the same as in `/sub` command.
32146

33-
`ITEM_ID` is a primary subscription item ID and is generally not exposed to an end-user.
34-
As such, manual execution of these commands is not possible.
35-
Instead, under every subscription state change message sent to all chat administrators there is a Suspend or Resume button.
147+
###### /list [CHAT_REF] [r]
36148

37-
#### status
149+
Lists subscriptions with buttons for suspending/resuming.
38150

39-
Usage: `/status`
151+
`CHAT_REF` is optional and is the same as in `/sub` command.
40152

41-
Simply returns an `OK` string.
153+
`r` is the option you can pass in order to list only active subscriptions.
154+
By default `/list` will return only suspended subscriptions if there are any,
155+
otherwise it will return all active subscriptions (so all subscriptions basically).
42156

43-
### Available services
157+
#### Example (with pictures)
44158

45-
| Service | Item | Item examples | Options |
46-
|---|---|---|---|
47-
| Dvach/Thread | Thread URL | `https://2ch.hk/b/res/12345678.html` | `m` for streaming only media files. |
48-
| Dvach/Catalog | Board URL or code (end slashes are optional) | `https://2ch.hk/b[/]` <br> `/b[/]` | Regexp for filtering threads. Defaults to `.*`. |
49-
| Reddit | Subreddit URL or code with optional sort | `https://reddit.com/r/meirl[/hot]` <br> `/r/meirl[/hot]` | Minimum amount of ups. Defaults to -1. |
159+
We start with a fresh channel. Below you can see that `/list` returns
160+
zero active subscriptions which means there are no subscriptions at all.
161+
Please ignore message time inconsistencies.
50162

163+
<img src="https://github.com/jfk9w/hikkabot/raw/master/doc/list-0-subs.png" height="150px"></img>
51164

52-
## Usage
165+
Let's subscribe our chat to `meirl` subreddit. `0.05` means that only top 5% of the posts
166+
should make it to our channel. In response we receive a notification which contains
167+
the button allowing to suspend the subscription:
53168

54-
### Prerequisites
55-
56-
An SQL database server should be installed (PostgreSQL recommended).
57-
58-
### Configuration
59-
60-
Hikkabot requires a JSON configuration file. The path must be provided as the first command-line argument.
61-
62-
You can use [this skeleton](https://github.com/jfk9w/hikkabot/blob/master/config.json) to build the configuration upon.
63-
64-
#### Description
65-
66-
```json5
67-
{
68-
// Your telegram user ID
69-
"AdminID": 12345678,
70-
"Storage": {
71-
// SQL driver to be used
72-
"Driver": "postgres",
73-
// Datasource connection string
74-
"Datasource": "postgres://postgres:postgres@postgres/postgres"
75-
},
76-
// How often should subscription update check occur
77-
"UpdateInterval": "10s",
78-
// Aliases allow to map certain CHAT_REFs to telegram chat IDs
79-
// May be useful for managing private channels and aliasing long channel usernames
80-
"Aliases": {
81-
"my_ref": -10000000000000000
82-
},
83-
"Telegram": {
84-
// Telegram Bot API token
85-
"Token": "2352626236:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
86-
},
87-
"Media": {
88-
"Aconvert": {
89-
// Maximum queue size fot aconvert.com client
90-
"QueueSize": 100,
91-
// Times a failed request will be retried
92-
"MaxRetries": 3,
93-
// Test file is used for aconvert.com client initialization
94-
// Should be a path to any media file supported by aconvert.com
95-
"TestFile": "test_media.webm",
96-
// Test format is used for aconvert.com client initialization
97-
// This is a target format for a media file specified above
98-
"TestFormat": "mp4"
99-
},
100-
// Number of maximum simultaneous media downloads
101-
"Workers": 13,
102-
// Temporary directory for media files
103-
"TempDir": "/tmp/hikkabot"
104-
},
105-
"Dvach": {
106-
// 2ch.hk client is updated with a cookie with this usercode
107-
// This may allow to access /e/, /hc/, etc.
108-
"Usercode": "fcdebafcde8642143"
109-
},
110-
"Reddit": {
111-
"ClientID": "iuYGY&b328fd",
112-
"ClientSecret": "a088HUUIHEAFjn9hesg",
113-
"Username": "username",
114-
"Password": "password",
115-
"UserAgent": "Your user agent 5.0",
116-
// Times a failed request will be retried
117-
"MaxRetries": 3
118-
}
119-
}
120-
```
169+
<img src="https://github.com/jfk9w/hikkabot/raw/master/doc/sub-ok.png" height="150px"></img>
121170

122-
### Installation and execution
171+
Check that `/list` returns the new subscription now.
172+
Note that `/list` outputs a button for each subscription.
173+
Button action depends on the context: if subscriptions are suspended,
174+
the button will resume the chosen one and vice versa.
123175

124-
Install using Go package manager:
176+
<img src="https://github.com/jfk9w/hikkabot/raw/master/doc/list-1-sub.png" height="150px"></img>
125177

126-
```bash
127-
$ go install github.com/jfk9w/hikkabot
128-
$ hikkabot config.json
129-
```
178+
We press the button and receive the notification below. Note that
179+
suspend and resume notifications are always sent only to the supervisor.
180+
181+
<img src="https://github.com/jfk9w/hikkabot/raw/master/doc/sub-suspended.png" height="150px"></img>
182+
183+
The suspend notification contains buttons to resume and delete the suspended subscription.
184+
We could use the latter, but instead (just to show off) let's use the `/clear` command
185+
(note how we infer the pattern from the error message above):
186+
187+
<img src="https://github.com/jfk9w/hikkabot/raw/master/doc/clear-1-sub.png" height="150px"></img>
188+
189+
That's it! Our channel is as good as new.
190+
Sorry for using the same pic.
191+
192+
<img src="https://github.com/jfk9w/hikkabot/raw/master/doc/list-0-subs.png" height="150px"></img>

‎config_template.yml‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
# datasource settings
2+
datasource:
3+
# either "sqlite3" or "postgres"
4+
driver: "sqlite3"
5+
# use "file::memory:?cache=shared" with sqlite3 in order to start in-memory
6+
# use "postgresql://user:pass@host:port/db" for postgres connection
7+
# current settings will write to a file
8+
conn: "/tmp/hikkabot.sqlite3"
9+
10+
# feed update interval in format of "10s", "5m1s", "2h45m", etc.
11+
interval: "10s"
12+
13+
# optional
14+
# prometheus settings
15+
#prometheus:
16+
# # the application publishes a prometheus metrics endpoint
17+
# address: "http://localhost:8092/metrics"
18+
19+
# media settings
20+
media:
21+
# directory to store temporary data
22+
directory: "/tmp/hikkabot"
23+
# max retries for each media request
24+
retries: 5
25+
# optional
26+
# if specified, curl will be used as fallback
27+
#curl: "/usr/bin/curl"
28+
29+
# telegram-related settings
30+
telegram:
31+
# Telegram Bot API token
32+
token: "1232:5326gasd"
33+
# this user will receive all subscription management notifications
34+
supervisor: 12345678
35+
# aliases are used to alias and manage channels with long names and private channels
36+
aliases:
37+
a: -1234566788
38+
b: -1234566789
39+
40+
# optional
41+
# 2ch.hk client settings
42+
#dvach:
43+
# # your 2ch.hk usercode, used to be required for accessing /e/, /hc/, etc.
44+
# usercode: "1235dfga"
45+
46+
# you can omit the following node completely
47+
# subreddit vendor will be disabled though
48+
# if present, required fields must be filled
49+
#reddit:
50+
# # required
51+
# clientid: "client_id"
52+
# # required
53+
# clientsecret: "client_secret"
54+
# # required
55+
# username: "your_reddit_username"
56+
# # required
57+
# password: "your_reddit_password"
58+
# # optional
59+
# # max retries for reddit requests
60+
# maxretries: 3
61+
# # optional
62+
# # used to compose user-agent header (as required by reddit)
63+
# # by default infers from username
64+
# owner: "your_another_username"

doc/subreddit-image-example.png renamed to doc/subreddit-image.png

File renamed without changes.

doc/subreddit-text-example.png renamed to doc/subreddit-text.png

File renamed without changes.

‎feed/media_client.go‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -61,7 +61,12 @@ type DefaultMediaClient struct {
6161

6262
func NewMediaClient(client *fluhttp.Client, curl string, retries int) DefaultMediaClient {
6363
main := StdLibClient{client}
64-
fallback := CURL{curl}
64+
var fallback MediaClient
65+
if curl != "" {
66+
fallback = CURL{curl}
67+
} else {
68+
fallback = main
69+
}
6570
return DefaultMediaClient{main, fallback, retries}
6671
}
6772

0 commit comments

Comments
 (0)