Skip to content

Commit 52f17b7

Browse files
committed
feat: update README.md
1 parent cfe0b45 commit 52f17b7

File tree

1 file changed

+12
-11
lines changed

1 file changed

+12
-11
lines changed

README.md

Lines changed: 12 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -4,25 +4,26 @@ Fastcrawl is a polite, configurable web-crawler focused on continuous streaming
44
Wikipedia example (`examples/wiki.rs`) that demonstrates how to plug custom link filters and crawl controls into the
55
core runtime.
66

7-
Current fastest speed, with default controls of `max-depth` 4, `max-links-per-page` 16, `politeness-ms` 250, and
8-
`partition-strategy` 'wiki-prefix' (instead of 'hash') is **52.69 pages/sec**.
7+
Current fastest speed, with default controls of `max-depth` 4, `max-links-per-page` 16, `politeness-ms` 250,
8+
`partition-strategy` 'wiki-prefix' (instead of 'hash'), and `duration-secs` 4 (it crawls for 4 seconds, but any enqued
9+
link is still awaited, so it runs for approxximately 30 sec.) is **68.98 pages/sec**.
910

1011
## Metrics
1112

12-
When running `cargo run --example wiki --features multi_thread -- --duration-secs 1 --partition wiki-prefix`:
13+
When running `cargo run --example wiki --features multi_thread -- --duration-secs 4 --partition wiki-prefix`:
1314

1415
```
15-
--- crawl metrics (4.00s) ---
16-
pages fetched: 211
17-
urls fetched/sec: 52.69
18-
urls discovered: 388
19-
urls enqueued: 207
20-
duplicate skips: 181
16+
--- crawl metrics (29.43s) ---
17+
pages fetched: 2030
18+
urls fetched/sec: 68.98
19+
urls discovered: 4844
20+
urls enqueued: 2026
21+
duplicate skips: 2818
2122
frontier rejects: 0
2223
http errors: 0
2324
url parse errors: 0
24-
local shard enqueues: 1301
25-
remote shard links: 277 (batches 114)
25+
local shard enqueues: 9315
26+
remote shard links: 3039 (batches 1442)
2627
```
2728

2829
## Highlights

0 commit comments

Comments
 (0)