[BUG] 优化 IPDB 数据库 断点续传 #1747
Description
版本号 - Version
v9.3.9
操作系统平台和系统架构 - OS and CPU Arch
OpenWrt Docker x64
部署方式 - Deploy methods
- Windows .exe 安装程序 (Windows .exe Installer)
- Windows .zip 绿色部署包 (Windows .zip Portable)
- Docker
- JAR 包直接启动 (Start via .JAR directly)
- 群晖 DSM 套件 (Synology .spk package)
- Debian 软件包 (Debian .deb package)
- RedHat/Fedora 软件包 (RedHat/Fedora .rpm package)
- FreeBSD 软件包(FreeBSD .pkg package)
添加的下载器 - Added Downloaders
- qBittorrent
- BiglyBT/Azureus/Vuze
- Deluge
- Transmission
- BitComet
下载器版本号 - Downloader Version
v5.1.4
问题描述 - Issue Description
程序在下载 IPDB 数据库时过了三分钟就尝试解压，但此时并未下载完毕，导致解压失败并使用备用源重试，且之前未完成的下载仍在进行
我猜测这个可能被用作下载速度慢时的自动换源机制，但我觉得这个实现不够优雅，浪费了一些流量和时间。应当对下载源进行测速，选择最优下载源，或者多源同时下载。
复现步骤 - Reproduce steps
在 IPDB 数据库过期后重启程序，等待程序自动更新
截图/日志文件 - Screenshot / Logs
[22:29:02] [DefaultDispatcher-worker-1/INFO]: GeoLite2-City 数据库已过期且需要更新，请等待 PBH 连接到服务器更新数据……
[22:29:05] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 6821/19849436 字节，进度：0.03%
[22:29:10] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 449301/19849436 字节，进度：2.26%
[22:29:20] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 865833/19849436 字节，进度：4.36%
[22:29:25] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 1095117/19849436 字节，进度：5.52%
[22:29:30] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 1342289/19849436 字节，进度：6.76%
[22:29:35] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 1536165/19849436 字节，进度：7.74%
[22:29:40] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 1772427/19849436 字节，进度：8.93%
[22:29:45] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 1974481/19849436 字节，进度：9.95%
[22:29:50] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 2207872/19849436 字节，进度：11.12%
[22:29:55] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 2418113/19849436 字节，进度：12.18%
[22:30:00] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 2676159/19849436 字节，进度：13.48%
[22:30:05] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 2893360/19849436 字节，进度：14.58%
[22:30:10] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 3103583/19849436 字节，进度：15.64%
[22:30:15] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 3300182/19849436 字节，进度：16.63%
[22:30:23] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 3350746/19849436 字节，进度：16.88%
[22:30:28] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 3436745/19849436 字节，进度：17.31%
[22:30:38] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 3539185/19849436 字节，进度：17.83%
[22:30:44] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 3582789/19849436 字节，进度：18.05%
[22:30:49] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 3590967/19849436 字节，进度：18.09%
[22:30:54] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 3611524/19849436 字节，进度：18.19%
[22:31:00] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 3619702/19849436 字节，进度：18.24%
[22:31:16] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 3622425/19849436 字节，进度：18.25%
[22:31:21] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 3742596/19849436 字节，进度：18.85%
[22:31:26] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 4014275/19849436 字节，进度：20.22%
[22:31:31] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 4346044/19849436 字节，进度：21.90%
[22:31:36] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 4569900/19849436 字节，进度：23.02%
[22:31:41] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 4750170/19849436 字节，进度：23.93%
[22:31:46] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 4916798/19849436 字节，进度：24.77%
[22:31:51] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 5110683/19849436 字节，进度：25.75%
[22:31:56] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 5322281/19849436 字节，进度：26.81%
[22:32:01] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 5543405/19849436 字节，进度：27.93%
[22:32:02] [ForkJoinPool.commonPool-worker-2/WARN]: IPDB 数据库 GeoLite2-City 解压失败
java.io.InterruptedIOException: timeout
at okhttp3.internal.connection.RealCall.timeoutExit(RealCall.kt:461)
at okhttp3.internal.connection.RealCall.callDone(RealCall.kt:423)
at okhttp3.internal.connection.RealCall.messageDone$okhttp(RealCall.kt:365)
at okhttp3.internal.connection.Exchange.bodyComplete(Exchange.kt:232)
at okhttp3.internal.connection.Exchange.bodyComplete$default(Exchange.kt:208)
at okhttp3.internal.connection.Exchange$ResponseBodySource.complete(Exchange.kt:395)
at okhttp3.internal.connection.Exchange$ResponseBodySource.read(Exchange.kt:370)
at okio.RealBufferedSource.read(RealBufferedSource.kt:215)
at okio.ForwardingSource.read(ForwardingSource.kt:27)
at com.ghostchu.peerbanhelper.util.HTTPUtil$ProgressResponseBody$1.read(HTTPUtil.java:279)
at okio.RealBufferedSource$inputStream$1.read(RealBufferedSource.kt:169)
at org.tukaani.xz.CountingInputStream.read(CountingInputStream.java:33)
at java.base/java.io.DataInputStream.readFully(Unknown Source)
at org.tukaani.xz.rangecoder.RangeDecoderFromBuffer.prepareInputBuffer(RangeDecoderFromBuffer.java:47)
at org.tukaani.xz.LZMA2InputStream.decodeChunkHeader(LZMA2InputStream.java:314)
at org.tukaani.xz.LZMA2InputStream.read(LZMA2InputStream.java:245)
at org.tukaani.xz.BlockInputStream.read(BlockInputStream.java:209)
at org.tukaani.xz.SingleXZInputStream.read(SingleXZInputStream.java:423)
at org.tukaani.xz.XZInputStream.read(XZInputStream.java:394)
at java.base/java.io.InputStream.read(Unknown Source)
at com.ghostchu.peerbanhelper.util.ipdb.IPDB.lambda$downloadFile$0(IPDB.java:385)
at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture$AsyncRun.exec(Unknown Source)
at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
Caused by: okhttp3.internal.http2.StreamResetException: stream was reset: CANCEL
at okhttp3.internal.http2.Http2Stream$FramingSource.read(Http2Stream.kt:398)
at okhttp3.internal.connection.Exchange$ResponseBodySource.read(Exchange.kt:346)
... 20 common frames omitted
[22:32:02] [ForkJoinPool.commonPool-worker-2/WARN]: IPDB 下载任务失败或超时，正在使用备用源重试……
[22:32:03] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 6594488/19849436 字节，进度：33.22%
[22:32:03] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 7643064/19849436 字节，进度：38.51%
[22:32:04] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 8691640/19849436 字节，进度：43.79%
[22:32:04] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 9740216/19849436 字节，进度：49.07%
[22:32:04] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 10788792/19849436 字节，进度：54.35%
[22:32:04] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 11837368/19849436 字节，进度：59.64%
[22:32:04] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 12885944/19849436 字节，进度：64.92%
[22:32:05] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 13934520/19849436 字节，进度：70.20%
[22:32:05] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 14983096/19849436 字节，进度：75.48%
[22:32:05] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 16031672/19849436 字节，进度：80.77%
[22:32:05] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 17080248/19849436 字节，进度：86.05%
[22:32:05] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 18128824/19849436 字节，进度：91.33%
[22:32:05] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 19177400/19849436 字节，进度：96.61%
[22:32:05] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已完成！共传输 19849436 字节的数据
[22:32:06] [ForkJoinPool.commonPool-worker-2/INFO]: 更新数据库 GeoLite2-City 成功！
[22:32:06] [DefaultDispatcher-worker-4/INFO]: GeoLite2-ASN 数据库已过期且需要更新，请等待 PBH 连接到服务器更新数据……
[22:32:10] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 1357/3690596 字节，进度：0.04%
[22:32:16] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 223819/3690596 字节，进度：6.06%
[22:32:21] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 245760/3690596 字节，进度：6.66%
[22:32:26] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 274392/3690596 字节，进度：7.43%
[22:32:37] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 290730/3690596 字节，进度：7.88%
[22:32:43] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 323554/3690596 字节，进度：8.77%
[22:32:49] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 329009/3690596 字节，进度：8.91%
[22:33:15] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 333098/3690596 字节，进度：9.03%
[22:33:21] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 335821/3690596 字节，进度：9.10%
[22:33:26] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 361712/3690596 字节，进度：9.80%
[22:33:31] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 442193/3690596 字节，进度：11.98%
[22:33:36] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 500917/3690596 字节，进度：13.57%
[22:33:41] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 584332/3690596 字节，进度：15.83%
[22:33:46] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 611571/3690596 字节，进度：16.57%
[22:33:51] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 668938/3690596 字节，进度：18.13%
[22:33:56] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 753544/3690596 字节，进度：20.42%
[22:34:01] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 814982/3690596 字节，进度：22.08%
[22:34:07] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 877804/3690596 字节，进度：23.78%
[22:34:12] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 918824/3690596 字节，进度：24.90%
[22:34:19] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 962392/3690596 字节，进度：26.08%
[22:34:24] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 966490/3690596 字节，进度：26.19%
[22:34:32] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 976205/3690596 字节，进度：26.45%
[22:34:38] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 1011617/3690596 字节，进度：27.41%
[22:34:45] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 1058065/3690596 字节，进度：28.67%
[22:34:50] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 1088027/3690596 字节，进度：29.48%
[22:34:55] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 1103174/3690596 字节，进度：29.89%
[22:35:03] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 1130413/3690596 字节，进度：30.63%
[22:35:09] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 1138591/3690596 字节，进度：30.85%
[22:35:09] [ForkJoinPool.commonPool-worker-2/WARN]: IPDB 数据库 GeoLite2-ASN 解压失败
java.io.InterruptedIOException: timeout
at okhttp3.internal.connection.RealCall.timeoutExit(RealCall.kt:461)
at okhttp3.internal.connection.RealCall.callDone(RealCall.kt:423)
at okhttp3.internal.connection.RealCall.messageDone$okhttp(RealCall.kt:365)
at okhttp3.internal.connection.Exchange.bodyComplete(Exchange.kt:232)
at okhttp3.internal.connection.Exchange.bodyComplete$default(Exchange.kt:208)
at okhttp3.internal.connection.Exchange$ResponseBodySource.complete(Exchange.kt:395)
at okhttp3.internal.connection.Exchange$ResponseBodySource.read(Exchange.kt:370)
at okio.RealBufferedSource.read(RealBufferedSource.kt:215)
at okio.ForwardingSource.read(ForwardingSource.kt:27)
at com.ghostchu.peerbanhelper.util.HTTPUtil$ProgressResponseBody$1.read(HTTPUtil.java:279)
at okio.RealBufferedSource$inputStream$1.read(RealBufferedSource.kt:169)
at org.tukaani.xz.CountingInputStream.read(CountingInputStream.java:33)
at java.base/java.io.DataInputStream.readFully(Unknown Source)
at org.tukaani.xz.rangecoder.RangeDecoderFromBuffer.prepareInputBuffer(RangeDecoderFromBuffer.java:47)
at org.tukaani.xz.LZMA2InputStream.decodeChunkHeader(LZMA2InputStream.java:314)
at org.tukaani.xz.LZMA2InputStream.read(LZMA2InputStream.java:245)
at org.tukaani.xz.BlockInputStream.read(BlockInputStream.java:209)
at org.tukaani.xz.SingleXZInputStream.read(SingleXZInputStream.java:423)
at org.tukaani.xz.XZInputStream.read(XZInputStream.java:394)
at java.base/java.io.InputStream.read(Unknown Source)
at com.ghostchu.peerbanhelper.util.ipdb.IPDB.lambda$downloadFile$0(IPDB.java:385)
at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture$AsyncRun.exec(Unknown Source)
at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
Caused by: okhttp3.internal.http2.StreamResetException: stream was reset: CANCEL
at okhttp3.internal.http2.Http2Stream$FramingSource.read(Http2Stream.kt:398)
at okhttp3.internal.connection.Exchange$ResponseBodySource.read(Exchange.kt:346)
... 20 common frames omitted
[22:35:09] [ForkJoinPool.commonPool-worker-2/WARN]: IPDB 下载任务失败或超时，正在使用备用源重试……
[22:35:10] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 2187246/3690596 字节，进度：59.27%
[22:35:10] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 3235822/3690596 字节，进度：87.68%
[22:35:10] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已完成！共传输 3690596 字节的数据
[22:35:10] [ForkJoinPool.commonPool-worker-2/INFO]: 更新数据库 GeoLite2-ASN 成功！
[22:35:10] [DefaultDispatcher-worker-4/INFO]: GeoCN (github.com/ljxi/GeoCN) 数据库已过期且需要更新，请等待 PBH 连接到服务器更新数据……
[22:36:43] [ForkJoinPool.commonPool-worker-2/WARN]: IPDB 下载任务失败或超时，正在使用备用源重试……
[22:36:43] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 8192/2082228 字节，进度：0.39%
[22:36:44] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已下载 1064906/2082228 字节，进度：51.14%
[22:36:44] [ForkJoinPool.commonPool-worker-2/INFO]: 下载进度：已完成！共传输 2082228 字节的数据
[22:36:44] [ForkJoinPool.commonPool-worker-2/INFO]: 更新数据库 GeoCN 成功！
额外信息 - Addition Information
No response
检查清单 - Check list
- PeerBanHelper 已更新到最新版本，非最新版本不接受任何错误反馈，任何非最新版本的 Issue 将被 立 刻 关 闭，不会有人给您提供任何支持 (I'm running the latest version of PBH that can be found in Github Relases, non-latest release won't receive any support)
- 所添加的下载器已满足 README 中的前置要求（如版本号和插件）(The downloaders that I've added already satisfied the requirements (E.g install plugins/adapters))
- 我已检查过 PBH 文档（特别是常见问题），且即使使用了搜索也没有找到与此有关的内容 (This not a question/or the question that not listed in README's FAQ or PBH WIKI)
- 我没有检查这个检查清单，只是闭眼选中了所有的复选框，请关闭这个 Issue (I have not read these checkboxes and therefore I just ticked them all, Please close this issue)
- 这不是一个安全漏洞，它可以被安全的公开报告。若需要报告安全漏洞，请在此报告 (This not a security related issue, can be safe report in public. If you want report a security exploit, please report it here)
Metadata
Metadata
Assignees
Labels
No labels