Skip to content

Commit c9ff089

Browse files
hastmuhastmu
authored
Merge pull request #4 from hastmu/feature-new-caching
Feature new caching of specific url
2 parents 020a4a4 + c418b40 commit c9ff089

File tree

3 files changed

+154
-65
lines changed

3 files changed

+154
-65
lines changed

‎README.md‎

Lines changed: 54 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,11 @@ Issues:
1111
* No check if the found proxy works for the requested target.
1212
* No longer active maintained (deprecated warnings all over the place)
1313

14+
What do you get?
15+
- [X] detect via avahi
16+
- [X] caching of URL specific working proxy (or none if none works)
17+
- [X] caching over reboot if possible
18+
1419
# How does it look?
1520

1621
Example output:
@@ -59,11 +64,17 @@ looks like:
5964
dev@dev:~$ curl -s https://raw.githubusercontent.com/hastmu/apt-proxy-detect/main/install.sh | sudo bash
6065
- check dependencies...
6166
- download latest to: /usr/local/bin/apt-proxy-detect.sh
67+
- BRANCH [main]
6268
- set permissions to a+rx
6369
- create/updating /etc/apt/apt.conf.d/30apt-proxy-detect.conf
6470
6571
```
6672

73+
if you like to specify a branch do it like:
74+
```
75+
curl -s https://raw.githubusercontent.com/hastmu/apt-proxy-detect/main/install.sh | sudo BRANCH=main bash
76+
```
77+
6778
or you just download the install.sh and run it locally.
6879

6980
# How does it work?
@@ -78,8 +89,8 @@ right or outdated for the new stuff to be checked out.
7889

7990
In case a proxy is not serving the URL it is drop for this dedicated URL.
8091

81-
The found (via avahi-browse _apt_proxy._tcp) proxies are cached under /tmp/.apt-proxy.$username
82-
in case this has the wrong owner it is ignored and a waring is issued.
92+
The found (via avahi-browse _apt_proxy._tcp) proxies are cached in case this has the
93+
wrong owner it is ignored and a waring is issued.
8394

8495
Thats it. Enjoy.
8596

@@ -103,41 +114,47 @@ looks like
103114
```
104115
dev@dev:~$ export DEBUG_APT_PROXY_DETECT=1
105116
dev@dev:~$ sudo apt update
106-
[ INFO]: apt-proxy-detect
107-
[ CACHE]: stored under: /tmp/.apt-proxy._apt
108-
[ TEST-URL]: URL: http://packages.microsoft.com/repos/code/dists/stable/InRelease
109-
[ AVAHI]: get cache entries for _apt_proxy._tcp
110-
[ AVAHI]: get non-cache entries for _apt_proxy._tcp
111-
[ CHECK]: Checking found proxy (http://192.168.0.27:3142) with testurl (http://packages.microsoft.com/repos/code/dists/stable/InRelease)
112-
Service[ER][apt-cacher-ng proxy on squid-deb-proxy]@http://192.168.0.27:3142
113-
[ CHECK]: Checking found proxy (http://192.168.0.27:8000) with testurl (http://packages.microsoft.com/repos/code/dists/stable/InRelease)
114-
[ CACHE]: Store (http://192.168.0.27:8000) in cache file (/tmp/.apt-proxy._apt)
115-
Service[OK][Squid deb proxy on squid-deb-proxy]@http://192.168.0.27:8000
116-
[ PROXY]: return http://192.168.0.27:8000
117-
[ INFO]: apt-proxy-detect
118-
[ CACHE]: stored under: /tmp/.apt-proxy._apt
119-
[ TEST-URL]: URL: http://download.proxmox.com/debian/pve/dists/bookworm/InRelease
120-
[ CHECK]: Checking cached proxy (http://192.168.0.27:8000) with testurl (http://download.proxmox.com/debian/pve/dists/bookworm/InRelease)
121-
[ WORKS]: give back cached proxy
122-
[ PROXY]: return http://192.168.0.27:8000
123-
[ INFO]: apt-proxy-detect
124-
[ CACHE]: stored under: /tmp/.apt-proxy._apt
125-
[ TEST-URL]: URL: http://local-repo.fritz.box/local-repo/dists/trunk/InRelease
126-
[ CHECK]: Checking cached proxy (http://192.168.0.27:8000) with testurl (http://local-repo.fritz.box/local-repo/dists/trunk/InRelease)
127-
[ WORKS]: give back cached proxy
128-
[ PROXY]: return http://192.168.0.27:8000
129-
[ INFO]: apt-proxy-detect
130-
[ CACHE]: stored under: /tmp/.apt-proxy._apt
131-
[ TEST-URL]: URL: http://security.debian.org/debian-security/dists/bookworm-security/InRelease
132-
[ CHECK]: Checking cached proxy (http://192.168.0.27:8000) with testurl (http://security.debian.org/debian-security/dists/bookworm-security/InRelease)
133-
[ WORKS]: give back cached proxy
134-
[ PROXY]: return http://192.168.0.27:8000
135-
[ INFO]: apt-proxy-detect
136-
[ CACHE]: stored under: /tmp/.apt-proxy._apt
137-
[ TEST-URL]: URL: http://deb.debian.org/debian/dists/bookworm/InRelease
138-
[ CHECK]: Checking cached proxy (http://192.168.0.27:8000) with testurl (http://deb.debian.org/debian/dists/bookworm/InRelease)
139-
[ WORKS]: give back cached proxy
140-
[ PROXY]: return http://192.168.0.27:8000
117+
# INFO-TAG MS : MESSAGE
118+
[ INFO][ 2]: ===--- apt-proxy-detect ---===
119+
[ TEST-URL][ 16]: URL: http://packages.microsoft.com/repos/code/dists/stable/InRelease
120+
[ HASH][ 25]: HASH: c0b917f192fa7cccb3f536f2c01b824d of (http://packages.microsoft.com)
121+
[ CACHE][ 30]: using stored under: /var/lib/apt/lists/auxfiles/.apt-proxy-detect._apt
122+
[ CACHE-AGE][ 33]: age: 687 sec
123+
[ CHECK-PROXY][ 35]: Checking proxy (http://192.168.0.27:8000) with testurl (http://packages.microsoft.com/repos/code/dists/stable/InRelease)
124+
[ WORKS][ 127]: give back cached proxy
125+
[ PROXY][ 131]: return http://192.168.0.27:8000
126+
[ INFO][ 3]: ===--- apt-proxy-detect ---===
127+
[ TEST-URL][ 20]: URL: http://download.proxmox.com/debian/pve/dists/bookworm/InRelease
128+
[ HASH][ 34]: HASH: 17b43db99b56eb6355d41861f4f304d0 of (http://download.proxmox.com)
129+
[ CACHE][ 41]: using stored under: /var/lib/apt/lists/auxfiles/.apt-proxy-detect._apt
130+
[ CACHE-AGE][ 44]: age: 687 sec
131+
[ CHECK-PROXY][ 47]: Checking proxy (http://192.168.0.27:8000) with testurl (http://download.proxmox.com/debian/pve/dists/bookworm/InRelease)
132+
[ WORKS][ 461]: give back cached proxy
133+
[ PROXY][ 464]: return http://192.168.0.27:8000
134+
[ INFO][ 3]: ===--- apt-proxy-detect ---===
135+
[ TEST-URL][ 17]: URL: http://local-repo.fritz.box/local-repo/dists/trunk/InRelease
136+
[ HASH][ 25]: HASH: 2bfbb1335aaf9d333a5c9498226eb208 of (http://local-repo.fritz.box)
137+
[ CACHE][ 32]: using stored under: /var/lib/apt/lists/auxfiles/.apt-proxy-detect._apt
138+
[ CACHE-AGE][ 37]: age: 687 sec
139+
[ CHECK-PROXY][ 39]: Checking proxy (http://192.168.0.27:8000) with testurl (http://local-repo.fritz.box/local-repo/dists/trunk/InRelease)
140+
[ WORKS][ 57]: give back cached proxy
141+
[ PROXY][ 60]: return http://192.168.0.27:8000
142+
[ INFO][ 2]: ===--- apt-proxy-detect ---===
143+
[ TEST-URL][ 19]: URL: http://security.debian.org/debian-security/dists/bookworm-security/InRelease
144+
[ HASH][ 28]: HASH: 3b68f7b6590a2da8625ff71f01d38ffb of (http://security.debian.org)
145+
[ CACHE][ 36]: using stored under: /var/lib/apt/lists/auxfiles/.apt-proxy-detect._apt
146+
[ CACHE-AGE][ 39]: age: 688 sec
147+
[ CHECK-PROXY][ 41]: Checking proxy (http://192.168.0.27:8000) with testurl (http://security.debian.org/debian-security/dists/bookworm-security/InRelease)
148+
[ WORKS][ 93]: give back cached proxy
149+
[ PROXY][ 95]: return http://192.168.0.27:8000
150+
[ INFO][ 2]: ===--- apt-proxy-detect ---===
151+
[ TEST-URL][ 24]: URL: http://deb.debian.org/debian/dists/bookworm/InRelease
152+
[ HASH][ 33]: HASH: efbfa0e2acaaa513c457b6698de83118 of (http://deb.debian.org)
153+
[ CACHE][ 41]: using stored under: /var/lib/apt/lists/auxfiles/.apt-proxy-detect._apt
154+
[ CACHE-AGE][ 44]: age: 687 sec
155+
[ CHECK-PROXY][ 47]: Checking proxy (http://192.168.0.27:8000) with testurl (http://deb.debian.org/debian/dists/bookworm/InRelease)
156+
[ WORKS][ 145]: give back cached proxy
157+
[ PROXY][ 147]: return http://192.168.0.27:8000
141158
Hit:1 http://local-repo.fritz.box/local-repo trunk InRelease
142159
Hit:2 http://deb.debian.org/debian bookworm InRelease
143160
Hit:3 http://security.debian.org/debian-security bookworm-security InRelease

‎apt-proxy-detect.sh‎

Lines changed: 95 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,22 @@
55
# v1 ... pure detect
66
# v2 ... cache value with check
77
# v3 ... add persistant caching for found proxies.
8+
# v4 ... extend caching to also cache no-found proxy state
9+
# and proxy checking based on testurl
10+
# + debug reveals ms execution time
11+
# + timeout for check_proxy
12+
# + cache age per entry not per whole file.
13+
# + cache leading url not full, as i get called for all packages also.
14+
15+
16+
# defaults
17+
service_name="_apt_proxy._tcp"
18+
cache_file_name=".apt-proxy-detect.$(id -un)"
19+
cache_file_none_retry_timeout=60
20+
declare -i cache_age=0
21+
22+
declare -A CACHED_PROXIES
23+
declare -A CACHED_PROXIES_AGE
824

925
declare -i debug
1026
[ -z "${DEBUG_APT_PROXY_DETECT}" ] && debug=0 || debug=1
@@ -15,20 +31,38 @@ then
1531
:
1632
}
1733
else
34+
declare -i start_time=0
35+
start_time=$(date +%s%N)
1836
function debug() {
19-
printf "[%10s]: %s\n" "$1" "$2" >&2
37+
printf "[%12s][%4s]: %s\n" "$1" "$(( ($(date +%s%N) - start_time) / 1000000 ))" "$2" >&2
2038
}
2139
fi
22-
debug "INFO" "apt-proxy-detect"
40+
debug "INFO" "===--- apt-proxy-detect ---==="
2341

2442
function check_proxy() {
25-
wget -e "http_proxy=$1" -e "https_proxy=$1" -qO - "$2" >> /dev/null
26-
return $?
43+
if [ "$1" == "NONE" ]
44+
then
45+
if [ ${cache_age} -gt ${cache_file_none_retry_timeout} ]
46+
then
47+
debug "CHECK-PROXY" "NONE-cached expired"
48+
return 1
49+
else
50+
debug "CHECK-PROXY" "NONE-cached"
51+
return 0
52+
fi
53+
else
54+
debug "CHECK-PROXY" "Checking proxy (${1}) with testurl (${2})"
55+
if [ $debug -gt 1 ]
56+
then
57+
wget -v --tries=1 -T 1 -e "http_proxy=$1" -e "https_proxy=$1" -O - "$2" >&2
58+
else
59+
wget -q --tries=1 -T 1 -e "http_proxy=$1" -e "https_proxy=$1" -O - "$2" >> /dev/null 2>&1
60+
fi
61+
return $?
62+
fi
2763
}
2864

29-
# cache file name
30-
cache_file_name=".apt-proxy-detect.$(id -un)"
31-
65+
# --- cache location ---
3266
# persistant cache file location
3367
# $HOME or special for system accounts
3468
declare -A CACHE_FILE_LOC
@@ -46,33 +80,57 @@ else
4680
fi
4781
fi
4882

49-
# check cache_file
50-
touch "${cache_file}" >> /dev/null 2>&1
83+
# check cache_file if not there create it.
84+
[ ! -e "${cache_file}" ] && touch "${cache_file}" >> /dev/null 2>&1
5185

86+
# check ownership of cache file and fetch age.
5287
skip_cache=0
5388
if [ "$(stat -c %u "${cache_file}")" != "$(id -u)" ]
5489
then
5590
skip_cache=1
5691
echo "E: wrong owner of cache file ${cache_file}, remove or reown to $(id -un)" >&2
5792
fi
58-
service_name="_apt_proxy._tcp"
93+
94+
# eval testurl
5995
[ -z "$1" ] && testurl="http://deb.debian.org/debian" || testurl="$1"
60-
debug "TEST-URL" "URL: ${testurl}"
96+
debug "TEST-URL" "URL: ${testurl}"
97+
testurl_hash_url="$(echo "${testurl}" | cut -d/ -f1-3)"
98+
testurl_hash="$(echo "${testurl_hash_url}" | md5sum)" ; testurl_hash="${testurl_hash%% *}"
99+
debug "HASH" "HASH: ${testurl_hash} of (${testurl_hash_url})"
61100

101+
# check out cached value
102+
now=$(date +%s)
62103
if [ -s "${cache_file}" ] && [ ${skip_cache} -eq 0 ]
63104
then
64-
debug "CACHE" "using stored under: ${cache_file}"
65-
proxy="$(cat "${cache_file}")"
66-
debug "CHECK" "Checking cached proxy (${proxy}) with testurl (${testurl})"
67-
if check_proxy "${proxy}" "${testurl}"
105+
# shellcheck disable=SC1090
106+
source "${cache_file}"
107+
# shellcheck disable=SC2181
108+
if [ $? -ne 0 ]
68109
then
69-
debug "WORKS" "give back cached proxy"
70-
debug "PROXY" "return ${proxy}"
71-
echo "${proxy}"
72-
exit 0
73-
else
74-
debug "FAILED" "remove cache file."
110+
# something is wrong with the cache file remove it.
111+
debug "CACHE" "invalid cachefile (cleanup): ${cache_file}"
75112
rm -f "${cache_file}"
113+
else
114+
debug "CACHE" "using stored under: ${cache_file}"
115+
fi
116+
cache_age=$(( now - ${CACHED_PROXIES_AGE[${testurl_hash}]:=0} ))
117+
debug "CACHE-AGE" "age: ${cache_age} sec"
118+
proxy="${CACHED_PROXIES[${testurl_hash}]}"
119+
if [ -n "${proxy}" ]
120+
then
121+
if check_proxy "${proxy}" "${testurl}"
122+
then
123+
debug "WORKS" "give back cached proxy"
124+
debug "PROXY" "return ${proxy}"
125+
[ "${proxy}" != "NONE" ] && echo "${proxy}"
126+
exit 0
127+
else
128+
debug "FAILED" "remove from cache file."
129+
# declare -p CACHED_PROXIES >&2
130+
unset 'CACHED_PROXIES["${testurl_hash}"]'
131+
unset 'CACHED_PROXIES_AGE["${testurl_hash}"]'
132+
# declare -p CACHED_PROXIES >&2
133+
fi
76134
fi
77135
fi
78136

@@ -100,11 +158,6 @@ do
100158
if [ -z "${ret}" ]
101159
then
102160
ret="${proxy}"
103-
if [ ${skip_cache} -eq 0 ]
104-
then
105-
debug "CACHE" "Store (${proxy}) in cache file (${cache_file})"
106-
echo "${proxy}" > "${cache_file}"
107-
fi
108161
fi
109162
stat="OK"
110163
else
@@ -113,9 +166,24 @@ do
113166
printf "Service[%s][%s]@%s \n" "${stat}" "${name}" "${proxy}" >&2
114167
done
115168

116-
debug "PROXY" "return ${ret}"
169+
debug "PROXY" "return :${ret}:"
170+
117171
if [ -n "${ret}" ]
118172
then
119173
echo "${ret}"
174+
else
175+
ret="NONE"
120176
fi
177+
178+
# write back cachefile finally.
179+
if [ ${skip_cache} -eq 0 ]
180+
then
181+
debug "CACHE" "Store (${ret}) in cache file (${cache_file})"
182+
CACHED_PROXIES[${testurl_hash}]="${ret}"
183+
CACHED_PROXIES_AGE[${testurl_hash}]="${now}"
184+
debug "CACHE" "Update cachefile."
185+
declare -p CACHED_PROXIES > "${cache_file}"
186+
declare -p CACHED_PROXIES_AGE >> "${cache_file}"
187+
fi
188+
121189
exit 0

‎install.sh‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,8 @@ BINS["grep"]=0
1313
BINS["cut"]=0
1414
BINS["["]=0
1515
BINS["printf"]=0
16+
BINS["date"]=0
17+
BINS["md5sum"]=0
1618

1719
for pitem in ${PATH//:/ }
1820
do
@@ -42,7 +44,9 @@ TARGET="/usr/local/bin/apt-proxy-detect.sh"
4244
# download latest
4345
echo "- download latest to: ${TARGET}"
4446

45-
if ! wget -q -O "${TARGET}" https://raw.githubusercontent.com/hastmu/apt-proxy-detect/main/apt-proxy-detect.sh
47+
48+
echo "- BRANCH [${BRANCH:=main}]"
49+
if ! wget -q -O "${TARGET}" https://raw.githubusercontent.com/hastmu/apt-proxy-detect/${BRANCH}/apt-proxy-detect.sh
4650
then
4751
echo "- download failed."
4852
[ -x "${TARGET}" ] && rm -f "${TARGET}"

0 commit comments

Comments
 (0)