build_parameters.py: use local cache and server cache to get files#7681
build_parameters.py: use local cache and server cache to get files#7681khancyr wants to merge 1 commit intoArduPilot:masterfrom
Conversation
peterbarker
left a comment
peterbarker
left a comment
There was a problem hiding this comment.
It would be nice if this was all tied up in a little object rather than being a bunch of methods, but I'm not going to insist.
|
|
||
| # Get the directory where this script is located | ||
| script_dir = os.path.dirname(os.path.abspath(__file__)) | ||
| default_cache_dir = os.path.join(script_dir, '.cache') |
There was a problem hiding this comment.
Maybe a bit generic a name
There was a problem hiding this comment.
default_http_request_cache_dir ok ?
| os.makedirs(cache_dir, exist_ok=True) | ||
|
|
||
| # Create cache filename from URL | ||
| cache_filename = re.sub(r'[^\w\-_.]', '_', url) + '.cache' |
There was a problem hiding this comment.
This is the wrong sort of regex to use here. You should be choosing which characters to keep, not which ones to throw away.
urllib has a quote method available.
There was a problem hiding this comment.
I didn't know about urllib.quote, using it instead of the regex
| response = session.get(firmware_url + vehicle, timeout=30) | ||
| response.raise_for_status() | ||
| content = response.text | ||
| content = get_cached_url(firmware_url + vehicle) |
There was a problem hiding this comment.
That's not what this really does.
khancyr
force-pushed
the
build_parameters_cache
branch
2 times, most recently
from
72882ab to
7e08525
Compare
|
@peterbarker comments addressed ! |
khancyr
force-pushed
the
build_parameters_cache
branch
from
7e08525 to
91076cd
Compare
|
@peterbarker , when cache is done in build_parameters.py and update.py, I will consolidate both script with a common lib (logging, http and git things) that will make things easier at a central place |
khancyr
force-pushed
the
build_parameters_cache
branch
2 times, most recently
from
99af786 to
678f075
Compare
khancyr
force-pushed
the
build_parameters_cache
branch
from
678f075 to
e0debf1
Compare
2 participants
add local cache as parameters files don't change often
use HEAD request before getting the file to know if it changed or not.
Cache work in 3 steps:
6h time is totally arbitrary. As we don't update often parameter that look sane to test on 4 time per day and reduce lot the request on the server !
We could work on clever way later for old parameters and perhaps reporting if a modification is noted on old files
From testing our server don't return http 304