-
Notifications
You must be signed in to change notification settings - Fork 1k
Preserve artifact cache unless stale #1918
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Do you think |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks a lot! Add your address to the PR description if you want another tip!
Why do you think so? AFAICT, |
By making it |
// FIXME eagr: extremely janky, please comment on the appropriate way of setting | ||
// * `last_time_needed` set as roughly around startup time | ||
// * `prepare_stats` set as Nones, since the metadata was lost |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
last_time_needed
looks good to me.
prepare_stats
may not even be necessary after the artifact has been prepared and the metrics logged. What happens when this field is removed?
Oh, I see what you mean. But I think it can be used in |
Yeah, but it's being used in the host mod for testing. |
Co-authored-by: Marcin S <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Almost there! 👍
/tip medium |
@mrcnski A medium (16 KSM) tip was successfully submitted for @eagr (FvpsvV1GQAAbwqX6oyRjemgdKV11QU5bXsMg9xsonD1FLGK on kusama). https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Fkusama-rpc.polkadot.io#/referenda |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks great! Thanks for all your hard work! @s0me0ne-unkn0wn want to take another look?
I see there's this
build-script-utils
crate. Do you think it should go there instead?
Yeah, could be useful for the follow-up I mentioned above, to have this in other crates.
get_wasmtime_version(); | ||
} | ||
|
||
pub fn get_wasmtime_version() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe add a comment that in case there are multiple wasmtime versions being used, is an error. Example:
$ cargo tree --package=aead --depth=0
error: There are multiple `aead` packages in your project, and the specification `aead` is ambiguous.
Please re-run this command with one of the following specifications:
[email protected]
[email protected]
[email protected]
I think an error in that case great, we definitely don't want to be mixing wasmtime versions anyway.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is somehow captured. Right now this shows a warning and the compilation will fail if they try to refer to the environment variable. Would that be enough?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I mean a comment would be useful for readers of the code. I was wondering what would happen if there were multiple versions of wasmtime, I didn't know it would be an error until I tried it.
Kudos to you! I know how demanding code review is, esp. at this level of fidelity. I really don't know how could you keep up with it. Really appreciate it man.
Let's do it in this PR, alright? @mrcnski |
Thanks! 😊 And thanks for putting up with all my comments, let me know if I'm being overly perfectionist...
I was thinking probably a code-owner of build-script-utils should review that change. I guess you can do it here and ask them to review just one file. |
There be some scary merge conflicts, fixing... |
The CI pipeline was cancelled due to failure one of the required jobs. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks very good! 👍 Left some minor comments.
const ARTIFACT_PREFIX: &str = | ||
concat_const!(RUNTIME_PREFIX, RUNTIME_VERSION, "_", NODE_PREFIX, NODE_VERSION); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
const ARTIFACT_PREFIX: &str = | |
concat_const!(RUNTIME_PREFIX, RUNTIME_VERSION, "_", NODE_PREFIX, NODE_VERSION); |
This is over the top. We don't need this macro, nor do we need to concat this at compile time. We are using format anyway to generate the name.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You mean we do something like this instead?
fn artifact_prefix() -> String {
format!("{}{}_{}{}", RUNTIME_PREFIX, RUNTIME_VERSION, NODE_PREFIX, NODE_VERSION)
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is the very first version. Now it still feels odd to compute the same thing over and over.. I understand this is not remotely close to hot and this isn't about performance. I understand that the macro thing may feel like too much at the first glance. When I looked at how much cleaner the code became with the const/static compared to a function call, I felt it's worth it. A const is just so much easier to use than a function, as when a value is static, it probably should be used as one. Besides, isn't it nice that drop/dealloc/format calls are wiped off from the assembly? :))
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If it's a function you can at least put it behind a OnceCell
or something. Otherwise I did a search and there are some crates that do this: const_format
, concat_const
, etc. But it would be sad to pull in a whole dependency for this. I wish std had this... Meh, let's just merge as-is (with the function). I'll press the green button once conflicts are resolved.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If it's a function you can at least put it behind a OnceCell or something.
Function, cell, lazy static, considered all that before doing the macro as it results in the simplest interface, a static str. I won't change it back. Let's get this merged. 🚀 I'm just trying to understand why is a function call better here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm just trying to understand why is a function call better here.
The problem with the macro is that it's a lot of code for readers to understand, just to declare a constant. TBH, all the options are kind of meh, but given that the code is cold, optimizing for maintainability with a plain function seems like the right call.
fn generate_dependency_version(dep: &str, env_var: &str) { | ||
// we only care about the root | ||
match std::process::Command::new("cargo") | ||
.args(["tree", "--depth=0", "--package", dep]) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
.args(["tree", "--depth=0", "--package", dep]) | |
.args(["tree", "--depth=0", "--locked", "--offline", "--package", dep]) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Turns out that --locked alone would achieve both.
.file_stem() | ||
.expect("path must contain a file name") | ||
.to_str() | ||
.expect("file name must be valid UTF-8"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you please still handle them gracefully. I mean this is here just by returning true.
return !file_name.ends_with(checksum.to_hex().as_str()) | ||
} | ||
} | ||
false |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should be true
right? Could use a test for this case if we don't have one yet.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oops! That's right. But this line of can only be reached when
- the path end in "/..", or
- the path contain non-Unicode
These are all screened out at the call site, so I don't even know how to write a test that could fail because of this line lol.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess is_corrupt
can be extracted and unit-tested, but whatever, looks good already!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should probably stick to testing the interfaces, this kind of tests often become maintenance liability (prone to change). 😉
Aaaand merged! 🎉 Thanks @eagr, awesome PR! I extracted out a couple follow-ups so they don't get lost. |
closes #685
cc @mrcnski @s0me0ne-unkn0wn
kusama address: FvpsvV1GQAAbwqX6oyRjemgdKV11QU5bXsMg9xsonD1FLGK