Skip to content

Conversation

eagr
Copy link
Contributor

@eagr eagr commented Oct 17, 2023

closes #685

cc @mrcnski @s0me0ne-unkn0wn

kusama address: FvpsvV1GQAAbwqX6oyRjemgdKV11QU5bXsMg9xsonD1FLGK

@eagr eagr marked this pull request as draft October 17, 2023 15:56
@eagr
Copy link
Contributor Author

eagr commented Oct 17, 2023

Do you think Artifacts::insert_prepared() should be made pub(crate)?

@eagr eagr marked this pull request as ready for review October 17, 2023 18:34
@mrcnski mrcnski added the T0-node This PR/Issue is related to the topic “node”. label Oct 18, 2023
Copy link
Contributor

@mrcnski mrcnski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot! Add your address to the PR description if you want another tip!

@mrcnski
Copy link
Contributor

mrcnski commented Oct 18, 2023

Do you think Artifacts::insert_prepared() should be made pub(crate)?

Why do you think so? AFAICT, Artifact is not available outside the crate anyway, so should be fine as-is.

@eagr
Copy link
Contributor Author

eagr commented Oct 18, 2023

Why do you think so?

By making it pub(crate), it could be used in Artifacts. Right now I'm using an ad hoc version of it there.

Comment on lines 231 to 233
// FIXME eagr: extremely janky, please comment on the appropriate way of setting
// * `last_time_needed` set as roughly around startup time
// * `prepare_stats` set as Nones, since the metadata was lost
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

last_time_needed looks good to me.

prepare_stats may not even be necessary after the artifact has been prepared and the metrics logged. What happens when this field is removed?

@mrcnski
Copy link
Contributor

mrcnski commented Oct 18, 2023

Why do you think so?

By making it pub(crate), it could be used in Artifacts. Right now I'm using an ad hoc version of it there.

Oh, I see what you mean. But I think it can be used in Artifacts even if it's private.

@eagr
Copy link
Contributor Author

eagr commented Oct 18, 2023

Oh, I see what you mean. But I think it can be used in Artifacts even if it's private.

Yeah, but it's being used in the host mod for testing.

Copy link
Contributor

@mrcnski mrcnski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Almost there! 👍

@mrcnski
Copy link
Contributor

mrcnski commented Oct 18, 2023

/tip medium

@substrate-tip-bot
Copy link

@mrcnski A medium (16 KSM) tip was successfully submitted for @eagr (FvpsvV1GQAAbwqX6oyRjemgdKV11QU5bXsMg9xsonD1FLGK on kusama).

https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Fkusama-rpc.polkadot.io#/referenda tip

Copy link
Contributor

@mrcnski mrcnski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! Thanks for all your hard work! @s0me0ne-unkn0wn want to take another look?

I see there's this build-script-utils crate. Do you think it should go there instead?

Yeah, could be useful for the follow-up I mentioned above, to have this in other crates.

get_wasmtime_version();
}

pub fn get_wasmtime_version() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add a comment that in case there are multiple wasmtime versions being used, is an error. Example:

 $ cargo tree --package=aead --depth=0
error: There are multiple `aead` packages in your project, and the specification `aead` is ambiguous.
Please re-run this command with one of the following specifications:
  [email protected]
  [email protected]
  [email protected]

I think an error in that case great, we definitely don't want to be mixing wasmtime versions anyway.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is somehow captured. Right now this shows a warning and the compilation will fail if they try to refer to the environment variable. Would that be enough?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean a comment would be useful for readers of the code. I was wondering what would happen if there were multiple versions of wasmtime, I didn't know it would be an error until I tried it.

@eagr
Copy link
Contributor Author

eagr commented Nov 14, 2023

Kudos to you! I know how demanding code review is, esp. at this level of fidelity. I really don't know how could you keep up with it. Really appreciate it man.

Yeah, could be useful for the follow-up I mentioned above, to have this in other crates.

Let's do it in this PR, alright? @mrcnski

@mrcnski
Copy link
Contributor

mrcnski commented Nov 14, 2023

Kudos to you! I know how demanding code review is, esp. at this level of fidelity. I really don't know how could you keep up with it. Really appreciate it man.

Thanks! 😊 And thanks for putting up with all my comments, let me know if I'm being overly perfectionist...

Let's do it in this PR, alright? @mrcnski

I was thinking probably a code-owner of build-script-utils should review that change. I guess you can do it here and ask them to review just one file.

@mrcnski
Copy link
Contributor

mrcnski commented Nov 14, 2023

There be some scary merge conflicts, fixing...

@paritytech-cicd-pr
Copy link

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable 3/3
Logs: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/4335309

Copy link
Contributor

@s0me0ne-unkn0wn s0me0ne-unkn0wn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks very good! 👍 Left some minor comments.

Comment on lines 113 to 114
const ARTIFACT_PREFIX: &str =
concat_const!(RUNTIME_PREFIX, RUNTIME_VERSION, "_", NODE_PREFIX, NODE_VERSION);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
const ARTIFACT_PREFIX: &str =
concat_const!(RUNTIME_PREFIX, RUNTIME_VERSION, "_", NODE_PREFIX, NODE_VERSION);

This is over the top. We don't need this macro, nor do we need to concat this at compile time. We are using format anyway to generate the name.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mean we do something like this instead?

fn artifact_prefix() -> String {
	format!("{}{}_{}{}", RUNTIME_PREFIX, RUNTIME_VERSION, NODE_PREFIX, NODE_VERSION)
}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the very first version. Now it still feels odd to compute the same thing over and over.. I understand this is not remotely close to hot and this isn't about performance. I understand that the macro thing may feel like too much at the first glance. When I looked at how much cleaner the code became with the const/static compared to a function call, I felt it's worth it. A const is just so much easier to use than a function, as when a value is static, it probably should be used as one. Besides, isn't it nice that drop/dealloc/format calls are wiped off from the assembly? :))

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it's a function you can at least put it behind a OnceCell or something. Otherwise I did a search and there are some crates that do this: const_format, concat_const, etc. But it would be sad to pull in a whole dependency for this. I wish std had this... Meh, let's just merge as-is (with the function). I'll press the green button once conflicts are resolved.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it's a function you can at least put it behind a OnceCell or something.

Function, cell, lazy static, considered all that before doing the macro as it results in the simplest interface, a static str. I won't change it back. Let's get this merged. 🚀 I'm just trying to understand why is a function call better here.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm just trying to understand why is a function call better here.

The problem with the macro is that it's a lot of code for readers to understand, just to declare a constant. TBH, all the options are kind of meh, but given that the code is cold, optimizing for maintainability with a plain function seems like the right call.

fn generate_dependency_version(dep: &str, env_var: &str) {
// we only care about the root
match std::process::Command::new("cargo")
.args(["tree", "--depth=0", "--package", dep])
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
.args(["tree", "--depth=0", "--package", dep])
.args(["tree", "--depth=0", "--locked", "--offline", "--package", dep])

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Turns out that --locked alone would achieve both.

.file_stem()
.expect("path must contain a file name")
.to_str()
.expect("file name must be valid UTF-8");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please still handle them gracefully. I mean this is here just by returning true.

return !file_name.ends_with(checksum.to_hex().as_str())
}
}
false
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be true right? Could use a test for this case if we don't have one yet.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops! That's right. But this line of can only be reached when

  • the path end in "/..", or
  • the path contain non-Unicode

These are all screened out at the call site, so I don't even know how to write a test that could fail because of this line lol.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess is_corrupt can be extracted and unit-tested, but whatever, looks good already!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should probably stick to testing the interfaces, this kind of tests often become maintenance liability (prone to change). 😉

@mrcnski
Copy link
Contributor

mrcnski commented Nov 19, 2023

Aaaand merged! 🎉 Thanks @eagr, awesome PR! I extracted out a couple follow-ups so they don't get lost.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

T0-node This PR/Issue is related to the topic “node”.

Projects

Status: Completed

Development

Successfully merging this pull request may close these issues.

PVF: Avoid clearing the artifacts cache on restart

5 participants