Skip to content

Sort tokens in CTLexerBuilder codegen for deterministic output #576

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ratmice merged 1 commit into from
May 22, 2025
Merged

Sort tokens in CTLexerBuilder codegen for deterministic output #576

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 20 additions & 7 deletions lrlex/src/lib/ctbuilder.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -782,7 +782,9 @@ where

let mut token_consts = TokenStream::new();
if let Some(rim) = self.rule_ids_map {
for (name, id) in rim {
let mut rim_sorted = Vec::from_iter(rim.iter());
rim_sorted.sort_by_key(|(k, _)| *k);
for (name, id) in rim_sorted {
if RE_TOKEN_ID.is_match(&name) {
let tok_ident = format_ident!("N_{}", name.to_ascii_uppercase());
let storaget =
Expand Down Expand Up @@ -815,9 +817,17 @@ where
};
// Try and run a code formatter on the generated code.
let unformatted = out_tokens.to_string();
let outs = syn::parse_str(&unformatted)
.map(|syntax_tree| prettyplease::unparse(&syntax_tree))
.unwrap_or(unformatted);
let mut outs = String::new();
// Record the time that this version of lrlex was built. If the source code changes and rustc
// forces a recompile, this will change this value, causing anything which depends on this
// build of lrlex to be recompiled too.
let timestamp = env!("VERGEN_BUILD_TIMESTAMP");
Copy link
Member

@ltratt ltratt May 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this environment variable always defined?

Copy link
Collaborator

@ratmice ratmice May 22, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, you can see it also used in the ct_token_map codegen at the top of the diff context in the first patch in this series. 719da09

I believe it is ensured by:
https://docs.rs/vergen/latest/vergen/struct.BuildBuilder.html#method.build_timestamp
https://github.com/softdevteam/grmtools/blob/master/lrlex/build.rs#L4

along with the compile time caching behavior of the env! macro:
https://www.cs.brandeis.edu/~cs146a/rust/doc-02-21-2015/std/macro.env!.html

So I think this is including a timestamp from the time when lrlex/build.rs is built to every token_map.rs,
so this addition syncs things so that applies to the main lex output as well!

Copy link
Member

@ltratt ltratt May 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, yes, I can see. @ratmice I think you're best placed to judge this PR.

write!(outs, "// lrlex build time: {}\n\n", quote!(#timestamp),).ok();
outs.push_str(
&syn::parse_str(&unformatted)
.map(|syntax_tree| prettyplease::unparse(&syntax_tree))
.unwrap_or(unformatted),
);
// If the file we're about to write out already exists with the same contents, then we
// don't overwrite it (since that will force a recompile of the file, and relinking of the
// binary etc).
Expand Down Expand Up @@ -1205,9 +1215,12 @@ pub fn ct_token_map<StorageT: Display + ToTokens>(
let timestamp = env!("VERGEN_BUILD_TIMESTAMP");
let mod_ident = format_ident!("{}", mod_name);
write!(outs, "// lrlex build time: {}\n\n", quote!(#timestamp),).ok();
let tokens = &token_map
.borrow()
.iter()
// Sort the tokens so that they're always in the same order.
// This will prevent unneeded rebuilds.
let mut token_map_sorted = Vec::from_iter(token_map.borrow().iter());
token_map_sorted.sort_by_key(|(k, _)| *k);
let tokens = &token_map_sorted
.into_iter()
.map(|(k, id)| {
let name = match rename_map {
Some(rmap) => *rmap.get(k.as_str()).unwrap_or(&k.as_str()),
Expand Down