Skip to content

Conversation

@nielstron
Copy link
Contributor

@nielstron nielstron commented Oct 21, 2025

This adds harder tests for fibonacci open.

The "prepacked" solutions by aiken and plutarch basically hardcode all testcases of the benchmark. This seems to me to somewhat improperly gaming the system.

Moreover, to really evaluate the power of linear solutions, larger inputs need to be tested.

This commit adds such larger inputs. Note that the aiken solution fails on this, because it just hardcodes results up until 25. Note also that all other submissions fail on this because they essentially time out. The latter case should be correctly handled by uplc-cape (i.e. by restricting the budget of the UPLC machine)

@nielstron
Copy link
Contributor Author

It is currently not explicitly specified anywhere but I guess it should be clarified that solutions should implement a generally correct implementation of the algorithm in question. Ideally, test cases should be designed to make this inevitable.

@KtorZ
Copy link
Contributor

KtorZ commented Oct 23, 2025

Haha; I wanted to suggest this; but I am glad someone else did. Although the prepacked versions demonstrates a practical solution when the problem space is known; I think we can agree they aren't really demonstrating language features 😬... (which was kind of the point I was trying to make).

@Unisay Unisay self-assigned this Oct 24, 2025
@Unisay
Copy link
Collaborator

Unisay commented Oct 24, 2025

The "prepacked" solutions by aiken and plutarch basically hardcode all testcases of the benchmark. This seems to me to somewhat improperly gaming the system.

While I can relate to the "gaming the system" feeling, it's not all that straightforward: the prepack strategy is legit and could be used by real validators. Its true that it won't scale to the higher input values, but its ok to apply it in general.

I am hesitant to approve this PR which changes the scenario tests for one reason: doing it will invalidate existing submissions.

However, I have a counter-offer for you:
I create a new fibonacci_max/factorial_max scenarios with modified rules of the "game" - the goal is to get correct result for increasing inputs: 1, 2, 3, 4, ... 25, ... the higher you get - the better it is.
(For this to work I'd need to modify the measurement framework first)

What do you guys think of this alternative idea?

Otherwise it might be not to late to ask authors of existing pre-packed submissions to update them for higher inputs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants