[Discussion] The state of LuckyFlow

[jwoertink]

For the last 2 weeks after releasing Lucky 1.4.0, I've noticed an increase in failed CI specs all related to LuckyFlow. Sometimes there's 1 failure, sometimes there's 2... Before finalizing the release I wanted to make sure this was fixed. Once I started digging in, I realized this hole was much larger than I expected.

When you generate a new Lucky browser-app, you're given LuckyFlow along with specs that ensure the generated app works. By default this is set to use headless-chrome.

• Generate new app
• Setup new app
• Run specs for very first time
• LuckyFlow tells Webdrivers to install chromedriver
• Specs boot HTTP server
• Chrome spins up in headless mode
• Specs run
• Chrome and HTTP server shut down

This is a great feature, and it gives you lots of examples on how to use it going forward.

The issues

In order to keep these alive, and cross-platform compatible, there's several things that have to be taken in to consideration:

• Selenium
• Chome / Chromedriver
• Firefox / Geckodriver
• Webless (a selenium alternative for faster specs)

This also means maintaining these shards:

• https://github.com/crystal-loot/selenium.cr
• https://github.com/crystal-loot/webdrivers.cr
• https://github.com/crystal-loot/webless
• and of course https://github.com/luckyframework/lucky_flow

At the time of this writing, the only active core team member is just me, so maintaining these is already tough. When new Crystal versions are released, there's potential for breaking changes that can turn in to lots of yak shaving.

Current state

Right now when you generate a Lucky app and run your specs, there's some race condition that happens where you may get 1 of a few different errors. Running the specs back to back without changes is a roll of dice whether the spec will pass or not.

Note

These are snippets from the errors I get from running crystal spec 3 times in a row with no changes between them

  1) Reset password flow works
     Failure/Error: PasswordResetRequestEmail.new(reset_user).should be_delivered

       Expected: #<PasswordResetRequestEmail:0x7db8cf291570> to be delivered, but no emails were delivered

     # spec/support/flows/reset_password_flow.cr:26

  1) Reset password flow works

       Expected to find element on page, but it was not found.
       
         ▸ looking for: [name='user:email'] (LuckyFlow::ElementNotFoundError)
         from lib/lucky_flow/src/lucky_flow/find_element.cr:57:5 in 'raise_element_not_found_error'
         from lib/lucky_flow/src/lucky_flow/find_element.cr:25:5 in 'run'
         from lib/lucky_flow/src/lucky_flow/find_element.cr:13:5 in 'run'
         from lib/lucky_flow/src/lucky_flow.cr:146:5 in 'el'
         from lib/lucky_flow/src/lucky_flow.cr:150:5 in 'field'
         from lib/lucky_flow/src/ext/avram.cr:18:7 in 'fill_form:email:password'


  1) Reset password flow works

       stale element reference: stale element reference: stale element not found
         (Session info: chrome=137.0.7151.103) (Selenium::Error)
         from lib/selenium/src/selenium/http_client.cr:33:7 in 'get'
         from lib/selenium/src/selenium/command_handler.cr:18:7 in 'execute'
         from lib/selenium/src/selenium/command_handler.cr:12:5 in 'execute'
         from lib/selenium/src/selenium/element.cr:103:12 in 'text'
         from lib/lucky_flow/src/lucky_flow/selenium/element.cr:11:5 in 'text'

At this point, I'm nearly 20 hours over the past 2 weeks in to trying to solve this with no end in sight.

Options

I think LuckyFlow is an amazing tool, and something we should definitely have in the ecosystem, but in order to keep Lucky moving forward, we may need to consider some alternate solutions.

1. We remove LuckyFlow as a default integration when generating a Lucky project. In doing this, we could leave in some files with commented code as a way to say "If you'd like to use this, here's a jumping off point".
2. We slim down LuckyFlow to only include Webless out of the box. This would still take a lot of work to get setup, but in theory shouldn't require much additional maintenance in the future since it's basically just piggy-backing off Crystal HTTP::Client. If you wanted to use Chrome and Firefox, they would be extra extensions that may or may not work without someone submitting some PRs
3. We keep things as-is, but just set the specs to pending in the generated apps. This is the cheater way out, but also the easiest solution to just move forward until we can figure out the root cause...
4. Some other option I'm not thinking about at the moment??

Let me hear your opinions on this.

[matthewmcgarvey]

I'm sorry luckyflow is causing problems! I think it's entirely up to you and whatever you can handle. My assumption is that taking it out of the project means LuckyFlow will die, but if it can't be maintained then it's no one's fault. I feel the itch to come back to Crystal quite often, I just have other commitments and can never seem to really pull the trigger on spare-time coding anymore.

[robacarp]

LuckyFlow is an amazing tool and it’s really useful. However I think it’s out of the weight class of the lucky community at the moment. There are already a burdensome count of shards to maintain regardless.

In the same way that Avram focuses on a single database, at most Flow should have well defined hooks for one automation platform. Supporting so many is a task for a dedicated team of maintainers.

But if there is no one ready to commit the time to maintain it even in that state, then the eventual conclusion will be deeper bitrot.

[jwoertink]

taking it out of the project means LuckyFlow will die

I have that worry too. I definitely don't want that to happen. I'm sure if I could figure out the weird race condition, and it can stay stable, then that would be the best option. I just have no clue what's causing it. It must be something recent, but I'm not sure.

I guess the other thing is, none of our production apps use Crystal on the front-end anymore, so we've ripped out Flow a while ago. I no longer use it for work which makes it extra difficult to keep up with.

[fernandes]

I no longer use it for work which makes it extra difficult to keep up with

yes, who takes care, needs to be invested, so don't worry about it

about the state of lucky from itself:

if you don't want to remove and let flow to die and maybe find a better future, a quick solution™ can be adding some feature flag to include flow when creating a new app and default to false? seems like a middle ground between total removal (and not having the burden of maintenance) and having to keep it up-to-date/drivers/...