feat(abr-testing): Mix Protocol with Automated Error Recovery

[rclarke0]

Overview

Protocol that mixes and handles stalls

Test Plan and Hands on Testing

• Tested protocol with 8ch 1000 ul, 200 ul tips, and water
• Initiated stall
• Watched successful error recovery

Changelog

• Protocol added to mix
• If stall occurs, pipette homes the z axis and attempts to blow_out() liquid in the tips
• If a PositionUnknownError occurs, all of the axises are homed excluding the pipette plungers
• If no other error occurs, the pipette continues mixing
• If another error does occur, the error is raised.

Review requests

Risk assessment

• Protocol will only work if error recovery is disabled on the robot
• Haven't tested with different liquid classes

[JonKlar-OT]

It would be better to catch the error types we are interested in explicitly like:

try:
  ...
except StallOrCollisionDetectedError as err:
  try:
    ...
  except PositionUnkknownError as second_err:
    ...
  finally: 
    ctx.comment(f"Total stalls: {stall_count}")

[rclarke0]

I did try catching the errors more explicitly and it did not work.

[sfoster1]

I don't know that this is going to work because of the way protocol execution happens.

Because the robot needs to talk to the app, the only thing that python protocol api calls do is create protocol engine commands and cause them to run. Any success or failure or state change caused by the commands is reflected by the protocol engine. In some cases, successes or failures of command execution might cause an exception to get eventually thrown in the protocol itself.

However, exceptions are thrown from the engine (i.e. because of protocol failures, rather than like syntax errors) to the protocol basically only for the purpose of causing the python interpreter to stop running the protocol - that's the only way to get the python interpreter to do that. By the time the exception gets thrown, the engine has already seen the error, shown the error to the user, and is actively in the middle of cancelling itself. You can't really handle errors that the engine raises to the protocol.

Because of this, we don't make it easy to catch exceptions - it might make people think that it's a good idea, when probably what it will do is leave the engine in a fairly busted state, where core assumptions about the protocol engine state are not true and things may work but haven't really been tested and will likely fail in surprising ways.

Anyway if you want to catch specific errors from the engine you should catch ProtocolCommandFailedError and check what the original_error field is. Success is really not guaranteed.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 24.35%. Comparing base (4c4cb09) to head (dd3efb6).

Additional details and impacted files

@@           Coverage Diff           @@
##             edge   #18910   +/-   ##
=======================================
  Coverage   24.35%   24.35%           
=======================================
  Files        3359     3359           
  Lines      294605   294605           
  Branches    31029    31029           
=======================================
  Hits        71756    71756           
  Misses     222828   222828           
  Partials       21       21           

Flag	Coverage Δ
step-generation	5.32% <ø> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[sfoster1]

I don't think this is a thing that will work super well because of how the engine works (see the inline comment) and won't be reliable if it does work for now, but if it tests out okay maybe it's fine?

[sfoster1]

I don't know that this is going to work because of the way protocol execution happens.

Because the robot needs to talk to the app, the only thing that python protocol api calls do is create protocol engine commands and cause them to run. Any success or failure or state change caused by the commands is reflected by the protocol engine. In some cases, successes or failures of command execution might cause an exception to get eventually thrown in the protocol itself.

However, exceptions are thrown from the engine (i.e. because of protocol failures, rather than like syntax errors) to the protocol basically only for the purpose of causing the python interpreter to stop running the protocol - that's the only way to get the python interpreter to do that. By the time the exception gets thrown, the engine has already seen the error, shown the error to the user, and is actively in the middle of cancelling itself. You can't really handle errors that the engine raises to the protocol.

Because of this, we don't make it easy to catch exceptions - it might make people think that it's a good idea, when probably what it will do is leave the engine in a fairly busted state, where core assumptions about the protocol engine state are not true and things may work but haven't really been tested and will likely fail in surprising ways.

Anyway if you want to catch specific errors from the engine you should catch ProtocolCommandFailedError and check what the original_error field is. Success is really not guaranteed.

[rclarke0]

I don't think this is a thing that will work super well because of how the engine works (see the inline comment) and won't be reliable if it does work for now, but if it tests out okay maybe it's fine?

I tested it on a robot and it worked