Skip to content

Conversation

@VoyTechnology
Copy link
Member

This is a functional fallback version of the new heracles architecture. It is designed to full job processing pipeline.

It contains all functionality of the normal manager (after #418), It is however slightly deficient and not as fully featured as the Rust version. The main are:

  • There is no support for chunks larger than 64MB
  • All progress checking is done using state store and filesystem notifications instead of message acknowledgements

Known issues, possibly originating and related to the old worker rewrite include:

  • If a task is send too quickly, the connection to the broker might not yet be established
  • If a task fails, it gets restarted indefinatelly
  • If a task breaks midway, it might hang the worker

Some of the above issues can be "fixed" by purging the RabbitMQ queue

Note: the logging is commented out in worker due to weird issues on my machine. They are serving informational purpose only and are non essential to the working of the project.

This is a functional version of the new heracles architecture.
@VoyTechnology VoyTechnology merged commit 4c872a4 into develop Apr 19, 2018
@VoyTechnology VoyTechnology deleted the feature/manager-fallback branch April 19, 2018 10:56
@VoyTechnology
Copy link
Member Author

Merged using admin privileges, without code review, for sprint purposes.

Please see #420 for tracking issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants