Simplify auto-scaling, support interoperability with mirai

crew 1.3.0

• Reinstate checks that were temporarily suppressed to help plugins transition to #217.
• Support reset_globals, reset_packages, reset_options, and garbage_collection in crew_controller_sequential() (#217).
• Change argument tls to tlscert in the call to mirai::daemon() (#227, @shikokuchuo).
• Use collections queues instead of custom queues (#229).
• Use collections dictionaries instead of hash environments to track tasks (#229).
• Remove the long-deprecated promise() method of the controller.
• Improve interoperability of crew's workers by avoiding nanonext::cv_value() for condition variables (#225). Consequences:
  • wait(mode = "all") no longer guarantees that a task is available for pop(). It just consumes a condition variable signal. pop() should always be checked for NULL return values.
  • Until mirai gains a threaded dispatcher, saturated() needs to avoid the overhead of calling status() to get task counts. So the definition of "saturated" has changed: a controller is saturated if the number of uncollected tasks is greater than or equal to the maximum number of workers. Previously, it was the number of unresolved tasks.
  • Internal counters .pushed and .popped are safely removed.
  • Active bindings pushed and popped are removed.
• Controllers no longer have a unpopped() method because it is unnecessary to export and it assumes only the controller submits tasks to the compute profile (affecting interoperability).
• Allow a custom later loop in autoscale().
• Deprecate the pids() methods in controllers and clients and stop babysitting the dispatcher process (#236).
• Deprecate crew_clean() in favor of crew_monitor_local() (#236).
• Trim down the argument list of launch_worker() to just include arguments call and name.
• Simplify the auto-scaling algorithm to support greater interoperability and the eventual threaded dispatcher in mirai (#232).
• Deprecate processes and async in crew launchers in favor of job arrays (#218, #237).
• Drop the unified controller group throttle. Let individual controllers handle their own throttling based on their own auto-scaling settings.
• When tasks_mask is finite, set seconds_min to seconds_interval in the throttle to give tasks enough time to accumulate.
• Use the counters in mirai::info() instead of mirai::status()$events (#232, @shikokuchuo).
• Allow custom compute profiles in crew_client(). This is mainly to support the use of the default compute profile to make crew more compatible with non-crew mirai-based workflows. It's a quick solution for single controllers but does not work for controller groups.
• crew_client()$start() now errors if a mirai compute profile of the same name is already active.
• Shorten the length of strings from crew_random_name().

CRAN patch: MKL 'Additional issues'

crew 1.2.1

• CRAN patch (address MKL "Additional issues").

Performance and compatibility

crew 1.2.0

• 22.7x speedup in controller$walk() with 100000 tasks: pre-compute the task list and defer auto-scaling until all tasks are pushed. Pushing a million tasks in walk() now takes just under 70 seconds on a local M2 Macbook.
• Use nanonext::ip_addr() instead of getip::getip() to get the default IP address of the host (#216, @shikokuchuo).
• Add a progress bar to walk() report progress when pushing tasks.
• Move arguments reset_globals, reset_packages, reset_options, and garbage_collection from the launcher to the controller (#217). crew_eval() now handles cleanup directly, and workflows running mirai 2.0.0 should mostly run faster because not all cleanup steps are automatically enabled.
• Avoid mirai::call_mirai_() in tests (#222, @shikokuchuo).

Performance

crew 1.1.2

• Speedup for targets: avoid superfluous processing in controller$pop().
• Speedup queue methods, especially queue$push(), by making the queue non-portable and using direct access and direct assignment in R6.
• Reduce the frequency of calls to mirai::status() in workloads with a large number of quick tasks. Instead of resetting the throttle every time there is task or worker activity, only reset if there is worker activity. This was achieved by giving relay objects their own throttles instead of having them use the launcher throttles.
• Revert to a logical value for the dispatcher argument of mirai::daemons() (#215, @shikokuchuo).

Fix documentation and links

crew 1.1.1

• Fix incorrect messages in documentation about deprecating the name and workers arguments of controllers. They are not deprecated at the controller level. They are only deprecated in crew_client().

Serialization and sequential controller

crew 1.1.0

• Change prefix in name_worker() from crew- to crew-worker-.
• Use call_mirai() instead of the now deprecated call_mirai_() (@shikokuchuo, #198).
• Add crew_controller_sequential() (#205).
• Add a serialization argument to crew_client() (#204).

Migrate to mirai 1.0.0

crew 1.0.0

• Avoid partial match in a call to crew_terminate_process().
• Migrate to mirai 2.0.0 with the pair 1 poly protocol (#193). Most notably, all workers now share the same URL instead of having different web sockets for different workers. This gets rid of crew's previous "slotted" model for workers.
• Prohibit duplicated task names in push(). (The task must be popped first before another task of the same name can be submitted.) This allows controllers to use hash tables to track tasks, which makes push() and pop() orders of magnitude faster. It is also needed for the new retry mechanism based on backup controllers and controller groups.
• Use exponential backoff in throttles.
• Share throttles between scale() and wait() to increase responsiveness.
• Use a unified throttle for controller groups, which requires refactoring wait(mode = "all") for controller groups.
• Name every task and record every command.
• Switch URLs from ws:// (and wss://) to tcp:// (and tls+tcp://) (#193).
• Change R-level error status code to -1 (previously 1).
• Add a queue class for resolved tasks to make pop() and collect() more efficient. pop() is 64 times faster.
• For clarity, rename the "launcher" column of the results to "controller".
• Log crashes and cancellations separately from ordinary errors in summary().

Improved crash detection

crew 0.10.2

• Eliminate spurious launch_max error from underutilized workers (#189).
• Deprecate launch_max in favor of crashes_error (#189).
• Look for crashes of all workers in rotate() instead of looking for crashes of a specific worker in launch() (#189).
• Add a crashes() launcher method to allow plugins to detect and respond to crashes more easily.
• Change default seconds_idle to 300.
• Move autometric to Suggests:.

Resource logging with autometric

crew 0.10.0

• Give full worker name in the worker launch error message (@multimeric).
• Start the controller automatically in more methods.
• Add an r_arguments argument to supply command line arguments to R for workers (#175, @rpruim).
• Add a cancel() controller method to cancel one or more tasks.
• Call cancel(all = TRUE) from terminate() as a safeguard.
• Make client$dispatcher a ps::ps_handle() handle and add a new client$client handle for the current process.
• To passively log memory usage when log_resources is given, the controller now calls log() as a side effect in most controller methods, with throttling to preserve speed.
• Return a status and status code from pop() etc.
• New internal function as_monad() makes error reporting more consistent.
• Use path.expand() on local log files.
• Switch to Rscript.
• Print session info from crew_worker() before attempting to dial into mirai.
• Add asyncdial = FALSE back to mirai::daemon() call.
• Deprecate local_log_directory and local_log_join in favor of crew_options_local() and the options_local argument.
• Add crew_options_metrics() and the options_metrics argument for recording resource metrics (#178).
• Add a new logging.Rmd vignette to explain best practices for logging and resource usage metrics (#178).

CRAN patch

crew 0.9.5

• CRAN patch.