Merge branch 'release/0.7.2'

Author: HenrikBengtsson

DESCRIPTION

@@ -1,8 +1,8 @@
 Package: future.batchtools
-Version: 0.7.1
+Version: 0.7.2
 Depends:
     R (>= 3.2.0),
-    future (>= 1.8.1)
+    future (>= 1.10.0)
 Imports:
     batchtools (>= 0.9.8)
 Suggests:
@@ -25,5 +25,5 @@ License: LGPL (>= 2.1)
 LazyLoad: TRUE
 URL: https://github.com/HenrikBengtsson/future.batchtools
 BugReports: https://github.com/HenrikBengtsson/future.batchtools/issues
-RoxygenNote: 6.0.1
+RoxygenNote: 6.1.1
 Roxygen: list(markdown = TRUE)

NAMESPACE

@@ -9,6 +9,7 @@ S3method(nbrOfWorkers,batchtools)
 S3method(print,BatchtoolsFuture)
 S3method(resolved,BatchtoolsFuture)
 S3method(result,BatchtoolsFuture)
+S3method(run,BatchtoolsFuture)
 S3method(status,BatchtoolsFuture)
 export("%resources%")
 export(BatchtoolsFuture)
@@ -55,6 +56,7 @@ importFrom(future,nbrOfWorkers)
 importFrom(future,plan)
 importFrom(future,resolved)
 importFrom(future,result)
+importFrom(future,run)
 importFrom(future,tweak)
 importFrom(utils,capture.output)
 importFrom(utils,file_test)

NEWS

@@ -1,16 +1,36 @@
 Package: future.batchtools
 ==========================
 
+Version: 0.7.2 [2018-12-03]
+
+DOCUMENTATION:
+
+ * Add a simple example(future_custom).
+
+FIXES:
+
+ * Made internal code agile to upcoming changes in the future package on
+   how a captured error is represented.
+
+SOFTWARE QUALITY:
+
+ * FYI: Every release is tested against one Torque/PBS and one SGE scheduler.
+ 
+BUG FIXES:
+
+ * resolve() on a lazy batchtools future would stall and never return.
+ 
+
 Version: 0.7.1 [2018-07-18]
 
 NEW FEATURES:
 
- o The batchtools_* backends support the handling of the standard output as
+ * The batchtools_* backends support the handling of the standard output as
    implemented in future (>= 1.9.0).
 
 BUG FIXES:
 
- o A bug was introduced in future.batchtools 0.7.0 that could result in "Error 
+ * A bug was introduced in future.batchtools 0.7.0 that could result in "Error 
    in readLog(id, reg = reg) : Log file for job with id 1 not available" when
    using one of the batchtools backends.  It occurred when the value was
    queried.  It was observered using 'batchtools_torque' but not when using
@@ -22,7 +42,7 @@ Version: 0.7.0 [2018-05-03]
 
 NEW FEATURES:
 
- o Argument 'workers' of future strategies may now also be a function, which
+ * Argument 'workers' of future strategies may now also be a function, which
    is called without argument when the future strategy is set up and used as
    is.  For instance, plan(callr, workers = halfCores) where
    halfCores <- function() { max(1, round(availableCores() / 2)) } will use
@@ -31,91 +51,91 @@ NEW FEATURES:
 
 CODE REFACTORING:
 
- o Preparing for futures to gather a richer set of results from batchtools
+ * Preparing for futures to gather a richer set of results from batchtools
    backends.
 
 
 Version: 0.6.0 [2017-09-10]
 
 NEW FEATURES:
 
- o If the built-in attempts of batchtools for finding a default template file
+ * If the built-in attempts of batchtools for finding a default template file
    fails, then system("templates", package = "future.batchtools") is searched
    for template files as well.  Currently, there exists a `torque.tmpl` file.
 
- o A job's name in the scheduler is now set as the future's label (requires
+ * A job's name in the scheduler is now set as the future's label (requires
    batchtools 0.9.4 or newer).  If no label is specified, the default job name
    is controlled by batchtools.
 
- o The period between each poll of the scheduler to check whether a future
+ * The period between each poll of the scheduler to check whether a future
    (job) is finished or not now increases geometrically as a function of number
    of polls.  This lowers the load on the scheduler for long running jobs.
 
- o The error message for expired batchtools futures now include the last few
+ * The error message for expired batchtools futures now include the last few
    lines of the logged output, which sometimes includes clues on why the future
    expired.  For instance, if a TORQUE/PBS job use more than the allocated
    amount of memory it might be terminated by the scheduler leaving the message
    "PBS: job killed: vmem 1234000 exceeded limit 1048576" in the output.
 
- o print() for BatchtoolsFuture returns the object invisibly.
+ * print() for BatchtoolsFuture returns the object invisibly.
 
 BUG FIXES:
 
- o Calling future_lapply() with functions containing globals part of non-default
+ * Calling future_lapply() with functions containing globals part of non-default
    packages would when using batchtools futures give an error complaining that
    the global is missing. This was due to updates in future (>= 1.4.0) that
    broke this package.
 
- o loggedOutput() for BatchtoolsFuture would always return NULL unless an error
+ * loggedOutput() for BatchtoolsFuture would always return NULL unless an error
    had occurred.
 
 
 Version: 0.5.0 [2017-06-02]
 
- o First version submitted to CRAN.
+ * First version submitted to CRAN.
 
 SOFTWARE QUALITY:
 
- o Added more tests; test coverage now at 93%.
+ * Added more tests; test coverage now at 93%.
 
 
 Version: 0.4.0 [2017-05-16]
 
 NEW FEATURES:
 
- o Added batchtools_custom() for specifying batchtools futures using any type
+ * Added batchtools_custom() for specifying batchtools futures using any type
    of batchtools cluster functions.
 
- o batchtools_template(pathname = NULL, type = <type>) now relies on the
+ * batchtools_template(pathname = NULL, type = <type>) now relies on the
    batchtools package for locating the <type> template file.
 
- o nbrOfWorkers() for batchtools futures now defaults to +Inf unless the
+ * nbrOfWorkers() for batchtools futures now defaults to +Inf unless the
    evaluator's 'workers' or 'cluster.functions' specify something else.
 
- o Renamed argument 'pathname' to 'template' for batchtools_<tmpl>() functions.
+ * Renamed argument 'pathname' to 'template' for batchtools_<tmpl>() functions.
 
 BUG FIXES:
 
- o Under plan(batchjobs_*), when being created futures would produce an error
+ * Under plan(batchjobs_*), when being created futures would produce an error
    on "all(is.finite(workers)) is not TRUE" due to an outdated sanity check.
 
 SOFTWARE QUALITY:
 
- o TESTS: Added test of future_lapply() for batchtools backends.
+ * TESTS: Added test of future_lapply() for batchtools backends.
 
- o TESTS: Added optional tests for batchjobs_* HPC schedulers listed in
+ * TESTS: Added optional tests for batchjobs_* HPC schedulers listed in
    environment variable 'R_FUTURE_TESTS_STRATEGIES'.
 
 CODE REFACTORING:
 
- o CLEANUP: Package no longer depends on R.utils.
+ * CLEANUP: Package no longer depends on R.utils.
 
 
 Version: 0.3.0 [2017-03-19]
 
 NEW FEATURES:
 
- o The number of jobs one can add to the queues of HPC schedulers is in
+ * The number of jobs one can add to the queues of HPC schedulers is in
    principle unlimited, which is why the number of available workers for such
    batchtools_* backends is reported as +Inf.  However, as the number of
    workers is used by future_lapply() to decide how many futures should be used
@@ -128,14 +148,14 @@ Version: 0.2.0 [2017-02-23]
 
 GLOBALS:
 
- o batchtools (>= 0.9.2) now supports exporting objects with any type of names
+ * batchtools (>= 0.9.2) now supports exporting objects with any type of names
    (previously only possible if they mapped to to strictly valid filenames).
    This allowed me to avoid lots of internal workaround code encoding and
    decoding globals.
 
 
 Version: 0.1.0 [2017-02-11]
 
- o Package created by porting the code of future.BatchJobs. This version passes
+ * Package created by porting the code of future.BatchJobs. This version passes
    'R CMD check --as-cran' with all OK after a minimal amount of adjustments
    to the ported code.

R/BatchtoolsFuture-class.R

@@ -198,8 +198,7 @@ status.BatchtoolsFuture <- function(future, ...) {
 
   result <- future$result
   if (inherits(result, "FutureResult")) {
-    condition <- result$condition
-    if (inherits(condition, "error")) status <- c("error", status)
+    if (result_has_errors(result)) status <- unique(c("error", status))
   }
 
   status
@@ -312,10 +311,9 @@ result.BatchtoolsFuture <- function(future, cleanup = TRUE, ...) {
 }
 
 
-run <- function(...) UseMethod("run")
-
-#' @importFrom future getExpression
+#' @importFrom future run getExpression
 #' @importFrom batchtools batchExport batchMap saveRegistry setJobNames
+#' @export
 run.BatchtoolsFuture <- function(future, ...) {
   if (future$state != "created") {
     label <- future$label
@@ -504,9 +502,7 @@ await.BatchtoolsFuture <- function(future, cleanup = TRUE,
         result[["batchtools_log"]] <- try({
           getLog(id = jobid, reg = reg)
         }, silent = TRUE)
-        if (inherits(result$condition, "error")) {
-          cleanup <- FALSE
-        }
+        if (result_has_errors(result)) cleanup <- FALSE
       }
     } else if ("error" %in% stat) {
       cleanup <- FALSE
@@ -637,7 +633,7 @@ delete.BatchtoolsFuture <- function(future,
     status <- status(future)
     res <- future$result
     if (inherits(res, "FutureResult")) {
-      if (inherits(res$condition, "error")) status <- "error"
+      if (result_has_errors(res)) status <- unique(c("error", status))
     }
     mdebug("delete(): status(<future>) = %s",
            paste(sQuote(status), collapse = ", "))

R/batchtools_custom.R

@@ -12,6 +12,8 @@
 #'
 #' @return An object of class `BatchtoolsFuture`.
 #'
+#' @example incl/batchtools_custom.R
+#'
 #' @export
 #' @importFrom utils file_test
 batchtools_custom <- function(expr, envir = parent.frame(), substitute = TRUE,

R/utils.R

@@ -170,3 +170,18 @@ tempvar <- function(prefix = "var", value = NA, envir = parent.frame()) {
   # Failed to find a unique temporary variable name
   stop(sprintf("Failed to generate a unique non-existing temporary variable with prefix '%s'", prefix)) #nolint
 }
+
+
+
+result_has_errors <- function(result) {
+  stop_if_not(inherits(result, "FutureResult"))
+
+  ## BACKWARD COMPATIBILITY: future (< 1.11.0)
+  if (inherits(result$condition, "error")) return(TRUE)
+  
+  for (c in result$conditions) {
+    if (inherits(c$condition, "error")) return(TRUE)
+  }
+  
+  FALSE
+}

README.md

@@ -213,7 +213,7 @@ Contributing to this package is easy.  Just send a [pull request](https://help.g
 
 ## Software status
 
-| Resource:     | CRAN        | Travis CI       | Appveyor         |
+| Resource:     | CRAN        | Travis CI       | AppVeyor         |
 | ------------- | ------------------- | --------------- | ---------------- |
 | _Platforms:_  | _Multiple_          | _Linux & macOS_ | _Windows_        |
 | R CMD check   | <a href="https://cran.r-project.org/web/checks/check_results_future.batchtools.html"><img border="0" src="http://www.r-pkg.org/badges/version/future.batchtools" alt="CRAN version"></a> | <a href="https://travis-ci.org/HenrikBengtsson/future.batchtools"><img src="https://travis-ci.org/HenrikBengtsson/future.batchtools.svg" alt="Build status"></a>   | <a href="https://ci.appveyor.com/project/HenrikBengtsson/future-batchtools"><img src="https://ci.appveyor.com/api/projects/status/github/HenrikBengtsson/future.batchtools?svg=true" alt="Build status"></a> |

cran-comments.md

@@ -1,64 +1,56 @@
-# CRAN submission future.batchtools 0.7.1
+# CRAN submission future.batchtools 0.7.2
 
-on 2018-07-18
+on 2019-01-03
 
-## Submission 1
+Thanks in advance.
 
-Thanks in advance
-
-## Submission 2
-
-Resubmission of future.batchtools 0.7.1 where the overall R CMD check time has been decreased significantly.
 
+## Notes not sent to CRAN
 
 ### R CMD check --as-cran validation
 
 The package has been verified using `R CMD check --as-cran` on:
 
-* Platform x86_64-apple-darwin15.6.0 (64-bit) [r-hub; single-core]:
-  - R version 3.5.0 (2018-04-23)
+* Platform x86_64-apple-darwin15.6.0 (64-bit) [Travis CI]:
+  - R version 3.4.4 (2017-01-27)
+  - R version 3.5.2 (2018-12-20)
 
 * Platform x86_64-unknown-linux-gnu (64-bit) [Travis CI]:
   - R version 3.4.4 (2017-01-27)
-  - R version 3.5.0 (2017-01-27)
-  - R Under development (unstable) (2018-06-20 r74923)
+  - R version 3.5.1 (2018-07-02)
+  - R Under development (unstable) (2019-01-02 r75944)
 
 * Platform x86_64-pc-linux-gnu (64-bit) [r-hub]:
   - R version 3.4.4 (2018-03-15)
-  - R Under development (unstable) (2018-07-16 r74967)
-
+  - R Under development (unstable) (2018-12-22 r75884)
+  
 * Platform x86_64-pc-linux-gnu (64-bit):
-  - R version 3.2.0 (2015-04-16)
-  - R version 3.3.0 (2016-05-03)
-  - R version 3.4.0 (2017-04-21)
-  - R version 3.5.0 (2018-04-23)
+  - R version 3.2.0 (2015-04-16) w/ Torque/PBS scheduler
+  - R version 3.3.0 (2016-05-03) w/ Torque/PBS scheduler
+  - R version 3.5.0 (2018-04-23) w/ Torque/PBS scheduler
   - R version 3.5.1 (2018-07-02)
+  - R version 3.5.1 (2018-07-02) w/ SGE scheduler
+  - R version 3.5.2 (2018-12-20) w/ Torque/PBS scheduler
 
-* Platform i386-w64-mingw32 (32-bit) (64-bit) [Appveyor CI]:
-  - R Under development (unstable) (2018-07-16 r74967)
+* Platform i686-pc-linux-gnu (32-bit):
+  - R version 3.4.4 (2018-03-15)
 
-* Platform x86_64-w64-mingw32/x64 (64-bit) [Appveyor CI]:
-  - R version 3.5.1 (2018-07-02)
-  - R Under development (unstable) (2018-07-16 r74967)
+* Platform i386-w64-mingw32 (32-bit) [Appveyor CI]:
+  - R Under development (unstable) (2018-10-12 r75434)
 
 * Platform x86_64-w64-mingw32 (64-bit) [r-hub]:
-  - R Under development (unstable) (2018-07-16 r74967)
+  - R Under development (unstable) (2018-12-26 r75909)
 
-* Platform x86_64-w64-mingw32/x64 (64-bit) [win-builder]:
-  - R version 3.5.1 (2018-07-02)
-  - R Under development (unstable) (2018-07-16 r74967)
-
-
-The following setups were skipped due to non-availability:
-
-* Platform x86_64-apple-darwin13.4.0 (64-bit) [Travis CI]:
-  - R version 3.4.4 (2017-01-27)
+* Platform x86_64-w64-mingw32/x64 (64-bit) [Appveyor CI]:
+  - R version 3.5.2 (2018-12-20)
+  - R Under development (unstable) (2018-12-29 r75924)
 
-* Platform x86_64-apple-darwin15.6.0 (64-bit) [Travis CI]:
-  - R version 3.5.0 (2018-04-23)
+* Platform x86_64-w64-mingw32/x64 (64-bit) [win-builder]:
+  - R version 3.5.2 (2018-12-20)
+  - R Under development (unstable) (2019-01-01 r75943)
 
-* Platform i686-pc-linux-gnu (32-bit):
-  - R version 3.4.4 (2018-03-15)
+Failed to test:
 
-* Platform i686-pc-linux-gnu (32-bit):
-  - R version 3.4.4 (2018-03-15)
+* Platform i386-pc-solaris2.10 (32-bit):
+  - R version 3.5.0 Patched (2018-04-30 r74674)
+  REASON: 'stringi' package failed to install

incl/batchtools_custom.R

@@ -0,0 +1,10 @@
+cf <- batchtools::makeClusterFunctionsInteractive(external = TRUE)
+plan(batchtools_custom, cluster.functions = cf)
+
+## Create explicit future
+f <- future({
+  cat("PID:", Sys.getpid(), "\n")
+  42L
+})
+v <- value(f)
+print(v)