Don't attempt reuseSorting on non-data.table in bmerge (#6502)

Author: MichaelChirico

NEWS.md

@@ -71,6 +71,8 @@ rowwiseDT(
 
 12. Using a namespace-qualified call on the RHS of `by=`, e.g. `DT[,.N,by=base::mget(v)]`, works again, fixing [#6493](https://github.com/Rdatatable/data.table/issues/6493). Thanks to @mmoisse for the report and @MichaelChirico for the fix.
 
+13. Restore some join operations on `x` and `i` (e.g. an anti-join `x[!i]`)  where `i` is an extended data.frame, but not a data.table (e.g. a `tbl`), [#6501](https://github.com/Rdatatable/data.table/issues/6501). Thanks @MichaelChirico for the report and PR.
+
 ## NOTES
 
 1. Tests run again when some Suggests packages are missing, [#6411](https://github.com/Rdatatable/data.table/issues/6411). Thanks @aadler for the note and @MichaelChirico for the fix.

inst/tests/tests.Rraw

@@ -19268,3 +19268,10 @@ test(2291.1, set(df, NULL, "new", "new"), error="attributes .* have been reassig
 DT = data.table(a = 1)
 test(2292.1, DT[, .N, by=base::mget("a")], data.table(a = 1, N = 1L))
 test(2292.2, DT[, .N, by=base::c("a")], data.table(a = 1, N = 1L))
+
+# don't try to re-use sorting on a non-data.table, #6501
+x = data.table(id = "aaa")
+y = data.frame(id = "bbb")
+test(2293.1, x[!y, on='id'], x)
+class(y) = c("tbl_df", "tbl", "data.frame")
+test(2293.2, x[!y, on = "id"], x)

src/bmerge.c

@@ -160,7 +160,8 @@ SEXP bmerge(SEXP idt, SEXP xdt, SEXP icolsArg, SEXP xcolsArg, SEXP xoArg, SEXP r
   protecti += 2;
 
   SEXP ascArg = PROTECT(ScalarInteger(1));
-  SEXP oSxp = PROTECT(forderReuseSorting(idt, icolsArg, /* retGrpArg= */ScalarLogical(FALSE), /* retStatsArg= */ScalarLogical(FALSE), /* sortGroupsArg= */ScalarLogical(TRUE), ascArg, /* naArg= */ScalarLogical(FALSE), /* reuseSortingArg= */ScalarLogical(TRUE))); protecti++;
+  SEXP reuseSortingArg = INHERITS(idt, char_datatable) ? ScalarLogical(TRUE) : ScalarLogical(FALSE);
+  SEXP oSxp = PROTECT(forderReuseSorting(idt, icolsArg, /* retGrpArg= */ScalarLogical(FALSE), /* retStatsArg= */ScalarLogical(FALSE), /* sortGroupsArg= */ScalarLogical(TRUE), ascArg, /* naArg= */ScalarLogical(FALSE), reuseSortingArg)); protecti++;
   UNPROTECT(2); // down stack to 'ascArg'
   PROTECT(oSxp);