`frollapply` doesn't handle zero-length output

[aitap]

Found while looking for unprotected calls to memcpy for #7051:

data.table::frollapply(list(integer()), 0, \(x) 1)

froll.c:436:15: runtime error: pointer index expression with base 0x000000000001 overflowed to 0xfffffffffffffff9
froll.c:436:21: runtime error: store to misaligned address 0xfffffffffffffff9 for type 'double', which requires 8 byte alignment
0xfffffffffffffff9: note: pointer points here
<memory cannot be printed>

 *** caught segfault ***
address (nil), cause 'unknown'

Traceback:
 1: data.table::frollapply(list(integer()), 0, function(x) 1)
An irrecoverable exception occurred. R is aborting now ...

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff3f2341c in frollapply (x=0x1, nx=0, w=w@entry=0x1, k=0, ans=ans@entry=0x55555618c070, align=align@entry=1, fill=fill@entry=nan(0x0000007a2), call=0x555558b09ae0, 
    rho=0x555558b09ed0, verbose=false) at froll.c:436
436         ans->dbl_v[k-1] = REAL(eval0)[0];
(gdb) bt
#0  0x00007ffff3f2341c in frollapply (x=0x1, nx=0, w=w@entry=0x1, k=0, ans=ans@entry=0x55555618c070, align=align@entry=1, fill=fill@entry=nan(0x0000007a2), 
    call=0x555558b09ae0, rho=0x555558b09ed0, verbose=false) at froll.c:436
#1  0x00007ffff3f24bf6 in frollapplyR (fun=0x555558b075d8, obj=0x5555585c8448, k=<optimized out>, fill=<optimized out>, align=<optimized out>, rho=0x555558b09ed0)
    at frollR.c:292

I was looking for calls of the form memcpy(<destination>, <invalid pointer>, 0) (which typically don't show any symptoms until additional sanitizer checks at CRAN find it; I think it also happens here), but frollapply() actually dereferences the invalid pointer and crashes the process.

[jangorecki]

It will be resolved by #5575 which moved frollapply implementation from C to R.

---

Actually it will be resolved even sooner, as PR #5441, preceding PR #5575, already adds an extra check:

data.table::frollapply(list(integer()), 0, function(x) 1)
#Error in froll(FUN = FUN, rho = rho, x = x, n = n, fill = fill, align = align,  : 
#  n must be positive integer values (> 0)

---

I will close this issue with extra unit test when merging #5575

frollapply(list(integer()), 0, function(x) 1)

[aitap]

Thank you very much! Does the froll2025 branch in your fork contain the latest code? Unfortunately I found another one:

frollapply(list(integer()), list(integer()), str, adaptive=TRUE)

Program received signal SIGSEGV, Segmentation fault.
maxk (k=0x1, len=0) at frollR.c:214
214       int mk = k[0];
(gdb) bt                           
#0  maxk (k=0x1, len=0) at frollR.c:214                                                                                                                                        
#1  0x00007ffff3f23c9b in frollapplyR (fun=0x555556af7ff0, xobj=0x555555cccfa8, kobj=<optimized out>, fill=<optimized out>, align=<optimized out>, adaptive=<optimized out>, 
    rho=0x555558b0b200) at frollR.c:313