Memory exhaustion when trying to fread the index.

[mookman288]

Hi,

First, I think this is related to #747.

I've been using Clockwork on a development app that I've been playing around with for a year or so. The app is a bit of a passion project, so thankfully this didn't happen in production. I'm currently testing extremely large media uploads and processing them. I'm trying to nail down 2-20+GB 4K UHD file uploads.

When testing last night I started receiving this error on every request:

Allowed memory size of 1073741824 bytes exhausted (tried to allocate 1242655144 bytes) at /vendor/itsgoingd/clockwork/Clockwork/Storage/FileStorage.php:256)

This is related to this line:

                $trimmed = $trimmedLength > 0 ? fread($this->indexHandle, $trimmedLength) : '';

I turned off debug mode, but because Clockwork was still enabled, it continued to occur.

So I figured at this point that it's related to the index file just being too large to fread, and I decided to investigate the clockwork storage folder:

du -sh clockwork
304G clockwork

Woops. At 304GB, I had a feeling something was truly off. I haven't come close to exhausting my local system's storage, so I just didn't notice this 300+GB folder (the storage folder is also larger than that, because I'm testing these massive uploads.)

Looking at the clockwork/index file:

1302535836 Oct 15 11:36 clockwork/index

It's absolutely massive, at 1.3GB. I figure this isn't normal. For reference, I set PHP to 1GB of memory, but it looks like it's actually using 1073MB.

I imagine that because I'm uploading large files through Laravel, that must be the reason why Clockwork has such a large storage directory. I don't mind it being large, but I think it's been recording data since I very first started using it, and that was over a year ago.

In looking at #747, I saw that there was a configuration declaration CLOCKWORK_STORAGE_EXPIRATION but I can't see anything in the documentation about this. Based on the comment in that thread though, it seems to be set to minutes, so I am going to try 40000+, since I figure a month or so is enough data to retain.

According to a different repo comment (ppy/osu-web#8420 (comment)) this is set by default to 1 week for them.

Is there a default set for Clockwork?

Did my system not honor that default?

I don't want to lose all the data, so I'm going to try and delete files older than 30 days. Hopefully that works.

If you have any suggestions, let me know, and I hope this helps the probably one other person this might happen to.