Any way to guess estimate the potential storage space required for a backup of slack , heuristic or math.

[PaulNewton]

spitballing through prep going through the options for exporting slack channels , so gonna be trying this tool first I think.
So curious if there's a way to estimate data's total space, hard drive disk space needed, that an entire slack channel might take up.
Just in raw text and relations no media/files mainly.

I'll should be starting first on a small subchannel as a test soon, to sample the archival sizes and the ongoing maintenance of that.
Then expand out to larger subchannels in order of priority.
Based on factors like moderator hygiene of the channel to minimize backing up spam or just general noise.

Files

For files slack has a ~1gb limit per file, so if files can be counted that can give one metrics possible MAX that way.
there's also total upload limits based on plan?, and sometimes it's viewers can only see that last X-gbs added i think.

Then if getting ALL file contents is unrealistic, will need work out how to decide which files are worth saving sight-unseen from slackland.
Then programmatically get only certain files.
spitball algorithm 💭 : e.g. bookmarked, matches keywords, mod posts/files, announcements, Posts with most replies/emojii for just the opening post, and-or OP replies to such posts with attachments, optionally other replies with files.

Though a concern I'm not clear yet on is if after getting text content you can then get files and still having a connection between them.
__
Then of course on top of the data also stowing this project itself ,or a viewer, long term which should be miniscule in comparison.
But worth accounting for; such as if made as an image to get running fast when you just want to export some slice.

---

searched keywords like hard drive, storage, space, total size, file size, capacity, uncompressed/compressed/compression but didn't find much on estimation preparation aside from Deltas & resume functionality, memory running out getting fixed :) , and exporting only parts of a channel .

[volker-fr]

I am not aware of any. Maybe programmatical check all files & their sizes (if reported by the API) and add sum it. Maybe the Slack admin interfaces shows it in the export setting.

I have one Slack with 247G, years of chats and images. Other ones are between 1GB and 15GB.

You should be able to download only text for everything and then text+images for individual channels. Alternatively stop downloads and increase the storage / move to a different disk.

[rusq]

Hey @PaulNewton, as @volker-fr said, it would be possible to determine the file download size by scanning all messages in the [already downloaded] archive or export or dump, and summing the file sizes. There's no built in function for this, but maybe it would be a good addition.

I did a quick check and there seems to be no API call to get the estimate.

I have made some changes to redownload tool to support any kind of archive (but not ZIP files!), so it would be possible to grab all messages, say, with

slackdump archive -files=false

and then, download all files with:

slackdump tools redownload <your_export_or_archive>

I'll create an issue to add a file download size calculator to tools.

[PaulNewton]

Okay wow, thanks, will have to look at the redownload if there turns out to be enough time, just doing a lot of back and forth yak shaving in regards to preservation with a time crunch while I also should be doing actual work 😏.

Does running archive on an existing folder w/ database hard overwrite the contents?
Right now I've got archive running at full rip for a set of channels I want to triage so very hesitant to stop the run.
As that command outputs the stream ids/timestamps so I've been able to convert that to dates and guesstimate channel[1] progress time/size.
I was using resume for a #general channel, but it's pretty blank on the output so didn't want to try that with a set.

For estimation not sure if it should be it's own thing to run separately, or during other command retry dwell time.
For running pre-emptively as it's own tool,
I was thinking more along the lines of sampling messages in channels from a given set; maybe with everything enabled(files,avatars) as default.
Then finding the first post to the last post then randomly samples in between dates.
If not fully hitting the mid point between shrinking sets of dates (?bisecting?).
Crudest I can think of not sure of a better probability approach for this.

Or for command retry dwell time, such as running as part of an archive run, or other commands.
During any rate limiting dwell time(retry_after) spit out stats; (during got rate limited, sleeping timeouts).
Though the later reeks of being more a notification thing, so stuff like metrics get left as a user responsibility.

And not sure if either should be straight stats or also track prior state for comparison, if not also a soft progress guesstimate.

[1] just noticed it seems archive doesn't processes the channels in the order given so I've got to remember to note the channel-ID or I did something wrong.

---

So far I think storage of messages, ?threads?, won't be as big an issue as I thought, it's more about the time it takes for messages.
Thought assets, _uploads/_avatars, for ALL channels could become problem; at least for my case so far.
Just don't have a good way to measure it all quickly.

In the here and now for estimation I've just started recording directory sizes, and writing a powershell script, so I can keep on eye on it if it gets too big, though I just noticed I think I got my size-on-disk code wrong /sigh.
etc etc Along with a bunch of other notes for a noob bumbling through slacks trapped walled garden at the last minute.

You've made a great tool here, so I'm probably making it harder on myself to learn it while trying to go full tilt.
Either I'm SOL in ~6hrs and hundreds if not thousands of years of human expertise vanishes, or might have a few more days not quite sure ¯_(ツ)_/¯.
But I'd prolly have waay more yak-shaving going on without a tool like this though.
Like doing something janky with browser-automation and html.
Not even sure if trying to preserve things is gonna having any long term utility at all lol.

[rusq]

archive won't operate on the existing directory, resume would. I would suggest for clarity make separate archives, there will be a merge functionality coming soon, that would be able to combine (and deduplicate) two or more archives into one, so in the end all those archives can be combined into a single large one, including files.

[rusq]

Released in v3.1.11

[rusq]

Issue created #582