Determine file storage strategy in API

[rekibnikufesin]

We need to determine how uploaded files are stored by the API.
Some initial thoughts, input welcome:

• API receives the file, submits the hash to protocol
• pulls any metadata/mime-types from the file
  • creates a thumbnail image --> sends to public S3 bucket for discoverability
  • in dynamodb: store mime-type, S3 URL, hash, thumbnail path, file size (bytes), metadata provided by supply user
• should we store this in S3? Or should it be a local file system to the API...?
  • S3 Pros:
    • S3 is cheap and can easily be used as a CDN source
    • delivery payload can be a signed URL with expiration
  • S3 Cons:
    • S3 is slow as a file system
  • File system pros:
    • fast 🐇
  • File system cons:
    • syncing with CDN can be a pain
    • need multiple file systems or shared file system for supporting highly available API instances
    • this rules out ECS Fargate. We'd either build on EC2 or EC2 based ECS

[ReidWilliams]

Hey @rekibnikufesin do you have a sense of when S3 being slower might have an impact on a user? Which step would take longer with S3 compared to a local filesystem?

Does it impact the upload speed itself (I'd guess that's dominated by the user's internet speed)
Is it the lag between API receipt of the full file and a hash being available to send to protocol?
Something else?

Something else that's probably relevant to this decision: we should think carefully about how we'd use a CDN, S3 or anything that puts a file behind a permanent URL. If we do that, it means that one user can buy access, see the file's permanent url, then share that url on the internet and give everyone else free access to the file.

I think the final delivery url, the place that leads to the raw file download via browser would need to be one time use or expire after a short amount of time (minutes or hours).

[ReidWilliams]

I read your comment more carefully, and expiring S3 urls do seem like a good thing to have.

[rekibnikufesin]

The lag would be between API receipt of the full file and a hash being available to send to protocol. Given that we're looking at ~15 seconds on Mainnet for a transaction anyway, I'm starting to think this is less of an issue than I originally thought.

[rekibnikufesin]

RE: Expiring URLs - we can have the time as little as a few minutes. I'm thinking of something like

• give the user a permanent URL on the api, like https://ffa.computablelabs.com/download/mylittlepony.png
• the API performs any validation we want/can do
• the API generates an S3 signed URL
• the API does an HTTP 301 redirect to the URL (maybe? that would be cool. POC needed)

[ReidWilliams]

Re: lag, good point, and there's the voting itself that creates a delay before a listing is available for use, so yeah filesystem lag doesn't seem to be a big issue to me either.