After starting to host my own private Mastodon instance it became clear that disk usage was going to be problem. I eventually moved all the assets to an S3 compatible Object Storage.
It's funny that all came down to a currency and payment issue (not pricing). Both providers are super cheap and high free tiers.
- Backblaze charges in USD ❌
- Scaleway charges in EUR ✅
- Backblaze speed for deleting objects: slow ❌
- Scaleway speed for deleting objects: faster ✅
- Both providers have data centers in the EU ✅✅
My credit card is linked to an Spanish bank and they charge me a fee for every purchase I do in currencies other than EUR. So the actual S3 Object Storage cost was being inflated by my bank's fees. That's no good.
In hindsight, I should've known this when I signed up for Backblaze
Want to know how I migrated? Here's how:
1. Create a snapshot of the bucket
From B2 you can create a zip (snapshot), then you just download it. It might take a few hours to generate depending on your bucket size, though. Mine was about 50Gb and took around 5 hours.
2. Point Mastodon to the new S3 bucket
Since I'm a single private instance I could this step now. While waiting for the zip to be generated with the existing assets I updated
~/live/.env.production with the new values from the Scaleway bucket (as well as the new API credentials generated within Scaleway, so Mastodon can upload objects) (More on this here).
⚠️ There's a difference on how to set the S3 configuration on your Mastodon instance for Scaleway, here's how it should look like (you may change
nl-ams for your bucket region):
By doing this all new avatars and headers fetched by Mastodon from this point on started being uploaded to the new bucket.
3. AWS S3 cli to the rescue
Now that you have all your assets on your machine (zip file), it's time to upload them to the new provider.
Since all these Object Storage providers support the Amazon S3 API you can make use of AWS's CLI to do all the hard work for you.
Here's the command I used, which syncs a local folder in my computer to the S3 bucket while setting the policy to public. ⚠️ This last part is very important because in Scaleway even if the bucket is public an uploaded file, by default, is set to private. Not sure why, but it works like that.
I used Scaleway's documentation to configure the aws client on my Mac. I only changed the default
max_concurrent_requests to 50 because otherwise my Mac would complain "too many files open" 🤷♂️
4. Update the CDN's origin
Because I'm using the fantastic (and free, because of their insanely high free tier) AWS's CloudFront I just needed to go into the distribution settings and change the origin S3 bucket.
Done, now my CDN starts pulling from the new bucket.
I was a little challenged at the begining but after some digging turns out is not so bad. The longest part is of course the syncing of all the assets from your local machine to the S3, which can take hours.
Otherwise very happy with the process and the change