Moving S3 from Backblaze B2 to Scaleway

After starting to host my own private Mastodon instance it became clear that disk usage was going to be problem. I eventually moved all the assets to an S3 compatible Object Storage.

Now I'm moving away from Backblaze B2 as a S3 provider to Scaleway. I wanted to highlight why and how I did it, which is trivial when you know what to do

Why?

It's funny that all came down to a currency and payment issue (not pricing). Both providers are super cheap and high free tiers.

• Backblaze charges in USD ❌
• Scaleway charges in EUR ✅

• Backblaze speed for deleting objects: slow ❌
• Scaleway speed for deleting objects: faster ✅

• Both providers have data centers in the EU ✅✅

My credit card is linked to an Spanish bank and they charge me a fee for every purchase I do in currencies other than EUR. So the actual S3 Object Storage cost was being inflated by my bank's fees. That's no good.
In hindsight, I should've known this when I signed up for Backblaze

Want to know how I migrated? Here's how:

1. Create a snapshot of the bucket

From B2 you can create a zip (snapshot), then you just download it. It might take a few hours to generate depending on your bucket size, though. Mine was about 50Gb and took around 5 hours.

2. Point Mastodon to the new S3 bucket

Since I'm a single private instance I could this step now. While waiting for the zip to be generated with the existing assets I updated ~/live/.env.production with the new values from the Scaleway bucket (as well as the new API credentials generated within Scaleway, so Mastodon can upload objects) (More on this here).

⚠️ There's a difference on how to set the S3 configuration on your Mastodon instance for Scaleway, here's how it should look like (you may change nl-ams for your bucket region):

S3_ENDPOINT=https://s3.nl-ams.scw.cloud
S3_HOSTNAME=s3.nl-ams.scw.cloud
S3_BUCKET=your-bucket-nameCode language: JavaScript (javascript)

By doing this all new avatars and headers fetched by Mastodon from this point on started being uploaded to the new bucket.

3. AWS S3 cli to the rescue

Now that you have all your assets on your machine (zip file), it's time to upload them to the new provider.

Since all these Object Storage providers support the Amazon S3 API you can make use of AWS's CLI to do all the hard work for you.

Here's the command I used, which syncs a local folder in my computer to the S3 bucket while setting the policy to public. ⚠️ This last part is very important because in Scaleway even if the bucket is public an uploaded file, by default, is set to private. Not sure why, but it works like that.

aws s3 sync . s3://my-bucket --acl public-readCode language: JavaScript (javascript)

I used Scaleway's documentation to configure the aws client on my Mac. I only changed the default max_concurrent_requests to 50 because otherwise my Mac would complain "too many files open" 🤷‍♂️

4. Update the CDN's origin

Because I'm using the fantastic (and free, because of their insanely high free tier) AWS's CloudFront I just needed to go into the distribution settings and change the origin S3 bucket.

Done, now my CDN starts pulling from the new bucket.

Conclusion

I was a little challenged at the begining but after some digging turns out is not so bad. The longest part is of course the syncing of all the assets from your local machine to the S3, which can take hours.

Otherwise very happy with the process and the change