After starting to host my own private Mastodon instance it became clear that disk usage was going to be problem. I eventually moved all the assets to an S3 compatible Object Storage.
Now I'm moving away from Backblaze B2 as a S3 provider to Scaleway. I wanted to highlight why and how I did it, which is trivial when you know what to do
Why?
It's funny that all came down to a currency and payment issue (not pricing). Both providers are super cheap and high free tiers.
- Backblaze charges in USD ❌
- Scaleway charges in EUR ✅
- Backblaze speed for deleting objects: slow ❌
- Scaleway speed for deleting objects: faster ✅
- Both providers have data centers in the EU ✅✅
My credit card is linked to an Spanish bank and they charge me a fee for every purchase I do in currencies other than EUR. So the actual S3 Object Storage cost was being inflated by my bank's fees. That's no good.
In hindsight, I should've known this when I signed up for Backblaze
Want to know how I migrated? Here's how:
1. Create a snapshot of the bucket
From B2 you can create a zip (snapshot), then you just download it. It might take a few hours to generate depending on your bucket size, though. Mine was about 50Gb and took around 5 hours.
2. Point Mastodon to the new S3 bucket
Since I'm a single private instance I could this step now. While waiting for the zip to be generated with the existing assets I updated ~/live/.env.production with the new values from the Scaleway bucket (as well as the new API credentials generated within Scaleway, so Mastodon can upload objects) (More on this here).
⚠️ There's a difference on how to set the S3 configuration on your Mastodon instance for Scaleway, here's how it should look like (you may change nl-ams for your bucket region):
S3_ENDPOINT=https://s3.nl-ams.scw.cloud
S3_HOSTNAME=s3.nl-ams.scw.cloud
S3_BUCKET=your-bucket-nameCode language: JavaScript (javascript)By doing this all new avatars and headers fetched by Mastodon from this point on started being uploaded to the new bucket.
3. AWS S3 cli to the rescue
Now that you have all your assets on your machine (zip file), it's time to upload them to the new provider.
Since all these Object Storage providers support the Amazon S3 API you can make use of AWS's CLI to do all the hard work for you.
Here's the command I used, which syncs a local folder in my computer to the S3 bucket while setting the policy to public. ⚠️ This last part is very important because in Scaleway even if the bucket is public an uploaded file, by default, is set to private. Not sure why, but it works like that.
aws s3 sync . s3://my-bucket --acl public-readCode language: JavaScript (javascript)I used Scaleway's documentation to configure the aws client on my Mac. I only changed the default max_concurrent_requests to 50 because otherwise my Mac would complain "too many files open" 🤷♂️
4. Update the CDN's origin
Because I'm using the fantastic (and free, because of their insanely high free tier) AWS's CloudFront I just needed to go into the distribution settings and change the origin S3 bucket.
Done, now my CDN starts pulling from the new bucket.
Conclusion
I was a little challenged at the begining but after some digging turns out is not so bad. The longest part is of course the syncing of all the assets from your local machine to the S3, which can take hours.
Otherwise very happy with the process and the change
@ricard nice! Thanks for including the write up also 🙂
@cambraca haha Thanks Camilo! South Korea is a tasty land!
@ricard and you, good sir, earn a follow because of (and in equal parts) 1. your excellent blog post and 2. that delicious cover photo.I mean, I *just* had lunch and you made me hungry again!
@ricard Storagebox is mounted to /home/mastodon/live/public/system via fusemount. So I just cd into the system dir and run `aws s3 sync . s3://my-bucket –acl public-read`, right?
@rolle thank you for the feedback!I don’t fully know how that storage mount works but you should be able to use a CLI to sync it to an S3. I’ve used AWS CLI and had no problems moving from local to S3 and the other way around.
@ricard I’m using Hetzner storagebox. I actually read your other blog post just now, great stuff!I’m not exactly sure how to move to S3 if I’m currently using a storagebox mount… Switching would lead to a lot of broken files? I’m not sure if there’s a wise way to move/re-fetch media etc.? Any tips appreciated.
@docent thanks!
@ricard fyi: https://www.idrive.com/e2/pricing
IDrive® e2 plans and pricing for cloud storage
Blackblaze is painfully slow. I hosted images and https requests were supper slow. Almost took 1s to load image.
I never noticed as I had Amazon’s CDN in front of it.
What I did notice however is how slow it was to delete objects. Although now that I’m at Scaleway it doesn’t seem to be fast either.
On the same note there’s a PR to do batch objects deletion in the works: https://github.com/mastodon/mastodon/pull/23302
Mastodon’s built-in CLI gives you the availability to clean attachments and previews from remote accounts, purging the disk cache. This is fantastic and you couldn’t possible survive with out it.
My current
crontabthat runs every 3 hours:
.wp-block-code {
border: 0;
padding: 0;
-webkit-text-size-adjust: 100%;
text-size-adjust: 100%;
}
.wp-block-code > span {
display: block;
overflow: auto;
}
.shcb-language {
border: 0;
clip: rect(1px, 1px, 1px, 1px);
-webkit-clip-path: inset(50%);
clip-path: inset(50%);
height: 1px;
margin: -1px;
overflow: hidden;
padding: 0;
position: absolute;
width: 1px;
word-wrap: normal;
word-break: normal;
}
.hljs {
box-sizing: border-box;
}
.hljs.shcb-code-table {
display: table;
width: 100%;
}
.hljs.shcb-code-table > .shcb-loc {
color: inherit;
display: table-row;
width: 100%;
}
.hljs.shcb-code-table .shcb-loc > span {
display: table-cell;
}
.wp-block-code code.hljs:not(.shcb-wrap-lines) {
white-space: pre;
}
.wp-block-code code.hljs.shcb-wrap-lines {
white-space: pre-wrap;
}
.hljs.shcb-line-numbers {
border-spacing: 0;
counter-reset: line;
}
.hljs.shcb-line-numbers > .shcb-loc {
counter-increment: line;
}
.hljs.shcb-line-numbers .shcb-loc > span {
padding-left: 0.75em;
}
.hljs.shcb-line-numbers .shcb-loc::before {
border-right: 1px solid #ddd;
content: counter(line);
display: table-cell;
padding: 0 0.75em;
text-align: right;
-webkit-user-select: none;
-moz-user-select: none;
-ms-user-select: none;
user-select: none;
white-space: nowrap;
width: 1%;
}
0 */3 * * * /bin/bash /home/mastodon/purge-media.shCode language: JavaScript (javascript)Code language: Bash (bash)
As of Mastodon 4.1.0, we have new available commands. Here’s the content of my
purge-media.shCode language: CSS (css)script:
#!/bin/bash # Prune remote accounts that never interacted with a local user RAILS_ENV=production /home/mastodon/live/bin/tootctl accounts prune; # Remove remote statuses that local users never interacted with older than 4 days RAILS_ENV=production /home/mastodon/live/bin/tootctl statuses remove --days 4; # Remove media attachments older than 4 days RAILS_ENV=production /home/mastodon/live/bin/tootctl media remove --days 4; # Remove all headers (including people I follow) RAILS_ENV=production /home/mastodon/live/bin/tootctl media remove --remove-headers --include-follows --days 0; # Remove link previews older than 4 days RAILS_ENV=production /home/mastodon/live/bin/tootctl preview_cards remove --days 4; # Remove files not linked to any post RAILS_ENV=production /home/mastodon/live/bin/tootctl media remove-orphans;Code language: PHP (php)Code language: PHP (php)
⚠️ If you’ve never run these commands before I’d suggest run them one by one (not in a cronjob) as they might take several hours (or days) to run each. The size of the cached media and database will depend on how many people you follow, how many are on your instance, how many relays you have added to your instance, etc.
Having like 10 relays added to my single-user instance with the bash script above I’m around 30Gb in my Object Storage S3.
Do you have any other tips on how to keep a Mastodon instance lean?
👋 Don’t miss the follow up post: Scaling Mastodon: moving media assets to Object Storage