One of our services at Monoceros is managing Mastodon instances that we host. Being a small business, we don’t have any on-premises servers but instead run everything on AWS. AWS is popular for many reasons, including speed and ease of deployment. But their pricing structure is full of traps that can easily inflate your bill, and services like Mastodon are heavy enough that you cannot get by on the basic options. So how do we keep our costs low enough to turn a profit selling managed hosting?

Most self-hosted instances are built on-site, using computer hardware that is already available. Even a mid-range raspberry Pi, with 4GB RAM and an external hard disk is more than capable of running a small instance. Larger instances with busier accounts need a bit more memory and storage, but even the most basic starter setup is expensive to run on AWS.

The problem

Consider what goes into minimal Mastodon: A PostgreSQL database, a Redis database, and the Mastodon application itself, which runs in Ruby On Rails. In the process of managing incoming and outgoing messages, it also needs graphical and video transcoding software. For smooth operation without too many delivery delays, it’s generally recommended to have 4GB of RAM. An on-demand T4g.Medium instance, with 2 vCPUs and 4GB of RAM, costs $0.0368 per hour in the eu-west-1 region. That comes to $26.50 per month, and is the cheapest compute instance in their catalogue that has 4GB RAM. It’s possible to pack your install into a smaller instance, but this requires a lot of tuning, and will be slow. You can expect frequent delivery delays as busy periods cause long backlogs of messages waiting to be processed and delivered.

Additionally, you’ll need plenty of storage space. Modern social media tends to be heavy on images and video clips, and all of these need to be stored within your instance. And it’s not just your own images that you’re attaching to your posts; Mastodon caches local copies of all attachments on every post shared by the accounts that you follow (and those of every other user on your instance). It adds up very quickly – even small instances can fill up hundreds of gigabytes of storage per month. This can be managed, but it’s an extra administrative task, and if you trim older content too aggressively then your users will become frustrated when they open older posts and find the attachments missing.

It’s a truism in IT that storage is cheap. But the default fixed storage option for an EC2 instance (gp3 volumes using the EBS service) is not – 100GB will cost over $60 per month, in eu-west-1. Your first 30GB-months are free, but that discount covers your entire AWS account – it is not per server.

The final hidden cost is data. On-premises equipment, on your local network, doesn’t cost anything for data – you wire up your network and your only concern is whether you’re saturating your switch or not. But in the cloud, you pay per bit. The price seems low – cents per gigabyte – but it adds up very quickly, in ways that are hard to calculate. Depending on your load, the cost might be negligible, or it might make the bulk of your entire bill.

Our approach

So given that we charge a base rate of $15 per instance, we’ve had to find as many optimizations as possible. The obvious first step is to take advantage of object storage, and move all those media files off of the server. This is supported out of the box by Mastodon, and is described in their official documentation. AWS S3 is extremely cheap, extremely reliable, and serves data at a good speed. If you’re not running your instance from a physical server with ample hard drives, then this is a no-brainer.

The next step is to try and share resources: Can we divide that cost between our instances? The official documentation does not cover this, so we’ve had to do some experimenting to figure it out ourselves. We’re not the only ones to do this, so a brief google will find various recipes, but we keep a separate home folder for each instance, containing it’s own ruby environment and it’s own copy of the Mastodon code. This is not only simpler to implement, it also makes it easier for us to customize instances (changing character limits, for example). The down side is that we need to provision more storage space, and we have to update each instance individually when a new mastodon patch level is released.

The standard Mastodon install includes a reverse proxy powered by nginx. This allows the various services running behind the scenes of a Mastodon installation to all function together under the same domain name. We feed all of our instances through the same nginx reverse proxy, using virtual domains to keep them separated.

Dividing the load

The next level of optimization is to separate out systems into their own environment. For us, the first step was removing PostgreSQL. Since database tuning is hard and we are not database specialists, we opted to use AWS’s managed database service: RDS. In testing we’ve found that the load on the database is very low. So far we have all of our instances sharing a database server, which is a a t3.micro, which costs $14.40 per month in eu-west-1. Currently this database is running almost idle, regardless of load, so we’re confident that it will support a great many more mastodon instances, making it quite cost effective.

Similarly, we found that offloading Redis gave us a big performance boost. In this case, however, we opted to manage our own installation. AWS has a service called ElastiCache which is managed hosting of several in-memory databases (including Redis), but it’s not cost effective at our scale. Besides, Redis is very easy to run and tune, so we simply spun up a dedicated EC2 instance to run our own Redis installation. We initially ran it on a t4.nano instance, but found that Redis kept running out of memory. We’ve since upgraded that server to a t4.micro, which has a full gigabyte of RAM, which eliminated the problem.

EFS

We’ve always anticipated that we will eventually need to support a very larger instance, which will require dividing that instance’s Sidekiq tasks between multiple hosts. To simplify implementation, we originally planned to store home directories on EFS (AWS’s managed NFS service), to ensure consistency between hosts. In fact, we implemented all our instances on an EFS shared /home folder as standard practice from the very beginning. Unfortunately, while the costs of storage space seemed manageable, the per-IO and data transfer costs escalated very quickly and ended up being too expensive to run. We opted to move /home to a dedicated EBS volume. We may return to EFS for individual instances that need to be split across hosts, but we’ll cross that bridge when we get to it.

Tuning

At this point, by sharing the load across a single EC2 instance, and splitting off different functions to a mix of self-hosted and managed services, we’ve gotten to a point where we will benefit from economies of scale. Each new client will reduce our per-client cost. The external services will all scale to much higher loads than we’re currently using, making the actual Mastodon host itself the last remaining cost to manage.

For this, we look to standard tuning advice. Unfortunately, there’s no such thing – every admin seems to have their own tips and tricks that they swear by, but few agree about what works. Here’s what we’ve settled on:

• Single Sidekiq process per instance. While it’s worth dividing Sidekiq queues across multiple processes, we’ve found that this doesn’t work on memory-limited hardware as each process has it’s own memory footprint. We’ve found that splitting queues up into separate processes provides a great performance boost (it’s a great trick to run temporarily when you’ve had an outage and need to clear a massive task backlog). But the cost is increased memory use, and our architecture is more vulnerable to memory exhaustion than anything else.
• Limited concurrency for Sidekiq and Puma. Sidekiq is the task scheduler, which runs background processes such as delivering a post to all your followers, or generating preview cards for links, and Puma is the web server. As above, this is all about reducing memory use. The default out-of-the-box settings are quite generous for a small instance, and we like to reduce them. For my personal instance (2 accounts – me and my bot), I have only 5 Sidekiq threads, and a single Puma process with 5 threads. For my paying customers, I provide 10 threads, which can be increased if needed. Our heaviest instance has a single user, but they have a very large follower count. They like to send out massive bursts of posts, most of which contain photographs. Their 20-thread instance typically handles about a half million tasks per day, without choking.
• Enable local caching in nginx. This is a cost optimization, rather than a performance tweak. Since all instances store their media files on object storage, we are at risk of a viral post generating a lot of traffic, which could get very expensive. Traffic is cheapest from EC2, so all media URLs point to our nginx reverse proxy, which caches those files locally.