Serverless applications can bring accidental complexity
I used to think the only downside of hosting your application on serverless infrastructure was cost. For predictable workloads, serverless carries a significant markup over classical architectures. Companies are willing to pay it because of the benefits: it's highly elastic, it gives you sandboxing and workload isolation, and serverless solutions tend to be more managed in nature, which makes them very easy to get started with. I have since realised there is more here than meets the eye. Serverless infrastructure can bring accidental complexity over the longer run. Here are two examples from my own experience.
First, something as simple as caching gets unexpectedly twisted. In classic infrastructure, if you want to cache a resource that is expensive to fetch repeatedly, you reach for an in-memory cache unless there is a strong reason to share the cache between instances. In serverless, the number of instances underlying your application is usually — but not always — a function of concurrency. Since each underlying instance is also much smaller, you end up with one to two orders of magnitude more of them. This quietly breaks many in-memory caching architectures, which rest on the fundamental assumption that the number of instances is low. You now have to reach for a shared cache like Redis, which means one more component in the stack that you have to manage, and more importantly, secure: it is a data store that can be independently compromised. An in-memory cache can only be reached through RCE, whereas a shared cache is exposed over the network and can be attacked through more channels.
The shared cache store also reduces application reliability. I believe the common wisdom that a cache store going down does not sacrifice application availability is overstated. In practice, performance outages can get as serious as availability outages. If your cache store is down, you either stop serving requests or bypass the cache and overload your resource server. Either way, this can quickly cascade into another failure mode.
Second, serverless imposes a connection management burden on resource servers. I've seen serverless architectures overwhelm our database with sheer connection count. So you need a connection pooler in between, which introduces its own drama — session pinning, for instance. Compare this to classic architectures, where you would simply keep the connection pool in memory.
I learned very recently that this goes beyond databases. We use LaunchDarkly at $WORK, and LaunchDarkly bills you on the number of connections to their service. Their Python SDK holds a persistent streaming connection, and on Lambda that means one connection per warm sandbox rather than one per host. If you run it that way, you are going to get a shock when you see the bill. The same QPS workload running on ECS produces a significantly cheaper LaunchDarkly bill.
Sometimes serverless is not a choice but a requirement, primarily when workload isolation or sandboxing is a hard constraint. In that case your best bet is to spin up a Firecracker microVM or a V8 isolate per request, and accept the cost. But mostly we reach for serverless for the convenience — and I no longer feel that convenience justifies the extra complexity you incur over the long run, once you inevitably run into these problems.