Continuous and discontinuous limits in software systems
Designing software systems for scale means understanding the limits of your system, and how the system reacts when it is approaching those. Complex software systems have hundreds of pieces all moving together, and reducing all software systems to just a handful of limits like CPU, memory is overly simplistic. At the same time, calculating the breaking point of hundreds of possible metrics and monitoring them all in a dashboard all the time is also exhausting. After some number of years of designing the system, you forget what the number even means. In my experience, it is better to develop an understanding of limits not as arbitrary numbers, but in terms of how they impact observable system behaviour. Noticing degradation in system behaviour is easier, and more well understood, and the problem of mapping the impaired behaviour to its underlying causal limit can be deferred to root cause analysis which cover only a slice of the system at a time.
When viewed from this lens of how limits influence system behaviour, my experience is that these limits are best categorized into two camps, which I am calling, for a lack of more formal term, continuous or discontinuous. Continuous limits are limits like CPU, database IOPS, bulkheads with queue (concurrency control). When system is close to approaching these, its behaviour degrades. It may start to slow down, and it will give you enough observable signals much before it actually fails. A 80% CPU utilisation already gives away signs. These limits are "nice", and more forgiving. You usually get sufficient time to react before a fire is set if the load is ramping up gradually. On the other hand are the discontinuous limits like memory, disk space, max QPS in some serverless dependency. System behaves almost normal even while it is about to be hit, but as soon as you actually hit it, the system fails, partly or wholly. These are ones which cause fires in production, and give no early signs of failure even for a gradually ramping load. In my experience, a bunch of discontinuous limits are artificially enforced software constraints rather than physical limits. Like your hyperscaler may impose some limit on max containers you can run in the VPC, and you realise it suddenly one day when your autoscaler fails to add more containers. Or, your serverless database/redis suddenly starts throttling requests because you have exceeded its configured QPS.
Discontinuous limits are difficult to work with, precisely because they don't give signs. You can put up an alarm on each of these numbers, but as I said it's easy to lose track. The next best option is to systemically solve for them in the design by taming them into becoming continuous. Adding queues is one of the best ways to do that. For example, postgres limits number of connection to database and this is a discontinuous limit usually. But using a connection pooler (even client-side) means extra connections can wait for their turn instead of getting rejected right away. This coverts system behaviour to degradation instead of failure which are more forgiving. Other possibility is to couple them with a continuous limit. Example: some language runtimes allow you to tune your garbage collector to run more aggressively as you approach memory limits. This couples memory pressure with cpu pressure. GC running more frequently will cause your system to degrade as it runs out of CPU first instead of crashing and failing over due to OOM.
Surprisingly, when poorly designed, continuous limits can also manifest as discontinuous. The concept of burst credits is such a mechanism. Burst credits can hide sustained increase in load for some time, and then suddenly when you run out of them the system gets clamped to a much lower capacity. A badly degraded system is identical to a failed one (continuity has its limits). Either the burst has to end before your system can enter a badly degraded state, or you don't rely on burst credits at all. Other examples usually arise when we over aggressively protect a continuous limit through discontinuous mechanisms like rejecting requests over some RPS (load shedding). Usually these protections like load-shedding are well-intentioned, but if not properly tuned they cause their own fires.
A system with large number of continuous limits is easier to reason about because you don't have to monitor a large number of metrics. Discontinuous metrics on the other hand demand your attention and separate monitoring so minimise them.