I'd love to hear some stories about how you or your organization is using Kubernetes for development! My team is experimenting with using it because our "platform" is getting into the territory of too large to run or manage on a single developer machine. We've previously used Docker Compose to enable starting things up locally, but that started getting complicated.
The approach we're trying now is to have a Helm chart to deploy the entire platform to a k8s namespace unique to each developer and then using Telepresence to connect a developer's laptop to the cluster and allow them to run specific services they're working on locally.
This seems to be working well, but now I'm finding myself concerned with resource utilization in the cluster as devs don't remember to uninstall or scale down their workloads when they're not active any more, leading to inflation of the cluster size.
Would love to hear some stories from others!
Right, the tradeoff here is that to maintain that state you're paying for envs even when they're not in use. Extended periods of "accumulated state" are definitely a thing, and you want some escape valve to enable them occasionally. But the way to reduce hosting costs is definitely to make them the exception rather than the rule, which involves adapting workflows to rely more on storing and offline analyzing telemetry rather than interactively debugging everything.
A other approach here is using something like EBS so that stateful pods can be stopped and then reattached to persistent disk. But unless you do some kind of deep hibernation you lose memory state, and even if you do that you lose socket and other environmental state. IMO telemetry is a stronger long-term strategy as it can capture this state that hibernation destroys.