Things we readily use

There are a few types of k8s resources we use without hesitation. I’m only listing resources here that we create explicitly; most of these resources implicitly create other resources (like Pods) that I will not mention but which we of course (indirectly) use.

Deployments : Most of our pods are created through deployments. Every deployment critical to our service functioning has multiple replicas and rolling updates.

Services : specifically, ClusterIP for internal services and LoadBalancer for external ones. We have avoided NodePort and ExternalName services, preferring for our DNS configuration to live outside of Kubernetes.

CronJobs: for cleanup scripts and that sort of thing.

ConfigMaps and Secrets : for passing data to the above resources.

Things we cautiously use

StatefulSet and PersistentVolumeClaim : we have used some StatefulSets. The configuration is a bit more convoluted than Deployments, but they can have a persistent volume across restarts. We prefer to persist important data in managed services outside of k8s. We don’t have a hard rule against volumes because sometimes it’s nice to persist e.g. a cache across a service restart, but I avoid them when possible because they can interact badly (deadlock) with rolling deploys.

RBAC : we have used this in a few places, e.g. to give a service permission to refresh a secret. It adds enough complexity to our small cluster that I mostly avoid it.

Things we actively avoid

Hand-writing YAML . YAML has enough foot-guns that I avoid it as much as possible. Instead, our Kubernetes resource definitions are created from TypeScript with Pulumi.

Non-built-in resources and operators . I’ve written before about how the control loop pattern is a double-edged sword: it’s the core idea that makes K8s robust, but it’s also a source of indirection and complexity. The operator pattern and custom resources allow third-party software to use Kubernetes’ robust infrastructure for its own control loops, which is a great idea in theory that I’ve found to be clunky in practice. Rather than cert-manager, we use Caddy’s certificate automation.

Helm . Helm is a no-go because of the operators and no YAML rules, but I also just think that using unstructured string templating to generate something machine-parsable means introducing fragility for no gain. nindent is like nails-on-a-chalkboard to me, I’m sorry.

Anything with “mesh” in the name. I guess they’re useful to somebody, but not me, and not this guy either.

Ingress resources . I don’t have any battle scars from these, and I know some people use them productively, but a theme of our successful use of Kubernetes is avoiding adding unnecessary layers of indirection. Configuring Caddy works for us, so we just do that.

Trying to replicate the entire k8s stack locally . Instead of using things like k3s or kind to replicate production exactly, we just use Docker Compose or our own scripts that start the subset of the system we actually care about in the moment.

from The hater’s guide to Kubernetes