Best practices to Deploy Microservices in k8s

Deploying and managing an application that has 100’s of microservices is an extremely difficult task. Kubernetes make the life of deploying microservice easy.

The below are some Anti-patterns to be followed to deploy microservices in the kubernetes cluster.

1. Run workload as a non-privileged user.
2. Use IAC tools.
3. Autoscale Workloads.
4. Define Pod Disruption Budget.
5. Define liveness and readiness probes.
6. Use volumeClaimTemplates for statefulsets.

Run workload as non-privileged user:

Even though Containers use namespace and cgroups to isolate resources, running the workloads as the privileged user provides an entry point to an attacker exploiting a security threat in your application. So it is one of the basic principles followed by many security professionals even before the adoption of kubernetes.

This can be done using securityContext in the resource definition.

securityContext:
  fsGroup: 1000
  runAsNonRoot: true
  runAsUser: 1000

Use IAC tools:

With hundreds of microservices to deploy using IAC tools will bring the following advantages.

• All the changes to the infrastructure are tracked.
• Deploying microservices to different environments is easy.
• Rollback and upgrades to the microservices are efficient.
• Developing a CD pipeline using IAC tools, reduces the time taken to release the application.

Autoscale Workloads:

It is very difficult to predict the number of replicas and size of your workloads at peak hours. Kubernetes allows using auto scalers based on metrics fed from the workloads such as resource utilization, number of hits. One can also use custom metrics hooked to HPA or VPA to autoscale the workloads in kubernetes. This allows us to meet the SLA of five 9’s.

Below is an example of HPA resource definition that allows deployment sample-apache when CPU utilization is greater than 80%.

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: sample-apache
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: sample-apache
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 80

Define Pod Disruption Budget:

Pod disruption budget allows one to define how many workloads that be down at any given point in time. It always keeps your workloads running especially during upgrades or rollbacks.

Sample resource definition of PDB with max unavailable workloads as 50%

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: sample-apache
spec:
  maxUnavailable: 50%
  selector:
    matchLabels:
      app: sample-apache

This ensures 50% of your defined workload capacity will always be running.

Define liveness and readiness probes:

liveness and readiness probes prevent user requests from ending up in faulty workloads. Requests are routed to workloads only if the liveness and readiness probes are successful.

Use volumeClaimTemplates for statefulsets:

Statefulsets are used to deploy stateful application to kubernetes, volumeClaimTemplates allows a unique persistent volume to mount on each replica in a Statefulset. Which allows all the pods in the stateful to maintain it own state.

Below is the snippet of defining volumeClaimTemplates in a statefulset.

.....
volumeClaimTemplates:
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: apache-data
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 5Gi
      storageClassName: gold
.....