TLDR
Build Price-Aware Applications
Check the Price History: In general, picking older generations of instances will result in lower net prices and fewer interruptions.
Use Multiple Capacity Pools: By having the ability to run across multiple pools, you reduce your application’s sensitivity to price spikes that affect a pool or two (in general, there is very little correlation between prices in different capacity pools). For example, if you run in five different pools your price swings and interruptions can be cut by 80%.
Symptom: autoscaler works (it can scale up) but for some reasons, it doesn’t scale down after the load goes away.
I spent sometimes debugging and turns out, it’s not really a bug per se. More of a bad luck pod placement on my Kubernetes cluster.
I first added --v=4 to get more verbose logging in cluster-autoscaler and watch kubectl get logs -f cluster-autoscaler-xxx. I notice this line from the logs
Fluentd Docker image to send Kuberntes logs to CloudWatch
Very easy to setup. Good option for centralized logging if all of your infrastructures are already in AWS.
echo -n "accesskeyhere" > aws_access_key echo -n "secretkeyhere" > aws_secret_key kubectl create secret --namespace=kube-system generic fluentd-secrets --from-file=aws_access_key --from-file=aws_secret_key kubectl apply -f fluentd-cloudwatch-daemonset.yaml On a side note, I think i will need to move fluend configuration file to secret as I just want to collect logs from certain namespace/filter.
How to build a custom Kubernetes scheduler by Mr. Kubernetes
A DaemonSet to be run on node instance and keep polling http://169.254.169.254/latest/meta-data/spot/termination-time for termination notice.
The daemonset will poll every 5 seconds which will give you approx 2 minutes to drain the spot node and migrate pods to another node.
A bit outdated post but many points stay valid.
kube-monkey is an implementation of Netflix’s Chaos Monkey for Kubernetes clusters. It randomly deletes Kubernetes pods in the cluster encouraging and validating the development of failure-resilient services.
Netflix’s Chaos Monkey for Kubernetes
Kube-Lego automatically requests certificates for Kubernetes Ingress resources from Let’s Encrypt
You can find complete example how to use this with GCE here
I’ve seen people bundle config file within source code. I’ve seen people bundle config when building Docker images. PLEASE DON’T.
Just use secrets / environment variables.
Here is a very detail tutorial on how to share secret data in Kubernetes.
Your DevOps Dashboard for Kubernetes
I’ve been looking for CI/DI that would complete the Kubernetes setup. This looks like a good fit for it.