Due contract restrictions with our customers, at Previnet we must run our applications in our datacenter. This session describes the setup we adopted to run an enterprise grade kubernetes cluster. We have 3 masters configured in HA mode deployed on VM. Masters runs only kubernetes core services and pods, while everything else is ran on bare metal worker nodes. Emergency dynamic cluster sizing is done through worker nodes deployed onto vm. Haproxy is used to run ingress controllers, along with MetalLB to expose loadblancers to the systems external to kubernetes. Kubernetes exposed services are proxied by external haproxy clusters. An external Ceph cluster is employed to provide storage to pods, through csi rdb persistent volumes for ReadWriteOnce filesystems and CephFs for ReadWriteMany. Kube-prometheus operator is used to monitor the entire infrastructure. Eventually we describe our chaos experiments and what went wrong and what we learned and we continuously learn