Patroni is repeatedly patching the ha cluster ip service #1206
Description
Report
Hello,
While investigating on an unrelated MetalLB issue, I've stumbled upon these logs :
{"caller":"service_controller.go:64","controller":"ServiceReconciler","level":"info","start reconcile":"everest-managed-databases/XXX-ha","ts":"2025-07-03T10:07:39Z"}
{"caller":"service_controller.go:115","controller":"ServiceReconciler","end reconcile":"everest-managed-databases/XXX-ha","level":"info","ts":"2025-07-03T10:07:39Z"}
{"caller":"service_controller.go:64","controller":"ServiceReconciler","level":"info","start reconcile":"everest-managed-databases/YYY-ha","ts":"2025-07-03T10:07:40Z"}
{"caller":"service_controller.go:115","controller":"ServiceReconciler","end reconcile":"everest-managed-databases/YYY-ha","level":"info","ts":"2025-07-03T10:07:40Z"}
{"caller":"service_controller.go:64","controller":"ServiceReconciler","level":"info","start reconcile":"everest-managed-databases/ZZZ-ha","ts":"2025-07-03T10:07:42Z"}
{"caller":"service_controller.go:115","controller":"ServiceReconciler","end reconcile":"everest-managed-databases/ZZZ-ha","level":"info","ts":"2025-07-03T10:07:42Z"}
Investigating further, I've found in K8S audit logs that Patroni was constantly patching the HA ClusterIP service every 10 seconds :
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"f7e08f90-25f9-4d5b-acd2-5f03f29161a7","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/everest-managed-databases/endpoints/ZZZ-ha","verb":"patch","user":{"username":"system:serviceaccount:everest-managed-databases:ZZZ-instance","uid":"a96b430e-e1cf-4dcb-9571-6576866e88d8","groups":["system:serviceaccounts","system:serviceaccounts:everest-managed-databases","system:authenticated"],"extra":{"authentication.kubernetes.io/credential-id":["JTI=0bc30e9f-1a7b-45e3-b5cb-d344f0ddb2fc"],"authentication.kubernetes.io/node-name":["XXXX"],"authentication.kubernetes.io/node-uid":["5126e92f-f833-4ffc-9659-0f37ee864315"],"authentication.kubernetes.io/pod-name":["ZZZ-instance1-dql9-0"],"authentication.kubernetes.io/pod-uid":["c0f54e93-a40c-4761-9e65-a0552f6a9f67"]}},"sourceIPs":["XXXX"],"userAgent":"Patroni/4.0.5 Python/3.9.21 Linux","objectRef":{"resource":"endpoints","namespace":"everest-managed-databases","name":"ZZZ-ha","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2025-07-03T09:46:22.360065Z","stageTimestamp":"2025-07-03T09:46:22.363984Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by RoleBinding \"ZZZ-instance/everest-managed-databases\" of Role \"ZZZ-instance\" to ServiceAccount \"ZZZ-instance/everest-managed-databases\""}}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"6db138f2-4533-4b3e-be27-b749e1775b53","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/everest-managed-databases/endpoints/ZZZ-ha","verb":"patch","user":{"username":"system:serviceaccount:everest-managed-databases:ZZZ-instance","uid":"a96b430e-e1cf-4dcb-9571-6576866e88d8","groups":["system:serviceaccounts","system:serviceaccounts:everest-managed-databases","system:authenticated"],"extra":{"authentication.kubernetes.io/credential-id":["JTI=0bc30e9f-1a7b-45e3-b5cb-d344f0ddb2fc"],"authentication.kubernetes.io/node-name":["XXXX"],"authentication.kubernetes.io/node-uid":["5126e92f-f833-4ffc-9659-0f37ee864315"],"authentication.kubernetes.io/pod-name":["ZZZ-instance1-dql9-0"],"authentication.kubernetes.io/pod-uid":["c0f54e93-a40c-4761-9e65-a0552f6a9f67"]}},"sourceIPs":["XXXX"],"userAgent":"Patroni/4.0.5 Python/3.9.21 Linux","objectRef":{"resource":"endpoints","namespace":"everest-managed-databases","name":"ZZZ-ha","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2025-07-03T09:46:32.363414Z","stageTimestamp":"2025-07-03T09:46:32.367206Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by RoleBinding \"ZZZ-instance/everest-managed-databases\" of Role \"ZZZ-instance\" to ServiceAccount \"ZZZ-instance/everest-managed-databases\""}}
Is this a normal behavior or is there a bug somewhere in my configuration ?
Antoine
More about the problem
The HA ClusterIP service is constantly patched by Patroni.
This is not a big deal but it generates some noise in audit logs and possibly other components monitoring services such as MetalLB or CNI.
Steps to reproduce
- Deploy a
postgresclusters.postgres-operator.crunchydata.comobject in your cluster - Enable Kubernetes audit logs
- Observe from audit logs that
XXX-hacluster ip is constantly patched by patroni.
Versions
- Kubernetes : 1.31.8
- Operator : PostgreSQL operator 2.6.0 / Everest 1.7.0
- Database : PostgreSQL 17.4
Anything else?
No response