feat: Add configurable liveness probe #1972
Conversation
davidspek
requested review from
sdudoladov,
Jan-M,
CyberDem0n,
FxKu,
jopadi and
idanovinda
as code owners
davidspek
changed the title
|
LivenessProbes are dangerous. 🔥 🔥 🔥 |
|
Imho the point of the article is not that Liveness probes are dangerous per se. You just have to use them right. I even would go so far to say that a deployment which can't properly crash and restart automatically because of a missing liveness probe is just broken, because that's one of the essential things using a container in kubernetes. Therefore it should be configurable at least even if not used by default. Currently we have to use an ugly workaround to get our postgres instances running reliably: https://github.com/monotek/patroni-selfheal |
|
I would still favor one pre-configured liveness probe that works for the operator and have only one toggle for users to switch. |
|
We actually developed this as we didn’t want to use patroni-selfheal, and have been using it in prod on multiple cluster for 2 months now without issues and solving the problem we had. The main problem with a single liveness probe toggle is that it will likely not work in many scenarios, similarly to the readiness probe. One example is if a replica needs to clone a very large DB from the master, the probes would need to be configurable for this use case specifically. In terms of errors configuring the probes, they have the same type and validation as Kubernetes so that should take care of part of it. We can also add docs, examples, etc to inform people how to configure the probe and give a good starting point they can use. |
|
@FxKu friendly bump since I believe this is a pretty essential feature for the operator. |
Signed-off-by: DavidSpek <vanderspek.david@gmail.com>
Signed-off-by: DavidSpek <vanderspek.david@gmail.com>
Signed-off-by: DavidSpek <vanderspek.david@gmail.com>
|
@FxKu Friendly bump since I've just rebased on the latest release. |
|
Hi, having this PR and feature merged seems a very good improvement. Any luck that it will happen? |
|
still valid :> |
Closes: #58, #1703
This PR adds a new field to the Operator configuration parameters and the Postgres spec allowing users too specify a liveness probe for the Spilo container. This is necessary too resolve errors with Postgres going down if communication with the K8s API temporarily fails, as described in this issue.
The Postgres CRD managed by the operator doesn't seem to have the new field added, trying to fix that still. Also, it doesn't seem like the statefulset is updated when the liveness probe is changed which still needs fixing.The CRD managed by the operator now includes the new field and changes to the liveness and readiness probes now properly update the statefulset.