Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Exclude evicted pods from termination #82

Closed
nsidhaye opened this issue Jun 21, 2018 · 4 comments
Closed

Exclude evicted pods from termination #82

nsidhaye opened this issue Jun 21, 2018 · 4 comments

Comments

@nsidhaye
Copy link

nsidhaye commented Jun 21, 2018

I tested your simple but effective chaoskube & I think it really useful for us.

Do you have any plans to exclude evicted pods from terminations ?

e.g.
kubectl get pods -n {your_namespace} -w

cassandra-db-fc85698c7-hsw6w                      1/1       Running            0          7m
cassandra-db-fc85698c7-l57ms                      0/1       Evicted            0          4h
cassandra-db-fc85698c7-pg8ps                      0/1       Evicted            0          4h
cassandra-db-fc85698c7-swpk9                      0/1       Evicted            0          4h
cassandra-db-fc85698c7-tv4nv                      0/1       Evicted            0          4h
cassandra-db-fc85698c7-zjvwz                      0/1       Evicted            0          4h

In this case chaoskube try to evict pods which already evicted.

If this feature already implemented then can anybody guide me in configuration?

Observation: pod which are in error state or not fully started victim of chaoskube.

@linki
Copy link
Owner

linki commented Jun 21, 2018

@nsidhaye Thanks, I'm glad you find it useful.

I did not consider this and similar cases at all. Evicted, Terminating, ContainerCreating, CrashLoopBackoff, etc. should probably all be ignored.

Do you think it makes sense to only consider Running pods as targets? Is there a valid reason to include any other state?

@nsidhaye
Copy link
Author

nsidhaye commented Jun 27, 2018

@linki : Sorry for late reply. I feel there is no point to killing pod which is already evicted or terminating & even CrashLoopBackoff state.

If we target only Running pods then we can effectively use chaoskube in chaose engineering. Assume chaoskube run interval is set to 120 min. If chaoskube is trying to kill pods which are not in healthy state then we are loosing one opportunity to create chaos.

Even better if we can configure state of pods. In case configuration then we are not changing default behavior of chaoskube.

@Aergonus
Copy link

kubemonkey has an option to kill-all types vs just running pods. It can be trivial to add but I don't think there's a use case for terminating evicted pods. I would say you should be using some kind of pod-reaper to remove evicted pods.

@nsidhaye
Copy link
Author

@Spellchaser : I don't want to spend cycles on trying to delete evicted pods. Instead I want chaoskube should only consider healthy pods to terminate.

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants