setting up colored logging 2023-02-06 11:10:39.366 INFO logger initialized using INFO log level 2023-02-06 11:10:39.366 INFO Creating hikaru monkey patches 2023-02-06 11:10:39.366 INFO Creating yaml monkey patch 2023-02-06 11:10:39.366 INFO Creating kubernetes ContainerImage monkey patch 2023-02-06 11:10:39.371 INFO watching dir /etc/robusta/playbooks/ for custom playbooks changes 2023-02-06 11:10:39.372 INFO watching dir /etc/robusta/config/active_playbooks.yaml for custom playbooks changes 2023-02-06 11:10:39.372 INFO Reloading playbook packages due to change on initialization 2023-02-06 11:10:39.373 INFO loading config /etc/robusta/config/active_playbooks.yaml 2023-02-06 11:10:39.432 INFO No custom playbooks defined at /etc/robusta/playbooks/storage 2023-02-06 11:10:39.432 INFO Importing actions package robusta.core.playbooks.internal 2023-02-06 11:10:39.433 INFO importing actions from robusta.core.playbooks.internal.discovery_events 2023-02-06 11:10:39.436 INFO Importing actions package robusta_playbooks 2023-02-06 11:10:39.437 INFO importing actions from robusta_playbooks.alerts_integration 2023-02-06 11:10:39.473 INFO importing actions from robusta_playbooks.argo_cd 2023-02-06 11:10:39.489 INFO importing actions from robusta_playbooks.autoscaler 2023-02-06 11:10:39.499 INFO importing actions from robusta_playbooks.babysitter 2023-02-06 11:10:39.508 INFO importing actions from robusta_playbooks.bash_enrichments 2023-02-06 11:10:39.510 INFO importing actions from robusta_playbooks.chaos_engineering 2023-02-06 11:10:39.512 INFO importing actions from robusta_playbooks.common_actions 2023-02-06 11:10:39.535 INFO importing actions from robusta_playbooks.configuration_ab_testing 2023-02-06 11:10:39.550 INFO importing actions from robusta_playbooks.cpu_throttling 2023-02-06 11:10:39.552 INFO importing actions from robusta_playbooks.daemonsets 2023-02-06 11:10:39.553 INFO importing actions from robusta_playbooks.deployment_enrichments 2023-02-06 11:10:39.554 INFO importing actions from robusta_playbooks.deployment_status_report 2023-02-06 11:10:39.563 INFO importing actions from robusta_playbooks.disk_benchmark 2023-02-06 11:10:39.571 INFO importing actions from robusta_playbooks.event_enrichments 2023-02-06 11:10:39.578 INFO importing actions from robusta_playbooks.git_change_audit 2023-02-06 11:10:39.586 INFO importing actions from robusta_playbooks.grafana_enrichment 2023-02-06 11:10:39.588 INFO importing actions from robusta_playbooks.http_actions 2023-02-06 11:10:39.605 INFO importing actions from robusta_playbooks.image_pull_backoff_enricher 2023-02-06 11:10:39.608 INFO importing actions from robusta_playbooks.java_pod_troubleshooting 2023-02-06 11:10:39.621 INFO importing actions from robusta_playbooks.job_actions 2023-02-06 11:10:39.653 INFO importing actions from robusta_playbooks.k8s_resource_enrichments 2023-02-06 11:10:39.655 INFO importing actions from robusta_playbooks.networking 2023-02-06 11:10:39.668 INFO importing actions from robusta_playbooks.node_cpu_analysis 2023-02-06 11:10:39.674 INFO importing actions from robusta_playbooks.node_disk_analysis 2023-02-06 11:10:39.690 INFO importing actions from robusta_playbooks.node_enrichments 2023-02-06 11:10:39.692 INFO importing actions from robusta_playbooks.oom_killer 2023-02-06 11:10:39.726 INFO importing actions from robusta_playbooks.overcommit_enrichments 2023-02-06 11:10:39.728 INFO importing actions from robusta_playbooks.persistent_data 2023-02-06 11:10:39.732 INFO importing actions from robusta_playbooks.persistent_volume_actions 2023-02-06 11:10:39.732 INFO importing actions from robusta_playbooks.pod_actions 2023-02-06 11:10:39.734 INFO importing actions from robusta_playbooks.pod_enrichments 2023-02-06 11:10:39.737 INFO importing actions from robusta_playbooks.pod_troubleshooting 2023-02-06 11:10:39.809 INFO importing actions from robusta_playbooks.prometheus_enrichments 2023-02-06 11:10:39.816 INFO importing actions from robusta_playbooks.prometheus_simulation 2023-02-06 11:10:39.837 INFO importing actions from robusta_playbooks.pvc_snapshots 2023-02-06 11:10:39.853 INFO importing actions from robusta_playbooks.restart_loop_reporter 2023-02-06 11:10:39.874 INFO importing actions from robusta_playbooks.silence 2023-02-06 11:10:39.918 INFO importing actions from robusta_playbooks.simple_examples 2023-02-06 11:10:39.925 INFO importing actions from robusta_playbooks.stress_tests 2023-02-06 11:10:39.944 INFO importing actions from robusta_playbooks.targetdown_enrichment 2023-02-06 11:10:39.946 INFO Adding sink named main_slack_sink 2023-02-06 11:10:40.006 INFO Adding sink named robusta_ui_sink 2023-02-06 11:10:40.020 INFO Supabase dal login 2023-02-06 11:10:40.202 INFO cluster status {'account_id': '1f8c96e6-9275-4e64-b81b-bcfbf444c5e3', 'cluster_id': 'atlantis', 'version': '0.10.10', 'light_actions': [], 'updated_at': 'now()'} 2023-02-06 11:10:40.412 INFO Initializing TopServiceResolver 2023-02-06 11:10:40.697 INFO Cluster discovery initialized 2023-02-06 11:10:40.825 INFO Loading RSA keys from /etc/robusta/auth 2023-02-06 11:10:40.830 INFO Loaded private key file /etc/robusta/auth/prv 2023-02-06 11:10:40.831 INFO Loaded public key file /etc/robusta/auth/pub 2023-02-06 11:10:40.831 INFO starting relay receiver 2023-02-06 11:10:40.831 INFO Telemetry set to include error info, Thank you for helping us improve Robusta. 2023-02-06 11:10:40.856 INFO Initialized task queue: 20 workers. Max size 500 2023-02-06 11:10:40.865 INFO Initialized task queue: 20 workers. Max size 500 * Serving Flask app 'robusta.runner.web' * Debug mode: off 2023-02-06 11:10:40.904 INFO Cluster already has historical data, No history pulled. 2023-02-06 11:10:40.950 INFO cluster status {'account_id': '1f8c96e6-9275-4e64-b81b-bcfbf444c5e3', 'cluster_id': 'atlantis', 'version': '0.10.10', 'light_actions': ['related_pods', 'prometheus_enricher', 'add_silence', 'delete_pod', 'delete_silence', 'get_silences', 'logs_enricher', 'pod_events_enricher', 'deployment_events_enricher', 'job_events_enricher', 'job_pod_enricher', 'get_resource_yaml', 'node_cpu_enricher', 'node_disk_analyzer', 'node_running_pods_enricher', 'node_allocatable_resources_enricher', 'node_status_enricher', 'node_graph_enricher', 'oomkilled_container_graph_enricher', 'pod_oom_killer_enricher', 'oom_killer_enricher', 'volume_analysis', 'python_profiler', 'pod_ps', 'python_memory', 'debugger_stack_trace', 'python_process_inspector', 'prometheus_alert', 'create_pvc_snapshot'], 'updated_at': 'now()'} 2023-02-06 11:10:40.951 INFO connecting to server as account_id=1f8c96e6-9275-4e64-b81b-bcfbf444c5e3; cluster_name=atlantis 2023-02-06 11:10:42.325 INFO Initializing services cache 2023-02-06 11:10:42.724 INFO Initializing nodes cache 2023-02-06 11:10:43.102 INFO Initializing jobs cache 2023-02-06 11:12:13.321 INFO cluster status {'account_id': '1f8c96e6-9275-4e64-b81b-bcfbf444c5e3', 'cluster_id': 'atlantis', 'version': '0.10.10', 'last_alert_at': '2023-02-06 11:11:02.769894', 'light_actions': ['related_pods', 'prometheus_enricher', 'add_silence', 'delete_pod', 'delete_silence', 'get_silences', 'logs_enricher', 'pod_events_enricher', 'deployment_events_enricher', 'job_events_enricher', 'job_pod_enricher', 'get_resource_yaml', 'node_cpu_enricher', 'node_disk_analyzer', 'node_running_pods_enricher', 'node_allocatable_resources_enricher', 'node_status_enricher', 'node_graph_enricher', 'oomkilled_container_graph_enricher', 'pod_oom_killer_enricher', 'oom_killer_enricher', 'volume_analysis', 'python_profiler', 'pod_ps', 'python_memory', 'debugger_stack_trace', 'python_process_inspector', 'prometheus_alert', 'create_pvc_snapshot'], 'updated_at': 'now()'} 2023-02-06 11:23:54.477 INFO Reloading playbook packages due to change on /etc/robusta/config/active_playbooks.yaml 2023-02-06 11:23:54.477 INFO loading config /etc/robusta/config/active_playbooks.yaml 2023-02-06 11:23:54.553 INFO No custom playbooks defined at /etc/robusta/playbooks/storage 2023-02-06 11:23:54.554 INFO Cloning git repo https://github.com/robusta-dev/kubernetes-chatgpt-bot.git. repo name kubernetes-chatgpt-bot Processing ./robusta-git/kubernetes-chatgpt-bot Preparing metadata (pyproject.toml): started Preparing metadata (pyproject.toml): finished with status 'done' Collecting openai==0.26.0 Downloading openai-0.26.0.tar.gz (54 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 54.7/54.7 kB 11.3 MB/s eta 0:00:00 Preparing metadata (pyproject.toml): started Preparing metadata (pyproject.toml): finished with status 'done' Requirement already satisfied: kubernetes<13.0.0,>=12.0.1 in /usr/local/lib/python3.9/site-packages (from chatgpt-robusta-actions==0.0.1) (12.0.1) Collecting tqdm Downloading tqdm-4.64.1-py2.py3-none-any.whl (78 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.5/78.5 kB 12.6 MB/s eta 0:00:00 Collecting aiohttp Downloading aiohttp-3.8.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.0 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.0/1.0 MB 29.0 MB/s eta 0:00:00 Requirement already satisfied: requests>=2.20 in /usr/local/lib/python3.9/site-packages (from openai==0.26.0->chatgpt-robusta-actions==0.0.1) (2.25.1) Requirement already satisfied: setuptools>=21.0.0 in /usr/local/lib/python3.9/site-packages (from kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (58.1.0) Requirement already satisfied: python-dateutil>=2.5.3 in /usr/local/lib/python3.9/site-packages (from kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (2.8.2) Requirement already satisfied: urllib3>=1.24.2 in /usr/local/lib/python3.9/site-packages (from kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (1.26.12) Requirement already satisfied: certifi>=14.05.14 in /usr/local/lib/python3.9/site-packages (from kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (2022.6.15) Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.9/site-packages (from kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (1.16.0) Requirement already satisfied: google-auth>=1.0.1 in /usr/local/lib/python3.9/site-packages (from kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (2.11.0) Requirement already satisfied: websocket-client!=0.40.0,!=0.41.*,!=0.42.*,>=0.32.0 in /usr/local/lib/python3.9/site-packages (from kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (1.3.3) Requirement already satisfied: pyyaml>=3.12 in /usr/local/lib/python3.9/site-packages (from kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (6.0) Requirement already satisfied: requests-oauthlib in /usr/local/lib/python3.9/site-packages (from kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (1.3.1) Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.9/site-packages (from google-auth>=1.0.1->kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (4.9) Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.9/site-packages (from google-auth>=1.0.1->kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (5.2.0) Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.9/site-packages (from google-auth>=1.0.1->kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (0.2.8) Requirement already satisfied: chardet<5,>=3.0.2 in /usr/local/lib/python3.9/site-packages (from requests>=2.20->openai==0.26.0->chatgpt-robusta-actions==0.0.1) (4.0.0) Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.9/site-packages (from requests>=2.20->openai==0.26.0->chatgpt-robusta-actions==0.0.1) (2.10) Collecting multidict<7.0,>=4.5 Downloading multidict-6.0.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (114 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 114.2/114.2 kB 7.8 MB/s eta 0:00:00 Collecting aiosignal>=1.1.2 Downloading aiosignal-1.3.1-py3-none-any.whl (7.6 kB) Collecting charset-normalizer<3.0,>=2.0 Using cached charset_normalizer-2.1.1-py3-none-any.whl (39 kB) Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.9/site-packages (from aiohttp->openai==0.26.0->chatgpt-robusta-actions==0.0.1) (22.1.0) Collecting frozenlist>=1.1.1 Downloading frozenlist-1.3.3-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (158 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 158.8/158.8 kB 28.7 MB/s eta 0:00:00 Collecting async-timeout<5.0,>=4.0.0a3 Downloading async_timeout-4.0.2-py3-none-any.whl (5.8 kB) Collecting yarl<2.0,>=1.0 Downloading yarl-1.8.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (264 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 264.6/264.6 kB 27.0 MB/s eta 0:00:00 Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.9/site-packages (from requests-oauthlib->kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (3.2.0) Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.9/site-packages (from pyasn1-modules>=0.2.1->google-auth>=1.0.1->kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (0.4.8) Building wheels for collected packages: chatgpt-robusta-actions, openai Building wheel for chatgpt-robusta-actions (pyproject.toml): started Building wheel for chatgpt-robusta-actions (pyproject.toml): finished with status 'done' Created wheel for chatgpt-robusta-actions: filename=chatgpt_robusta_actions-0.0.1-py2.py3-none-any.whl size=2079 sha256=d668ebce24a80b033960f258b87f39ac4384d1987a3f53da84deb88d371edf61 Stored in directory: /root/.cache/pip/wheels/dc/11/de/1137460e08cac026926087b9febd6dcaa9e62964e468dc5fee Building wheel for openai (pyproject.toml): started Building wheel for openai (pyproject.toml): finished with status 'done' Created wheel for openai: filename=openai-0.26.0-py3-none-any.whl size=66855 sha256=39a5c1aed4c9d2833c979cec49f3121e45383fe175c30e944a47c630ae3e9907 Stored in directory: /root/.cache/pip/wheels/ff/93/6c/6661ee49921d0c80d49b2438d5b78af119be6c2acf31213d25 Successfully built chatgpt-robusta-actions openai Installing collected packages: tqdm, multidict, frozenlist, charset-normalizer, async-timeout, yarl, aiosignal, aiohttp, openai, chatgpt-robusta-actions Successfully installed aiohttp-3.8.3 aiosignal-1.3.1 async-timeout-4.0.2 charset-normalizer-2.1.1 chatgpt-robusta-actions-0.0.1 frozenlist-1.3.3 multidict-6.0.4 openai-0.26.0 tqdm-4.64.1 yarl-1.8.2 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv [notice] A new release of pip available: 22.3.1 -> 23.0 [notice] To update, run: pip install --upgrade pip 2023-02-06 11:24:01.406 INFO Importing actions package robusta.core.playbooks.internal 2023-02-06 11:24:01.407 INFO importing actions from robusta.core.playbooks.internal.discovery_events 2023-02-06 11:24:01.410 INFO Importing actions package robusta_playbooks 2023-02-06 11:24:01.411 INFO importing actions from robusta_playbooks.alerts_integration 2023-02-06 11:24:01.413 INFO importing actions from robusta_playbooks.argo_cd 2023-02-06 11:24:01.413 INFO importing actions from robusta_playbooks.autoscaler 2023-02-06 11:24:01.418 INFO importing actions from robusta_playbooks.babysitter 2023-02-06 11:24:01.419 INFO importing actions from robusta_playbooks.bash_enrichments 2023-02-06 11:24:01.420 INFO importing actions from robusta_playbooks.chaos_engineering 2023-02-06 11:24:01.422 INFO importing actions from robusta_playbooks.common_actions 2023-02-06 11:24:01.423 INFO importing actions from robusta_playbooks.configuration_ab_testing 2023-02-06 11:24:01.424 INFO importing actions from robusta_playbooks.cpu_throttling 2023-02-06 11:24:01.425 INFO importing actions from robusta_playbooks.daemonsets 2023-02-06 11:24:01.426 INFO importing actions from robusta_playbooks.deployment_enrichments 2023-02-06 11:24:01.427 INFO importing actions from robusta_playbooks.deployment_status_report 2023-02-06 11:24:01.428 INFO importing actions from robusta_playbooks.disk_benchmark 2023-02-06 11:24:01.429 INFO importing actions from robusta_playbooks.event_enrichments 2023-02-06 11:24:01.430 INFO importing actions from robusta_playbooks.git_change_audit 2023-02-06 11:24:01.431 INFO importing actions from robusta_playbooks.grafana_enrichment 2023-02-06 11:24:01.432 INFO importing actions from robusta_playbooks.http_actions 2023-02-06 11:24:01.433 INFO importing actions from robusta_playbooks.image_pull_backoff_enricher 2023-02-06 11:24:01.434 INFO importing actions from robusta_playbooks.java_pod_troubleshooting 2023-02-06 11:24:01.435 INFO importing actions from robusta_playbooks.job_actions 2023-02-06 11:24:01.436 INFO importing actions from robusta_playbooks.k8s_resource_enrichments 2023-02-06 11:24:01.437 INFO importing actions from robusta_playbooks.networking 2023-02-06 11:24:01.438 INFO importing actions from robusta_playbooks.node_cpu_analysis 2023-02-06 11:24:01.439 INFO importing actions from robusta_playbooks.node_disk_analysis 2023-02-06 11:24:01.440 INFO importing actions from robusta_playbooks.node_enrichments 2023-02-06 11:24:01.441 INFO importing actions from robusta_playbooks.oom_killer 2023-02-06 11:24:01.442 INFO importing actions from robusta_playbooks.overcommit_enrichments 2023-02-06 11:24:01.443 INFO importing actions from robusta_playbooks.persistent_data 2023-02-06 11:24:01.444 INFO importing actions from robusta_playbooks.persistent_volume_actions 2023-02-06 11:24:01.444 INFO importing actions from robusta_playbooks.pod_actions 2023-02-06 11:24:01.445 INFO importing actions from robusta_playbooks.pod_enrichments 2023-02-06 11:24:01.446 INFO importing actions from robusta_playbooks.pod_troubleshooting 2023-02-06 11:24:01.448 INFO importing actions from robusta_playbooks.prometheus_enrichments 2023-02-06 11:24:01.449 INFO importing actions from robusta_playbooks.prometheus_simulation 2023-02-06 11:24:01.450 INFO importing actions from robusta_playbooks.pvc_snapshots 2023-02-06 11:24:01.450 INFO importing actions from robusta_playbooks.restart_loop_reporter 2023-02-06 11:24:01.451 INFO importing actions from robusta_playbooks.silence 2023-02-06 11:24:01.452 INFO importing actions from robusta_playbooks.simple_examples 2023-02-06 11:24:01.453 INFO importing actions from robusta_playbooks.stress_tests 2023-02-06 11:24:01.454 INFO importing actions from robusta_playbooks.targetdown_enrichment 2023-02-06 11:24:01.455 INFO Importing actions package chatgpt_robusta_actions 2023-02-06 11:24:01.456 INFO importing actions from chatgpt_robusta_actions.chat_gpt 2023-02-06 11:24:01.566 WARNING Docstring contains description of token which is not a field on the model 2023-02-06 11:24:01.571 WARNING Docstring contains description of token which is not a field on the model 2023-02-06 11:26:02.905 INFO Successfully loaded Kubernetes resource atlantis-0 for alert KubePodNotReady 2023-02-06 11:26:03.053 INFO Successfully loaded Kubernetes resource atlantis-0 for alert KubePodNotReady 2023-02-06 11:26:03.174 WARNING Docstring contains description of token which is not a field on the model 2023-02-06 11:26:03.180 WARNING Docstring contains description of token which is not a field on the model 2023-02-06 11:26:03.613 INFO Successfully loaded Kubernetes resource atlantis-0 for alert KubePodNotReady 2023-02-06 11:26:04.270 WARNING Docstring contains description of token which is not a field on the model 2023-02-06 11:26:04.278 WARNING Docstring contains description of token which is not a field on the model 2023-02-06 11:26:04.552 INFO Successfully loaded Kubernetes resource atlantis-0 for alert KubePodNotReady 2023-02-06 11:26:17.274 INFO Successfully loaded Kubernetes resource atlantis for alert KubeStatefulSetReplicasMismatch 2023-02-06 11:26:17.400 INFO Successfully loaded Kubernetes resource atlantis for alert KubeStatefulSetReplicasMismatch 2023-02-06 11:26:17.749 WARNING Docstring contains description of token which is not a field on the model 2023-02-06 11:26:17.766 WARNING Docstring contains description of token which is not a field on the model 2023-02-06 11:26:18.109 INFO Successfully loaded Kubernetes resource atlantis for alert KubeStatefulSetReplicasMismatch 2023-02-06 11:27:06.659 INFO Error loading kubernetes pod default/example-pod. reason: Not Found status: 404 2023-02-06 11:27:06.672 INFO Error loading kubernetes pod default/example-pod. reason: Not Found status: 404 2023-02-06 11:27:06.819 WARNING Docstring contains description of token which is not a field on the model 2023-02-06 11:27:06.828 WARNING Docstring contains description of token which is not a field on the model 2023-02-06 11:27:06.845 INFO Error loading kubernetes pod default/example-pod. reason: Not Found status: 404 2023-02-06 11:27:06.848 ERROR cannot run pod_events_enricher on alert with no pod object: PrometheusKubernetesAlert(sink_findings=defaultdict(, {'main_slack_sink': [], 'robusta_ui_sink': []}), named_sinks=['main_slack_sink', 'robusta_ui_sink'], response={'success': True}, stop_processing=False, _scheduler=, _context=ExecutionContext(account_id='1f8c96e6-9275-4e64-b81b-bcfbf444c5e3', cluster_name='atlantis'), obj=None, alert=PrometheusAlert(endsAt=datetime.datetime(2023, 2, 6, 11, 27, 6, 636077), generatorURL='', startsAt=datetime.datetime(2023, 2, 6, 11, 27, 6, 636082), fingerprint='', status='firing', labels={'severity': 'error', 'namespace': 'default', 'alertname': 'KubePodCrashLooping', 'pod': 'example-pod'}, annotations={}), alert_name='KubePodCrashLooping', alert_severity='error', label_namespace='default', node=None, pod=None, deployment=None, job=None, daemonset=None, statefulset=None) 2023-02-06 11:27:07.210 WARNING Docstring contains description of token which is not a field on the model 2023-02-06 11:27:07.216 WARNING Docstring contains description of token which is not a field on the model 2023-02-06 11:27:07.237 INFO Error loading kubernetes pod default/example-pod. reason: Not Found status: 404 2023-02-06 11:27:22.428 INFO cluster status {'account_id': '1f8c96e6-9275-4e64-b81b-bcfbf444c5e3', 'cluster_id': 'atlantis', 'version': '0.10.10', 'last_alert_at': '2023-02-06 11:27:06.640497', 'light_actions': ['related_pods', 'prometheus_enricher', 'add_silence', 'delete_pod', 'delete_silence', 'get_silences', 'logs_enricher', 'pod_events_enricher', 'deployment_events_enricher', 'job_events_enricher', 'job_pod_enricher', 'get_resource_yaml', 'node_cpu_enricher', 'node_disk_analyzer', 'node_running_pods_enricher', 'node_allocatable_resources_enricher', 'node_status_enricher', 'node_graph_enricher', 'oomkilled_container_graph_enricher', 'pod_oom_killer_enricher', 'oom_killer_enricher', 'volume_analysis', 'python_profiler', 'pod_ps', 'python_memory', 'debugger_stack_trace', 'python_process_inspector', 'prometheus_alert', 'create_pvc_snapshot'], 'updated_at': 'now()'} 2023-02-06 11:27:56.073 INFO Callback `show_chat_gpt_search` chat_gpt_token: sk-***KtI, search_term: How***es?, model: tex***003 2023-02-06 11:27:56.094 INFO message='Request to OpenAI API' method=post path=https://api.openai.com/v1/completions 2023-02-06 11:27:56.321 INFO message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=None request_id=cda81f1097cbadfd65e134fe6361f06a response_code=429 2023-02-06 11:27:56.321 INFO error_code=None error_message='You exceeded your current quota, please check your plan and billing details.' error_param=None error_type=insufficient_quota message='OpenAI API error received' stream_error=False 2023-02-06 11:27:56.321 ERROR Failed to execute action show_chat_gpt_search cluster_name: atl***tis, account_id: 1f8***5e3, chat_gpt_token: sk-***KtI, grafana_api_key: , grafana_dashboard_uid: , grafana_url: , prometheus_url: , signing_key: 17f***802, search_term: How***es?, model: tex***003, named_sinks: ['m***k'] 2023-02-06 11:28:15.809 INFO Callback `show_chat_gpt_search` chat_gpt_token: sk-***KtI, search_term: How***es?, model: tex***003 2023-02-06 11:28:15.809 INFO message='Request to OpenAI API' method=post path=https://api.openai.com/v1/completions 2023-02-06 11:28:15.873 INFO message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=None request_id=b80cf73405cb40a7e8a4dab55d6f25a4 response_code=429 2023-02-06 11:28:15.873 INFO error_code=None error_message='You exceeded your current quota, please check your plan and billing details.' error_param=None error_type=insufficient_quota message='OpenAI API error received' stream_error=False 2023-02-06 11:28:15.874 ERROR Failed to execute action show_chat_gpt_search cluster_name: atl***tis, account_id: 1f8***5e3, chat_gpt_token: sk-***KtI, grafana_api_key: , grafana_dashboard_uid: , grafana_url: , prometheus_url: , signing_key: 17f***802, search_term: How***es?, model: tex***003, named_sinks: ['m***k'] 2023-02-06 11:29:03.300 INFO Callback `show_chat_gpt_search` chat_gpt_token: sk-***KtI, search_term: How***es?, model: tex***003 2023-02-06 11:29:03.301 INFO message='Request to OpenAI API' method=post path=https://api.openai.com/v1/completions 2023-02-06 11:29:03.356 INFO message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=None request_id=c8da61675ce028a358a68d7a5a669d83 response_code=429 2023-02-06 11:29:03.357 INFO error_code=None error_message='You exceeded your current quota, please check your plan and billing details.' error_param=None error_type=insufficient_quota message='OpenAI API error received' stream_error=False 2023-02-06 11:29:03.357 ERROR Failed to execute action show_chat_gpt_search cluster_name: atl***tis, account_id: 1f8***5e3, chat_gpt_token: sk-***KtI, grafana_api_key: , grafana_dashboard_uid: , grafana_url: , prometheus_url: , signing_key: 17f***802, search_term: How***es?, model: tex***003, named_sinks: ['m***k'] 2023-02-06 11:29:34.124 INFO Callback `related_pods` name: atl***tis, namespace: def***ult, kind: Sta***Set 2023-02-06 11:29:34.965 INFO Callback `prometheus_enricher` promql_query: sum***od), duration: {'d***40} 2023-02-06 11:29:34.980 INFO discovered service with label-selector: `app=kube-prometheus-stack-prometheus` at url: `http://robusta-kube-prometheus-st-prometheus.default.svc.cluster.local:9090` 2023-02-06 11:29:35.014 INFO Callback `prometheus_enricher` promql_query: sum***od), duration: {'d***40} 2023-02-06 11:29:35.045 INFO Callback `prometheus_enricher` promql_query: sum***od), duration: {'d***40} 2023-02-06 11:29:35.075 INFO Callback `prometheus_enricher` promql_query: sum***od), duration: {'d***40} 2023-02-06 11:32:05.348 INFO Callback `show_chat_gpt_search` chat_gpt_token: sk-***KtI, search_term: How***es?, model: tex***003 2023-02-06 11:32:05.350 INFO message='Request to OpenAI API' method=post path=https://api.openai.com/v1/completions 2023-02-06 11:32:05.495 INFO message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=None request_id=284b7db9ddc18ad50f50013c5c1f5558 response_code=429 2023-02-06 11:32:05.495 INFO error_code=None error_message='You exceeded your current quota, please check your plan and billing details.' error_param=None error_type=insufficient_quota message='OpenAI API error received' stream_error=False 2023-02-06 11:32:05.495 ERROR Failed to execute action show_chat_gpt_search cluster_name: atl***tis, account_id: 1f8***5e3, chat_gpt_token: sk-***KtI, grafana_api_key: , grafana_dashboard_uid: , grafana_url: , prometheus_url: , signing_key: 17f***802, search_term: How***es?, model: tex***003, named_sinks: ['m***k'] 2023-02-06 11:35:42.921 INFO Callback `show_chat_gpt_search` chat_gpt_token: sk-***KtI, search_term: How***es?, model: tex***003 2023-02-06 11:35:42.924 INFO message='Request to OpenAI API' method=post path=https://api.openai.com/v1/completions 2023-02-06 11:35:43.124 INFO message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=None request_id=d607a4eb140982697a4e66184e8d2339 response_code=429 2023-02-06 11:35:43.124 INFO error_code=None error_message='You exceeded your current quota, please check your plan and billing details.' error_param=None error_type=insufficient_quota message='OpenAI API error received' stream_error=False 2023-02-06 11:35:43.124 ERROR Failed to execute action show_chat_gpt_search cluster_name: atl***tis, account_id: 1f8***5e3, chat_gpt_token: sk-***KtI, grafana_api_key: , grafana_dashboard_uid: , grafana_url: , prometheus_url: , signing_key: 17f***802, search_term: How***es?, model: tex***003, named_sinks: ['m***k'] 2023-02-06 11:39:07.911 INFO Callback `show_chat_gpt_search` chat_gpt_token: sk-***KtI, search_term: How***es?, model: tex***003 2023-02-06 11:39:07.913 INFO message='Request to OpenAI API' method=post path=https://api.openai.com/v1/completions 2023-02-06 11:39:08.119 INFO message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=None request_id=94865efdfb464f9dda8f319c286534b6 response_code=429 2023-02-06 11:39:08.120 INFO error_code=None error_message='You exceeded your current quota, please check your plan and billing details.' error_param=None error_type=insufficient_quota message='OpenAI API error received' stream_error=False 2023-02-06 11:39:08.120 ERROR Failed to execute action show_chat_gpt_search cluster_name: atl***tis, account_id: 1f8***5e3, chat_gpt_token: sk-***KtI, grafana_api_key: , grafana_dashboard_uid: , grafana_url: , prometheus_url: , signing_key: 17f***802, search_term: How***es?, model: tex***003, named_sinks: ['m***k'] 2023-02-06 11:42:30.097 INFO cluster status {'account_id': '1f8c96e6-9275-4e64-b81b-bcfbf444c5e3', 'cluster_id': 'atlantis', 'version': '0.10.10', 'last_alert_at': '2023-02-06 11:27:06.640497', 'light_actions': ['related_pods', 'prometheus_enricher', 'add_silence', 'delete_pod', 'delete_silence', 'get_silences', 'logs_enricher', 'pod_events_enricher', 'deployment_events_enricher', 'job_events_enricher', 'job_pod_enricher', 'get_resource_yaml', 'node_cpu_enricher', 'node_disk_analyzer', 'node_running_pods_enricher', 'node_allocatable_resources_enricher', 'node_status_enricher', 'node_graph_enricher', 'oomkilled_container_graph_enricher', 'pod_oom_killer_enricher', 'oom_killer_enricher', 'volume_analysis', 'python_profiler', 'pod_ps', 'python_memory', 'debugger_stack_trace', 'python_process_inspector', 'prometheus_alert', 'create_pvc_snapshot'], 'updated_at': 'now()'} 2023-02-06 11:46:16.202 INFO Error loading kubernetes pod default/example-pod. reason: Not Found status: 404 2023-02-06 11:46:16.223 INFO Error loading kubernetes pod default/example-pod. reason: Not Found status: 404 2023-02-06 11:46:16.243 INFO Error loading kubernetes pod default/example-pod. reason: Not Found status: 404 2023-02-06 11:46:16.245 ERROR cannot run pod_events_enricher on alert with no pod object: PrometheusKubernetesAlert(sink_findings=defaultdict(, {'main_slack_sink': [], 'robusta_ui_sink': []}), named_sinks=['main_slack_sink', 'robusta_ui_sink'], response={'success': True}, stop_processing=False, _scheduler=, _context=ExecutionContext(account_id='1f8c96e6-9275-4e64-b81b-bcfbf444c5e3', cluster_name='atlantis'), obj=None, alert=PrometheusAlert(endsAt=datetime.datetime(2023, 2, 6, 11, 46, 16, 167831), generatorURL='', startsAt=datetime.datetime(2023, 2, 6, 11, 46, 16, 167836), fingerprint='', status='firing', labels={'severity': 'error', 'namespace': 'default', 'alertname': 'KubePodCrashLooping', 'pod': 'example-pod'}, annotations={}), alert_name='KubePodCrashLooping', alert_severity='error', label_namespace='default', node=None, pod=None, deployment=None, job=None, daemonset=None, statefulset=None) 2023-02-06 11:46:16.266 INFO Error loading kubernetes pod default/example-pod. reason: Not Found status: 404 2023-02-06 11:46:32.725 INFO Callback `show_chat_gpt_search` chat_gpt_token: sk-***KtI, search_term: How***es?, model: tex***003 2023-02-06 11:46:32.725 INFO message='Request to OpenAI API' method=post path=https://api.openai.com/v1/completions 2023-02-06 11:46:32.903 INFO message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=None request_id=3776e10d5f593f35fed94f05d4cde8a8 response_code=429 2023-02-06 11:46:32.903 INFO error_code=None error_message='You exceeded your current quota, please check your plan and billing details.' error_param=None error_type=insufficient_quota message='OpenAI API error received' stream_error=False 2023-02-06 11:46:32.906 ERROR Failed to execute action show_chat_gpt_search cluster_name: atl***tis, account_id: 1f8***5e3, chat_gpt_token: sk-***KtI, grafana_api_key: , grafana_dashboard_uid: , grafana_url: , prometheus_url: , signing_key: 17f***802, search_term: How***es?, model: tex***003, named_sinks: ['m***k'] 2023-02-06 11:53:14.330 INFO Callback `show_chat_gpt_search` chat_gpt_token: sk-***KtI, search_term: How***es?, model: tex***003 2023-02-06 11:53:14.331 INFO message='Request to OpenAI API' method=post path=https://api.openai.com/v1/completions 2023-02-06 11:53:14.511 INFO message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=None request_id=3e2347cff203be09119ee05047b992bf response_code=429 2023-02-06 11:53:14.512 INFO error_code=None error_message='You exceeded your current quota, please check your plan and billing details.' error_param=None error_type=insufficient_quota message='OpenAI API error received' stream_error=False 2023-02-06 11:53:14.512 ERROR Failed to execute action show_chat_gpt_search cluster_name: atl***tis, account_id: 1f8***5e3, chat_gpt_token: sk-***KtI, grafana_api_key: , grafana_dashboard_uid: , grafana_url: , prometheus_url: , signing_key: 17f***802, search_term: How***es?, model: tex***003, named_sinks: ['m***k'] 2023-02-06 11:57:37.696 INFO cluster status {'account_id': '1f8c96e6-9275-4e64-b81b-bcfbf444c5e3', 'cluster_id': 'atlantis', 'version': '0.10.10', 'last_alert_at': '2023-02-06 11:46:16.176798', 'light_actions': ['related_pods', 'prometheus_enricher', 'add_silence', 'delete_pod', 'delete_silence', 'get_silences', 'logs_enricher', 'pod_events_enricher', 'deployment_events_enricher', 'job_events_enricher', 'job_pod_enricher', 'get_resource_yaml', 'node_cpu_enricher', 'node_disk_analyzer', 'node_running_pods_enricher', 'node_allocatable_resources_enricher', 'node_status_enricher', 'node_graph_enricher', 'oomkilled_container_graph_enricher', 'pod_oom_killer_enricher', 'oom_killer_enricher', 'volume_analysis', 'python_profiler', 'pod_ps', 'python_memory', 'debugger_stack_trace', 'python_process_inspector', 'prometheus_alert', 'create_pvc_snapshot'], 'updated_at': 'now()'} 2023-02-06 11:59:36.426 INFO Callback `show_chat_gpt_search` chat_gpt_token: sk-***KtI, search_term: How***es?, model: tex***003 2023-02-06 11:59:36.427 INFO message='Request to OpenAI API' method=post path=https://api.openai.com/v1/completions 2023-02-06 11:59:36.627 INFO message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=None request_id=e5b81f279e3a6cd1214a17f33b71360b response_code=429 2023-02-06 11:59:36.628 INFO error_code=None error_message='You exceeded your current quota, please check your plan and billing details.' error_param=None error_type=insufficient_quota message='OpenAI API error received' stream_error=False 2023-02-06 11:59:36.628 ERROR Failed to execute action show_chat_gpt_search cluster_name: atl***tis, account_id: 1f8***5e3, chat_gpt_token: sk-***KtI, grafana_api_key: , grafana_dashboard_uid: , grafana_url: , prometheus_url: , signing_key: 17f***802, search_term: How***es?, model: tex***003, named_sinks: ['m***k'] 2023-02-06 12:05:05.472 INFO Successfully loaded Kubernetes resource example-pod for alert KubePodCrashLooping 2023-02-06 12:05:05.539 INFO Successfully loaded Kubernetes resource example-pod for alert KubePodCrashLooping 2023-02-06 12:05:05.583 INFO Successfully loaded Kubernetes resource example-pod for alert KubePodCrashLooping 2023-02-06 12:05:05.679 INFO Successfully loaded Kubernetes resource example-pod for alert KubePodCrashLooping 2023-02-06 12:05:21.920 INFO Callback `show_chat_gpt_search` chat_gpt_token: sk-***KtI, search_term: How***es?, model: tex***003 2023-02-06 12:05:21.921 INFO message='Request to OpenAI API' method=post path=https://api.openai.com/v1/completions 2023-02-06 12:05:22.088 INFO message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=None request_id=85ce1a156271d680940971a9e4562207 response_code=429 2023-02-06 12:05:22.088 INFO error_code=None error_message='You exceeded your current quota, please check your plan and billing details.' error_param=None error_type=insufficient_quota message='OpenAI API error received' stream_error=False 2023-02-06 12:05:22.089 ERROR Failed to execute action show_chat_gpt_search cluster_name: atl***tis, account_id: 1f8***5e3, chat_gpt_token: sk-***KtI, grafana_api_key: , grafana_dashboard_uid: , grafana_url: , prometheus_url: , signing_key: 17f***802, search_term: How***es?, model: tex***003, named_sinks: ['m***k'] 2023-02-06 12:12:44.920 INFO cluster status {'account_id': '1f8c96e6-9275-4e64-b81b-bcfbf444c5e3', 'cluster_id': 'atlantis', 'version': '0.10.10', 'last_alert_at': '2023-02-06 12:05:05.431894', 'light_actions': ['related_pods', 'prometheus_enricher', 'add_silence', 'delete_pod', 'delete_silence', 'get_silences', 'logs_enricher', 'pod_events_enricher', 'deployment_events_enricher', 'job_events_enricher', 'job_pod_enricher', 'get_resource_yaml', 'node_cpu_enricher', 'node_disk_analyzer', 'node_running_pods_enricher', 'node_allocatable_resources_enricher', 'node_status_enricher', 'node_graph_enricher', 'oomkilled_container_graph_enricher', 'pod_oom_killer_enricher', 'oom_killer_enricher', 'volume_analysis', 'python_profiler', 'pod_ps', 'python_memory', 'debugger_stack_trace', 'python_process_inspector', 'prometheus_alert', 'create_pvc_snapshot'], 'updated_at': 'now()'} 2023-02-06 12:12:56.729 INFO Reloading playbook packages due to change on /etc/robusta/config/active_playbooks.yaml 2023-02-06 12:12:56.729 INFO loading config /etc/robusta/config/active_playbooks.yaml 2023-02-06 12:12:56.771 INFO No custom playbooks defined at /etc/robusta/playbooks/storage 2023-02-06 12:12:56.771 INFO Deleting local repo before init /app/robusta-git/kubernetes-chatgpt-bot 2023-02-06 12:12:56.775 INFO Cloning git repo https://github.com/robusta-dev/kubernetes-chatgpt-bot.git. repo name kubernetes-chatgpt-bot Processing ./robusta-git/kubernetes-chatgpt-bot Preparing metadata (pyproject.toml): started Preparing metadata (pyproject.toml): finished with status 'done' Requirement already satisfied: openai==0.26.0 in /usr/local/lib/python3.9/site-packages (from chatgpt-robusta-actions==0.0.1) (0.26.0) Requirement already satisfied: kubernetes<13.0.0,>=12.0.1 in /usr/local/lib/python3.9/site-packages (from chatgpt-robusta-actions==0.0.1) (12.0.1) Requirement already satisfied: aiohttp in /usr/local/lib/python3.9/site-packages (from openai==0.26.0->chatgpt-robusta-actions==0.0.1) (3.8.3) Requirement already satisfied: tqdm in /usr/local/lib/python3.9/site-packages (from openai==0.26.0->chatgpt-robusta-actions==0.0.1) (4.64.1) Requirement already satisfied: requests>=2.20 in /usr/local/lib/python3.9/site-packages (from openai==0.26.0->chatgpt-robusta-actions==0.0.1) (2.25.1) Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.9/site-packages (from kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (1.16.0) Requirement already satisfied: websocket-client!=0.40.0,!=0.41.*,!=0.42.*,>=0.32.0 in /usr/local/lib/python3.9/site-packages (from kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (1.3.3) Requirement already satisfied: urllib3>=1.24.2 in /usr/local/lib/python3.9/site-packages (from kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (1.26.12) Requirement already satisfied: pyyaml>=3.12 in /usr/local/lib/python3.9/site-packages (from kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (6.0) Requirement already satisfied: requests-oauthlib in /usr/local/lib/python3.9/site-packages (from kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (1.3.1) Requirement already satisfied: certifi>=14.05.14 in /usr/local/lib/python3.9/site-packages (from kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (2022.6.15) Requirement already satisfied: setuptools>=21.0.0 in /usr/local/lib/python3.9/site-packages (from kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (58.1.0) Requirement already satisfied: google-auth>=1.0.1 in /usr/local/lib/python3.9/site-packages (from kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (2.11.0) Requirement already satisfied: python-dateutil>=2.5.3 in /usr/local/lib/python3.9/site-packages (from kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (2.8.2) Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.9/site-packages (from google-auth>=1.0.1->kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (5.2.0) Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.9/site-packages (from google-auth>=1.0.1->kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (4.9) Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.9/site-packages (from google-auth>=1.0.1->kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (0.2.8) Requirement already satisfied: chardet<5,>=3.0.2 in /usr/local/lib/python3.9/site-packages (from requests>=2.20->openai==0.26.0->chatgpt-robusta-actions==0.0.1) (4.0.0) Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.9/site-packages (from requests>=2.20->openai==0.26.0->chatgpt-robusta-actions==0.0.1) (2.10) Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.9/site-packages (from aiohttp->openai==0.26.0->chatgpt-robusta-actions==0.0.1) (6.0.4) Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.9/site-packages (from aiohttp->openai==0.26.0->chatgpt-robusta-actions==0.0.1) (4.0.2) Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.9/site-packages (from aiohttp->openai==0.26.0->chatgpt-robusta-actions==0.0.1) (1.3.1) Requirement already satisfied: charset-normalizer<3.0,>=2.0 in /usr/local/lib/python3.9/site-packages (from aiohttp->openai==0.26.0->chatgpt-robusta-actions==0.0.1) (2.1.1) Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.9/site-packages (from aiohttp->openai==0.26.0->chatgpt-robusta-actions==0.0.1) (1.8.2) Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.9/site-packages (from aiohttp->openai==0.26.0->chatgpt-robusta-actions==0.0.1) (22.1.0) Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.9/site-packages (from aiohttp->openai==0.26.0->chatgpt-robusta-actions==0.0.1) (1.3.3) Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.9/site-packages (from requests-oauthlib->kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (3.2.0) Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.9/site-packages (from pyasn1-modules>=0.2.1->google-auth>=1.0.1->kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (0.4.8) Building wheels for collected packages: chatgpt-robusta-actions Building wheel for chatgpt-robusta-actions (pyproject.toml): started Building wheel for chatgpt-robusta-actions (pyproject.toml): finished with status 'done' Created wheel for chatgpt-robusta-actions: filename=chatgpt_robusta_actions-0.0.1-py2.py3-none-any.whl size=2079 sha256=d668ebce24a80b033960f258b87f39ac4384d1987a3f53da84deb88d371edf61 Stored in directory: /root/.cache/pip/wheels/dc/11/de/1137460e08cac026926087b9febd6dcaa9e62964e468dc5fee Successfully built chatgpt-robusta-actions Installing collected packages: chatgpt-robusta-actions Attempting uninstall: chatgpt-robusta-actions Found existing installation: chatgpt-robusta-actions 0.0.1 Uninstalling chatgpt-robusta-actions-0.0.1: Successfully uninstalled chatgpt-robusta-actions-0.0.1 Successfully installed chatgpt-robusta-actions-0.0.1 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv [notice] A new release of pip available: 22.3.1 -> 23.0 [notice] To update, run: pip install --upgrade pip 2023-02-06 12:12:59.183 INFO Importing actions package robusta.core.playbooks.internal 2023-02-06 12:12:59.184 INFO importing actions from robusta.core.playbooks.internal.discovery_events 2023-02-06 12:12:59.187 INFO Importing actions package robusta_playbooks 2023-02-06 12:12:59.188 INFO importing actions from robusta_playbooks.alerts_integration 2023-02-06 12:12:59.189 INFO importing actions from robusta_playbooks.argo_cd 2023-02-06 12:12:59.190 INFO importing actions from robusta_playbooks.autoscaler 2023-02-06 12:12:59.191 INFO importing actions from robusta_playbooks.babysitter 2023-02-06 12:12:59.192 INFO importing actions from robusta_playbooks.bash_enrichments 2023-02-06 12:12:59.193 INFO importing actions from robusta_playbooks.chaos_engineering 2023-02-06 12:12:59.194 INFO importing actions from robusta_playbooks.common_actions 2023-02-06 12:12:59.195 INFO importing actions from robusta_playbooks.configuration_ab_testing 2023-02-06 12:12:59.196 INFO importing actions from robusta_playbooks.cpu_throttling 2023-02-06 12:12:59.197 INFO importing actions from robusta_playbooks.daemonsets 2023-02-06 12:12:59.198 INFO importing actions from robusta_playbooks.deployment_enrichments 2023-02-06 12:12:59.199 INFO importing actions from robusta_playbooks.deployment_status_report 2023-02-06 12:12:59.200 INFO importing actions from robusta_playbooks.disk_benchmark 2023-02-06 12:12:59.201 INFO importing actions from robusta_playbooks.event_enrichments 2023-02-06 12:12:59.202 INFO importing actions from robusta_playbooks.git_change_audit 2023-02-06 12:12:59.203 INFO importing actions from robusta_playbooks.grafana_enrichment 2023-02-06 12:12:59.204 INFO importing actions from robusta_playbooks.http_actions 2023-02-06 12:12:59.204 INFO importing actions from robusta_playbooks.image_pull_backoff_enricher 2023-02-06 12:12:59.205 INFO importing actions from robusta_playbooks.java_pod_troubleshooting 2023-02-06 12:12:59.206 INFO importing actions from robusta_playbooks.job_actions 2023-02-06 12:12:59.207 INFO importing actions from robusta_playbooks.k8s_resource_enrichments 2023-02-06 12:12:59.208 INFO importing actions from robusta_playbooks.networking 2023-02-06 12:12:59.209 INFO importing actions from robusta_playbooks.node_cpu_analysis 2023-02-06 12:12:59.210 INFO importing actions from robusta_playbooks.node_disk_analysis 2023-02-06 12:12:59.211 INFO importing actions from robusta_playbooks.node_enrichments 2023-02-06 12:12:59.212 INFO importing actions from robusta_playbooks.oom_killer 2023-02-06 12:12:59.213 INFO importing actions from robusta_playbooks.overcommit_enrichments 2023-02-06 12:12:59.214 INFO importing actions from robusta_playbooks.persistent_data 2023-02-06 12:12:59.215 INFO importing actions from robusta_playbooks.persistent_volume_actions 2023-02-06 12:12:59.215 INFO importing actions from robusta_playbooks.pod_actions 2023-02-06 12:12:59.216 INFO importing actions from robusta_playbooks.pod_enrichments 2023-02-06 12:12:59.217 INFO importing actions from robusta_playbooks.pod_troubleshooting 2023-02-06 12:12:59.218 INFO importing actions from robusta_playbooks.prometheus_enrichments 2023-02-06 12:12:59.219 INFO importing actions from robusta_playbooks.prometheus_simulation 2023-02-06 12:12:59.220 INFO importing actions from robusta_playbooks.pvc_snapshots 2023-02-06 12:12:59.221 INFO importing actions from robusta_playbooks.restart_loop_reporter 2023-02-06 12:12:59.222 INFO importing actions from robusta_playbooks.silence 2023-02-06 12:12:59.223 INFO importing actions from robusta_playbooks.simple_examples 2023-02-06 12:12:59.224 INFO importing actions from robusta_playbooks.stress_tests 2023-02-06 12:12:59.225 INFO importing actions from robusta_playbooks.targetdown_enrichment 2023-02-06 12:12:59.226 INFO Importing actions package chatgpt_robusta_actions 2023-02-06 12:12:59.226 INFO importing actions from chatgpt_robusta_actions.chat_gpt 2023-02-06 12:19:51.458 INFO Successfully loaded Kubernetes resource example-pod for alert KubePodCrashLooping 2023-02-06 12:19:51.525 INFO Successfully loaded Kubernetes resource example-pod for alert KubePodCrashLooping 2023-02-06 12:19:51.564 INFO Successfully loaded Kubernetes resource example-pod for alert KubePodCrashLooping 2023-02-06 12:19:51.659 INFO Successfully loaded Kubernetes resource example-pod for alert KubePodCrashLooping 2023-02-06 12:20:07.292 INFO Callback `show_chat_gpt_search` chat_gpt_token: sk-***iyg, search_term: How***es?, model: tex***003 2023-02-06 12:20:07.294 INFO message='Request to OpenAI API' method=post path=https://api.openai.com/v1/completions 2023-02-06 12:20:18.012 INFO message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=10426 request_id=f357659c6eaedcaf0a31e2f6e478bdc6 response_code=200 2023-02-06 12:20:18.654 INFO Callback `show_chat_gpt_search` chat_gpt_token: sk-***iyg, search_term: How***es?, model: tex***003 2023-02-06 12:20:18.656 INFO message='Request to OpenAI API' method=post path=https://api.openai.com/v1/completions 2023-02-06 12:20:35.405 INFO message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=16630 request_id=bf7ae2a7be14c9f36688667e3958901e response_code=200 2023-02-06 12:25:07.966 INFO Callback `show_chat_gpt_search` chat_gpt_token: sk-***iyg, search_term: How***es?, model: tex***003 2023-02-06 12:25:07.966 INFO message='Request to OpenAI API' method=post path=https://api.openai.com/v1/completions 2023-02-06 12:25:20.078 INFO message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=11883 request_id=8bec1eff610bf56cfedea8704776e82b response_code=200 2023-02-06 12:26:11.590 INFO Callback `show_chat_gpt_search` chat_gpt_token: sk-***iyg, search_term: How***es?, model: tex***003 2023-02-06 12:26:11.591 INFO message='Request to OpenAI API' method=post path=https://api.openai.com/v1/completions 2023-02-06 12:26:49.741 INFO message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=38082 request_id=e4d5ac0fdc678fe107be7e3f2df228ab response_code=200 2023-02-06 12:27:51.748 INFO cluster status {'account_id': '1f8c96e6-9275-4e64-b81b-bcfbf444c5e3', 'cluster_id': 'atlantis', 'version': '0.10.10', 'last_alert_at': '2023-02-06 12:19:51.405680', 'light_actions': ['related_pods', 'prometheus_enricher', 'add_silence', 'delete_pod', 'delete_silence', 'get_silences', 'logs_enricher', 'pod_events_enricher', 'deployment_events_enricher', 'job_events_enricher', 'job_pod_enricher', 'get_resource_yaml', 'node_cpu_enricher', 'node_disk_analyzer', 'node_running_pods_enricher', 'node_allocatable_resources_enricher', 'node_status_enricher', 'node_graph_enricher', 'oomkilled_container_graph_enricher', 'pod_oom_killer_enricher', 'oom_killer_enricher', 'volume_analysis', 'python_profiler', 'pod_ps', 'python_memory', 'debugger_stack_trace', 'python_process_inspector', 'prometheus_alert', 'create_pvc_snapshot'], 'updated_at': 'now()'} 2023-02-06 12:36:02.830 INFO Successfully loaded Kubernetes resource example-pod for alert KubePodNotReady 2023-02-06 12:36:02.871 INFO Successfully loaded Kubernetes resource atlantis-0 for alert KubePodNotReady 2023-02-06 12:36:02.888 INFO Successfully loaded Kubernetes resource example-pod for alert KubePodNotReady 2023-02-06 12:36:02.931 INFO Successfully loaded Kubernetes resource example-pod for alert KubePodNotReady 2023-02-06 12:36:03.101 INFO Successfully loaded Kubernetes resource example-pod for alert KubePodNotReady 2023-02-06 12:36:03.297 INFO Successfully loaded Kubernetes resource atlantis-0 for alert KubePodNotReady 2023-02-06 12:36:03.402 INFO Successfully loaded Kubernetes resource atlantis-0 for alert KubePodNotReady 2023-02-06 12:36:03.615 INFO Successfully loaded Kubernetes resource atlantis-0 for alert KubePodNotReady 2023-02-06 12:38:25.309 INFO Callback `show_chat_gpt_search` chat_gpt_token: sk-***iyg, search_term: How***es?, model: tex***003 2023-02-06 12:38:25.310 INFO message='Request to OpenAI API' method=post path=https://api.openai.com/v1/completions 2023-02-06 12:38:26.575 INFO message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=1075 request_id=6c67cb5f92b332fab9e9c06becdba4fb response_code=429 2023-02-06 12:38:26.576 INFO error_code=None error_message='The server had an error while processing your request. Sorry about that!' error_param=None error_type=server_error message='OpenAI API error received' stream_error=False 2023-02-06 12:38:26.576 ERROR Failed to execute action show_chat_gpt_search cluster_name: atl***tis, account_id: 1f8***5e3, chat_gpt_token: sk-***iyg, grafana_api_key: , grafana_dashboard_uid: , grafana_url: , prometheus_url: , signing_key: 17f***802, search_term: How***es?, model: tex***003, named_sinks: ['m***k'] 2023-02-06 12:38:38.078 INFO Callback `show_chat_gpt_search` chat_gpt_token: sk-***iyg, search_term: How***es?, model: tex***003 2023-02-06 12:38:38.081 INFO message='Request to OpenAI API' method=post path=https://api.openai.com/v1/completions 2023-02-06 12:39:21.126 INFO message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=42944 request_id=79ee8375bd8080a9842f433acd30ad14 response_code=200 2023-02-06 12:39:21.226 INFO Relay websocket error: ping/pong timed out 2023-02-06 12:39:21.259 INFO relay websocket closed 2023-02-06 12:39:24.327 INFO connecting to server as account_id=1f8c96e6-9275-4e64-b81b-bcfbf444c5e3; cluster_name=atlantis 2023-02-06 12:39:51.183 INFO Callback `show_chat_gpt_search` chat_gpt_token: sk-***iyg, search_term: How***es?, model: tex***003 2023-02-06 12:39:51.184 INFO message='Request to OpenAI API' method=post path=https://api.openai.com/v1/completions 2023-02-06 12:40:01.932 INFO message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=10654 request_id=b768e1d7459e7f67306cb83b84bb3a1a response_code=200 2023-02-06 12:42:58.231 INFO cluster status {'account_id': '1f8c96e6-9275-4e64-b81b-bcfbf444c5e3', 'cluster_id': 'atlantis', 'version': '0.10.10', 'last_alert_at': '2023-02-06 12:36:02.776016', 'light_actions': ['related_pods', 'prometheus_enricher', 'add_silence', 'delete_pod', 'delete_silence', 'get_silences', 'logs_enricher', 'pod_events_enricher', 'deployment_events_enricher', 'job_events_enricher', 'job_pod_enricher', 'get_resource_yaml', 'node_cpu_enricher', 'node_disk_analyzer', 'node_running_pods_enricher', 'node_allocatable_resources_enricher', 'node_status_enricher', 'node_graph_enricher', 'oomkilled_container_graph_enricher', 'pod_oom_killer_enricher', 'oom_killer_enricher', 'volume_analysis', 'python_profiler', 'pod_ps', 'python_memory', 'debugger_stack_trace', 'python_process_inspector', 'prometheus_alert', 'create_pvc_snapshot'], 'updated_at': 'now()'} 2023-02-06 12:58:06.540 INFO cluster status {'account_id': '1f8c96e6-9275-4e64-b81b-bcfbf444c5e3', 'cluster_id': 'atlantis', 'version': '0.10.10', 'last_alert_at': '2023-02-06 12:36:02.776016', 'light_actions': ['related_pods', 'prometheus_enricher', 'add_silence', 'delete_pod', 'delete_silence', 'get_silences', 'logs_enricher', 'pod_events_enricher', 'deployment_events_enricher', 'job_events_enricher', 'job_pod_enricher', 'get_resource_yaml', 'node_cpu_enricher', 'node_disk_analyzer', 'node_running_pods_enricher', 'node_allocatable_resources_enricher', 'node_status_enricher', 'node_graph_enricher', 'oomkilled_container_graph_enricher', 'pod_oom_killer_enricher', 'oom_killer_enricher', 'volume_analysis', 'python_profiler', 'pod_ps', 'python_memory', 'debugger_stack_trace', 'python_process_inspector', 'prometheus_alert', 'create_pvc_snapshot'], 'updated_at': 'now()'} 2023-02-06 13:08:55.161 ERROR could not fetch logs from container: crashpod. logs were  2023-02-06 13:13:15.961 INFO cluster status {'account_id': '1f8c96e6-9275-4e64-b81b-bcfbf444c5e3', 'cluster_id': 'atlantis', 'version': '0.10.10', 'last_alert_at': '2023-02-06 12:36:02.776016', 'light_actions': ['related_pods', 'prometheus_enricher', 'add_silence', 'delete_pod', 'delete_silence', 'get_silences', 'logs_enricher', 'pod_events_enricher', 'deployment_events_enricher', 'job_events_enricher', 'job_pod_enricher', 'get_resource_yaml', 'node_cpu_enricher', 'node_disk_analyzer', 'node_running_pods_enricher', 'node_allocatable_resources_enricher', 'node_status_enricher', 'node_graph_enricher', 'oomkilled_container_graph_enricher', 'pod_oom_killer_enricher', 'oom_killer_enricher', 'volume_analysis', 'python_profiler', 'pod_ps', 'python_memory', 'debugger_stack_trace', 'python_process_inspector', 'prometheus_alert', 'create_pvc_snapshot'], 'updated_at': 'now()'} 2023-02-06 13:17:25.068 INFO Reloading playbook packages due to change on /etc/robusta/config/active_playbooks.yaml 2023-02-06 13:17:25.068 INFO loading config /etc/robusta/config/active_playbooks.yaml 2023-02-06 13:17:25.107 INFO No custom playbooks defined at /etc/robusta/playbooks/storage 2023-02-06 13:17:25.116 INFO Deleting local repo before init /app/robusta-git/kubernetes-chatgpt-bot 2023-02-06 13:17:25.122 INFO Cloning git repo https://github.com/robusta-dev/kubernetes-chatgpt-bot.git. repo name kubernetes-chatgpt-bot Processing ./robusta-git/kubernetes-chatgpt-bot Preparing metadata (pyproject.toml): started Preparing metadata (pyproject.toml): finished with status 'done' Requirement already satisfied: openai==0.26.0 in /usr/local/lib/python3.9/site-packages (from chatgpt-robusta-actions==0.0.1) (0.26.0) Requirement already satisfied: kubernetes<13.0.0,>=12.0.1 in /usr/local/lib/python3.9/site-packages (from chatgpt-robusta-actions==0.0.1) (12.0.1) Requirement already satisfied: requests>=2.20 in /usr/local/lib/python3.9/site-packages (from openai==0.26.0->chatgpt-robusta-actions==0.0.1) (2.25.1) Requirement already satisfied: aiohttp in /usr/local/lib/python3.9/site-packages (from openai==0.26.0->chatgpt-robusta-actions==0.0.1) (3.8.3) Requirement already satisfied: tqdm in /usr/local/lib/python3.9/site-packages (from openai==0.26.0->chatgpt-robusta-actions==0.0.1) (4.64.1) Requirement already satisfied: pyyaml>=3.12 in /usr/local/lib/python3.9/site-packages (from kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (6.0) Requirement already satisfied: certifi>=14.05.14 in /usr/local/lib/python3.9/site-packages (from kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (2022.6.15) Requirement already satisfied: google-auth>=1.0.1 in /usr/local/lib/python3.9/site-packages (from kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (2.11.0) Requirement already satisfied: python-dateutil>=2.5.3 in /usr/local/lib/python3.9/site-packages (from kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (2.8.2) Requirement already satisfied: urllib3>=1.24.2 in /usr/local/lib/python3.9/site-packages (from kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (1.26.12) Requirement already satisfied: requests-oauthlib in /usr/local/lib/python3.9/site-packages (from kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (1.3.1) Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.9/site-packages (from kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (1.16.0) Requirement already satisfied: websocket-client!=0.40.0,!=0.41.*,!=0.42.*,>=0.32.0 in /usr/local/lib/python3.9/site-packages (from kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (1.3.3) Requirement already satisfied: setuptools>=21.0.0 in /usr/local/lib/python3.9/site-packages (from kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (58.1.0) Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.9/site-packages (from google-auth>=1.0.1->kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (4.9) Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.9/site-packages (from google-auth>=1.0.1->kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (5.2.0) Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.9/site-packages (from google-auth>=1.0.1->kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (0.2.8) Requirement already satisfied: chardet<5,>=3.0.2 in /usr/local/lib/python3.9/site-packages (from requests>=2.20->openai==0.26.0->chatgpt-robusta-actions==0.0.1) (4.0.0) Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.9/site-packages (from requests>=2.20->openai==0.26.0->chatgpt-robusta-actions==0.0.1) (2.10) Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.9/site-packages (from aiohttp->openai==0.26.0->chatgpt-robusta-actions==0.0.1) (1.3.3) Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.9/site-packages (from aiohttp->openai==0.26.0->chatgpt-robusta-actions==0.0.1) (1.8.2) Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.9/site-packages (from aiohttp->openai==0.26.0->chatgpt-robusta-actions==0.0.1) (22.1.0) Requirement already satisfied: charset-normalizer<3.0,>=2.0 in /usr/local/lib/python3.9/site-packages (from aiohttp->openai==0.26.0->chatgpt-robusta-actions==0.0.1) (2.1.1) Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.9/site-packages (from aiohttp->openai==0.26.0->chatgpt-robusta-actions==0.0.1) (1.3.1) Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.9/site-packages (from aiohttp->openai==0.26.0->chatgpt-robusta-actions==0.0.1) (4.0.2) Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.9/site-packages (from aiohttp->openai==0.26.0->chatgpt-robusta-actions==0.0.1) (6.0.4) Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.9/site-packages (from requests-oauthlib->kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (3.2.0) Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.9/site-packages (from pyasn1-modules>=0.2.1->google-auth>=1.0.1->kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (0.4.8) Building wheels for collected packages: chatgpt-robusta-actions Building wheel for chatgpt-robusta-actions (pyproject.toml): started Building wheel for chatgpt-robusta-actions (pyproject.toml): finished with status 'done' Created wheel for chatgpt-robusta-actions: filename=chatgpt_robusta_actions-0.0.1-py2.py3-none-any.whl size=2079 sha256=d668ebce24a80b033960f258b87f39ac4384d1987a3f53da84deb88d371edf61 Stored in directory: /root/.cache/pip/wheels/dc/11/de/1137460e08cac026926087b9febd6dcaa9e62964e468dc5fee Successfully built chatgpt-robusta-actions Installing collected packages: chatgpt-robusta-actions Attempting uninstall: chatgpt-robusta-actions Found existing installation: chatgpt-robusta-actions 0.0.1 Uninstalling chatgpt-robusta-actions-0.0.1: Successfully uninstalled chatgpt-robusta-actions-0.0.1 Successfully installed chatgpt-robusta-actions-0.0.1 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv [notice] A new release of pip available: 22.3.1 -> 23.0 [notice] To update, run: pip install --upgrade pip 2023-02-06 13:17:28.191 INFO Importing actions package robusta.core.playbooks.internal 2023-02-06 13:17:28.192 INFO importing actions from robusta.core.playbooks.internal.discovery_events 2023-02-06 13:17:28.194 INFO Importing actions package robusta_playbooks 2023-02-06 13:17:28.195 INFO importing actions from robusta_playbooks.alerts_integration 2023-02-06 13:17:28.198 INFO importing actions from robusta_playbooks.argo_cd 2023-02-06 13:17:28.200 INFO importing actions from robusta_playbooks.autoscaler 2023-02-06 13:17:28.202 INFO importing actions from robusta_playbooks.babysitter 2023-02-06 13:17:28.203 INFO importing actions from robusta_playbooks.bash_enrichments 2023-02-06 13:17:28.205 INFO importing actions from robusta_playbooks.chaos_engineering 2023-02-06 13:17:28.208 INFO importing actions from robusta_playbooks.common_actions 2023-02-06 13:17:28.209 INFO importing actions from robusta_playbooks.configuration_ab_testing 2023-02-06 13:17:28.210 INFO importing actions from robusta_playbooks.cpu_throttling 2023-02-06 13:17:28.211 INFO importing actions from robusta_playbooks.daemonsets 2023-02-06 13:17:28.213 INFO importing actions from robusta_playbooks.deployment_enrichments 2023-02-06 13:17:28.214 INFO importing actions from robusta_playbooks.deployment_status_report 2023-02-06 13:17:28.215 INFO importing actions from robusta_playbooks.disk_benchmark 2023-02-06 13:17:28.216 INFO importing actions from robusta_playbooks.event_enrichments 2023-02-06 13:17:28.217 INFO importing actions from robusta_playbooks.git_change_audit 2023-02-06 13:17:28.218 INFO importing actions from robusta_playbooks.grafana_enrichment 2023-02-06 13:17:28.220 INFO importing actions from robusta_playbooks.http_actions 2023-02-06 13:17:28.220 INFO importing actions from robusta_playbooks.image_pull_backoff_enricher 2023-02-06 13:17:28.221 INFO importing actions from robusta_playbooks.java_pod_troubleshooting 2023-02-06 13:17:28.223 INFO importing actions from robusta_playbooks.job_actions 2023-02-06 13:17:28.224 INFO importing actions from robusta_playbooks.k8s_resource_enrichments 2023-02-06 13:17:28.225 INFO importing actions from robusta_playbooks.networking 2023-02-06 13:17:28.226 INFO importing actions from robusta_playbooks.node_cpu_analysis 2023-02-06 13:17:28.227 INFO importing actions from robusta_playbooks.node_disk_analysis 2023-02-06 13:17:28.228 INFO importing actions from robusta_playbooks.node_enrichments 2023-02-06 13:17:28.229 INFO importing actions from robusta_playbooks.oom_killer 2023-02-06 13:17:28.230 INFO importing actions from robusta_playbooks.overcommit_enrichments 2023-02-06 13:17:28.232 INFO importing actions from robusta_playbooks.persistent_data 2023-02-06 13:17:28.234 INFO importing actions from robusta_playbooks.persistent_volume_actions 2023-02-06 13:17:28.234 INFO importing actions from robusta_playbooks.pod_actions 2023-02-06 13:17:28.235 INFO importing actions from robusta_playbooks.pod_enrichments 2023-02-06 13:17:28.237 INFO importing actions from robusta_playbooks.pod_troubleshooting 2023-02-06 13:17:28.239 INFO importing actions from robusta_playbooks.prometheus_enrichments 2023-02-06 13:17:28.240 INFO importing actions from robusta_playbooks.prometheus_simulation 2023-02-06 13:17:28.242 INFO importing actions from robusta_playbooks.pvc_snapshots 2023-02-06 13:17:28.243 INFO importing actions from robusta_playbooks.restart_loop_reporter 2023-02-06 13:17:28.244 INFO importing actions from robusta_playbooks.silence 2023-02-06 13:17:28.245 INFO importing actions from robusta_playbooks.simple_examples 2023-02-06 13:17:28.247 INFO importing actions from robusta_playbooks.stress_tests 2023-02-06 13:17:28.248 INFO importing actions from robusta_playbooks.targetdown_enrichment 2023-02-06 13:17:28.249 INFO Importing actions package chatgpt_robusta_actions 2023-02-06 13:17:28.277 INFO importing actions from chatgpt_robusta_actions.chat_gpt 2023-02-06 13:17:28.281 ERROR Action chat_gpt_enricher cannot be triggered by  2023-02-06 13:17:28.290 ERROR unknown error reloading playbooks. will try again when they next change Traceback (most recent call last): File "/app/src/robusta/runner/config_loader.py", line 241, in __reload_playbook_packages (sinks_registry, playbooks_registry) = self.__prepare_runtime_config( File "/app/src/robusta/runner/config_loader.py", line 308, in __prepare_runtime_config playbooks_registry = PlaybooksRegistryImpl( File "/app/src/robusta/model/config.py", line 151, in __init__ raise Exception(msg) Exception: Action chat_gpt_enricher cannot be triggered by  2023-02-06 13:17:50.409 WARNING Failed to parse delete response data 2023-02-06 13:26:02.805 INFO Error loading kubernetes pod default/example-pod. reason: Not Found status: 404 2023-02-06 13:26:02.835 INFO Error loading kubernetes pod default/example-pod. reason: Not Found status: 404 2023-02-06 13:26:02.890 INFO Successfully loaded Kubernetes resource atlantis-0 for alert KubePodNotReady 2023-02-06 13:26:02.967 INFO Successfully loaded Kubernetes resource atlantis-0 for alert KubePodNotReady 2023-02-06 13:26:03.059 INFO Successfully loaded Kubernetes resource atlantis-0 for alert KubePodNotReady 2023-02-06 13:26:03.199 INFO Successfully loaded Kubernetes resource atlantis-0 for alert KubePodNotReady 2023-02-06 13:26:17.215 INFO Error loading kubernetes pod default/crashpod-847bc5cbfb-gkkp2. reason: Not Found status: 404 2023-02-06 13:26:17.229 INFO Error loading kubernetes pod default/crashpod-847bc5cbfb-gkkp2. reason: Not Found status: 404 2023-02-06 13:26:17.325 INFO Successfully loaded Kubernetes resource atlantis for alert KubeStatefulSetReplicasMismatch 2023-02-06 13:26:17.460 INFO Successfully loaded Kubernetes resource atlantis for alert KubeStatefulSetReplicasMismatch 2023-02-06 13:26:17.580 INFO Successfully loaded Kubernetes resource atlantis for alert KubeStatefulSetReplicasMismatch 2023-02-06 13:26:48.412 INFO Callback `show_chat_gpt_search` chat_gpt_token: sk-***iyg, search_term: How***es?, model: tex***003 2023-02-06 13:26:48.413 INFO message='Request to OpenAI API' method=post path=https://api.openai.com/v1/completions 2023-02-06 13:27:38.350 INFO message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=49688 request_id=39e2e3c0bc7e9db613b0874d1508c40c response_code=200 2023-02-06 13:27:38.458 INFO Callback `show_chat_gpt_search` chat_gpt_token: sk-***iyg, search_term: How***es?, model: tex***003 2023-02-06 13:27:38.459 INFO message='Request to OpenAI API' method=post path=https://api.openai.com/v1/completions 2023-02-06 13:27:58.383 INFO Successfully loaded Kubernetes resource example-pod for alert KubePodCrashLooping 2023-02-06 13:27:58.441 INFO Successfully loaded Kubernetes resource example-pod for alert KubePodCrashLooping 2023-02-06 13:27:58.484 INFO Successfully loaded Kubernetes resource example-pod for alert KubePodCrashLooping 2023-02-06 13:27:58.590 INFO Successfully loaded Kubernetes resource example-pod for alert KubePodCrashLooping 2023-02-06 13:28:18.561 INFO message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=40009 request_id=f3ed15aac17ac37f9181bfe92c784603 response_code=200 2023-02-06 13:28:18.664 INFO Relay websocket error: ping/pong timed out 2023-02-06 13:28:18.694 INFO relay websocket closed 2023-02-06 13:28:21.775 INFO connecting to server as account_id=1f8c96e6-9275-4e64-b81b-bcfbf444c5e3; cluster_name=atlantis 2023-02-06 13:28:26.558 INFO cluster status {'account_id': '1f8c96e6-9275-4e64-b81b-bcfbf444c5e3', 'cluster_id': 'atlantis', 'version': '0.10.10', 'last_alert_at': '2023-02-06 13:27:58.324266', 'light_actions': ['related_pods', 'prometheus_enricher', 'add_silence', 'delete_pod', 'delete_silence', 'get_silences', 'logs_enricher', 'pod_events_enricher', 'deployment_events_enricher', 'job_events_enricher', 'job_pod_enricher', 'get_resource_yaml', 'node_cpu_enricher', 'node_disk_analyzer', 'node_running_pods_enricher', 'node_allocatable_resources_enricher', 'node_status_enricher', 'node_graph_enricher', 'oomkilled_container_graph_enricher', 'pod_oom_killer_enricher', 'oom_killer_enricher', 'volume_analysis', 'python_profiler', 'pod_ps', 'python_memory', 'debugger_stack_trace', 'python_process_inspector', 'prometheus_alert', 'create_pvc_snapshot'], 'updated_at': 'now()'} 2023-02-06 13:28:38.733 INFO Callback `show_chat_gpt_search` chat_gpt_token: sk-***iyg, search_term: How***es?, model: tex***003 2023-02-06 13:28:38.734 INFO message='Request to OpenAI API' method=post path=https://api.openai.com/v1/completions 2023-02-06 13:28:51.670 INFO message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=12813 request_id=af5a4406ce56f2cc5b25a1a019de0416 response_code=200 2023-02-06 13:29:25.529 INFO Callback `show_chat_gpt_search` chat_gpt_token: sk-***iyg, search_term: How***es?, model: tex***003 2023-02-06 13:29:25.529 INFO message='Request to OpenAI API' method=post path=https://api.openai.com/v1/completions 2023-02-06 13:29:38.188 INFO message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=12562 request_id=b20ac616b429594cafddcd61f6987f5c response_code=200 2023-02-06 13:39:11.791 INFO Reloading playbook packages due to change on /etc/robusta/config/active_playbooks.yaml 2023-02-06 13:39:11.791 INFO loading config /etc/robusta/config/active_playbooks.yaml 2023-02-06 13:39:11.842 INFO No custom playbooks defined at /etc/robusta/playbooks/storage 2023-02-06 13:39:11.843 INFO Deleting local repo before init /app/robusta-git/kubernetes-chatgpt-bot 2023-02-06 13:39:11.846 INFO Cloning git repo https://github.com/robusta-dev/kubernetes-chatgpt-bot.git. repo name kubernetes-chatgpt-bot Processing ./robusta-git/kubernetes-chatgpt-bot Preparing metadata (pyproject.toml): started Preparing metadata (pyproject.toml): finished with status 'done' Requirement already satisfied: kubernetes<13.0.0,>=12.0.1 in /usr/local/lib/python3.9/site-packages (from chatgpt-robusta-actions==0.0.1) (12.0.1) Requirement already satisfied: openai==0.26.0 in /usr/local/lib/python3.9/site-packages (from chatgpt-robusta-actions==0.0.1) (0.26.0) Requirement already satisfied: aiohttp in /usr/local/lib/python3.9/site-packages (from openai==0.26.0->chatgpt-robusta-actions==0.0.1) (3.8.3) Requirement already satisfied: tqdm in /usr/local/lib/python3.9/site-packages (from openai==0.26.0->chatgpt-robusta-actions==0.0.1) (4.64.1) Requirement already satisfied: requests>=2.20 in /usr/local/lib/python3.9/site-packages (from openai==0.26.0->chatgpt-robusta-actions==0.0.1) (2.25.1) Requirement already satisfied: requests-oauthlib in /usr/local/lib/python3.9/site-packages (from kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (1.3.1) Requirement already satisfied: websocket-client!=0.40.0,!=0.41.*,!=0.42.*,>=0.32.0 in /usr/local/lib/python3.9/site-packages (from kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (1.3.3) Requirement already satisfied: pyyaml>=3.12 in /usr/local/lib/python3.9/site-packages (from kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (6.0) Requirement already satisfied: google-auth>=1.0.1 in /usr/local/lib/python3.9/site-packages (from kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (2.11.0) Requirement already satisfied: setuptools>=21.0.0 in /usr/local/lib/python3.9/site-packages (from kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (58.1.0) Requirement already satisfied: urllib3>=1.24.2 in /usr/local/lib/python3.9/site-packages (from kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (1.26.12) Requirement already satisfied: certifi>=14.05.14 in /usr/local/lib/python3.9/site-packages (from kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (2022.6.15) Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.9/site-packages (from kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (1.16.0) Requirement already satisfied: python-dateutil>=2.5.3 in /usr/local/lib/python3.9/site-packages (from kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (2.8.2) Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.9/site-packages (from google-auth>=1.0.1->kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (4.9) Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.9/site-packages (from google-auth>=1.0.1->kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (0.2.8) Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.9/site-packages (from google-auth>=1.0.1->kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (5.2.0) Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.9/site-packages (from requests>=2.20->openai==0.26.0->chatgpt-robusta-actions==0.0.1) (2.10) Requirement already satisfied: chardet<5,>=3.0.2 in /usr/local/lib/python3.9/site-packages (from requests>=2.20->openai==0.26.0->chatgpt-robusta-actions==0.0.1) (4.0.0) Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.9/site-packages (from aiohttp->openai==0.26.0->chatgpt-robusta-actions==0.0.1) (1.3.3) Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.9/site-packages (from aiohttp->openai==0.26.0->chatgpt-robusta-actions==0.0.1) (1.3.1) Requirement already satisfied: charset-normalizer<3.0,>=2.0 in /usr/local/lib/python3.9/site-packages (from aiohttp->openai==0.26.0->chatgpt-robusta-actions==0.0.1) (2.1.1) Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.9/site-packages (from aiohttp->openai==0.26.0->chatgpt-robusta-actions==0.0.1) (22.1.0) Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.9/site-packages (from aiohttp->openai==0.26.0->chatgpt-robusta-actions==0.0.1) (6.0.4) Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.9/site-packages (from aiohttp->openai==0.26.0->chatgpt-robusta-actions==0.0.1) (1.8.2) Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.9/site-packages (from aiohttp->openai==0.26.0->chatgpt-robusta-actions==0.0.1) (4.0.2) Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.9/site-packages (from requests-oauthlib->kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (3.2.0) Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.9/site-packages (from pyasn1-modules>=0.2.1->google-auth>=1.0.1->kubernetes<13.0.0,>=12.0.1->chatgpt-robusta-actions==0.0.1) (0.4.8) Building wheels for collected packages: chatgpt-robusta-actions Building wheel for chatgpt-robusta-actions (pyproject.toml): started Building wheel for chatgpt-robusta-actions (pyproject.toml): finished with status 'done' Created wheel for chatgpt-robusta-actions: filename=chatgpt_robusta_actions-0.0.1-py2.py3-none-any.whl size=2079 sha256=d668ebce24a80b033960f258b87f39ac4384d1987a3f53da84deb88d371edf61 Stored in directory: /root/.cache/pip/wheels/dc/11/de/1137460e08cac026926087b9febd6dcaa9e62964e468dc5fee Successfully built chatgpt-robusta-actions Installing collected packages: chatgpt-robusta-actions Attempting uninstall: chatgpt-robusta-actions Found existing installation: chatgpt-robusta-actions 0.0.1 Uninstalling chatgpt-robusta-actions-0.0.1: Successfully uninstalled chatgpt-robusta-actions-0.0.1 Successfully installed chatgpt-robusta-actions-0.0.1 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv [notice] A new release of pip available: 22.3.1 -> 23.0 [notice] To update, run: pip install --upgrade pip 2023-02-06 13:39:15.845 INFO Importing actions package robusta.core.playbooks.internal 2023-02-06 13:39:15.846 INFO importing actions from robusta.core.playbooks.internal.discovery_events 2023-02-06 13:39:15.849 INFO Importing actions package robusta_playbooks 2023-02-06 13:39:15.850 INFO importing actions from robusta_playbooks.alerts_integration 2023-02-06 13:39:15.852 INFO importing actions from robusta_playbooks.argo_cd 2023-02-06 13:39:15.853 INFO importing actions from robusta_playbooks.autoscaler 2023-02-06 13:39:15.855 INFO importing actions from robusta_playbooks.babysitter 2023-02-06 13:39:15.856 INFO importing actions from robusta_playbooks.bash_enrichments 2023-02-06 13:39:15.858 INFO importing actions from robusta_playbooks.chaos_engineering 2023-02-06 13:39:15.859 INFO importing actions from robusta_playbooks.common_actions 2023-02-06 13:39:15.860 INFO importing actions from robusta_playbooks.configuration_ab_testing 2023-02-06 13:39:15.861 INFO importing actions from robusta_playbooks.cpu_throttling 2023-02-06 13:39:15.862 INFO importing actions from robusta_playbooks.daemonsets 2023-02-06 13:39:15.864 INFO importing actions from robusta_playbooks.deployment_enrichments 2023-02-06 13:39:15.865 INFO importing actions from robusta_playbooks.deployment_status_report 2023-02-06 13:39:15.867 INFO importing actions from robusta_playbooks.disk_benchmark 2023-02-06 13:39:15.868 INFO importing actions from robusta_playbooks.event_enrichments 2023-02-06 13:39:15.870 INFO importing actions from robusta_playbooks.git_change_audit 2023-02-06 13:39:15.871 INFO importing actions from robusta_playbooks.grafana_enrichment 2023-02-06 13:39:15.872 INFO importing actions from robusta_playbooks.http_actions 2023-02-06 13:39:15.873 INFO importing actions from robusta_playbooks.image_pull_backoff_enricher 2023-02-06 13:39:15.874 INFO importing actions from robusta_playbooks.java_pod_troubleshooting 2023-02-06 13:39:15.875 INFO importing actions from robusta_playbooks.job_actions 2023-02-06 13:39:15.878 INFO importing actions from robusta_playbooks.k8s_resource_enrichments 2023-02-06 13:39:15.879 INFO importing actions from robusta_playbooks.networking 2023-02-06 13:39:15.880 INFO importing actions from robusta_playbooks.node_cpu_analysis 2023-02-06 13:39:15.882 INFO importing actions from robusta_playbooks.node_disk_analysis 2023-02-06 13:39:15.883 INFO importing actions from robusta_playbooks.node_enrichments 2023-02-06 13:39:15.884 INFO importing actions from robusta_playbooks.oom_killer 2023-02-06 13:39:15.885 INFO importing actions from robusta_playbooks.overcommit_enrichments 2023-02-06 13:39:15.887 INFO importing actions from robusta_playbooks.persistent_data 2023-02-06 13:39:15.888 INFO importing actions from robusta_playbooks.persistent_volume_actions 2023-02-06 13:39:15.889 INFO importing actions from robusta_playbooks.pod_actions 2023-02-06 13:39:15.890 INFO importing actions from robusta_playbooks.pod_enrichments 2023-02-06 13:39:15.891 INFO importing actions from robusta_playbooks.pod_troubleshooting 2023-02-06 13:39:15.893 INFO importing actions from robusta_playbooks.prometheus_enrichments 2023-02-06 13:39:15.894 INFO importing actions from robusta_playbooks.prometheus_simulation 2023-02-06 13:39:15.895 INFO importing actions from robusta_playbooks.pvc_snapshots 2023-02-06 13:39:15.897 INFO importing actions from robusta_playbooks.restart_loop_reporter 2023-02-06 13:39:15.898 INFO importing actions from robusta_playbooks.silence 2023-02-06 13:39:15.900 INFO importing actions from robusta_playbooks.simple_examples 2023-02-06 13:39:15.901 INFO importing actions from robusta_playbooks.stress_tests 2023-02-06 13:39:15.902 INFO importing actions from robusta_playbooks.targetdown_enrichment 2023-02-06 13:39:15.903 INFO Importing actions package chatgpt_robusta_actions 2023-02-06 13:39:15.903 INFO importing actions from chatgpt_robusta_actions.chat_gpt 2023-02-06 13:43:35.709 INFO cluster status {'account_id': '1f8c96e6-9275-4e64-b81b-bcfbf444c5e3', 'cluster_id': 'atlantis', 'version': '0.10.10', 'last_alert_at': '2023-02-06 13:27:58.324266', 'light_actions': ['related_pods', 'prometheus_enricher', 'add_silence', 'delete_pod', 'delete_silence', 'get_silences', 'logs_enricher', 'pod_events_enricher', 'deployment_events_enricher', 'job_events_enricher', 'job_pod_enricher', 'get_resource_yaml', 'node_cpu_enricher', 'node_disk_analyzer', 'node_running_pods_enricher', 'node_allocatable_resources_enricher', 'node_status_enricher', 'node_graph_enricher', 'oomkilled_container_graph_enricher', 'pod_oom_killer_enricher', 'oom_killer_enricher', 'volume_analysis', 'python_profiler', 'pod_ps', 'python_memory', 'debugger_stack_trace', 'python_process_inspector', 'prometheus_alert', 'create_pvc_snapshot'], 'updated_at': 'now()'} 2023-02-06 13:49:05.862 INFO Successfully loaded Kubernetes resource example-pod for alert KubePodCrashLooping 2023-02-06 13:49:05.933 INFO Successfully loaded Kubernetes resource example-pod for alert KubePodCrashLooping 2023-02-06 13:49:05.984 INFO Successfully loaded Kubernetes resource example-pod for alert KubePodCrashLooping 2023-02-06 13:49:06.129 INFO Successfully loaded Kubernetes resource example-pod for alert KubePodCrashLooping 2023-02-06 13:50:37.598 INFO Callback `show_chat_gpt_search` chat_gpt_token: sk-***iyg, search_term: How***es?, model: tex***003 2023-02-06 13:50:37.598 INFO message='Request to OpenAI API' method=post path=https://api.openai.com/v1/completions 2023-02-06 13:50:50.303 INFO message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=12475 request_id=0acb93aa79a326c4cb8a198c776c0fc6 response_code=200 2023-02-06 13:58:45.228 INFO cluster status {'account_id': '1f8c96e6-9275-4e64-b81b-bcfbf444c5e3', 'cluster_id': 'atlantis', 'version': '0.10.10', 'last_alert_at': '2023-02-06 13:49:05.830388', 'light_actions': ['related_pods', 'prometheus_enricher', 'add_silence', 'delete_pod', 'delete_silence', 'get_silences', 'logs_enricher', 'pod_events_enricher', 'deployment_events_enricher', 'job_events_enricher', 'job_pod_enricher', 'get_resource_yaml', 'node_cpu_enricher', 'node_disk_analyzer', 'node_running_pods_enricher', 'node_allocatable_resources_enricher', 'node_status_enricher', 'node_graph_enricher', 'oomkilled_container_graph_enricher', 'pod_oom_killer_enricher', 'oom_killer_enricher', 'volume_analysis', 'python_profiler', 'pod_ps', 'python_memory', 'debugger_stack_trace', 'python_process_inspector', 'prometheus_alert', 'create_pvc_snapshot'], 'updated_at': 'now()'} 2023-02-06 14:02:28.674 INFO Successfully loaded Kubernetes resource crashpod-847bc5cbfb-ck7vt for alert KubePodCrashLooping 2023-02-06 14:02:28.750 INFO Successfully loaded Kubernetes resource crashpod-847bc5cbfb-ck7vt for alert KubePodCrashLooping 2023-02-06 14:02:28.909 INFO Successfully loaded Kubernetes resource crashpod-847bc5cbfb-ck7vt for alert KubePodCrashLooping 2023-02-06 14:02:29.118 INFO Successfully loaded Kubernetes resource crashpod-847bc5cbfb-ck7vt for alert KubePodCrashLooping 2023-02-06 14:02:50.043 INFO Callback `show_chat_gpt_search` chat_gpt_token: sk-***iyg, search_term: How***es?, model: tex***003 2023-02-06 14:02:50.044 INFO message='Request to OpenAI API' method=post path=https://api.openai.com/v1/completions 2023-02-06 14:03:03.224 INFO message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=12969 request_id=6d9da349012e5a4dcaad0666926ef953 response_code=200 2023-02-06 14:05:01.415 INFO for Event(involvedObject=ObjectReference(apiVersion='v1', fieldPath=None, kind='Pod', name='nginx', namespace='default', resourceVersion='706033', uid='ed9a23e5-9d31-4145-b9ed-35522f7fe1f0'), metadata=ObjectMeta(clusterName=None, creationTimestamp='2023-02-06T14:04:47Z', deletionGracePeriodSeconds=None, deletionTimestamp=None, generateName=None, generation=None, name='nginx.17414141d39a11dc', namespace='default', resourceVersion='706035', selfLink=None, uid='5dc65ce3-0742-46d8-a13b-b28c9056d15a', annotations={}, finalizers=[], labels={}, managedFields=[ManagedFieldsEntry(apiVersion='v1', fieldsType='FieldsV1', manager='kube-scheduler', operation='Update', time='2023-02-06T14:04:47Z', fieldsV1={'f:count': {}, 'f:firstTimestamp': {}, 'f:involvedObject': {}, 'f:lastTimestamp': {}, 'f:message': {}, 'f:reason': {}, 'f:source': {'f:component': {}}, 'f:type': {}})], ownerReferences=[]), action=None, apiVersion='v1', count=1, eventTime=None, firstTimestamp='2023-02-06T14:04:47Z', kind='Event', lastTimestamp='2023-02-06T14:04:47Z', message='Successfully assigned default/nginx to ip-172-31-40-146.ec2.internal', reason='Scheduled', related=None, reportingComponent='', reportingInstance='', series=None, source=EventSource(component='default-scheduler', host=None), type='Normal') got message: None 2023-02-06 14:05:01.416 INFO for Event(involvedObject=ObjectReference(apiVersion='v1', fieldPath='spec.containers{nginx}', kind='Pod', name='nginx', namespace='default', resourceVersion='706034', uid='ed9a23e5-9d31-4145-b9ed-35522f7fe1f0'), metadata=ObjectMeta(clusterName=None, creationTimestamp='2023-02-06T14:04:48Z', deletionGracePeriodSeconds=None, deletionTimestamp=None, generateName=None, generation=None, name='nginx.17414141fdbbec2d', namespace='default', resourceVersion='706092', selfLink=None, uid='e6a37cc3-1989-464a-903e-0d51c8a1b2f8', annotations={}, finalizers=[], labels={}, managedFields=[ManagedFieldsEntry(apiVersion='v1', fieldsType='FieldsV1', manager='kubelet', operation='Update', time='2023-02-06T14:04:48Z', fieldsV1={'f:count': {}, 'f:firstTimestamp': {}, 'f:involvedObject': {}, 'f:lastTimestamp': {}, 'f:message': {}, 'f:reason': {}, 'f:source': {'f:component': {}, 'f:host': {}}, 'f:type': {}})], ownerReferences=[]), action=None, apiVersion='v1', count=2, eventTime=None, firstTimestamp='2023-02-06T14:04:48Z', kind='Event', lastTimestamp='2023-02-06T14:05:00Z', message='Pulling image "nginxx"', reason='Pulling', related=None, reportingComponent='', reportingInstance='', series=None, source=EventSource(component='kubelet', host='ip-172-31-40-146.ec2.internal'), type='Normal') got message: None 2023-02-06 14:05:01.417 INFO for Event(involvedObject=ObjectReference(apiVersion='v1', fieldPath='spec.containers{nginx}', kind='Pod', name='nginx', namespace='default', resourceVersion='706034', uid='ed9a23e5-9d31-4145-b9ed-35522f7fe1f0'), metadata=ObjectMeta(clusterName=None, creationTimestamp='2023-02-06T14:04:48Z', deletionGracePeriodSeconds=None, deletionTimestamp=None, generateName=None, generation=None, name='nginx.17414142006594ee', namespace='default', resourceVersion='706041', selfLink=None, uid='cdd08682-214f-4f5a-8208-dc21f7befac5', annotations={}, finalizers=[], labels={}, managedFields=[ManagedFieldsEntry(apiVersion='v1', fieldsType='FieldsV1', manager='kubelet', operation='Update', time='2023-02-06T14:04:48Z', fieldsV1={'f:count': {}, 'f:firstTimestamp': {}, 'f:involvedObject': {}, 'f:lastTimestamp': {}, 'f:message': {}, 'f:reason': {}, 'f:source': {'f:component': {}, 'f:host': {}}, 'f:type': {}})], ownerReferences=[]), action=None, apiVersion='v1', count=1, eventTime=None, firstTimestamp='2023-02-06T14:04:48Z', kind='Event', lastTimestamp='2023-02-06T14:04:48Z', message='Failed to pull image "nginxx": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/nginxx:latest": failed to resolve reference "docker.io/library/nginxx:latest": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed', reason='Failed', related=None, reportingComponent='', reportingInstance='', series=None, source=EventSource(component='kubelet', host='ip-172-31-40-146.ec2.internal'), type='Warning') got message: failed to pull and unpack image "docker.io/library/nginxx:latest": failed to resolve reference "docker.io/library/nginxx:latest": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed 2023-02-06 14:05:01.423 INFO reason is: ImagePullBackoffReason.ImageDoesntExist|NotAuthorized 2023-02-06 14:05:41.819 INFO Successfully loaded Kubernetes resource nginx for alert KubePodErrImagePull 2023-02-06 14:05:41.919 INFO Successfully loaded Kubernetes resource nginx for alert KubePodErrImagePull 2023-02-06 14:05:42.081 INFO Successfully loaded Kubernetes resource nginx for alert KubePodErrImagePull 2023-02-06 14:06:07.179 INFO Callback `show_chat_gpt_search` chat_gpt_token: sk-***iyg, search_term: How***es?, model: tex***003 2023-02-06 14:06:07.180 INFO message='Request to OpenAI API' method=post path=https://api.openai.com/v1/completions 2023-02-06 14:06:15.296 INFO message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=7771 request_id=a1bfeec5493d830012d220a896292cfc response_code=200 2023-02-06 14:10:48.576 ERROR failed to get pod logs crashpod-847bc5cbfb-ck7vt default crashpod Traceback (most recent call last): File "/app/src/robusta/integrations/kubernetes/api_client_utils.py", line 154, in get_pod_logs resp = core_v1.read_namespaced_pod_log( File "/usr/local/lib/python3.9/site-packages/kubernetes/client/api/core_v1_api.py", line 22928, in read_namespaced_pod_log return self.read_namespaced_pod_log_with_http_info(name, namespace, **kwargs) # noqa: E501 File "/usr/local/lib/python3.9/site-packages/kubernetes/client/api/core_v1_api.py", line 23043, in read_namespaced_pod_log_with_http_info return self.api_client.call_api( File "/usr/local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 348, in call_api return self.__call_api(resource_path, method, File "/usr/local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 180, in __call_api response_data = self.request( File "/usr/local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 373, in request return self.rest_client.GET(url, File "/usr/local/lib/python3.9/site-packages/kubernetes/client/rest.py", line 239, in GET return self.request("GET", url, File "/usr/local/lib/python3.9/site-packages/kubernetes/client/rest.py", line 233, in request raise ApiException(http_resp=r) kubernetes.client.exceptions.ApiException: (400) Reason: Bad Request HTTP response headers: HTTPHeaderDict({'Audit-Id': '6c0e9aac-e1dc-4bdd-a89e-74d8a5a32d53', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Mon, 06 Feb 2023 14:10:48 GMT', 'Content-Length': '204'}) HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"previous terminated container \\"crashpod\\" in pod \\"crashpod-847bc5cbfb-ck7vt\\" not found","reason":"BadRequest","code":400}\n'  2023-02-06 14:10:48.627 ERROR could not fetch logs from container: crashpod. logs were  2023-02-06 14:13:55.841 INFO cluster status {'account_id': '1f8c96e6-9275-4e64-b81b-bcfbf444c5e3', 'cluster_id': 'atlantis', 'version': '0.10.10', 'last_alert_at': '2023-02-06 14:05:41.728178', 'light_actions': ['related_pods', 'prometheus_enricher', 'add_silence', 'delete_pod', 'delete_silence', 'get_silences', 'logs_enricher', 'pod_events_enricher', 'deployment_events_enricher', 'job_events_enricher', 'job_pod_enricher', 'get_resource_yaml', 'node_cpu_enricher', 'node_disk_analyzer', 'node_running_pods_enricher', 'node_allocatable_resources_enricher', 'node_status_enricher', 'node_graph_enricher', 'oomkilled_container_graph_enricher', 'pod_oom_killer_enricher', 'oom_killer_enricher', 'volume_analysis', 'python_profiler', 'pod_ps', 'python_memory', 'debugger_stack_trace', 'python_process_inspector', 'prometheus_alert', 'create_pvc_snapshot'], 'updated_at': 'now()'} 2023-02-06 14:26:02.827 INFO Error loading kubernetes pod default/atlantis-0. reason: Not Found status: 404 2023-02-06 14:26:02.844 INFO Error loading kubernetes pod default/atlantis-0. reason: Not Found status: 404 2023-02-06 14:26:17.223 INFO Error loading kubernetes statefulset default/atlantis. reason: Not Found status: 404 2023-02-06 14:26:17.238 INFO Error loading kubernetes statefulset default/atlantis. reason: Not Found status: 404 2023-02-06 14:29:04.390 INFO cluster status {'account_id': '1f8c96e6-9275-4e64-b81b-bcfbf444c5e3', 'cluster_id': 'atlantis', 'version': '0.10.10', 'last_alert_at': '2023-02-06 14:26:17.203103', 'light_actions': ['related_pods', 'prometheus_enricher', 'add_silence', 'delete_pod', 'delete_silence', 'get_silences', 'logs_enricher', 'pod_events_enricher', 'deployment_events_enricher', 'job_events_enricher', 'job_pod_enricher', 'get_resource_yaml', 'node_cpu_enricher', 'node_disk_analyzer', 'node_running_pods_enricher', 'node_allocatable_resources_enricher', 'node_status_enricher', 'node_graph_enricher', 'oomkilled_container_graph_enricher', 'pod_oom_killer_enricher', 'oom_killer_enricher', 'volume_analysis', 'python_profiler', 'pod_ps', 'python_memory', 'debugger_stack_trace', 'python_process_inspector', 'prometheus_alert', 'create_pvc_snapshot'], 'updated_at': 'now()'} 2023-02-06 14:34:53.956 INFO Successfully loaded Kubernetes resource nginx for alert KubePodOomKilled 2023-02-06 14:34:54.084 INFO Successfully loaded Kubernetes resource nginx for alert KubePodOomKilled 2023-02-06 14:34:54.228 INFO Successfully loaded Kubernetes resource nginx for alert KubePodOomKilled 2023-02-06 14:35:22.615 INFO Callback `show_chat_gpt_search` chat_gpt_token: sk-***iyg, search_term: How***es?, model: tex***003 2023-02-06 14:35:22.615 INFO message='Request to OpenAI API' method=post path=https://api.openai.com/v1/completions 2023-02-06 14:35:33.114 INFO message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=10261 request_id=bdc3f77b72b786abd854a2c8831ba06d response_code=200 2023-02-06 14:44:14.738 INFO cluster status {'account_id': '1f8c96e6-9275-4e64-b81b-bcfbf444c5e3', 'cluster_id': 'atlantis', 'version': '0.10.10', 'last_alert_at': '2023-02-06 14:34:53.879151', 'light_actions': ['related_pods', 'prometheus_enricher', 'add_silence', 'delete_pod', 'delete_silence', 'get_silences', 'logs_enricher', 'pod_events_enricher', 'deployment_events_enricher', 'job_events_enricher', 'job_pod_enricher', 'get_resource_yaml', 'node_cpu_enricher', 'node_disk_analyzer', 'node_running_pods_enricher', 'node_allocatable_resources_enricher', 'node_status_enricher', 'node_graph_enricher', 'oomkilled_container_graph_enricher', 'pod_oom_killer_enricher', 'oom_killer_enricher', 'volume_analysis', 'python_profiler', 'pod_ps', 'python_memory', 'debugger_stack_trace', 'python_process_inspector', 'prometheus_alert', 'create_pvc_snapshot'], 'updated_at': 'now()'}