Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Fetch intermediate log async GKEStartPod #39348

Merged
merged 5 commits into from
May 29, 2024

Conversation

pankajastro
Copy link
Member

@pankajastro pankajastro commented May 1, 2024

This PR introduces a parameter that enables the retrieval of intermediate logs for the GKEStartPod asynchronous operator.

  • Add param last_log_time and logging_interval in GKEStartPodTrigger serialize
  • Add optional param last_log_time in method invoke_defer_method

Example DAG:

start_pod = GKEStartPodOperator(
        task_id="start_pod",
        project_id=PROJECT_ID,
        location=LOCATION,
        cluster_name=GKE_CLUSTER_NAME,
        do_xcom_push=True,
        namespace=GKE_NAMESPACE,
        image="ubuntu:jammy",
        cmds=["sh", "-c", "timeout 300 bash -c 'while true; do echo \"meow\"; sleep 30; done'"],
        name="test-sleep",
        in_cluster=False,
        on_finish_action="delete_pod",
        deferrable=True,
        get_logs=True,
        logging_interval=5,
        gcp_conn_id=GCP_CONN_ID
    )
Screenshot 2024-05-01 at 6 53 50 PM

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@boring-cyborg boring-cyborg bot added area:providers provider:cncf-kubernetes Kubernetes provider related issues provider:google Google (including GCP) related issues labels May 1, 2024
@pankajastro pankajastro marked this pull request as ready for review May 1, 2024 13:32
@pankajastro pankajastro force-pushed the logging_interval_gke branch from 9dd6061 to e594417 Compare May 1, 2024 13:43
Copy link
Contributor

@dirrao dirrao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Can you add the test case with last_log_time and last_log_time?

@jonathonbattista
Copy link

Any updates on this?

@pankajastro
Copy link
Member Author

Any updates on this?

I'll try address the pending review by end of this week

@jonathonbattista
Copy link

Bump on this 🙏

@jonathonbattista
Copy link

@pankajastro Any ETA on this? Thanks

@pankajastro pankajastro force-pushed the logging_interval_gke branch from 786ba83 to 3c60132 Compare May 24, 2024 19:42
@pankajastro
Copy link
Member Author

@pankajastro Any ETA on this? Thanks

CI is green now. Hopefully, we will merge soon. In the meantime, could you please test it once?

@pankajastro pankajastro merged commit a2c09d2 into apache:main May 29, 2024
41 checks passed
@pankajastro pankajastro deleted the logging_interval_gke branch May 29, 2024 12:39
fdemiane pushed a commit to fdemiane/airflow that referenced this pull request Jun 6, 2024
* Fetch intermediate log in async GKEStartPod

This PR introduces a parameter that enables the retrieval of intermediate logs for the GKEStartPod asynchronous operator.

Add param last_log_time and logging_interval in GKEStartPodTrigger serialize
Add optional param last_log_time in method invoke_defer_method
Example DAG:

start_pod = GKEStartPodOperator(
        task_id="start_pod",
        project_id=PROJECT_ID,
        location=LOCATION,
        cluster_name=GKE_CLUSTER_NAME,
        do_xcom_push=True,
        namespace=GKE_NAMESPACE,
        image="ubuntu:jammy",
        cmds=["sh", "-c", "timeout 300 bash -c 'while true; do echo \"meow\"; sleep 30; done'"],
        name="test-sleep",
        in_cluster=False,
        on_finish_action="delete_pod",
        deferrable=True,
        get_logs=True,
        logging_interval=5,
        gcp_conn_id=GCP_CONN_ID
    )
romsharon98 pushed a commit to romsharon98/airflow that referenced this pull request Jul 26, 2024
* Fetch intermediate log in async GKEStartPod

This PR introduces a parameter that enables the retrieval of intermediate logs for the GKEStartPod asynchronous operator.

Add param last_log_time and logging_interval in GKEStartPodTrigger serialize
Add optional param last_log_time in method invoke_defer_method
Example DAG:

start_pod = GKEStartPodOperator(
        task_id="start_pod",
        project_id=PROJECT_ID,
        location=LOCATION,
        cluster_name=GKE_CLUSTER_NAME,
        do_xcom_push=True,
        namespace=GKE_NAMESPACE,
        image="ubuntu:jammy",
        cmds=["sh", "-c", "timeout 300 bash -c 'while true; do echo \"meow\"; sleep 30; done'"],
        name="test-sleep",
        in_cluster=False,
        on_finish_action="delete_pod",
        deferrable=True,
        get_logs=True,
        logging_interval=5,
        gcp_conn_id=GCP_CONN_ID
    )
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
area:providers provider:cncf-kubernetes Kubernetes provider related issues provider:google Google (including GCP) related issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants