Workflow in pending phase possibly not reported by argo workflows REST API as pending?

[yousefmoazzam]

Hi all 🙂

In working with the argo workflows REST API in an application, and specifically, testing this application against the different phases a workflow can have to ensure we handle the different cases correctly, we came across some unexpected behaviour that I'd like to ask the community for some input.

One piece of info to mention is that we're working with argo workflows v3.6.2 when we observed this behaviour.

To test against a pending workflow, we wanted to somehow deterministically get a submitted workflow in a pending state (not a task within a workflow being pending, but the workflow itself). One of our team members suggested taking down the workflow controller by scaling the number of replicas in the ReplicaSet down to 0, such that the workflow would never be picked up by the controller.

After doing this and submitting a workflow, it indeed looked like the workflow was pending from the web UI, in the sense that the single task inside it didn't even get to a pending phase, and the workflow wasn't in the "Running" phase. We thought this did the trick, and queried the argo workflows REST API for that particular workflow using the /api/v1/workflows/{namespace}/{name} endpoint, expecting to see the status.phase field with the value "Pending".

However, the response we got back was unexpectedly missing the status.phase field altogether. The full response was the following:

{
  "metadata": {
    "name": "numpy-benchmark-pwtgn",
    "generateName": "numpy-benchmark-",
    "namespace": "mg36964-1",
    "uid": "fcffd6b5-aeb5-4228-98a1-ececfe0563c7",
    "resourceVersion": "233729873",
    "generation": 1,
    "creationTimestamp": "2025-01-22T16:38:08Z",
    "labels": {
      "submit-from-ui": "true",
      "workflows.argoproj.io/cluster-workflow-template": "numpy-benchmark",
      "workflows.argoproj.io/creator": "d2cc5e04-14e7-453b-9622-e20bea96ccf5",
      "workflows.argoproj.io/creator-email": "yousef.moazzam.at.diamond.ac.uk",
      "workflows.argoproj.io/creator-preferred-username": "twi18192",
      "workflows.argoproj.io/resubmitted-from-workflow": "numpy-benchmark-k8jpn"
    },
    "annotations": {
      "workflows.argoproj.io/pod-name-format": "v2"
    },
    "managedFields": [
      {
        "manager": "argo",
        "operation": "Update",
        "apiVersion": "argoproj.io/v1alpha1",
        "time": "2025-01-22T16:38:08Z",
        "fieldsType": "FieldsV1",
        "fieldsV1": {
          "f:metadata": {
            "f:annotations": {
              ".": {},
              "f:workflows.argoproj.io/pod-name-format": {}
            },
            "f:generateName": {},
            "f:labels": {
              ".": {},
              "f:submit-from-ui": {},
              "f:workflows.argoproj.io/cluster-workflow-template": {},
              "f:workflows.argoproj.io/creator": {},
              "f:workflows.argoproj.io/creator-email": {},
              "f:workflows.argoproj.io/creator-preferred-username": {},
              "f:workflows.argoproj.io/resubmitted-from-workflow": {}
            }
          },
          "f:spec": {},
          "f:status": {}
        }
      }
    ]
  },
  "spec": {
    "entrypoint": "numpy-test",
    "arguments": {
      "parameters": [
        {
          "name": "size",
          "value": "1000"
        },
        {
          "name": "memory",
          "value": "10Gi"
        }
      ]
    },
    "podGC": {
      "strategy": "OnPodCompletion",
      "deleteDelayDuration": "60s"
    },
    "workflowTemplateRef": {
      "name": "numpy-benchmark",
      "clusterScope": true
    }
  },
  "status": {
    "startedAt": null,
    "finishedAt": null,
    "storedTemplates": {
      "cluster/numpy-benchmark/numpy-test": {
        "name": "numpy-test",
        "inputs": {
          "parameters": [
            {
              "name": "size",
              "value": "2000"
            },
            {
              "name": "memory",
              "value": "20Gi"
            }
          ]
        },
        "outputs": {},
        "metadata": {},
        "script": {
          "name": "",
          "image": "gcr.io/diamond-privreg/ptypy/test_openmpi_full:0.1",
          "command": [
            "python"
          ],
          "env": [
            {
              "name": "MKL_NUM_THREADS",
              "value": "1"
            },
            {
              "name": "NUMEXPR_NUM_THREADS",
              "value": "1"
            },
            {
              "name": "OMP_NUM_THREADS",
              "value": "1"
            }
          ],
          "resources": {},
          "source": "import numpy as np\nimport time\n\nn = int(\"{{ inputs.parameters.size }}\")\nA = np.random.randn(n,n).astype('float64')\nB = np.random.randn(n,n).astype('float64')\nstart_time = time.time()\nnrm = np.linalg.norm(A@B)\nprint(\" took {} seconds \".format(time.time() - start_time))\nprint(\" norm = \",nrm)\nprint(np.__config__.show())\n"
        },
        "podSpecPatch": "containers:\n- name: main\n  resources:\n    requests:\n      cpu: \"1\"\n      memory: \"{{ inputs.parameters.memory }}\"\n    limits:\n      cpu: \"1\"\n      memory: \"{{ inputs.parameters.memory }}\"\n"
      }
    }
  }
}

I wanted to ask if anyone else had seen this behaviour of the argo workflows REST API regarding a workflow being in a pending phase? Was this workflow in a pending phase, but the argo workflows REST API was producing an incorrect response? Was the strategy of "taking down" the workflow controller not a valid way to get a submitted workflow into a pending phase? Or any other possibilities?

Let me know if I should provide any other info, thanks!

[tczhao]

This behaviour is expected.
status.phase is empty by default until controller performs operation on it.
You can try force pending workflow by setting parallelism limit to 0

[yousefmoazzam]

Thanks for the reply!

We tried setting parallelism to zero (tried both the value "0" and 0), and observed that a submitted workflow ran to completion. We confirmed that the configmap generated during redeployment had the parallelism field set, so on a first glance, it looks like that field is perhaps not being adhered to. Any ideas?