Skip to content

Commit 96876d5

Browse files
promiseofcakeandyhorner-chime
promiseofcake
and
andyhorner-chime
authoredApr 16, 2024
V3 (#17)
* Intrapipeline Queueing (#16) * documentation: update comments * feat: add internal queue * Job, Command, Renames / Recontextulization (#18) * feat: renames * feat: add ability to ignore specific workflows * fix: formatting * fix: shellcheck * fix: job name * fix: shellcheck * Validate Queue Behavior (#19) * feat: ensure we can validate the correct ordering of jobs * fix: rename to the correct pipeline to ignore * debugging * fix: debug * walk back validation * feat: update documentation * Update src/jobs/global-queue.yml Co-authored-by: Andy Horner <114095421+andyhorner-chime@users.noreply.github.com> * Update src/commands/global_block.yml Co-authored-by: Andy Horner <114095421+andyhorner-chime@users.noreply.github.com> --------- Co-authored-by: Andy Horner <114095421+andyhorner-chime@users.noreply.github.com>
1 parent 369a5f8 commit 96876d5

16 files changed

+409
-161
lines changed
 

‎.circleci/test-deploy.yml

+34-8
Original file line numberDiff line numberDiff line change
@@ -7,25 +7,51 @@ filters: &filters
77
tags:
88
only: /.*/
99

10+
jobs:
11+
sleep:
12+
docker:
13+
- image: cimg/base:edge
14+
parameters:
15+
time:
16+
type: integer
17+
steps:
18+
- run: echo "sleeping << parameters.time >>s" && sleep << parameters.time >>
19+
1020
workflows:
11-
test-deploy:
21+
test-pipeline-a:
1222
jobs:
13-
- orb-tools/pack:
14-
filters: *filters
15-
- workflow-queue/queue:
23+
- workflow-queue/pipeline-queue:
24+
context: orb-publishing
25+
debug: true
26+
test-pipeline-b:
27+
jobs:
28+
- sleep:
29+
time: 5
30+
test-pipeline-c:
31+
jobs:
32+
- sleep:
33+
time: 20
34+
35+
test-global-queue:
36+
jobs:
37+
- workflow-queue/global-queue:
38+
context: orb-publishing
39+
debug: true
40+
ignored-workflows: "test-pipeline-a" # given we are in one pipeline, don't block on ourselves
1641
filters:
1742
tags:
1843
ignore: /.*/
19-
context: orb-publishing
20-
requires: [orb-tools/pack]
44+
test-deploy:
45+
jobs:
46+
- orb-tools/pack:
47+
filters: *filters
2148
- orb-tools/publish:
49+
context: orb-publishing
2250
orb-name: promiseofcake/workflow-queue
2351
vcs-type: << pipeline.project.type >>
2452
pub-type: production
2553
requires:
2654
- orb-tools/pack
27-
- workflow-queue/queue
28-
context: orb-publishing
2955
filters:
3056
branches:
3157
ignore: /.*/

‎Makefile

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
test-global-queue:
2+
./src/scripts/test.sh global-queue.sh
3+
4+
test-pipeline-queue:
5+
./src/scripts/test.sh pipeline-queue.sh

‎README.md

+4-2
Original file line numberDiff line numberDiff line change
@@ -4,9 +4,11 @@
44

55
## Introduction
66

7-
Forked from <https://github.com/eddiewebb/circleci-queue> and updated to reduce the use-cases, and migrate to the CircleCI V2 API
7+
Originally forked from <https://github.com/eddiewebb/circleci-queue> and updated to reduce the use-cases, and migrate to the CircleCI V2 API
88

9-
The purpose of this Orb is to add a concept of a queue to specific branch workflow tasks in CircleCi. The main use-case is to isolate a set of changes to ensure that one set of a thing is running at one time. Think of smoke-tests against a nonproduction environment as a promotion gate.
9+
The purpose of this Orb is to add a concept of a queue to specific branch's workflow tasks in CircleCi. The main use-case is to isolate a set of changes to ensure that one set of a thing is running at one time. Think of smoke-tests against a nonproduction environment as a promotion gate.
10+
11+
Additional use-cases are for queueing workflows within a given pipeline (a feature missing today from CircleCi).
1012

1113
## Configuration Requirements
1214

‎src/README.md

-26
This file was deleted.

‎src/commands/block_execution.yml

-43
This file was deleted.

‎src/commands/global_block.yml

+48
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
description: >
2+
When used, this command blocks the assigned workflow from running until all previous (global) workflows have completed.
3+
This ensures only one instance of a given workflow is running at a time across all defined branches.
4+
parameters:
5+
debug:
6+
type: boolean
7+
default: false
8+
description: "When enabled, additional debug logging with be output."
9+
time:
10+
type: string
11+
default: "10"
12+
description: "Number of minutes to wait for a lock before giving up."
13+
dont-quit:
14+
type: boolean
15+
default: false
16+
description: "If true, forces the job through once time expires instead of failing."
17+
only-on-branch:
18+
type: string
19+
default: "*"
20+
description: "Only queue on specified branch"
21+
confidence:
22+
type: string
23+
default: "1"
24+
description: >
25+
Due to concurrency issues, the number of times should we requery the pipeline list to ensure previous jobs are "pending",
26+
but not yet active. This number indicates the threshold for API returning no previous pending pipelines.
27+
Default is `1` confirmation, increase if you see issues.
28+
ignored-workflows:
29+
type: string
30+
default: ""
31+
description: Comma separated list of workflow names to ignore as blocking workflows for the global queue.
32+
include-on-hold:
33+
type: boolean
34+
default: false
35+
description: Consider `on-hold` workflows waiting for approval as running and include them in the queue.
36+
37+
steps:
38+
- run:
39+
name: Block execution until the current workflow is at the front of the line
40+
environment:
41+
CONFIG_DEBUG_ENABLED: "<< parameters.debug >>"
42+
CONFIG_TIME: "<< parameters.time >>"
43+
CONFIG_DONT_QUIT: "<< parameters.dont-quit >>"
44+
CONFIG_ONLY_ON_BRANCH: "<< parameters.only-on-branch >>"
45+
CONFIG_CONFIDENCE: "<< parameters.confidence >>"
46+
CONFIG_IGNORED_WORKFLOWS: "<< parameters.ignored-workflows >>"
47+
CONFIG_INCLUDE_ON_HOLD: " << parameters.include-on-hold >>"
48+
command: <<include(scripts/global-queue.sh)>>

‎src/commands/pipeline_block.yml

+23
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
description: >
2+
This command blocks execution of a workflow within the context of a given pipeline.
3+
4+
parameters:
5+
debug:
6+
type: boolean
7+
default: false
8+
description: "When enabled, additional debug logging with be output."
9+
confidence:
10+
type: string
11+
default: "1"
12+
description: >
13+
Due to concurrency issues, the number of times should we requery the pipeline list to ensure previous jobs are "pending",
14+
but not yet active. This number indicates the threshold for API returning no previous pending pipelines.
15+
Default is `1` confirmation, increase if you see issues.
16+
17+
steps:
18+
- run:
19+
name: Blocking execution until the current workflow is the last to run in the given pipeline.
20+
environment:
21+
CONFIG_DEBUG_ENABLED: "<< parameters.debug >>"
22+
CONFIG_CONFIDENCE: "<< parameters.confidence >>"
23+
command: <<include(scripts/pipeline-queue.sh)>>

‎src/examples/example.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ description: >
33
usage:
44
version: 2.1
55
orbs:
6-
workflow-queue: promiseofcake/workflow-queue@1
6+
workflow-queue: promiseofcake/workflow-queue@2
77
workflows:
88
example:
99
jobs:

‎src/jobs/global-queue.yml

+49
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
description: >
2+
This job ensures only a single defined global workflow is running at a given point in time.
3+
4+
docker:
5+
- image: cimg/base:stable
6+
resource_class: small
7+
8+
parameters:
9+
debug:
10+
type: boolean
11+
default: false
12+
description: "When enabled, additional debug logging with be output."
13+
time:
14+
type: string
15+
default: "10"
16+
description: "Number of minutes to wait for a lock before giving up."
17+
dont-quit:
18+
type: boolean
19+
default: false
20+
description: "If true, forces the job through once time expires instead of failing."
21+
only-on-branch:
22+
type: string
23+
default: "*"
24+
description: "Only queue on specified branch"
25+
confidence:
26+
type: string
27+
default: "1"
28+
description: >
29+
Due to concurrency issues, the number of times should we requery the pipeline list to ensure previous jobs are "pending",
30+
but not yet active. This number indicates the threshold for API returning no previous pending pipelines.
31+
Default is `1` confirmation, increase if you see issues.
32+
ignored-workflows:
33+
type: string
34+
default: ""
35+
description: Comma separated list of workflow names to ignore as blocking workflows for the global queue.
36+
include-on-hold:
37+
type: boolean
38+
default: false
39+
description: Consider `on-hold` workflows waiting for approval as running and include them in the queue.
40+
41+
steps:
42+
- global_block:
43+
debug: << parameters.debug >>
44+
time: << parameters.time >>
45+
dont-quit: << parameters.dont-quit >>
46+
only-on-branch: << parameters.only-on-branch >>
47+
confidence: << parameters.confidence >>
48+
ignored-workflows: << parameters.ignored-workflows >>
49+
include-on-hold: << parameters.include-on-hold >>

‎src/jobs/pipeline-queue.yml

+24
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
description: >
2+
This job prevents a workflow within a given pipeline from running until all previous workflows have completed.
3+
4+
docker:
5+
- image: cimg/base:stable
6+
resource_class: small
7+
8+
parameters:
9+
debug:
10+
type: boolean
11+
default: false
12+
description: "If enabled, DEBUG messages will be logged."
13+
confidence:
14+
type: string
15+
default: "1"
16+
description: >
17+
Due to concurrency issues, how many times should we requery the pipeline list to ensure previous jobs are "pending",
18+
but not yet active. This number indicates the threshold for API returning no previous pending pipelines.
19+
Default is one confirmation, increase if you see issues.
20+
21+
steps:
22+
- pipeline_block:
23+
debug: <<parameters.debug>>
24+
confidence: <<parameters.confidence>>

‎src/jobs/queue.yml

-44
This file was deleted.

0 commit comments

Comments
 (0)