[Concourse] VSCode extension doesn't understand glob patterns in group config

[ringods]

Describe the bug
The Concourse extension in VSCode complains on glob patterns in groups configuration.

https://concourse-ci.org/pipelines.html#schema.group_config

To Reproduce

Use the sample from below. Although incomplete to be used in a real Concourse setup, it shows the error being reported clearly:

On top of that, the reverse problem of a job not being part of a group is also reported:

Sample

groups:
- name: build
  jobs:
    - "ci-*"

jobs:
- name: ci-project-one
  plan:
  - task: gradle-build
    file: gradle-build-dcind.yml

[kdvolder]

I looked at the documentation that is linked but I do not see anything there that says you can use glob patterns in this way. Nor does the doc have any detailed explanation about exact syntax and interpretation of the glob patterns.

Nevertheless I don't think it should be too hard to support something like this. @ringods if you have any more detailed information about this feature and how it works that might be helpful.

In the mean time I'll just play around with a bit and add support for simple string/glob patterns containing one ore more *.

[kdvolder]

I tried this out on our concourse instance and it seems the fly CLI doesn't like the use of these glob patterns.

Here's the output from my little test:

$ fly -t tools set-pipeline -p test-kdvolder -c pipeline.yml 
groups:
  group build has been added:
+ jobs:
+ - ci-*
+ name: build
  
jobs:
  job ci-project-one has been added:
+ name: ci-project-one
+ plan:
+ - file: gradle-build-dcind.yml
+   task: gradle-build
  
apply configuration? [yN]: y
error: invalid pipeline config:
invalid groups:
	group 'build' has unknown job 'ci-*'

Is this perhaps a rather new feature in concourse? If so, it may be that the concourse instance that I have access to doesn't support it yet.

We can definitely implement support for this if this is indeed a real feature. But neither the doc nor the fly cli / concourse version that I tested this with seems to support this. So in that case the editor really shouldn't support it either.

The version of concourse I tested with is v6.7.4.

[kdvolder]

Digging through concourse release notes: https://github.com/concourse/concourse/releases

I found that this feature seems to have been added in version 7.0.0. Corresponding issue ticket in concourse:

concourse/concourse#6060

[kdvolder]

Sadly the glob syntax appears to be rather complex as it uses a package called 'gobwas' found here: https://github.com/gobwas/glob

As our language server is implemented in Java that is unfortunately not a library I can use. So there's a bit of dilemma here as I really don't want to implement a copy of globwas package in Java myself. I googled around a bit to try and find a similar package for Java but didn't immediately find one.

So what I will probably do instead is:

• support only simple patterns that contain just the use of a single *
• try and detect whether more complex pattern is being used and, if so, disable the validations related to groups (i.e. so we don't get false error/warnings in this case).

Looking through examples on globwas readme I made a list of 'special' characters. Seeing any of these characters in a pattern could be taken as a hint that 'this looks like a too complex pattern for the editor to handle': {, }, [, !, ?.

[ringods]

Hello @kdvolder, the official docs for this grouping can be found here:

https://concourse-ci.org/pipelines.html#schema.group_config

If you click on the arrow left of jobs: [job.name], the description on how to use it becomes visible.

EDIT: added screenshot of docs. glob link points to: https://www.man7.org/linux/man-pages/man7/glob.7.html

(Side question: do we know each other from VUB? 😉 )

[kdvolder]

Yeah I did click there (I think) and a few other places but somehow I still cannot find any hint anywhere of using glob patterns in job names. No matter, I think I know enough from the issue ticket I dug out to make sense of it.

And about VUB.... yes indeed. What a small world! :-)

[kdvolder]

This should now be fixed in latest snapshot. If you want to try it out it can be downloaded from here: http://dist.springsource.com/snapshot/STS4/nightly-distributions.html

I implemented my own simple glob pattern matcher which only supports the use of '*'. As it wasn't too hard to do, it actually does support the use of multiple stars in a single pattern. Any other types of patterns are not supported. When the validation sees them it basically just assumes that it is a 'complex' pattern which it doesn't know how to match. So it conservatively assumes the pattern could match anything. The effect of that is that all validation checks around these patterns will automatically pass and there will be no false errors (but also any real errors will be supressed).

More patterns could be supported in the future either by making the implementation of the glob matcher more complete or by finding some 'good enough' Java library that can be used to replace the simple matcher and is considered as 'close enoungh' in terms of supporting similar syntax as concourse's library does.

Note: Java actually has a built-in glob matcher (https://docs.oracle.com/javase/8/docs/api/java/nio/file/PathMatcher.html). I didn't like to use it because it seems to be intended specifically for matching file names and paths, and job names are not file names.