SEP 38: a stupid idea for a pillar fileserver.

[whytewolf]

I know this isn't the most intelligent idea. and is not fully fleshed out currently. I welcome all criticisms. including the bad English.

[welcome]

Hello! Thank you for submitting a Salt Enhancement Proposal! Our process is detailed in the README.md and more about the SEP Life-cycle. An Open Core Team member will be assigned to follow up and help guide this SEP soon and you will find the this in the Community Slack channel #sep.
Please be sure to review our Code of Conduct.
You can also check out some of our community
resources:
- Community Wiki
- Salt’s Contributor Guide
- Join our Community Slack
- IRC on LiberaChat
- SaltStack YouTube channel
- SaltStackInc Twitch channel

[waynew]

This is SEP 38 - feel free to update the contents of your file!

[waynew]

I probably need to read through this a few times but... I actually really like the fundamental idea happening here.

I've noticed as I've dug into several of the bits of Salt that there are places where we should just have a nice, well-defined API, and the backing service is just kinda like 🤷 cool whatever.

But we end out with some weird tweaks instead, for particular backing systems.

It would be pretty amazing, though, to have a more generic fileserver for pillars (and more?)

[dmurphy18]

Made some points about use of pygit2 and GitPython in associated originating issue saltstack/salt#61508, which is closed since all discussion should occur here

[whytewolf]

Those points you made in saltstack/salt#61508 are awesome for the git pfs, When i get some time I'll look into adding those into the draft, in part with examples of how to work with the fileserver from the backends perspective.

[dmurphy18]

@whytewolf Thanks, thought - with Tiamat we can control the versions of pygit2/libgit2 used, so the issues encountered with the rpm/deb packages and the environments in use should be mitigated. So the SEP might only apply for Tiamat based functionality which implies a parting of the ways from regular OS distributed packages.

[whytewolf]

well this sep actually covers a lot more then just git_pillar. it separates out the file server fetch from the render completely. and adds an interface over top that lets any backend that works with files be used as a root pillar filesystem much like the normal salt fileserver. calling out the git functionality directly is a nice to have but it isn't the main focus. so much of the problems your describing with libgit and all that were never a consideration for this SEP.

This SEP was born more out of having 500 different ways of doing what amounts to the same thing with regards to handling pillar in different systems. forcing a splitting of tool functionality just to handle all the cases and not everything working together in a cohesive manner.

that being said by the time it is implemented I am sure we will be on a tiamat build as the core system anyway. so I do expect the libgit issues to be covered by then.

and as @waynew pointed out this kind of starts down a solid path of removing hacks and starting to work more towards functional APIs. as well as allowing more entry points for salt extensions.