Support external repositories and move supported packages to one

[natanjunges]

This is a long-term proposal.

The list of supported packages is growing very fast, and hardcoding it into the main script does not scale well:

• Loading metadata of all supported packages when only a few are installed could inflate the package size and degrade performance;
• Changes in packages are only available in the next release;
• Changes in packages get mixed with changes in the actual script;
• As mentioned in Improve the way software is listed #269, the README gets polluted with the list of supported packages.

To address those issues, I propose adding support to external repositories, and moving all supported packages to one. A repository would consist of:

• A manifest file listing all supported packages;
• A script file for each supported package.

When a repository is added, only the manifest file is downloaded. The script for a certain package would only be downloaded when required. It would also only be loaded in the main script when required, as the manifest can be used to get the list of supported packages. The wrapping deb_ functions could even be removed from the scripts.

[flexiondotorg]

I agree that we need to separate the package definitions from the main script 👍 Making a monolithic script was a conscious choice at the very beginning, in order to quickly bootstrap the project.

Also, I did not anticipate just how many packages would be available to include in deb-get 😁

I'd like to make one exception, and that is for deb-get itself. I want deb-get to be able to install/update itself without needing to query an "external" manifest.

I would also like to get Debian and Codename support in deb-get before we work on this

[natanjunges]

Yes, I agree that deb-get should update itself.

[Fuseteam]

i was just thinking filing an issue about this xd

[natanjunges]

Nice, if you have some suggestions, feel free to share.

[Fuseteam]

i do have an idea but i think i need to work it out first

[natanjunges]

@flexiondotorg after #466 is merged I believe everything related to Debian and codename support should be sorted out, so I'll start working on this implementation.

[natanjunges]

@flexiondotorg an implementation is available at https://github.com/natanjunges/deb-get/tree/extrepo, but I'll wait for all other pending PR's to be merged before opening this one, so all conflicts are sorted out. Also, the repository for the main external repository should be created first.

[peutch]

+1 for this

And should deb-get project be the place to collect any external repositories like for: https://packages.icinga.com/?
:-)

[peutch]

On Debian Wiki there is a list: https://wiki.debian.org/DebianRepository/Unofficial

[flexiondotorg]

@natanjunges I think it is time to move forward with this.

Trying to merge the ever-growing list of new packages into the monolithic script is no longer maintainable 😭

I think we'll just have to bite the bullet. Migrate to a repositories collection and close outstanding pull requests add new apps and ask contributors to resubmit them in the new format.

I would prefer that this repo be used to host the repositories collection in a suitable directory in /etc.

[natanjunges]

I would prefer that this repo be used to host the repositories collection in a suitable directory in /etc.

What do you mean by this? Do you want to keep both the deb-get script and its packages repository in this same repository?

[flexiondotorg]

What do you mean by this? Do you want to keep both the deb-get script and its packages repository in this same repository?

My first thought was Yes; I would like to keep deb-get and the package data in this git repository.

• Pros
  • Simple to package
  • Existing deb-get package can be installed as it is today and all supported apps will just work.
  • One place for contributors to submit their work.
• Cons
  • deb-get and its data will always be released in lockstep.

As for a separate repo for the deb-get repository data:

• Pros
  • deb-get and the repo data can be released independently of each other.
• Cons
  • The curl | bash install of deb-get will need to automatically handle installing the deb-get-data package too.
  • The packaging for deb-get is slightly more complex which might require Breaks/Replaces in the future if the data structure/format changes and requires a specific deb-get version.

On balance, I think I would like to keep the deb-get data in this git repository. This presents less risk and we still have the option of splitting the data into a separate git repository in the future should we desire 🙂

Thoughts?

[natanjunges]

  * The `curl | bash` install of `deb-get` will need to automatically handle installing the `deb-get-data` package too.

The deb-get-data package you are talking of is a .deb package with the main package metadata? The way I implemented this feature does not work that way. It is just a collection of files served by an HTTP server (the GitHub server in this case), so any changes made to them are immediately available the next time the user runs deb-get update (a rolling release of sorts). And I added a postinst script that runs deb-get update to download and initialize the main repository whenever the package is installed/updated.

  * `deb-get` and its data will always be released in lockstep.

As mentioned above, this would not be a problem. In this regard, keeping them together or splitting them makes no difference.

I agree that keeping everything in the same repository is a good idea, I just had not though of that before. I would just recommend keeping the package metadata and some of the documentation in a 01-main subfolder to keep things organized.

[Fuseteam]

sorry about that i didn't find the time to work out the idea i had.

i do think keeping it one repo does make contributions and releases either, how would the latest metadata be served? would it just pull the raw files from the branch tip?

[natanjunges]

how would the latest metadata be served?

GitHub can serve raw files through this URL scheme:

https://raw.githubusercontent.com/<user-organization>/<repository>/<branch>/<path>

would it just pull the raw files from the branch tip?

Yes, but we could work on a dedicated development branch and only merge to the main branch once everything is properly reviewed and tested, thus keeping the package metadata as stable as possible.

[Fuseteam]

Oh interesting that sounds promising to me

[natanjunges]

Closed by #523.