0.3.7 Calling github API excessively for each cache update.

[OMEGARAZER]

Reporting a bug 🐛

Expected behavior

Reduced API calls

Actual behavior

On an update with 0.3.7 the refresh_supported_cache_lists function removes the supported.list file from the cache which causes a full refresh of information through the lowest branch of the list_debs function calling validate_deb on every app that is listed as available through installed repos like 01-main with no regard to arch support or installed status. Because of this any incompatible apps were causing #752 (which can be mitigated by #756) and when not hit with #752 will cause 103 calls to the Github API which is well over the 60 available without providing a DEBGET_TOKEN.

Steps to reproduce

let cache files age over 60 minutes or deb-get clean && deb-get update without a DEBGET_TOKEN set and without issue in #752.

System information

/etc/os-release

PRETTY_NAME="Ubuntu 22.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.1 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

/etc/lsb-release

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.1 LTS"

deb-get version

0.3.7

deb-get update

  [+] Updating /etc/deb-get/01-main
  [+] Updating /var/cache/deb-get/bat.json
  [+] Updating cache of supported apps in the background
  [+] Updating /var/cache/deb-get/deb-get.json
  [+] Updating /var/cache/deb-get/du-dust.json
  [+] Updating /var/cache/deb-get/duf.json
  [+] Updating /var/cache/deb-get/fd.json
  [+] Updating /var/cache/deb-get/gh.json
  [+] Updating /var/cache/deb-get/github-desktop.json

API x-ratelimit-used

Before update: x-ratelimit-used: 1 (used by the check)
After update: x-ratelimit-used: 105 (2 used by checks)

[KlfJoat]

I have the same issue. I bashed my head against it off and on for weeks.

Part of the problem was that I didn't see my error messages anywhere in the Issues. So for posterity, this bug manifest for me as repeated errors in the form...

$ deb-get reinstall deb-get
  [*] WARNING! Cached file /var/cache/deb-get/deb-get.json is empty or missing.
/var/cache/deb-get/: Is a directory                                                                 
  [!] ERROR! Failed to download . Deleting /var/cache/deb-get/...

or

$ deb-get install sniffnet                                     
  [+] Updating /var/cache/deb-get/sniffnet.json                                                     
  [*] WARNING! Updating /var/cache/deb-get/sniffnet.json failed.                                    
  [*] WARNING! Cached file /var/cache/deb-get/sniffnet.json is empty or missing.                    
/var/cache/deb-get/: Is a directory                                                                 
  [!] ERROR! Failed to download . Deleting /var/cache/deb-get/...  

As a side-note, it's obvious that the devs use the "optional" DEBGET_TOKEN config 100% of the time. Because they can't tell that the token config has gone from optional to necessary.

A DEBGET_TOKEN is not optional with deb-get anymore in its current form.

[dreamcat4]

$ deb-get install sniffnet                                     
  [+] Updating /var/cache/deb-get/sniffnet.json                                                     
  [*] WARNING! Updating /var/cache/deb-get/sniffnet.json failed.                                    
  [*] WARNING! Cached file /var/cache/deb-get/sniffnet.json is empty or missing.                    
/var/cache/deb-get/: Is a directory                                                                 
  [!] ERROR! Failed to download . Deleting /var/cache/deb-get/...  

A DEBGET_TOKEN is not optional with deb-get anymore in its current form.

Hello wonder if you can answer this! (as same error here):

Trying to find out what permissions github PAT token geb-get uses... it's documented somewhere other than main README.md landing page?

Also github lets create both "classic" or "fine graned" token. Tried adding both kinds, but with 0 "extra" manual opt-in access permissions. So IDK if that is correct here.

[dreamcat4]

... i suppose to debug this issue requires the following command?

sudo -E bash -x deb-get install sniffnet

Here is mine:

https://gist.github.com/dreamcat4/0cb372af70b10813c0ca5e26ff137077

[edit]

ah interesting... +1 hour later

sudo deb-get install notable
  [+] Updating /var/cache/deb-get/notable.json
/var/cache/deb-get/notable_1.8.4_amd6  61%[============================================>            /var/cache/deb-get/notab 100%[==================================>]  48.38M  4.94MB/s    in 11s  

So that does indeed confer with those API rate limiting etc.