Daily files download from Tromador's server not updating

[rmb4253]

Normal procedure used to be the download of the TD backbone data files of Station, System, Item, Category.csv etc. This used to be at about 0030. Then the listings-live.csv would download at approximately 5-10 minute intervals throughout the rest of the day until the next day's download took place. This no longer seems to be happening.

Whenever, I start TD the download of the majority of the filles always seems to be about 2 hours previously and listings-live seems to be about 30-60 minutes after but no longer updating. Checking on elite.tromador.com throughout my gameplay shows that sometimes after another hour or 2, all the files will be downloaded again and listings-live starts its usual 5-10 minute later download which will continue for another hour or so before all stopping again.

Has something happened which has necessitated this change or has it just gone a bit wrong?

[eyeonus]

It probably just means the server crashed again. It happens infrequently, and I've still not found the source of the problem.

Based on your statement that things went back to expected after an hour or two, that probably either means Trom noticed and restarted it, or he's set up some auto-detection script that did that for him.

[eyeonus]

While it is true that Tromador's server gets data from Spansh, this is mostly just to establish a daily baseline, so the once-daily updates to 'listings.csv' puts Tromador's server in parity with the most recent Spansh update.

The vast majority of the data is acquired via EDDN, and it is these EDDN updates which provide the 'listings-live.csv' 5-ish minute updates.

[eyeonus]

@Tromador it does look like it's down again, current timestamp for the live file is nearly 30 min ago.

[Tromador]

Seems fine to me, it's running a spansh update. Odd time of day perhaps, but that would account for the delay.

[eyeonus]

No worries then

[Tromador]

I caught it running spansh again. I've restarted the listener, and it does say a spansh update is available, so I'll let it run this latest update and see what happens later.

Somehow it's clearly confused about Spansh, or Spansh itself is confused, though the timestamps on the website look sane. So hopefully it'll parse the update, then resume normal operations, but I'll keep an eye on it and post here with a conclusion.

[eyeonus]

Here's hoping it isn't some bs with the timestamps

I remember having that problem with our files, because of time zone stupidity

[Tromador]

Yeah, it's upset about something and redoing spansh over and over. Please advise :)

[eyeonus]

Okay, I'll see if I can reproduce, can you send me the log? I need to look at the bits related to the update checker process

…

On Sat, Feb 22, 2025, 01:18 Stefan Morrell ***@***.***> wrote: Yeah, it's upset about something and redoing spansh over and over. Please advise :) — Reply to this email directly, view it on GitHub <#194 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AANHHYBYYMKNC4MOVA5F6B32RAXGLAVCNFSM6AAAAABXTQCRWOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNZWGA4DOMBWHA> . You are receiving this because you modified the open/close state.Message ID: ***@***.***> [image: Tromador]*Tromador* left a comment (eyeonus/Trade-Dangerous#194) <#194 (comment)> Yeah, it's upset about something and redoing spansh over and over. Please advise :) — Reply to this email directly, view it on GitHub <#194 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AANHHYBYYMKNC4MOVA5F6B32RAXGLAVCNFSM6AAAAABXTQCRWOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNZWGA4DOMBWHA> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>

[Tromador]

Dunno what to tell you. I says

Live listings exporter acknowledging busy signal.
Message processor acknowledging busy signal.
Busy signal acknowledged, performing update.
Downloading prices from remote URL:
https://downloads.spansh.co.uk/galaxy_stations.json
NOTE: Requesting https://downloads.spansh.co.uk/galaxy_stations.json
NOTE: Downloading   4.9GB gziped data

Basically the log looks 100% normal, except it's saying there is a spansh update far too often to make any sense. I guess we'll have to do some debug.

What flags etc would you like me to run with and I'll produce some. You really don't want yesterday's log. It's 0.5GB and from my looksee, nothing there to learn.

[eyeonus]

Well, it determines whether or not it needs to do an update based on the timestamp of the file on the website.
It's possible that the timestamp of the file when you download it has its modified time set to when it was downloaded, and because of time zone shenanigans the timestamp of the local file is always 'less recent' than the file on Spansh.

I can confirm that my test is doing the same thing.

[Tromador]

Quick and dirty workaround:

Add a minimum back off time (12 hours?) during which it won't download regardless.

Give breathing space to work out what the real problem is.

[eyeonus]

Yeah, just change check_update_every_x_min to something high enough for now

[Tromador]

Ok, I can do that in config. Being as your test has the same symptoms, I will wait to hear back from you.

[eyeonus]

Well, it's not us, it's Spansh.
For some reason the file we use, and only the file we use, is getting updated multiple times a day. I went on the webpage just before starting a test and it said the file we use was generated 21 minutes prior to me starting my test run. an hour later when the first update from the test finished and it immediately started another update, I went back on the site and after refreshing the page, it said the file was generated 2 minutes ago.

I don't know why it's getting updated so often, but that's super freaking annoying.

[rmb4253]

Yes, that is annoying. However, since your suggestion to add a back off time was implemented by Tromador, it seems to be behaving itself. The listener is still going currently 9 hours after the Spansh update at 0405 so we could probably live with that! Perhaps a 24 hour back off would give us similar results to what we had before this problem started?

[eyeonus]

On that note, @Tromador, I have just pushed an update to the listener

In my efforts to track down the source of this, I made it so the listener downloads the Spansh file, and then runs the import using that file, rather than having the plugin download the file when it's run, which is how it was doing it.

The benefit to this is that the DB doesn't need to be locked during the download process, allowing the other threads to continue operating while the update thread is waiting for the download to complete, and the busy signal to perform the update isn't needed to be turned on until it's actually time to run the import plugin.

(That said, still want to set the update checker config to 1440 minutes, because of what it turns out the source of the problem actually is.)

[Tromador]

Maybe it's updating when an FC moves. Have you asked @spansh?

Alternatively, switch to using a different file. https://downloads.spansh.co.uk/galaxy_populated.json.gz is on the same schedule the majority of the files and I presume would fit our needs (assuming it contains the FC info).

It's a bit late to push a new version right now, as I'm shortly heading for bed, but I'll do it tomorrow probably.

[eyeonus]

No worries, it's not vital, it's a whenever you feel like thing.

It would probably have the FCs in populated systems, but not in any that are not populated, so it would definitely miss some. I'd have to actually look at the file, with my eyes, to see if it contains market data at all, although running the spansh import with any level of debug will, assuming it has an entry for it, spit out in the /tmp folder just the Shinrarta Dezhra system (in formatted json), which would make it somewhat easier to see an example system without having to cat or what-have-you an entire unformatted 10GB file and try to make sense of it.

As far as using one of the other files, if you think it'll be a better workaround to my suggestion of setting the update check delay to be once per day within the config:
https://github.com/eyeonus/TradeDangerous-listener/blob/33fb0ab1d4d0185a6dbb8a825c65fe0b7608bc7f/tradedangerous_listener.py#L40C1-L41C37

_SPANSH_FILE = "galaxy_systems.json"
_SOURCE_URL = f'https://downloads.spansh.co.uk/{_SPANSH_FILE}'

That's one of the changes I made in the latest push

[eyeonus]

I'm running Spansh with the populated one just to see what happens.

EDIT: Yup, seems to work fine using populated instead, but keep in mind that any FCs in unpopulated systems will be missed.

[spansh]

The reason that galaxy_stations.json.gz is updating so frequently is that it's now on a new schedule, and is generated every 30 minutes (In preparation for something else I'm working on which is using it and I want to be more up to date with regards to commodity data).

If you don't need it that frequently then set it to download on a schedule you prefer.