Query Info picks the wrong mod for some large files (MD5 result overflow)

[jmdavis]

The problem:

When Query Info is used on two of the three main files from https://www.nexusmods.com/skyrimspecialedition/mods/18115, MO decides that they're from https://www.nexusmods.com/skyrim/mods/92886 (which has been deleted) instead of the correct mod.

To Reproduce:

Steps to reproduce the behavior:

1. Download the main files from https://www.nexusmods.com/skyrimspecialedition/mods/18115?tab=files For 1.0, that's

    Music - HQ-18115-0-1.7z
    Voices EN  - Part 2-18115-0-1.7z
    Voices EN - Part 1-18115-0-1.7z
   

2. Copy them into the Downloads tab in MO2.

3. Right-click and select "Query Info" for each of the three files. Music - HQ-18115-0-1.7z will be correctly selected as being from "Unofficial High Definition Audio Project" (https://www.nexusmods.com/skyrimspecialedition/mods/18115), whereas the other two will be selected as being from the deleted mod https://www.nexusmods.com/skyrim/mods/92886.

Environment:

• Mod Organizer Version that exhibits the issue: 2.2.0
• Last Mod Organizer Verison that did not exhibit the issue (if applicable): Probably whichever version used the old Nexus API, though since that had other issues, it may not have worked properly there either, and it's not like it could be checked now, since the old API has been shut down.
• Desktop OS/version used to run Mod Organizer: 64-bit Windows 10

Details:

The steps to reproduce should be pretty self-explanatory for showing the problem.

As for why, it's failing, based on the log, it looks like the current Nexus API searches based on the MD5 of the file, and for some reason MO2 is searching for the wrong MD5. These are the MD5s of the files.

MD5 (Music - HQ-18115-0-1.7z) = 9651d5ec4cc0b2e1b8efccaa9dffb09d
MD5 (Voices EN  - Part 2-18115-0-1.7z) = 5082f7d6b644d468f1f09fd357e3736d
MD5 (Voices EN - Part 1-18115-0-1.7z) = c93ab67c100b63c2ed279c62db4a9162

The line

20:47:58 [D] Searching SkyrimSE for MD5 of 9651d5ec4cc0b2e1b8efccaa9dffb09d

from the log therefore looks correct for "Music - HQ-18115-0-1.7z," and it correctly identifies the mod that the file comes from. However, for both of the others, MO2 seems to be searching for the same, wrong MD5.

20:48:00 [D] Searching SkyrimSE for MD5 of d41d8cd98f00b204e9800998ecf8427e
20:48:01 [D] Searching Skyrim for MD5 of d41d8cd98f00b204e9800998ecf8427e

20:48:03 [D] Searching SkyrimSE for MD5 of d41d8cd98f00b204e9800998ecf8427e
20:48:03 [D] Searching Skyrim for MD5 of d41d8cd98f00b204e9800998ecf8427e

So, it at least seems like the problem is that somehow, MO2 is searching for the wrong MD5 for both of those files. But I haven't looked at the code at all, just what the log says and what the result in the .meta file is.

I did try changing the corresponding .meta files to the correct mod, but for better or worse, "Query Info" seems to just throw away what's in the .meta file, though I doubt that whoever worked on that bit of code was expecting someone to manually change the .meta file to try to get "Query Info" to look at the correct mod, and I couldn't fully fix the .meta file, because I have no clue what the fileIDs are. If I did, then MO probably wouldn't think that "Query Info" needed to be run again and wouldn't display the red exclamation sign.

Link to Mod Organizer logs:

The relevant part of mo_interface.log seems to be

20:47:31 [D] login successful
20:47:39 [D] Downloads after refresh: 2
20:47:42 [D] Downloads after refresh: 1
20:47:44 [D] Downloads after refresh: 0
20:47:54 [D] Downloads after refresh: 3
20:47:58 [D] Searching SkyrimSE for MD5 of 9651d5ec4cc0b2e1b8efccaa9dffb09d
20:48:00 [D] Information updated
20:48:00 [D] Downloads after refresh: 3
20:48:00 [D] Searching SkyrimSE for MD5 of d41d8cd98f00b204e9800998ecf8427e
20:48:01 [D] Searching Skyrim for MD5 of d41d8cd98f00b204e9800998ecf8427e
20:48:01 [D] Multiple active files found during MD5 search.  Defaulting to time stamps...
20:48:02 [W] request failed: Error transferring https://api.nexusmods.com/v1/games/skyrim/mods/92886/files - server replied: Forbidden
20:48:02 [D] Failed to request file info from nexus: Error transferring https://api.nexusmods.com/v1/games/skyrim/mods/92886/files - server replied: Forbidden
20:48:02 [D] Downloads after refresh: 3
20:48:03 [D] Searching SkyrimSE for MD5 of d41d8cd98f00b204e9800998ecf8427e
20:48:03 [D] Searching Skyrim for MD5 of d41d8cd98f00b204e9800998ecf8427e
20:48:04 [D] Multiple active files found during MD5 search.  Defaulting to time stamps...
20:48:04 [W] request failed: Error transferring https://api.nexusmods.com/v1/games/skyrim/mods/92886/files - server replied: Forbidden
20:48:04 [D] Failed to request file info from nexus: Error transferring https://api.nexusmods.com/v1/games/skyrim/mods/92886/files - server replied: Forbidden
20:48:04 [D] Downloads after refresh: 3

This is from copying the three files into MO2 together, and then using "Query Info" on them in the order of

Music - HQ-18115-0-1.7z
Voices EN - Part 1-18115-0-1.7z
Voices EN  - Part 2-18115-0-1.7z

Curiously, before using "Query Info," when the UI lists the file names instead of the mod names, it seems to list two spaces in a row as one space, showing the file name for "Part 2" with one space instead of two after the EN, which is presumably why the UI then lists Part 1 before Part 2, whereas otherwise, lexical ordering would put Part 2 before Part 1 (which is how File Explorer shows them). So, the Downloads tab UI doesn't actually show the exact file names for some reason, but I suppose that that isn't relevant to this bug, just odd.

[jmdavis]

I just had the exact same thing happen with https://www.nexusmods.com/skyrimspecialedition/mods/6096 - including it using the exact same MD5 in the log.

[jmdavis]

Aha. d41d8cd98f00b204e9800998ecf8427e is the MD5 hash of an empty file, and I'd copied the file from https://www.nexusmods.com/skyrimspecialedition/mods/6096 while it was still downloading (with how Firefox does downloads, it creates an empty file with the target file name, downloads the data to a file with the same name except that it has .part as an extension, and thenit overwrites the target file with the .part file when it's done). So, that's why I got the problem with https://www.nexusmods.com/skyrimspecialedition/mods/6096, and once I did "Query Info" with the fully downloaded file, it worked correctly.

However, the problem happens with the two files from https://www.nexusmods.com/skyrimspecialedition/mods/18115 as I originally reported even with the files fully downloaded. So, the original problem was not due to copying empty files, and MO2 does appear to be doing something wrong. However, the fact that MO2 is trying the hash for an empty file when it's querying the Nexus may give a clue as to what's wrong.

[bodrick]

So I believe its the size of the files that are the issue, I tried a couple of files and one that was around 2gb worked, but larger files failed and gave the blank md5. I suspect its an overflow bug since it looks like its caused by using an int which has a max value of 2,147,483,647 which is approx 2gb

[LostDragonist]

Fixed by PR #844 in version 2.2.2