[BUG] SyntaxError: Invalid magic, got b')DV\x07' #462

WinterPhoenix · 2024-06-27T06:26:02Z

Description
Steam Client Beta seems to be rolling out a new format again.

Steps to Reproduce the behavior

>>> from steam.utils.appcache import parse_appinfo
>>> header, apps = parse_appinfo(open(r"C:\Users\Winter\Desktop\WIP\appinfo.vdf", "rb"))

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "D:\Python311_64\Lib\site-packages\steam\utils\appcache.py", line 73, in parse_appinfo
    raise SyntaxError("Invalid magic, got %s" % repr(magic))
SyntaxError: Invalid magic, got b')DV\x07'

Example appinfo.vdf, like last time:
appinfo.zip

Versions Report

steam: 1.4.4

Dependencies:
                 vdf: 3.4
            protobuf: 4.21.11
            requests: 2.31.0
          cachetools: 5.2.0
              gevent: Not Installed
 gevent-eventemitter: Not Installed
       pycryptodomex: 3.16.0
              enum34: Not Installed
       win-inet-pton: Not Installed

Python runtime:
          executable: D:\Python311_64\python.exe
             version: 3.11.0 (main, Oct 24 2022, 18:26:48) [MSC v.1933 64 bit (AMD64)]
            platform: win32

System info:
              system: Windows
             machine: AMD64
             release: 10
             version: 10.0.19045

The text was updated successfully, but these errors were encountered:

sonic2kk · 2024-06-28T00:30:05Z

Looks like the magic changed again for appinfo.vdf, happened a little while ago as well: #418.

Matoking · 2024-06-28T08:28:08Z

Had a look at the new appinfo.vdf format. Not a case of just the magic number being changed:

#463 (comment)

The new Steam beta introduced a new `appinfo.vdf` version. This appinfo.vdf V29 introduces a new binary VDF format which does not include field keys in binary VDF segments as-is. Instead, each key is represented by a 32-bit integer which needs to be mapped to an actual string using a table are stored at the end of the `appinfo.vdf` file. This also means the binary VDF segments are no longer self-contained. Also see SteamDatabase/SteamAppinfo#56b1fec7f5ce6be961c3e44cf9baf117e363ad91 Refs ValvePython/steam#462

The new Steam beta introduced a new `appinfo.vdf` version. This appinfo.vdf V29 introduces a new binary VDF format which does not include field keys in binary VDF segments as-is. Instead, each key is represented by a 32-bit integer which needs to be mapped to an actual string using a table are stored at the end of the `appinfo.vdf` file. This also means the binary VDF segments in this case are no longer self-contained. Also see SteamDatabase/SteamAppinfo#56b1fec7f5ce6be961c3e44cf9baf117e363ad91 Refs ValvePython/steam#462

The new Steam beta introduced a new `appinfo.vdf` version. This appinfo.vdf V29 introduces a new binary VDF format which does not include field keys in binary VDF segments as-is. Instead, each key is represented by a 32-bit integer which needs to be mapped to an actual string using a table stored at the end of the `appinfo.vdf` file. This also means the binary VDF segments in this case are no longer self-contained. The developer can parse and provide the table themselves or instead use the `ValvePython/steam` library which contains a function to parse `appinfo.vdf` files. Also see SteamDatabase/SteamAppInfo#56b1fec7f5ce6be961c3e44cf9baf117e363ad91 Refs ValvePython/steam#462

The new Steam beta introduced a new `appinfo.vdf` version. This appinfo.vdf V29 introduces a new binary VDF format which does not include field keys in binary VDF segments as-is. Instead, each key is represented by a 32-bit integer which needs to be mapped to an actual string using a table stored at the end of the `appinfo.vdf` file. This also means the binary VDF segments in this case are no longer self-contained. The developer can parse and provide the table themselves or instead use the `ValvePython/steam` library which contains a function to parse `appinfo.vdf` files. Also see SteamDatabase/SteamAppInfo@56b1fec Refs ValvePython/steam#462

appinfo.vdf V29 was introduced in Steam beta. This new version introduces a space-saving optimization: instead of encoding each key name in the binary VDF segment directly, an int64 identifier is instead used for each key, with a table at the end of the 'appinfo.vdf' file providing the mapping to actual key names. Also see SteamDatabase/SteamAppInfo@56b1fec Fixes ValvePython#462 Co-authored-by: Eamonn Rea <eamonnrea@gmail.com>

The new Steam beta introduced a new `appinfo.vdf` version. This appinfo.vdf V29 introduces a new binary VDF format which does not include field keys in binary VDF segments as-is. Instead, each key is represented by a 32-bit integer which needs to be mapped to an actual string using a table stored at the end of the `appinfo.vdf` file. This also means the binary VDF segments in this case are no longer self-contained. The developer can parse and provide the table themselves or instead use the `ValvePython/steam` library which contains a function to parse `appinfo.vdf` files. Also see SteamDatabase/SteamAppInfo@56b1fec Refs ValvePython/steam#462

appinfo.vdf V29 was introduced in Steam beta. This new version introduces a space-saving optimization: instead of encoding each key name in the binary VDF segment directly, an int64 identifier is instead used for each key, with a table at the end of the 'appinfo.vdf' file providing the mapping to actual key names. Also see SteamDatabase/SteamAppInfo@56b1fec Fixes ValvePython#462 Co-authored-by: Eamonn Rea <eamonnrea@gmail.com>

WinterPhoenix · 2024-07-08T08:23:14Z

@rossengeorgiev Please review the relevant PRs. This should really be fixed sooner rather than later.

WinterPhoenix · 2024-07-25T02:33:20Z

I made a fork that incorporates the PR that fixes this, as well as several other pending PRs in this repo. I don't really particularly want to be the new maintainer for this, but if I'm still using it I may as well. PRs on the fork welcome.

https://github.com/solsticegamestudios/steam

Matoking · 2024-07-29T16:15:49Z

One option would be to vendor the relevant code and/or libraries, though I don't know if that is possible for projects using GPL and other similar licenses like Protontricks. Debian also doesn't like vendored code to my knowledge.

I've instead opted to push a version to Flathub beta branch with the fix. For pip users I've instructed to install the libraries from GitHub directly. Other than that, the current options are limited as vdf and steam libraries are packaged by upstream distros.

@rossengeorgiev has had some sporadic activity on GitHub, but these two ValvePython projects haven't had any activity in a while. I have to emphasize that they're not obligated to work on these libraries - open source development is often a thankless job and maintainers are prone to burnout, after all - though it would be appreciated if they could state whether these projects are in maintenance mode or abandoned.

appinfo.vdf V29 was introduced in Steam beta. This new version introduces a space-saving optimization: instead of encoding each key name in the binary VDF segment directly, an int64 identifier is instead used for each key, with a table at the end of the 'appinfo.vdf' file providing the mapping to actual key names. Also see SteamDatabase/SteamAppInfo@56b1fec Fixes ValvePython#462 Co-authored-by: Eamonn Rea <eamonnrea@gmail.com>

ample-samples · 2024-11-15T21:18:07Z

~~I would post this on @WinterPhoenix 's fork (https://github.com/solsticegamestudios/steam) but it doesn't allow issues:~~

I've installed the package with this command:
python -m pip install -U "git+https://github.com/solsticegamestudios/steam#egg=steam"

Running this code:

from steam.utils.appcache import parse_appinfo
header, apps = parse_appinfo(open('c:\\Steam\\appcache\\appinfo.vdf', 'rb'))
print(next(apps))

raises a TypeError:

Traceback (most recent call last):
  File "c:\Projects\tests\appinfo_test.py", line 3, in <module>
    print(next(apps))
          ^^^^^^^^^^
  File "C:\Users\{my_name}\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\local-packages\Python312\site-packages\steam\utils\appcache.py", line 137, in apps_iter
    app['data'] = binary_load(fp, key_table=key_table, mapper=mapper)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\{my_name}\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\local-packages\Python312\site-packages\vdf\__init__.py", line 326, in binary_load
    if not issubclass(mapper, Mapping):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen abc>", line 123, in __subclasscheck__
TypeError: issubclass() arg 1 must be a class

Printing the header comes back fine:
{'magic': b')DV\x07', 'universe': 1}

Could the Valve have broken this again?

Nevermind, running:
header, apps = parse_appinfo(open('c:\\Steam\\appcache\\appinfo.vdf', 'rb'), mapper=dict)
works completely fine

WinterPhoenix · 2024-11-15T22:13:59Z

Issues are enabled… What does it say when you try to create one, specifically?

ample-samples · 2024-11-15T22:38:21Z

Nevermind it actually does, I was looking at another steam fork

WinterPhoenix added bug needs-review labels Jun 27, 2024

This was referenced Jun 28, 2024

appcache: add new magic for Steam Client Beta (June 2024) #463

Closed

appinfo.vdf Magic Changed on Steam Client Beta (June 2024) - Needs Resolved Upstream in ValvePython/steam DavidoTek/ProtonUp-Qt#415

Closed

WinterPhoenix mentioned this issue Jun 29, 2024

Latest Steam Beta Update causes script to fail solsticegamestudios/GModCEFCodecFix#117

Closed

This was referenced Jun 30, 2024

Add deserialization support for new binary VDF w/ key table ValvePython/vdf#61

Open

Add deserialization support for appinfo.vdf V29 #464

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] SyntaxError: Invalid magic, got b')DV\x07' #462

[BUG] SyntaxError: Invalid magic, got b')DV\x07' #462

WinterPhoenix commented Jun 27, 2024 •

edited

Loading

sonic2kk commented Jun 28, 2024

Matoking commented Jun 28, 2024

WinterPhoenix commented Jul 8, 2024

WinterPhoenix commented Jul 25, 2024 •

edited

Loading

Matoking commented Jul 29, 2024

ample-samples commented Nov 15, 2024 •

edited

Loading

WinterPhoenix commented Nov 15, 2024 •

edited

Loading

ample-samples commented Nov 15, 2024

[BUG] SyntaxError: Invalid magic, got b')DV\x07' #462

[BUG] SyntaxError: Invalid magic, got b')DV\x07' #462

Comments

WinterPhoenix commented Jun 27, 2024 • edited Loading

sonic2kk commented Jun 28, 2024

Matoking commented Jun 28, 2024

WinterPhoenix commented Jul 8, 2024

WinterPhoenix commented Jul 25, 2024 • edited Loading

Matoking commented Jul 29, 2024

ample-samples commented Nov 15, 2024 • edited Loading

WinterPhoenix commented Nov 15, 2024 • edited Loading

ample-samples commented Nov 15, 2024

WinterPhoenix commented Jun 27, 2024 •

edited

Loading

WinterPhoenix commented Jul 25, 2024 •

edited

Loading

ample-samples commented Nov 15, 2024 •

edited

Loading

WinterPhoenix commented Nov 15, 2024 •

edited

Loading