Fully switch from steamy-vdf

[CosmicHorrorDev]

Fixes #6

This is the initial work for fully switching from steamy-vdf. This is done by using keyvalues-serde to convert the entire manifest file into an internal struct that finally gets converted to a SteamApp (the whole InternalSteamApp -> SteamApp was done to accommodate path which uses information that is external to the manifest file)

I also decided to add a super basic example to show off the SteamApp changes. Here is the result of running it on Inscryption's manifest

$ cargo run --example appmanifest -- 1092790
Found app - SteamApp {
    app_id: 1092790,
    path: "/path/to/my/first/library/common/Inscryption",
    name: "Inscryption",
    universe: 1,
    state_flags: 4,
    last_updated: 1650903140,
    update_result: 0,
    size_on_disk: 3513686670,
    build_id: 8386576,
    bytes_to_download: 64990864,
    bytes_downloaded: 64990864,
    bytes_to_stage: 3443365760,
    bytes_staged: 3443365760,
    staging_size: None,
    auto_update_behavior: 0,
    allow_other_downloads_while_running: 0,
    scheduled_auto_update: 0,
    installed_depots: {
        1092791: Depot {
            manifest: 4877467984383706702,
            size: 3513686670,
        },
    },
    user_config: {
        "language": "english",
        "platform_override_dest": "linux",
        "platform_override_source": "windows",
    },
    mounted_config: {},
    install_scripts: {},
    shared_depots: {
        228990: 228980,
    },
    last_user: 12312312312312312,
}

Follow-up work

• I'm not at all confident that these are all the fields possible in a manifest file, so I'll be going through my full steam library to harvest manifest files Almost all fields are now optional
• Be more user friendly! I'm planning on parsing a lot of the values into nicer types (universe, auto_update_behavior, etc. to enums, state_flags to a bitfield, etc.) This is now being done
  • I was initially going to include that in this PR, but it ended up being quite a lot, so I figured 2 smaller PRs was better
  • I found some scattered info on what some of the values mean, but haven't found anything comprehensive sadly ;-;

[WilliamVenner]

I gave the library a whirl in my current project and it failed to find Squad D:

libraryfolders.vdf

"libraryfolders"
{
	"0"
	{
		"path"		"C:\\Program Files (x86)\\Steam"
		"label"		""
		"contentid"		"135561836896929224"
		"totalsize"		"0"
		"update_clean_bytes_tally"		"33971334451"
		"time_last_update_corruption"		"0"
		"apps"
		{
			"730"		"30313029175"
			"22370"		"9347423268"
			"108600"		"4996040626"
			"228980"		"1193618916"
			"294100"		"408774349"
			"380870"		"3105130078"
			"380880"		"177214887"
		}
	}
	"1"
	{
		"path"		"D:\\Steam"
		"label"		""
		"contentid"		"3489870999083003877"
		"totalsize"		"500090007552"
		"update_clean_bytes_tally"		"75360774707"
		"time_last_update_corruption"		"0"
		"apps"
		{
			"4000"		"4288831370"
			"590830"		"6364427755"
		}
	}
	"2"
	{
		"path"		"F:\\Steam"
		"label"		""
		"contentid"		"4237462245967358517"
		"totalsize"		"3000679526400"
		"update_clean_bytes_tally"		"3489366132"
		"time_last_update_corruption"		"0"
		"apps"
		{
			"620"		"12751189338"
			"311210"		"120306250285"
			"331670"		"1396836515"
			"397460"		"751358817"
			"434170"		"1713411684"
			"494150"		"1507751382"
			"546560"		"72900756522"
			"610180"		"1101849737"
			"774461"		"2391723561"
			"1005300"		"2314420906"
			"1172380"		"55362496540"
		}
	}
	"3"
	{
		"path"		"G:\\Steam"
		"label"		""
		"contentid"		"5299832385336737214"
		"totalsize"		"2000262524928"
		"update_clean_bytes_tally"		"62870402092"
		"time_last_update_corruption"		"0"
		"apps"
		{
			"215"		"2811981136"
			"240"		"4628887753"
			"320"		"3059493671"
			"2400"		"1687152485"
			"6060"		"10296173294"
			"17390"		"4170359084"
			"17440"		"4268390543"
			"17520"		"825351189"
			"24720"		"6150697965"
			"42710"		"8580096"
			"200170"		"1713583360"
			"219150"		"565298931"
			"220200"		"4761830219"
			"221640"		"33743267"
			"221910"		"2096720157"
			"222880"		"9228115593"
			"225840"		"2511105162"
			"240720"		"1393495247"
			"242760"		"5950216105"
			"264710"		"7930209677"
			"274170"		"616372827"
			"298110"		"35454880783"
			"302590"		"713504988"
			"320240"		"12106518271"
			"351510"		"128756023"
			"362890"		"30064617865"
			"377160"		"37409240201"
			"450390"		"16625763869"
			"476620"		"94166758085"
			"552520"		"71193712331"
			"555160"		"61592016905"
			"578620"		"5800016212"
			"617830"		"1884334812"
			"768100"		"442487634"
			"823500"		"16532235889"
			"848450"		"10217432467"
			"868020"		"793838884"
			"916840"		"31524223590"
			"960910"		"36949470843"
			"960990"		"46332929339"
			"1082680"		"10763731655"
			"1190460"		"67802322583"
			"1222140"		"63077359331"
			"1426210"		"46895638288"
		}
	}
	"4"
	{
		"path"		"Q:\\Steam"
		"label"		""
		"contentid"		"5299832385336737214"
		"totalsize"		"2000262524928"
		"update_clean_bytes_tally"		"29788415420"
		"time_last_update_corruption"		"1649511932"
		"apps"
		{
			"215"		"2811981136"
			"220"		"3096281443"
			"240"		"4628887753"
			"320"		"3059493671"
			"380"		"1148800633"
			"400"		"4184279115"
			"420"		"3197069303"
			"440"		"24747954193"
			"550"		"14944811606"
			"2400"		"1687152485"
			"6060"		"10296173294"
			"17390"		"4170359084"
			"17440"		"4268390543"
			"17520"		"825351189"
			"24720"		"6150697965"
			"42700"		"9617628587"
			"42710"		"8580096"
			"105600"		"465676180"
			"200170"		"1713583360"
			"202970"		"3580495282"
			"202990"		"4194334258"
			"212910"		"4010389671"
			"219150"		"565298931"
			"220200"		"4761830219"
			"221640"		"33743267"
			"221910"		"2096720157"
			"222880"		"9228115593"
			"225840"		"2511105162"
			"233450"		"536784311"
			"240720"		"1393495247"
			"242760"		"5950216105"
			"244850"		"36619490715"
			"252490"		"20754758448"
			"252950"		"23898311653"
			"264710"		"7930209677"
			"271590"		"112917643584"
			"274170"		"616372827"
			"298110"		"35454880783"
			"302590"		"713504988"
			"320240"		"12106518271"
			"346010"		"2823611005"
			"351510"		"128756023"
			"362890"		"30064617865"
			"365670"		"703561386"
			"377160"		"37409240201"
			"393380"		"78678038927"
			"438100"		"833102126"
			"450390"		"16625763869"
			"457140"		"1993277636"
			"466240"		"15039883962"
			"471710"		"6756129973"
			"476620"		"94166758085"
			"493520"		"11816699896"
			"532210"		"11402414915"
			"552520"		"71193712331"
			"555160"		"62754833589"
			"578620"		"5800016212"
			"617830"		"2112749054"
			"620980"		"1425186449"
			"700330"		"5216457684"
			"739630"		"18124452918"
			"768100"		"442487634"
			"823500"		"16532235889"
			"848450"		"10217432467"
			"868020"		"793838884"
			"892970"		"1258614002"
			"916840"		"31524223590"
			"945360"		"402221874"
			"960910"		"36949470843"
			"960990"		"46332929339"
			"1019550"		"8048352232"
			"1062090"		"1652954854"
			"1082680"		"10763731655"
			"1190460"		"67802397772"
			"1222140"		"63077359331"
			"1302240"		"11203361865"
			"1426210"		"46895638288"
			"1444480"		"94223076"
			"1782380"		"456344376"
			"1902490"		"4654605096"
		}
	}
}

appmanifest_393380.acf

"AppState"
{
	"appid"		"393380"
	"Universe"		"1"
	"LauncherPath"		"C:\\Program Files (x86)\\Steam\\steam.exe"
	"name"		"Squad"
	"StateFlags"		"4"
	"installdir"		"Squad"
	"LastUpdated"		"1652906353"
	"SizeOnDisk"		"78678038927"
	"StagingSize"		"0"
	"buildid"		"8751158"
	"LastOwner"		"76561198040894045"
	"AutoUpdateBehavior"		"0"
	"AllowOtherDownloadsWhileRunning"		"0"
	"ScheduledAutoUpdate"		"0"
	"InstalledDepots"
	{
		"393381"
		{
			"manifest"		"5060222892107978912"
			"size"		"72681537471"
		}
		"1123920"
		{
			"manifest"		"9205097207890293247"
			"size"		"5996501456"
			"dlcappid"		"1123920"
		}
	}
	"SharedDepots"
	{
		"228984"		"228980"
		"228986"		"228980"
		"228990"		"228980"
	}
	"UserConfig"
	{
		"language"		"english"
	}
	"MountedConfig"
	{
		"language"		"english"
	}
}

[CosmicHorrorDev]

That's quite the library you have there!

• b2453d5 - It turns out that several of the fields I assumed were always present (UpdateResult, BytesToDownload, BytesDownloaded, BytesToStage, and BytesStaged) were optional
• b1e2c20 - After fixing that I noticed that there were a couple more fields included that I didn't have so I added those. keyvalues-serde does the same thing as serde_json does where it skips fields that it doesn't use, so it's easy for those to fly under the radar

I'll definitely have to start harvesting more manifest files since there's likely plenty of other things in them that don't match this PR still

[WilliamVenner]

Extreme defensive programming is definitely required here. We should assume that Valve can rug pull us any day. Missing fields, extra unknown fields, fields with unexpected data types etc should be handled to the best of our ability within reason.

I wish we had a lot of manifest files to look at, but I would imagine it's not the easiest thing to get a big dataset for :p

[CosmicHorrorDev]

I think it's important to strike a balance between end-user ergonomics and defensive programming

Valve may very well make breaking changes to all of this in ways that can't really be managed, but I would prefer a best effort along with returning helpful error messages that can be reported back to us over an API that's so defensive that it's unwieldy to use

[WilliamVenner]

Yeah I guess you are right - a better user experience would be better

[CosmicHorrorDev]

Sorry, didn't mean to be so terse earlier. To explain more:

If it makes you feel more comfortable I can switch more of the fields to be Optional which handles more of the "missing fields" portion, and "extra unknown fields" will already be ignored. The part that becomes very hairy to handle in a non-irritating way would be "fields with unexpected data types" since I'm planning on expressing some of the fields as enums, bit-fields, NaiveDateTimes, etc.. I can include an unknown variant that tries to act as a catch-all for the expected type in case there is a variant that is added or just not known about like so:

enum AutoUpdateBehavior {
    Always,
    OnlyWhenLaunched,
    HighPriority,
    Unknown(u8),
}

But being able to handle something like it also being a string or also including negative numbers becomes a pain to handle in an ergonomic way

---

I'm also currently trying to get some of my friends to send me their app manifests since some of their libraries go back quite a ways and have a wide range of games. I'll definitely feel more comfortable when I've run through a few hundred different manifests to see what differences there are

[CosmicHorrorDev]

I went ahead and switched this over to a draft until I can get a reasonably large corpus of manifest files

42be47a did add in some more missing fields that I found from some of the files I've gotten

[mtkennerly]

Here's the example output for all 45 manifests I have locally on Windows 11. Happy to provide more if needed: https://gist.github.com/mtkennerly/6bb41d32f33762bd181b40518df75ae3

[CosmicHorrorDev]

Great! I'll sift over things tomorrow, but everything looks good at a quick glance

[mtkennerly]

I started some more installs to give you more data to work with. Here's 206 manifests, my libraryfolders.vdf, and the examples output: during-installs.zip. I'll do a diff once the installs finish to see if anything changes.

Edit: Here's everything after the installations completed: after-installs.zip

Sample diff:

[CosmicHorrorDev]

No unknowns and no failed parsing is a good sign!

@WilliamVenner this should be ready for another review now. I made the parsing much more robust by switching almost all returned fields to be optional along with using a sprinkling of #[non_exhaustive] on structs and fallback variants for enums. Feel free to give it another pass at your library and let me know if there are any issues!

[CosmicHorrorDev]

Oh I also switched to targeting a branch for v2.0, so that there can still be changes and bugfixes on v1

[CosmicHorrorDev]

Thanks for merging @WilliamVenner!

Do you think we could put out a 2.0.0-alpha.0 version based off the current v2.0-dev branch, so that people can test/use these changes early?

[dreamer]

@CosmicHorrorDev @WilliamVenner - thank you so much for fixing this, and for providing nice, modern replacement for the unmaintained steamy package - great job! 😄

In Luxtorpeda I tested the functionality against appid 1070560 (Steam Linux Runtime), on Linux - worked correctly.

Releasing 2.0.0-alpha.0 would be very much appreciated.

[CosmicHorrorDev]

Copying my comment from #19. Apologies to people that are subscribed to both 😁

---

2.0.0-alpha.0 was just published and includes the changes to fully drops steamy-vdf including its dependency on nom v1.2.4

There will be significant changes before the actual v2 release for steamlocate, but this should let people have access to these changes early without having to rely on git dependencies