[Request]Add CHD format support for compressed disc

[rubenjavier]

Hello there, love the project, can we get some CHD love for the pcsx2 as a compressed disk format?

CHD or "Compressed Hunks of Data" - it's a MAME format for storing disk images.

With libchdr from @rtissera this should be doable like the new retroarch cores do
https://github.com/rtissera/libchdr

Some of the benefits the CHD support have:
.-lossless compression
.-15-18% better compression than pbp (this makes a bin cue uncompressed collection almost half its size)
.-its a streamable format, it doesnt need to decompress, just play
.-its increadibly easy to convert entire folders filled with uncompressed psx games with a batch file
.-every disc turns into just one file
.-supports redbook audio in flac compression
.-its becoming the defacto standard for libretro cores with support in mame, saturn, pcengine/turbografx, segacd/megacd, psx beetle and future support for dreamcast (in a few days)

Twice the fun, same space

Thanks in advance
Best regards

[RinMaru]

this is already requested

[rubenjavier]

Sorry if it was already requested, but I cloudn't find a CHD support request here on git before writing this one down.
Thanks in advance, Best regards

[mirh]

Mhh PBP (TIL) is already requested, not CHD.

Said this.. I feel like I'm missing something if CSO is regarded as lossy?
And is there some documentation anywhere? Your handful of points up there is basically the best description of this that I could see on the net.

[weirdbeardgame]

@mirh Oh no CSO's much much better just ask refraction this request would be better suited for PS1 mode cause there's already ps1 emu's with chd support plus I think chd handles bin cue better? (Don't quote me on that one.)

[refractionpcsx2]

CSO is not lossy. Lossy compression on ISO's would be hugely stupid.

[rubenjavier]

CSO for PS2 is cosidered a lossy format, since after the compression, if you recreate the ISO it wont be exactly like the original, and wont have the same signature, but CHD is lossless in every other console I've mentioned (in Dreamcast was lossy until the past week, when CHD format v5 was implemented and its now in beta reicast)
You can read about it here:
http://emulation.gametechwiki.com/index.php/Save_Disk_Space_for_ISOs

CHD is not only for PSX, its used on all the retroarch cores I mentioned before and I have all of my collections on them transformed to CHD with a simple batch file. Now Im about to do the same with my Dreamcast collection of GDI images and reduce the size to a half without loosing any data, Aas I said, its becoming the defacto standard for compressed discs in emulation

CHD was originally created for Arcade games that had HDD like Killer Instinct 1 and 2 or Street Fighter III

Thanks in advance
Best regards

[willkuer]

Actually CSO is explicitly stated to be lossless in the reference you cited. I also couldn‘t find any info that CSO could be lossy somewhere else.

[weirdbeardgame]

@rubenjavier CSO losslessly compresses much better the CHD does as a whole and I'm aware it's not just for psx games I was just suggesting that it's use would probably best fit ps1 mode whilst CSO would be best for ps2 games

[rubenjavier]

@willkuer Actually no, maybe you read another CSO section
In the PS2 section says:
CSO (aka CISO)
.-Archive-quality dump? No (missing data)
*maybe somebody with more technical knowledge than me could explain how maxcso works, but I've seen it referred as not being able to create the same original image with the same checksum, that would make it good for compression but bad for archiving

@kenshen112 its easier with a well implemented lossy algorithm to compress better than with a lossless one, but you loose the original image, thats what makes CHD so good as a lossless format, in its many iterations it got a good compression ratio (better than pbp) and the originals images can be recovered 1 to 1 whenever you want to with version 5, which is the one used now

Im just saying that this is proving to be good format for emulation and archiving

[mirh]

CSO (aka CISO) [...] I've seen it referred as not being able to create the same original image with the same checksum

@meoow @notaz @mnadareski welp

[willkuer]

Ah I read the ‚can be reverted: yes, lossless‘ part.

Apparently the read stream from the compressed file will be different to the read stream of the uncompressed file while the uncompressed file still can be reconstructed from the compressed file without loss (obviously not from the same read stream). Slightly surprising to me but possibly reasonable.

[RinMaru]

so if I uncompressed my CSO and recompress to CHDv5 they would still run perfectly on pcsx2? (If chd support comes) what exactly is lost in cso compression and does it really matter outside 1.1 copys? does it corrupt anything or is there a chance they can become unreadable with something that never supported cso?

[rubenjavier]

@RinMaru if the CSO version was working then the CHD version would work too, its only gonna read the same info on the fly from a more compressed file. About what is lost with CSO, we need someone with more expertise in maxcso than me.

Right now Im at work, but when I get some free time at home, Im gonna do some testing compressing and decompressing small and large files like Soul Calibur II and God of War II to compare formats, I read that someone did test with psp games with CHD and CSO compressions and CHD was better, but I want to check by my self.

psp CSO vs CHD:
hrydgard/ppsspp#10417

In this post you can see the real life compression a user got from etire collections of discs:
https://forums.libretro.com/t/converting-to-chds-for-certain-cores/13479/27

And the support for Dreamcast in the coming days:
skmp/reicast-emulator#1363

Thanks
Best regards

[mirh]

It's mentioned "with CSOPlus you will have the best compression but that's because it will delete UPDATE partition", but that seems something to do with the tool, not the format.
@unknownbrackets's maxcso in fact strive for 1:1 afaiu

I'm also not sure you appreciate (for as much as pcsx2 supports cso) the current golden standard for ps2 compression here is plain xz of the normal iso #2424
And it's hard to beat it for just about anything.
On the other hand, being able to actually compress CDDA audio sounds really really cool feature. I'm not sure any game needs it though.

[unknownbrackets]

Some misinformation control:

• There are two formats sometimes referred to as CSO. This causes confusion.

• The "CISO" Wii format is a LOSSY format. It is supported by Dolphin and involves deleting parts of the disc that have random data and are not accessible by games anyway. They call this "scrubbing."

• The "CSO" format, used primarily for PSP and PS2 games, is LOSSLESS and totally separate. It basically applies raw zlib compression at an adjustable block size level + embedded index. If you decompress a CSO, you get the original ISO - just as with zlib.

• There are some CSO creation tools that are ALSO able to strip files (such as movies) from discs. This is a destructive change, but such a tool could exist for CHD or gzip files. The existence of such tools doesn't really make CSO itself lossy.

• CSO, at least maxcso and the support in PCSX2, doesn't support files not a clean multiple of 2048 bytes. Therefore it isn't really appropriate for Mode 2 PS1 games (see ISO file not aligned to sector size unknownbrackets/maxcso#13.)

Just to note: CHD is likely to decompress slower than CSO. On powerful desktops, this probably won't matter, but that's why I wasn't very excited about this change - personally - for PPSSPP in hrydgard/ppsspp#10417. As it is, CSO support in PCSX2 (or PPSSPP) is quite fast, and only gets slower if you layer in the complicated caching layer used in PCSX2's GZ support.

Edit: that the XZ pull uses a file in /tmp to decompress to as a "cache" is slightly horrifying, but I'm sure the support will improve. It does seem like a good standard.

-[Unknown]

[refractionpcsx2]

Thanks for clearing that up UB :)

[rubenjavier]

Thanks for the info @unknownbrackets

[gregory38]

Well xz uncompress the data in a tmp file. Because I didn't bother to implement something super clever. It was only a proof of concept.
And my tmp is a ram disk, so it is free to use and nothing can beat it for speed ;)

[bryc]

Would be nice, but does CHD even support DVD? Dolphin doesn't support it and has their own GCZ format.

[rubenjavier]

CHD support DVD, CD, GD roms (dreamcast), HDD images, right now its used on the reicast emulator for this GD roms wich are basically 1.5GB DVDs, its a compression format so the original media is not important, just that the emulator have the code to read it, so far its been implemented on PSX, Saturn, SegaCD, Dreamcast, PCengine, MAME and its working great, Ive tested it on every platform, the space saving its hughe and the loading times are incredible

[Sanaki]

While I do use gcz for gamecube/wii, is there a reason to prefer it over chd other than emulator support?

[mirh]

Absolutely not.
Gcz uses zlib internally AFAIK (which is compression prehistory), with most of the data saving happening just because wii disks contain a lot of useless padding.

I guess like if you really wanted something fancy there is zstd and brotli.. but at this point it would just be better if a "decent general maintained format" could be supported and end of it.

[Sanaki]

At this point it seems like chd is taking over as the cd format of choice, and it's pretty clear why. I'd prefer to be able to stick with that for everything as well. I only use gcz for dolphin because that's what it supports. The same is true of cso for ppsspp (though that's due to cso being usable on the console itself of course). If one day both of those gained support for chd, I'd be quite happy to leave it at that.

[ryz]

@Sanaki Seconded. So far I've been using CHD for PS1, Dreamcast, Saturn and PC Engine CD games and it works great (mainly due to a concerted effort by the RetroArch crew to push it to their cores I guess). so the more emulators adopt this format the better, IMO.

[oblivioncth]

I agree that support for this feature would be useful. The only thing really comparable to CHD right now is CSO, but I've found slightly better compression ratios using GZIP for my PS2 games, though this has the issue of requiring index generation. Additionally, ISO/CSO doesn't work for games with CD audio as that would require multiple files (ISO does not support multi-track). This is also a problem for GZIP since it can't compress more than 1 file (not sure if PCSX2 supports .tar.gz, I don't think it does).

CHD has had stellar compression across all of my romsets for systems with emulators that support it. The tools that can manipulate CHDs are robust and flexible as well.

IMO it is pretty much the best at everything it seeks to be and makes it really straightforward to have a collection with one file per-game with no need to worry about CD vs DVD games or wasted space due to a format with better compression but worse features being available since CHD is already near the top in terms of its typical ratios.

[bajolzas]

So with the release of 1.6.0, is there any plans to support CHD in the near future?

[rodw-wilkins]

I too would very much like to see CHD support for the reasons mentioned!

[romprod]

I'll ditto this! :)

[jfmherokiller]

if this has not been directly implemented in the main branch is there a fork kept updated with it implimented?

[RedDevilus]

I haven't heard of a fork that has CHD support.

[jfmherokiller]

might see if i can add it myself out of curiosity

[jfmherokiller]

#pragma once
#include "AsyncFileReader.h"
#include <chd.h>

class ChdrFileReader : public AsyncFileReader
{
    DeclareNoncopyableObject(ChdrFileReader);
public:
    virtual ~ChdrFileReader(void) { Close(); };

    static bool CanHandle(const wxString &fileName);
    bool Open(const wxString &fileName) override;

    int ReadSync(void *pBuffer, uint sector, uint count) override;

    void BeginRead(void *pBuffer, uint sector, uint count) override;
    int FinishRead(void) override;
    void CancelRead(void) override {};

    void Close(void) override;
    uint GetBlockSize() const;
    uint GetBlockCount(void) const override;
    ChdrFileReader(void);

private:
    chd_file *ChdrFile;
    const chd_header *ChdrHeader;
};

#include "PrecompiledHeader.h"
#include "ChdrFileReader.h"

#include "CDVD/CompressedFileReaderUtils.h"


bool ChdrFileReader::CanHandle(const wxString &fileName)
{
    if (!wxFileName::FileExists(fileName) || !fileName.Lower().EndsWith(L".chd")) {
        return false;
    }
    return true;
}

bool ChdrFileReader::Open(const wxString &fileName)
{
    Close();
    const chd_error error = chd_open(fileName, CHD_OPEN_READ, nullptr, &ChdrFile);
    if (error != CHDERR_NONE) {
        return false;
    }
    ChdrHeader = static_cast<chd_header *>(malloc(sizeof(chd_header)));
    const auto *const tempvar2 = chd_get_header(ChdrFile);
    memcpy(const_cast<chd_header *>( ChdrHeader), tempvar2, sizeof(chd_header));

    return true;
}

int ChdrFileReader::ReadSync(void *pBuffer, uint sector, uint count)
{
    return chd_read(ChdrFile, sector, pBuffer);
}

void ChdrFileReader::BeginRead(void *pBuffer, uint sector, uint count)
{
}

int ChdrFileReader::FinishRead()
{
	return 0;
}

void ChdrFileReader::Close()
{
    chd_close(ChdrFile);
}

uint ChdrFileReader::GetBlockSize() const
{
    return ChdrHeader->hunkbytes;
}

uint ChdrFileReader::GetBlockCount() const
{
    return ChdrHeader->hunkcount;
}
ChdrFileReader::ChdrFileReader(void):
    ChdrFile(),ChdrHeader(0)
{

};

This is what i came up with it correctly identifies the chd file and "attempts to load it" but i don't know how to handle the "count" argument because I don't know if the "hunks" are contiguous

[bajolzas]

@jfmherokiller How is that handled on other emulators that suport it? (I have no idea if it can be compared, I'm just wondering)

[jfmherokiller]

@jfmherokiller How is that handled on other emulators that suport it? (I have no idea if it can be compared, I'm just wondering)

I dont know how its handled on other emulators.

[Tuxie]

@jfmherokiller Without having looked at neither the pcsx2 code nor the libchd code at all, I am pretty sure that these functions are supposed to return what the original file would had returned, and hide the CHD details. ChdrFileReader::ReadSync(void *pBuffer, uint sector, uint count) should read count number of sectors from the original file, starting from original sector number sector into pBuffer.

The chd_read() function seems to read a single CHD hunk into a buffer, so you need to translate the PS2 ISO sector number range into the CHD hunk range, then extract the CHD hunks in a loop into the target pBuffer.

Since the CHD hunk size can be different from the PS2 ISO sector size (2048 bytes), you probably need to chd_read() each chunk into a temporary buffer and memcpy() it into pBuffer, discarding the first part of the first hunk and the last part of the last hunk if they are not aligned with the PS2 ISO.

GetBlockSize() should probably return 2048 for DVD images and 2352 for CD images. Just guessing here but this function is probably not supposed to be overloaded. ChdrFileReader::GetBlockCount() should however return something like ChdrHeader->originalFileSize / GetBlockSize() (pseudocode).

[tastyratz]

Just wanted to tap in here and say the same that I as well would love to see CHD support. CHD is becoming inevitable it seems in emulation and with good reason. The desire seems to only be growing in time.

[whocares0101]

I think I got this working, only tested with x64 release build using VS2019.
The games I tested all boot up, I didn't test much further than the title screen.
Ico (USA) > CD
Breath of Fire - Dragon Quarter (USA) > DVD
God of War (USA) > DL-DVD

A few notes on how to get this going
add ChdFileReader.h and ChdFileReader.cpp to the project, best place pcsx2/CDVD/
pcsx2/gui/MainMenuClicks.cpp needs the .chd extension added in _DoSelectIsoBrowser.
pcsx2/CDVD/CompressedFileReader.cpp needs ChdFileReader added so it can be used.

The following libraries need to be added FLAC.lib, lzma.lib, and chdr-static.lib.
https://github.com/rtissera/libchdr
How to compile rtissera/libchdr#39
I had problems loading CHD files >2GiB but this can be fixed rtissera/libchdr#40

If someone else could add this properly to PCSX2 with all the dependencies that would great :)

ChdFileReader.zip

[refractionpcsx2]

@whocares0101 Can you not make a PR yourself? I tried messing around getting this in for ages and it was a dependency nightmare.

As we already have a version of lzma in there (in the xz folder) it would be nice to link that in, and of course we have zlib too. But what I would really like is for there not to be a dependency folder inside libchdr's folder, so any stuff like FLAC etc appear in the 3rdparty folder. Also note this needs to work for x86 and linux distro's too.

if you can pull that off and create a PR, then we can add it.

[0x90shell]

This is on the cusp of being resolved. Very exciting.

[Papermanzero]

at least for the ps2 bin/cue files chd would be perfect.

[liberodark]

This pr #4314 will be able to close this request.

[refractionpcsx2]

This has now been implemented in master