Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tools: Port refraction's bulk compression script to Python #11574

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

TheTechnician27
Copy link
Contributor

Description of Changes

Converts refraction's bulk compression script which was presented as a Windows-only batch file into a cross-platform Python script.

Rationale behind Changes

Makes the tool cross-platform, putting Windows, Linux, and macOS users (freeBSD?) on a more equal footing and allowing us to ship the tool with PCSX2 instead of tucked away in Red's PCloud.

Suggested Testing Steps

Try all 9 options and try them with and without deleting source items as well as different block sizes if compressing. Presently needs tested on all three major OSes, and it may work in freeBSD, but there's no guarantee at all for that. Until it has been properly tested by multiple people, USE THIS IN A CONTROLLED TESTING ENVIRONMENT WITH JUST A FEW FILES WHICH YOU HAVE BACKED UP ELSEWHERE.

tools/bulk_compression.py Outdated Show resolved Hide resolved
tools/bulk_compression.py Outdated Show resolved Hide resolved
bin/utils/bulk_compression.py Outdated Show resolved Hide resolved
@TheTechnician27 TheTechnician27 force-pushed the compressiontools branch 4 times, most recently from cd96ba3 to 77843da Compare July 18, 2024 15:49
@TheTechnician27 TheTechnician27 marked this pull request as draft July 18, 2024 21:25
@TheTechnician27 TheTechnician27 force-pushed the compressiontools branch 3 times, most recently from acc04c0 to 1251bfc Compare July 19, 2024 01:21
case _:
print("You definitely should not be here right now.")
sys.exit(1)

Copy link

@fizwidget fizwidget Jul 31, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’ve been using createdvd with my games. Just wondering why createraw is used here - is there a benefit, or much of a muchness?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made the script before createdvd was an option, but also I find createraw tends to give slightly better compression, from my own tests.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also they share similar options, createraw has more versatility in the way it has more options like createdvd doesn't have No --unitsize Option:'

Both have the same compression types (default is lzma,zlib,huff,flac).

But with createdvd you can easier give the correct hunksize. Createraw will default to correct one but you can change to whatever for more or less compression.

Generally looking at the options they do have the same compression powers just differently handled with the code logic.

@crashGG
Copy link
Contributor

crashGG commented Sep 23, 2024

Is there any way to modify the compression judgment condition of chdman? According to my observation, when chdman currently selects multiple compression algorithms for mixed compression, the final compression algorithm is only roughly judged based on the final block size.This will give the smallest final chd size, but it's not smart enough.
If there is ,for example, when the compressed block size exceeds 95% of the original block, the compression is abandoned and the block is stored directly. When the lzma compressed block size is less than 90% of the zstd compressed block, it is used, otherwise the zstd compressed block is still used. This won't be the smallest size, but it's the most balanced approach, giving you a good balance between size and cost. Maybe.

@lightningterror lightningterror added this to the Release 2.4 milestone Oct 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants