[Code Help] How do we read/parse a character's name out of Spawn/Diablo/Hellfire SP and MP files?

[donotsdubba]

Hello, my apologies if I missed seeing this being discussed elsewhere before.

I'm creating a read-only helper tool for all vanilla versions of Diablo, Diablo Spawn and Hellfire, and the only point I'm stuck at is figuring out how to extract a character/hero/player's name from a save file. Using C#, I believe I can easily read the data using some FileStream-like class and read it all as an array of bytes, then parse the exact location accordingly.

Thing is, when I used a hex editor, I saw that all the character data inside a file is "encrypted", although there is a hint at the top (header?) that it is a MPQ-format file, like many other Diablo files? In any case, I peeked and poked around both Devilution's and DevilutionX' C++ source codes, and while I think I located some of the relevant portions (MPQ file parser logic? Character data parsing logic?), I honestly could not figure out how it was actually doing all that...

Rather than reinvent the wheel, I thought I'd ask you guys who work on the code what is the simplest approach to parsing such files. All I really need is the character name, nothing more...

Incidentally, I believe I have seen somewhere before that the character name is supposed to be stored in a char array of 32 positions. Is that right?

Thank you very much for any help in advance,
Dubba

[qndel]

even to get "just the character name" you have to decrypt the MPQ

[donotsdubba]

Makes sense, thanks. Would you be able to describe what steps need to be taken programmatically to decrypt it, store its decrypted data, and extract the hero name out of it?

[qndel]

google is your friend, find some existing implementation of unpacking mpqs, use it

[donotsdubba]

Naturally I websearched first, specifically for D1 save file data extraction rather than general MPQ extraction, since MPQ-extracting logic is already in DevilutionX, yet, like I said in the original post, I couldn't understand the implementation. (Plus the fact I wasn't sure that character files are MPQ ones.)

This time, I went and searched specifically for MPQ tools, yet all I see are tools that can extract the multiple inner files an MPQ contains, but nothing that allowed me to peek inside the data of any of the extracted files in a way I know how to work out. (It looks encrypted, still?)

To name one such tool, I came across "DevilutionX MPQ tools", and I see it successfully extracts a file called "hero" from a Hellfire Single Player file (but not ".drv" Multi Player ones), yet the "hero" file cannot be coherently figured out from opening it with a text editor nor hex editor. Peeking at its source code, I am as stumped about understanding how it is extracting the hero data and making sense out of it as when I was looking at the Devilution/DevilutionX sources.

Perhaps someone else may chime in?

[StephenCWills]

I see it successfully extracts a file called "hero" from a Hellfire Single Player file (but not ".drv" Multi Player ones)

DevilutionX MPQ tools uses libmpq under the covers. It should be able to unpack the .drv files just the same. However, note that instead of using a hardcoded password, the .drv files use the machine name as the password.

https://github.com/diasurgical/devilution/blob/bbda8dd586c65b03028ec75c52f8ea8627eb9ff5/Source/pfile.cpp#L190

There are some gotchas here. First, GetComputerName() returns the name of the machine in all caps. Second, their buffer is only 16-characters, which means it should only read the first 15 characters if the machine name is longer than that. Third, the codec_init_key() function will expand the password to 64 bytes by repeating the machine name over and over again, but the way that DevilutionX optimizes this function doesn't work for any arbitrary-length password. So be careful if you are trying to use the DevilutionX functions to decode .drv files.

[AJenbo]

.drv are the old file format that is encrypted to the specific computer name, hellfire doesn't have official multiplayer and never got the updates to make the saves portable. But maybe you could implement a tool to convert them. If you decide to try and tackle them I would wait til after you have figured .sv out. Also worth noting is that hero is the same format in both games, but game is slightly different.

I don't know much about C# but it feels like it's not a particular good fit for the task, you woulg have to interface with C code since I don't think there are mpq implementations for it and it isn't as portable as C++ (try making a GUI that also works on Linux).
And you won't be able to use any examples, or draw on help form most who knows how things work since they work with C/C++.

As for "it looks encrypted" did you provide a decryption key when unpacking the files, if not you can be pretty sure you haven't performed the decryption yet.

[StephenCWills]

This is the function that decodes the encrypted files.

devilutionX/Source/codec.cpp

Lines 91 to 127 in 1078598

	std::size_t codec_decode(std::byte *pbSrcDst, std::size_t size, const char *pszPassword)
	{
	uint32_t buf[BlockSize];
	uint32_t dst[SHA1HashSize];
	SHA1Context context = CodecInitKey(pszPassword);
	if (size <= SignatureSize)
	return 0;
	size -= SignatureSize;
	if (size % BlockSize != 0)
	return 0;
	for (size_t i = 0; i < size; pbSrcDst += BlockSizeBytes, i += BlockSizeBytes) {
	memcpy(buf, pbSrcDst, BlockSizeBytes);
	ByteSwapBlock(buf);
	SHA1Result(context, dst);
	XorBlock(dst, buf);
	SHA1Calculate(context, buf);
	ByteSwapBlock(buf);
	memcpy(pbSrcDst, buf, BlockSizeBytes);
	}
	memset(buf, 0, sizeof(buf));
	const CodecSignature sig = GetCodecSignature(pbSrcDst);
	if (sig.error > 0) {
	return 0;
	}
	SHA1Result(context, dst);
	if (sig.checksum != dst[0]) {
	LogError("Checksum mismatch signature={} vs calculated={}", sig.checksum, dst[0]);
	memset(dst, 0, sizeof(dst));
	return 0;
	}
	size += sig.lastChunkSize - BlockSizeBytes;
	return size;
	}

You'll need to roll your own SHA-1 implementation, I think, because of the note in SHA1CircularShift().

devilutionX/Source/sha.cpp

Lines 18 to 19 in 1078598

	// The SHA-like algorithm as originally implemented treated word as a signed value and used arithmetic right shifts
	// (sign-extending). This results in the high 32-`bits` bits being set to 1.