Support for bigger maps

[suns-echoes]

Hello. First of all I would like to thank you for this awesome work :) I learn od M.A.X. just this year and I love it!

I would like to suggest, or ask, if it would be possible to implement support for bigger maps than original 112 x 112. I think that for baseline it would be acceptable for manual replacing map files with custom ones.

Yes, I already tried it, but game just crashed :(

[klei1984]

Hello, maps of arbitrary sizes are planned to be supported, but there are various limitations and design constraints to iron out first in the source code. Current focus is on fixing original game defects and reimplementation issues in the source code. Mid term goals are found on the roadmap page.

I may look into loading issues of non 112x112 maps out of curiosity after I finish with the current defect that I am analyzing, but I make no promises.

Back in March a rough long term plan that would enable arbitrary maps looked like this:

As soon as GUI library is replaced with nuklear, and renderer is changed to opengl, and internal help system is rebased on json format using nlohmann library, and in-game configuration interface is added, and game assets are converted to the new formats, a new saved game format specification will be introduced that allows "infinite" worlds, maps, units stats, custom mission goals, custom campaign sequences, etc. to be saved.

You can find the non published long term road map goals here. They will either happen or not. Future will tell.

M.A.X. has way too many defects to fix that block most other things 😞

[klei1984]

@suns-echoes , do you have a non 112 by 112 map that I could use for testing?

[suns-echoes]

@klei1984 Yes, I will upload it here.
I'm working on WRL maps editor and it was resize by this software.
It is 160 x 160 regular SNOW_1 map but padded with the same tiles that were on the edges.
Internal size is set to 160x160, minimap is also enlarged.
SNOW_1.WRL.zip

[klei1984]

🤣

The map tile decoder causes the first crash. That is an algorithm that was super duper overly optimized and was written entirely in assembly. It will be quite a challenge to rewrite it in a more flexible, less optimized way considering that I have absolutely no clue how it works. I rewrote the assembly in C without understanding the wild stuff it actually does...

[suns-echoes]

Well, I know that tiles are 64x64 pixels, that is 4096 unsigned 8 bit arrays embedded in map and each pixel is an index to embedded palette color. But you probably already known that 🙂

ps. It was even worse with 200x200 map :D

[klei1984]

There is a single algorithm for the normal resolution tile loader and for the low resolution one where tiles are resampled for 32x32 pixel size on the fly using address masking horror tricks on strictly aligned buffers. I hope that we will find a simple hard coded offset somewhere that could be easily fixed in that messy code... otherwise it will take a lot of time to redo what it does in a human fathomable way. Again, the original devs were true evil geniuses.

[suns-echoes]

Unfortunately tile count comes after minimap and map and tiles right after that.

[klei1984]

By the way, did you know that the word at file offset 3 is the map layer count with value range 1 - 12? All official maps have 1 single layer only though. Even more wicked, in theory each layer could have a dedicated minimap preview where the default is the highest layer index available. But instead of using the layer count from offset 3 to iterate over the minimaps, the released game eventually replaces the layer count and hard codes a value of 1.

Another interesting map file format fact is that half of the 256 color RGB666 palette stored in the world file is just junk. Only colors between index 64 - 159 are loaded, the rest are scrapped. Even more, only colors 64 - 95 and 128-159 can be used for land surface and vegetation as colors 96-102,103-109,110-116,117-122,123-127 are used for water caustic effects. Even more, the system palette needs to adopt these colors also for all units and GUI graphics assets which means that on the snow worlds the GUI is a bit brighter compared to the desert world.

[suns-echoes]

Hmm. What about campaign maps? Maybe they use layers for buildings and units? Or maybe it was planned for something else that was never implemented.

This is crater 1 map extract :)

This is snow 1 :)

it's interesting how many optimizations on such limited hardware had to be done, and how clever tricks were implemented in old games. Yet, it's so hard to fix, change, enhance such optimized code. Not event to say, disassemble and change in any way.

[klei1984]

What about campaign maps?

Eventually the released game does not use the layers. We will never know what was its real purpose. The campaign missions are just hacked save game files. The messages from the Reports menu is a good evidence for that. The campaign mission win loss conditions are hard coded into the game logic. E.g. if we swap mission 1 and 2, their loss conditions do not get swapped. Scenarios have no special win loss conditions.

Not even to say, disassemble and change in any way.

Actually we are very lucky. M.A.X. 1 was compiled with (most) optimizations disabled. I assume they did this as the original smart container classes violated certain C++ principles, like the this pointer can be assumed to be never nullptr by the optimizer thus as soon as they enabled certain optimizations the game just crashed. This made it possible to learn the compiler's C++ constructs and to decipher the disassembly much easier. I can easily read the assembly as if it would be C code. The only part of the game that had optimizations enabled are the C libraries like Tim Cain's GNW engine or the MVE video decoder library (most of it was assembly though). But the GNW engine is compiled with full debug symbols inside Fallout 2's mapper without optimizations so the GNW95 version was easy to replicate and then it was easy to decipher the M.A.X. 1 version as well.

it's interesting how many optimizations on such limited hardware had to be done, and how clever tricks were implemented in old games.

Fully agree! E.g. palette manipulations were simple memory copy operations that the VGA HW swapped effortlessly or instantly. So instead of changing texture or pixel buffers, games used color animations by changing palette colors. M.A.X. 1 implements several glow effects, the factory smoke, 3 water caustic effects, the coastal water wave effect and complete brightness control, e.g. shadows, this way. Then in M.A.X. 2 they switched to 16 bit color space and at that time emulating the 256 color palette animations by a software renderer was out of question so all of the cool effects were gone for good. The assembly hacks Interplay did within the MVE video player is even more stunning. I.e. they implemented self modifying code like viruses of the era!

[suns-echoes]

Speaking of campaign and hacked save files. Are there somewhere any resources that allow to learn save files format? I was considering extending my map editor (after finishing main features) with ability to edit save files to make more customized games.

[klei1984]

This is the save file format documentation. It can be considered now a bit inaccurate in some minor aspects, but nothing that the game's source code would not clarify. The ImHex profile implements the format parser which you could take as basis. If required I can share the updated profile which works with latest ImHex release version.

[suns-echoes]

Wow, that's great job.

[klei1984]

Made some progress. The game hard coded the map dimensions in a dozen places and certain map size dependent buffers were allocated once assuming the fixed 112x112 size.

It seems that water tiles normally follow a certain pattern. I did not realize this until I saw the resized mini map.

I refactored the tile decoder algorithm too. Now the error is clear. The map file contains a 16 bit tile index table. The Wrldisasm tool from RuMaxClub exports this segment as bigmap.dat. This segment is corrupted in the file. There are only 337 tiles in the map, thus the biggest tile index in a 16 bit word could be 336d or 150h. In little endian mode this is stored as 50 01.
Here we have 09 09 or 909h in the first word:

And at 160*160 offset, something strange happens in the data pattern:

Just for reference, the same file segment in the original map file looks as follows:

Could you please verify whether the tile index table is correctly generated?

Update
Made further progress using a different map from M.A.X. Reloaded. The tile decoder works fine.

The game still crashes at some point, but this is good progress, and further proof that the tile index table is corrupted in the modified snow map.

[suns-echoes]

Okay... maybe I made wrong assumptions.

this is how I read original map:

const buf = new BufferReader(wrl);
const header = buf.readBytes(5, 'header');
const width = buf.readUInt16LE('width');
const height = buf.readUInt16LE('height');
const minimap = buf.readBytes(width * height, 'minimap');
const bigmap = buf.readBytes(width * height * 2, 'bigmap');
const tileCount = buf.readUInt16LE('tileCount');
const tiles = new Array(tileCount);
for (let i = 0; i < tileCount; i++)
	tiles[i] = buf.readBytes(64 * 64, `tile[${i}]`);
const palette = buf.readBytes(256 * 3, 'palette');
const passtab = buf.readBytes(tileCount, 'passtab');

And I assumed that when I resize map I should also resize minimap.

So:

reading <header>: 5 bytes from 00000000
reading <width>: UInt16LE from 00000005
reading <height>: UInt16LE from 00000007
reading <minimap>: 12544 bytes from 00000009
reading <bigmap>: 25088 bytes from 00003109
reading <tile count>: UInt16LE from 00009309

reading <tile[0]>: 4096 bytes from 0000930B
reading <tile[1]>: 4096 bytes from 0000A30B
...
reading <tile[335]>: 4096 bytes from 0015830B
reading <tile[336]>: 4096 bytes from 0015930B

reading <palette>: 768 bytes from 0015A30B
reading <pass table>: 337 bytes from 0015A60B

and after resize:

writing <header>: 5 bytes at 00000000
writing <width>: UInt16LE at 00000005
writing <height>: UInt16LE at 00000007
writing <minimap>: 25600 bytes at 00000009
writing <bigmap>: 51200 bytes at 00006409
writing <tileCount>: UInt16LE at 00012C09

writing <tile[0]>: 4096 bytes at 00012C0B
writing <tile[1]>: 4096 bytes at 00013C0B
...
writing <tile[335]>: 4096 bytes at 00161C0B
writing <tile[336]>: 4096 bytes at 00162C0B

writing <palette>: 768 bytes at 00163C0B
writing <pass table>: 337 bytes at 00163F0B

[suns-echoes]

I'm checking data segments content. I thing I did something wrong there.

[klei1984]

Both minimap and bigmap need to use width * height for dimensions, that is correct. But while minimap has byte elements, bigmap has 16 bit words. If you read bigmap as a byte array (w * h * 2), you need to convert that to an array of size width * height with array element type UInt16LE.

[suns-echoes]

Yes. The bigmap is converted in higher layer for displaying/editing the map.

Can you explain more about this corruption in tile index table? Do you mean "bigmap" or "list of tile images"? I'm trying to understand this problem.

I attach zip with WRL file and all chunks for easier debugging.

SNOW_1_160_PARTS.zip

[klei1984]

The tile images are fine. I meant the "bigmap". From the snow_1_160_parts.zip this corresponds to segment inside 4.bigmap.bin. The big map is a 1 dimension array, or a vector, where a grid position corresponds to array_of_uint16le_words[width * y + x]. The array elements are 16 bit unsigned words, or UInt16LEs.

Inside snow_1_160_parts.zip the segment 4.bigmap.bin is correctly stored. But inside the assembled SAVED_MAP.WRL it is wrong.

Corrrect 4.bigmap.bin contents at start of file:

Corrupted content inside assembled SAVED_MAP.WRL at file offset 6409h:

[suns-echoes]

Thanks, I will investigate this further. It looks like writing uint16 went wrong.

[suns-echoes]

Okay, there was unintentional conversion from 16 bit to 8 bit array while saving file. Here I upload fixed map :)

SAVED_MAP_FIXED.WRL.zip

[klei1984]

Its working good.

The version I used here is not up on GitHub yet as minimap part is still broken in various ways.

[suns-echoes]

Great :) I can't wait till maps can be big enough so players/cpus bases defenses won't overlap :)

[klei1984]

Yeah... well... then it would be more obvious how stupid the computer AI is :D

Computers use transporter units very inefficiently and as soon as transporters become available they do not build bridges. They plan strategic assaults as a mess without any unit formations or much coordination between participating units. Most just escort an arbitrary assault leader unit that is usually slow, stalling all faster units, and those escorts do not protect each other proactively. If they lose their single recon unit they select another from the other end of the world... I am not even starting to rant about the unit task prioritization issues and how they steal each others' transport units. Oh and I did not mention how computer agents above expert difficulty are cheating all around which will lead to even bigger bases and more units standing inside those bases as defense reserve against another computer that is idling on the other side of the map doing the same arms race.

[suns-echoes]

Speaking of dumb A.I.
Don't even think about getting feet into this puddle! 😂

[klei1984]

In case you consider to support original color animations and water caustics generation for new or modified tile sets, I could write an article on the palette format. In the mean time this might be useful in itself:

The color animation plays out as per the table below:

struct ColorCycleData {
    uint16_t start_index;
    uint16_t end_index;
    uint8_t rotate_direction;
    uint32_t time_limit;
    uint32_t time_stamp;
};

static struct ColorCycleData GameManager_ColorCycleTable[] = {
    COLOR_CYCLE_DATA(9, 12, 0, TIMER_FPS_TO_MS(9), 0),     COLOR_CYCLE_DATA(13, 16, 1, TIMER_FPS_TO_MS(6), 0),
    COLOR_CYCLE_DATA(17, 20, 1, TIMER_FPS_TO_MS(9), 0),    COLOR_CYCLE_DATA(21, 24, 1, TIMER_FPS_TO_MS(6), 0),
    COLOR_CYCLE_DATA(25, 30, 1, TIMER_FPS_TO_MS(2), 0),    COLOR_CYCLE_DATA(31, 31, 1, TIMER_FPS_TO_MS(6), 0),
    COLOR_CYCLE_DATA(96, 102, 1, TIMER_FPS_TO_MS(8), 0),   COLOR_CYCLE_DATA(103, 109, 1, TIMER_FPS_TO_MS(8), 0),
    COLOR_CYCLE_DATA(110, 116, 1, TIMER_FPS_TO_MS(10), 0), COLOR_CYCLE_DATA(117, 122, 1, TIMER_FPS_TO_MS(6), 0),
    COLOR_CYCLE_DATA(123, 127, 1, TIMER_FPS_TO_MS(6), 0),
};

For example water caustics 1 ranges between color indices 96 to 102 which is defined in the above table as COLOR_CYCLE_DATA(96, 102, 1, TIMER_FPS_TO_MS(8), 0) where rotation direction 1 means rotate from left to right and time limit of 8 FPS means rotate one color every 8 frames per second so rotate once every 125 milliseconds.

One of the rules in the above table that have a single color as color range is handled in a special way. COLOR_CYCLE_DATA(31, 31, 1, TIMER_FPS_TO_MS(6), 0) is used for the blinking green user messages like "Research Complete" or "Unit Ready to Deploy". Palette colors use a pixel format of RGB666. Index 31 has a single green color component set to maximum, its red and blue components are off (or zero). This rule halves its green color component at every 6 frames per second or ~166 milliseconds until the value drops below 8d in which case it resets back to maximum level (of brightness or intensity) to 63d.

[klei1984]

The build instructions are not updated for 0.7.0 yet. I am working on the how to install the game article currently.

The version on master branch now complies to the XDG Base Directory Specification and the Deb package follows the official Debian packaging guidelines and is not anymore produced by CMake's CPack module. For Linux systems now we have Deb, Flatpak and Arch Linux Binary packages that are built by their respective official toolchains and are ready to be submitted to official package repositories like Debian's GitLab. I will not actually submit anything. I will not become a package maintainer for any OS.

You can take a look at the GitHub Actions page where you can download the latest Deb, Flatpak, Arch binary packages from their dedicated GitHub Actions jobs. If you want to build the Deb package locally from source, you can take a look on the GH Actions script here which installs all required tools and invokes the appropriate scripts.

Simply building the game on Linux is not anymore sufficient to run it as the game follows the XDG folder structure and requires to know where to find original MS-DOS M.A.X. game data files too. I actually never tried to invoke CMake's GNU Make install command directly with this new system it might not even work... if you are on Ubuntu x64, just download the Deb from GH Actions or build the package based on the above referenced GH Actions script. Then install as usual e.g. sudo apt install -y ./max-port_0.6.1-1_amd64.deb. Ignore the old version number...

After installation, the game shall be started with the command max-port or with the deployed Desktop icon M.A.X. Port. The entry point is actually a shell script that verifies XDG user folders and if needed invokes a text interface GUI to configure the game. On later runs the text interface is not called, its a one time configuration normally.

If the XDG stuff exists, directly invoking the max elf executable would also work of course, but I am not getting into the boring details... just use the new Dektop icon and done... unless you want to debug the game under linux, then its a different story.

[klei1984]

The updated build instructions and installation guideline are published on the home page.

[suns-echoes]

Thank you for that update :) I'm creating and testing new map 224 x 224. It looks good so far. However there is one thing.

I'm using 3440x1440 resolution and this makes lower part of the map unselectable.

[klei1984]

On the landing zone selection screen mouse hover is not implemented. You need to use keyboard arrow keys to move around. On laptops you may need to toggle Num Lock to have arrow keys.

[suns-echoes]

Okay :) Numlock did the job :)

I will upload the map when it's finished.

[suns-echoes]

I have finished mapping all tiles for valid adjacent tiles. Grouping them by category, merging all tiles sets based on planet type. Partially done tile templates based on how they are used in game. This will be super useful for auto-shore, fill and random functions in map editor.

But here arise new question. Is there a limit of how many tile types can be used in one map? @klei1984

[suns-echoes]

Curious case. I might be wrong but I think I saw animating pixel on another tile. If I spot it again I will let you know.

[klei1984]

By the way you might be interested in the internal test build of M.A.X. that was leaked by a gaming magazine as it contains some alternative maps and more tiles. E.g. SNOW_2 in the retail version has 407 tiles while in the internal test build it had 472. Maybe the additional tiles are just duplicates, but who knows... The internal test build was leaked by the Secret Service gaming magazine and it can be found on Cover CD # 38 from September, 1996. Its easy to find.

SNOW_1.WRL in the V1.00 press release demo is the same as the one found in the above test build. But there was an updated v1.01 demo too which has one more version of SNOW_1.WRL which is not found anywhere else. The last v1.03a demo has the same map revision as the retail game.

[suns-echoes]

@klei1984 I found this CD, but I was only able to find txt file and some pcx images related to M.A.X.

Is this internal build somehow hidden there?

[klei1984]

Oh sorry, my mistake. It is on Secret Service magazine Cover CD #40 - November, 1996.

[suns-echoes]

Got it :) I will investigate it tomorrow :)

[suns-echoes]

ANNOUNCEMENT:

I present the first public preview of M.A.X. Mechanized Assault & Exploration Map Editor. This is still a work in progress. Updates will be posted on this site:

https://suns-echoes.github.io/max-game-map-editor/

[suns-echoes]

I hope this is legit to post here :)

[klei1984]

Sure, but I will post it on our discord server too, just in case...

[suns-echoes]

@klei1984 we have Discord and I don't know about it?

[klei1984]

I admit, its just a badge on the GitHub front page, but I have actual links at other locations.

This is the link to it: https://discord.gg/TCn8DpeBaY