Unvanquished Forum

I've recently put all of my map assets up on my GitHub account. These are all of the uncompressed assets and original source files for all but a few things (such as the source .blend files for a few of my easter-egg models and the midi tracks for Spacetrack's elevators).

So far, using git for managing my map projects has proven very useful.

Pros

• Easy to keep multiple versions of maps for different releases (alphas, betas, etc)

• Works great with the text-based map format, with a caveat.*

• Other developers and users can see the map progress and try it out at any time; development is more open

• Provides a repository for the original map assets, so the map can be easily updated in the future without having to worry about lossless compression for textures or other similar issues.

• Could theoretically set up a CI server to automatically build the .bsp files from the .map file as commits are pushed.

• The caveat is that Radiant sticks comments into the map file with ever-changing brush numbers, which throws off diff. Not sure if this is an issue or not. I haven't tried merging or anything with .map files :)

Cons

• Your map might get stolen by kharnov and sold to raise money to buy real life grangers.

• Difficult to version large files

Handling really large files, like .psd/.xcf texture files or some .blend models, isn't optimal. Git isn't great for dealing with huge files. And while github offers their large-file extension, it offers a measly 1GB storage/1GB bandwidth per month, which could get eaten up fast. So really large files will need to be archived elsewhere...

We had a brief discussion about compressed assets and how best to handle them. I think that despite the large file issue, using something like a github repo is a strong solution; as Fuma suggests, we could use a Make-like system to automatically compile, compress, and package a map, making "building from source" a snap. We just need a way to deal with huge assets.

I wrote a few more points in favor of using github and a more open mapping process for the Freeway announcement on my blog. While this might not suit everybody, I think mappers should at least consider an approach like this for maps that don't have an inordinate number of new assets.

Thoughts?

Unrelated, here's the .gitignore that I use for my maps:

maps/*.autosave.map
maps/*.bak
maps/chasm/
maps/*.srf
maps/*.prt
maps/*.bsp

I only want the .map file in the repo. No .bsp, no lightmaps, no temporary files produced by the compiler. If you adopt this github, be sure to change "maps/chasm/" to your map's lightmap directory (same name as the .bsp).

I've been using git to version maps for a while, it's great except for when you version your own textures as that makes the repo's size explode.

Supertanker wrote:

The caveat is that Radiant sticks comments into the map file with ever-changing brush numbers, which throws off diff. Not sure if this is an issue or not. I haven't tried merging or anything with .map files :)

I tried to remove the brush number comments from the map file via a git hook but found out that those can only prevent you from commiting a commented map but not alter the commit's or file's content on the fly (if you know a way, let me know).

Supertanker wrote:

as Fuma suggests, we could use a Make-like system to automatically compile, compress, and package a map, making "building from source" a snap. We just need a way to deal with huge assets.

I'm doing just that, except for the BSP compilation that I still do by hand as I always test the results before packaging. My script also compresses the textures on the fly, sorts out unused ones and edits the BSP to include the proper map version. It's not really a portable make system as I developed it over the years to fit my personal needs but I can certainly recommend the concept. I think an actual make toolchain that also builds the map with settings from a config file would be nice to allow people to play around with maps.

The caveat is that Radiant sticks comments into the map file with ever-changing brush numbers, which throws off diff. Not sure if this is an issue or not. I haven't tried merging or anything with .map files :)

I've noticed that many maps I've read don't have correctly numbered brushes, so it can add more noise furthermore.

My map_cutter is able to sanitize a map file: fixing the spacing and renumbering the brushes. I can add an option to strip numbers if you need it.

It's not really a portable make system as I developed it over the years to fit my personal needs but I can certainly recommend the concept. I think an actual make toolchain that also builds the map with settings from a config file would be nice to allow people to play around with maps.

I've also a dirty PoC that does that too, with the ability to convert soundfiles too and fix the filepath inside the bsp if needed.

A build based on a Makefile could work, but I think about a specific tool that can just build your own package from scratch without writing the make rules, only edit those autogenerated if you want to customize them.

The idea is to scan the source directory and recognize each file and take the good decision for each one: lossless conversion, lossy conversion, map compilation, and triggers some extra actions (like the minimap/navmesh compilation when the bsp is ready). My current PoC is able to convert/compile only files that are newer (so, if you only change the map, it will only recompile it and not reconvert assets).

I'm writing this tool for my effort to port foreign maps to Unvanquished, it's better to write tools than doing all by hand: the tool can be reused.
I hope to be able to rewrite my PoC builder in august using Python, to add it to my Granger's Toolbox.

sorts out unused [textures]

Nice idea, since my bsp_cutter is already able to list textures, it's easy to implement. I'm adding that to my TODO list, but this option will be optionnal (to not drop all the textures when you pack a tex package without maps :p)

edits the BSP to include the proper map version

What do you mean?

illwieckz wrote:

Viech wrote:

sorts out unused [textures]

Nice idea, since my bsp_cutter is already able to list textures, it's easy to implement. I'm adding that to my TODO list, but this option will be optionnal (to not drop all the textures when you pack a tex package without maps :p)

Keep in mind some of those names name shaders and you need to parse all shader files for textures, too. Here's the python tool. It also strips shader files. It was designed to be run on a temporary directory so be careful not to call it on your development folder!

illwieckz wrote:

Viech wrote:

edits the BSP to include the proper map version

What do you mean?

The map message is part of the BSP and not the arena file, so I use a placeholder string in the MAP file and replace that with the proper version in the generated BSP. Note that if a script would also compile the map, it may be cleaner to change it in the MAP file first.

# change version in .bsp file
echo ":: Writing map message to .bsp file"
$updateentities ./maps/${mapname}.bsp > entities
sed -i "s,<mapmessage>,$longname," entities
$updateentities ./maps/${mapname}.bsp entities
rm -f ./entities ./maps/${mapname}.bsp.bak