Alpha 27 map upgrade

Maps and texture sets.
Post Reply
User avatar
Viech
Project Head
Posts: 2139
Joined: Fri Aug 03, 2012 11:50 pm UTC
Location: Berlin

Alpha 27 map upgrade

Post by Viech »

Hello fellow mappers! For the upcoming Alpha 27 release (on 2014-05-04), there's a bit of work to do for us. If your name is Emperorjack, Pevel or Supertanker, please leave a read receipt and give an estimate whether you will be able to get this done before the target release. (Yes, you need to read all of this. No, you will resist from making TL;DR jokes because I wrote this for the three of you and believe me this took much longer than you will need to read it. :p) Also please use this thread to give any feedback or error reports regarding the resources or manuals I provide you with.

Stuff that should be done for the next release of your map, ideally for Alpha 27

Shared texture packages

They have arrived! Atleast preview versions of them. If you don't hit any issues using those, they will most likely be the first versions to be shipped with the game.

For now there are packages available for the following sets:

  • ex by evillair. This is the main set used on Parpax and Thunder.

  • exm by me. These are additions to ex used on Parpax. (evillair granted us the right to create derivate works.)

  • pk01 & pk02 by philipk. These are used on Plat23, Chasm, Yocto, Parpax, not sure about Thunder.

  • trak5 by TraK. A big set with many color variations that is currently lacking specular maps. Used on crates on Parpax and quite a few other places.

  • space by Ingar. These are the space skyboxes that were posted in this thread.

  • tech by cr4zyb4st4rd. This is the set from this thread in the findings subforum.

[b]Advantages of shared packages and texture compression, skip if you want[/b] wrote:

First of all, as the name suggests, these textures can be shared among the maps, without one map depending on the other. This means that map sizes will be significantly smaller which makes updates less painful for everyone. Obviously this alone will already reduce the total (download) size of an Unvanquished installation. New mappers will also have a quicker start given that they don't need to download and cleanup the individual texture sets or write shaders and readme file entries for them. However, the biggest advantage is in the compression itself. With regard to file size, the compressed packages are much smaller than the zip-compressed source format sets. While the latter are 335 MB total, the compressed sets total up to 109 MB, so literally less than a third. The second big advantage is in the map load time. From what I know, we spent a lot of time, if not most of the time during the map load process, decoding the different texture formats. This part is now completely gone since the new format can be loaded directly onto the GPU. I haven't measured myself but previous tests with compressed base assets showed that the load times were reduced significantly. Pepople have also reported reduced memory usage since we compressed the base assets. Last but not least, I have found that compressed textures unexpectedly look a lot better since the effect of normal mapping is more strongly defined. All in all, I think the upgrade is very much worth the effort.

In order to upgrade your maps, I heavily recommend using Chameleon. The reason is that I tried to use a consistent naming scheme on all the sets. While this will sort the textures in your radiant preview in a meaningful order, this means that you cannot just run sed on your map file to replace the names of the sets. It would also be a PITA to use the very limited texture replacement capabilities of netradiant. The extra effort to install the Python environment and get Chameleon to run will be worthwhile. Make sure to use the latest version of Chameleon with added support for compressed tga, webp and many more formats.

More precisely, what I ask you to do is replace every usage of one of those textures in your map with the textures found in the shared packages. There are different approaches to do so: The first approach is simply using Chameleon to replace the shaders in your map with the corresponding shaders in the shared_* namespace. This should work for most textures. However, in case of lights with custom colors, you will want to ship your own shaders. There are two ways to do so: You can edit your old shaders to point them to the new source files (simple but doesn't produce viable feedback for me), you can extract and modify the shader files inside the packages (recommended; make sure to change the namespace!) or you can use Sloth to generate a whole new shader file from the sources, which I can provide you with when you message me on IRC (they will end up on our upcoming asset host eventually). Unfortunately, sloth cannot currently read .crn files and will fail when used on the compressed texture folder.

Download the latest versions of the preview packages here. You can put them in the pkg directory in the Unvanquished home folder. (In case you received some work in progress version of any of these pk3s before, overwrite them.) Be aware that the new filesystem requires you to explicitly specify dependencies in a file named DEPS in the root of your pk3(dir). For example, if you wanted to use the ex and the pk01 set in addition to the common shaders (which I will write on in the next section), the file would look like this:

Code: Select all

tex-common
tex-ex
tex-pk01

I also added a convenience package that will simply include all other packages, including the common shaders. In your DEPS file, just add the single line:

Code: Select all

tex-all

Before you start retexturing your map, also read the section about gamma renormalization, as you migth want to use lights of higher intensity.

Official™ common shader package

Technically, this is just another texture package but the importance of this set (and the amount of work I had to put in it) made me create a seperate section.

The tex-common_<version>.pk3 package contains both the common.shader file that hosts all the special shaders needed by mappers as well as the corresponding tetxures displayed inside radiant. It does not include any model support files, so buildables will show up incorrectly when you replace Ingar's support package with this. Please do so anyway, so the shaders don't clash. The model support will need to be shipped in a seperate package that shows our new models, ideally together with a fresh entities.def.

I wrote the common.shader file from scratch and it is ridiculous how outdated and wrong the old one was. Many keywords were superfluous (e.g. nomarks is implied by either nodraw or nonsolid, nolightmap is implied by nodraw, etc.) and quite a few of them were simply wrong. There was no difference between nodraw and nodrawnonsolid (both were non-solid) and a few shaders were entirely useless, such as missileclip, noimpact and nolightmap. Here's a list of rules for replacing the old shaders with new ones:

  • clip → Now named playerclip.

  • weapclip, full_clip → Were the same and are now named fullclip.

  • missileclip, nolightmap, noimpact → Removed.

  • nodraw → Is now solid. If you want a non-solid version, use nodrawnonsolid.

  • fullclip → It actually works now but the downside is that radiant will currently not filter it properly. (It previously used the playerclip keyword to lure radiant into treating it as a clip, but this made the shader work like a player clip in return…)

There were quite a few other fixes and changes that I forgot about but shouldn't pose any issues to you (if they do, post about it!). There is also a new shader named black that is basically a solid wall that will always appear pitch black. Should be useful for abysses and cracks. In addition, the shader file is now commented and explains what the different shaders do and can be used for.

Thanks to Ingar for providing the radiant texture sources. The entire content of the package is now in the public domain (more precisely under CC0) so you can use, sample, modify and distribute the content without worrying about anything, not even about attribution.

Please be very considerate that you never ship shaders using the textures/common/ namespace with your map, not even the unmodified ones! While the Daemon engine will not load data from external maps if not explicitly asked for via the DEPS file, radiant and q3map2 could be using your versions of the shaders instead of those that will be shipped with Unvanquished, which can pose hard-to-debug problems to mappers. Ideally, the shaders you find in the package should be sufficient and working for you. If you find any errors or are missing something, I'd appreciate it if you tell me instead of just shipping a custom shader file, so that I can make the modifications available to all mappers.

Gamma renormalization

We have been using a default gamma value of 1.3 so far. This was a very bad decision because obviously mappers didn't want their maps to look washed out and thus adjusted to the new setting, not only making it useless in the first place but also reducing the resolution of the overall brightness distribution as well as the player's clearance for further gamma adjustments (artifacts show up sooner). We aim to correct this with the release of Alpha 27 by resetting it to 1.0 for all clients. In addition, we will also reset r_specularScale to 1.0. These changes mean that all our maps will need to be recompiled with different lights or compile settings or else they would look too dark.

So in short the goal is to make your map look good when r_gamma and r_specularScale are both set to 1.0.

Personally, I found that either passing -scale 2.0 to q3map2 in the lighting phase or using shader lights that have twice the old intensity (which is technically the same) did the trick. Have a look at my experiments with Parpax to see the results. In future maps, you can either add this switch to your default compile options or simply use bigger lamps. Don't use q3map2's -gamma switch, it will screw up the lightgrid. Also please talk to me before using any compile switch apart from -scale so I can test whether this endangers stylistic consistency. Note that increased light intensities can increase map compilation time, but the results are bound to look better than any simple filter applied to the lightmap.

If you don't find the time to complete all the bullet points before Alpha 27, please atleast complete this one so we can introduce the gamma change without your map looking too dark.

Tone mapping optimization

Tone mapping, also known as color grading, is used on most if not all our maps but there are some quirks I'd like to get rid of.

First of all, make sure your tone map is saved in a lossless format. They are very fragile and should not be compressed in a lossy manner. Pevel had to find out that his webp export plugin wasn't doing a lossless export even if the slider was set to 100% quality, leading to color artifacts ingame. If you are unsure, use a different format that doesn't support lossy compression (tga and png should be safe). There are also reports from Mac users that webp images will generally cause tone mapping to fail. I haven't tested the crn format in-depth yet. It's lossy, but maps compressed with it looked fine at a quick glance anyway.

The second point is about what should or should not be achieved with tone mapping. Obviously, they should be used to change the color tone of your map (duh!). I once suggested to Pevel that he should use it to increase contrast, which lead to yocto looking inconsistent with the rest of our map base. I would now advise strongly against changing contrast with tone maps. Please also don't adjust brightness with them. Currently you cannot disable map color grading, but it is not unlikely that users will be allowed to disable this effect in the future. This should, however, not influence how bright the game appears to them.

Levelshot

Ideally as your last step but not before you adjusted to neutral gamma (read section above) please take a new levelshot so it shows with appropriate brightness and matches the new appearance of your map.

Reverb

If your map doesn't yet use reverb, add it! There is a wiki article on how to do it and there is a huge set of presets that you can use, potentially with reduced or increased intensity.

Navmeshes

Bot navigation meshes should be shipped within the map from now on. You can find a tutorial on the generation process here. If you need help compiling them for your map, ask me or ds on IRC.

Stuff postponed for the next upgrade wave
(You can stop reading here if you like.)

Entitiy and keyword names

Tim has rewritten the entity system a while ago and he used a new naming scheme for map entities and their keywords. While the old names still work, you will get deprecated warnings when your map loads. Eventually we want to get rid of the old naming scheme but right now this would be inconvenient to do as we just began working on a clean and up-to-date entities definition file. If you want to help out with that, please refer to this thread.

Map objects

We will want those in the future and we will want them to recurr among the official maps to create a more consistent atmosphere. The reason why we can't add them right now is because no high quality map objects seem to exist whatsoever. If you think you know a resource, please use the findings subforum. However, I'm pretty sure the only viable path will be using our main asset pipline (concept art → modeling → animation) once all the game objects are complete, so that we get models of similiar quality as our buildables.

Responsible for: Arch Linux package & torrent distribution, Parpax (map), Chameleon (map texture editor), Sloth (material file generator), gameplay design & programming, artistic direction

User avatar
Viech
Project Head
Posts: 2139
Joined: Fri Aug 03, 2012 11:50 pm UTC
Location: Berlin

Re: Alpha 27 map upgrade

Post by Viech »

ex 1.02: Added missing texture floor_grate_3.

Get the latest versions of all packages here.

Responsible for: Arch Linux package & torrent distribution, Parpax (map), Chameleon (map texture editor), Sloth (material file generator), gameplay design & programming, artistic direction

User avatar
Viech
Project Head
Posts: 2139
Joined: Fri Aug 03, 2012 11:50 pm UTC
Location: Berlin

Re: Alpha 27 map upgrade

Post by Viech »

common 1.02: Converted all textures to PNG24 for greater compatibility.
exm 1.02: Added three textures (transparent versions of shared_ex/floor_grate*).
pk02 1.02, trak5 1.02: Fixed typos in "surfaceparm alphashadow".

Thanks to Pevel for the reports.

Responsible for: Arch Linux package & torrent distribution, Parpax (map), Chameleon (map texture editor), Sloth (material file generator), gameplay design & programming, artistic direction

User avatar
Viech
Project Head
Posts: 2139
Joined: Fri Aug 03, 2012 11:50 pm UTC
Location: Berlin

Re: Alpha 27 map upgrade

Post by Viech »

exm 1.03, pk02 1.03, trak5 1.03: Don't use "surfaceparm alphashadow" for now since q3map2 can't read the compressed textures. Use "alphaFunc GE128" instead of "blend blend" where applicable.

Responsible for: Arch Linux package & torrent distribution, Parpax (map), Chameleon (map texture editor), Sloth (material file generator), gameplay design & programming, artistic direction

User avatar
Viech
Project Head
Posts: 2139
Joined: Fri Aug 03, 2012 11:50 pm UTC
Location: Berlin

Re: Alpha 27 map upgrade

Post by Viech »

pk02 1.04: Desaturated light01 and added a glow map to it.

Responsible for: Arch Linux package & torrent distribution, Parpax (map), Chameleon (map texture editor), Sloth (material file generator), gameplay design & programming, artistic direction

User avatar
Viech
Project Head
Posts: 2139
Joined: Fri Aug 03, 2012 11:50 pm UTC
Location: Berlin

Re: Alpha 27 map upgrade

Post by Viech »

Added a section for the levelshot:

Viech wrote:

Ideally as your last step but not before you adjusted to neutral gamma (read section above) please take a new levelshot so it shows with appropriate brightness and matches the new appearance of your map.

Responsible for: Arch Linux package & torrent distribution, Parpax (map), Chameleon (map texture editor), Sloth (material file generator), gameplay design & programming, artistic direction

User avatar
Viech
Project Head
Posts: 2139
Joined: Fri Aug 03, 2012 11:50 pm UTC
Location: Berlin

Re: Alpha 27 map upgrade

Post by Viech »

Some relevant bits from my discussion with EmperorJack:

Viech wrote:

The pk3 structure looks very good, I'd only change two things:

  • Move plat23.txt to about/map-plat23.txt. I want to establish this as another standard for readme files, so that extracting a number of maps doesn't lead to text file spam.

  • Add scripts/shaderlist.txt so that newbes can have a proper look at the map just by extracting the .map file. Unlike older editors, netradiant does support shaderlists inside pk3s. Without a matching shaderlist entry the shaders won't display at all and I'm not planning to ship a global "default" list as trem did, given that it wouldn't even prevent badly setup maps to create spam inside radiant and at the same time is rather inconvenient to maintain. With a clean namespace setup, shipping the shaderlist inside the map appears to be the better choice.

When making preview versions of maps, you should use the "~" seperator: The sorting order would be

Code: Select all

map-plat23_b12~1.pk3 < map-plat23_b12~2.pk3 < … < map-plat23_b12.pk3 < map-plat23_b12+1.pk3 < …

so that you can release an infinite amount of pk3s that will appear older than the target version but are still ordered according to the sequence numbers. This way you can make public preview releases without telling people to "overwrite the file of the same name".

With "clean namespace" I mean that every shader namespace has the form textures/<mapname><setname>/, e.g. textures/plat23_custom/. Source files should be in the seperate folder textures/<mapname><setname>_src/ so that you can conveniently browse the usable materials and hide their sources.

The next release of ex(m) won't have metalsteps on the base* materials but I'm not releasing it yet since this is a minor change (and not relevant at map compile time).

I'm still undecided with regards to in-editor-translucent caulk. Some people love it, some hate it. Therefor I opened a poll here.

Responsible for: Arch Linux package & torrent distribution, Parpax (map), Chameleon (map texture editor), Sloth (material file generator), gameplay design & programming, artistic direction

Post Reply