On the road to my Tremulous map pack

[illwieckz]

I use this thread to track all I've done or other people have done and all must be done to allow me to release my Tremulous map pack.

TODO list

• define package naming and versioning good practices: done by viech.
• write a tool to be able to dump & merge bsp textures list lump to fix broken path: done.
• write a tool to be able to dump & merge bsp entities lump to fix deprecated entities: done.
• write a tool to be able to dump & merge internal lightmaps inside bsp entities lump to fix bad lighting: done, thanks viech and `ishq.
• write a tool to be able to merge external layout file inside bsp entities lump to fix broken internal layout: work in progress.
• determine better picture/sound format to use in release pk3: done, thanks to viech and kangz.
• write a substitution rule for deprecated entities: done, thanks to viech.
• fix gamma: done (use colorgrading).
• fix windows-style path in textures and entities lump: done.
• fix case sensitive path problem in textures and entities lump: done.
• fix missing textures and sound paths: done.
• deduplicate textures files in tremulous res package and give substitute shader for deduplicated path: done.
• write a tool to build pk3 from source: done.
  
  • automatically extend the file containing build rules: done.
  • convert textures to crn (and webp for lightmaps, colorgrades and skyboxes): done.
  • convert sounds to opus: done.
  • build map: done, needs some patches in q3map2 (1, 2).
  • build minimap: done.
  • build navmeshes: done.

• write some acceptable workaround for deprecated shaders: done.
• fix unvanquished’s map list to display long map names and to be able to scroll map list: fix written, pull request merged.
• fix unvanquished’s map list to sort by name and not by color code: fix written, pull request merged.
• fix water bug: found: it’s deformVertexes’ fault, gimahel has a fix, branch merged.
• fix new entities: work in progress, needs a fix.
• allow daemon to load extension independent sound paths: done

[illwieckz]

Map status

Legend:

R package released
[V] confirmed deformVertexes bug (fixed, some remnants but does not prevent to play).
[E] confirmed entities bug (many other maps can have this bug, looking for this bug means testing all triggers, so only a few are reported here, but it's a blocking bug for many).
[T] unresolved shader bug (temple will be not released if I can't fix that, it's just a very good looking map but not a good playable map).
[W] weird bug on ethereal, difficult to explain, affects player moves, if I can't fix that, I will not release this map too.

Map list:

• map 1984 b4
• map ancient-remains r1 [V]
• map arachnid2 r1.2 R
• map atcshd r1.2 R
• map atcszalpha b2
• map bluedragon2 b3 [V]
• map citadel r1 [V]
• map cleanser r1
• map compact r1
• map cruz b6
• map derelictb b4
• map eden b3
• map ethereal r1 [V] [W]
• map flap r1
• map gamma-core r1
• map gloom2 b2
• map groundsource r1 [V]
• map hamunaptra r1 [V]
• map karith r1.2 R
• map medusa-ctm b1
• map meep b1
• map methane b1
• map metro b1.2 [V]
• map minetest2 r1
• map nano r1
• map nexus6 r1.2 [V] R
• map niveus r1.2 R
• map orion r1
• map outpost-p4 b1
• map powergrid b25
• map procyon r1
• map pulse b3 [V] [E]
• map sirius r1
• map sokolov b1.1 [V]
• map temple r0.5.5 [T]
• map thermal r1 [V]
• map transit r1.2 [E] R
• map tremor r1.2 R
• map uncreation r1.2 R
• map utcs r1
• map veddak r1
• map vega b1 [V] (Released by Ingar himself)
• map wrecktify b2

Resource package:

• res tremulous r1.2 R

[illwieckz]

Enhancement needed by this project.

• extension independent sound file name: issue opened, pull request merged.
• fixed undeprecated entity code issue opened.
• out-of-tree q3map2 build: issue opened, patches included, merge pending.
• allow q3map2 to load arbitrary pakpaths: issue opened (thanks to Melanosuchius), patches written, merge pending.
• import navmesh generation from daemonmap to q3map2: issue opened, patches written, merge pending.

Other suggested enhancement not blocking for this project but that could be good to do too.

Related

• oga extension support for ogg audio files.
• common ogg decoder able to discriminate opus and vorbis (currently .ogg extension is used for vorbis and .opus for opus).
• external builtin layout (in pk3 in layouts/mapname alongside maps/mapname) and perhaps multiple built-in layouts: topic opened.
• being able to force the engine to never end a game when we load an incomplete map (spawn entities missing) for development purpose: topic opened.
• allow q3map2 to write the .minimap file when generating a minimap: issue opened.

Other bugs revealed by this project:

• bsp parser read out of boundaries if entities lump is not NULL terminated even if the BSP knows the entity lump size: issue opened, fixed by kangz.
• logs.logLevel.audio debug corrupts itself: issue opened, fixed by kangz.

Functionalities that helped a lot this project:

• -pakpath: allows to use source/test/release pkg dir for development, and allows to use one directory for unvanquished maps and another directory for tremulous maps on server.

[Viech]

Good job so far! With regards to the deprecated/new entities, if a byproduct would be a script that replaces these inside .map files, too, we could actually do it on our new maps.

[Tom]

Excellent!

[illwieckz]

Good job so far! With regards to the deprecated/new entities, if a byproduct would be a script that replaces these inside .map files, too, we could actually do it on our new maps.

To me .map parsing is a bit more complex than bsp. BSP lumps are just indexes/lists stored on different offsets, you can query textures without taking care of entities for example. MAP files are whole trees to parse, you can't for example only parse entities without parsing the whole file. But since it's a mostly a key-value file, perhaps this sed script works too (and perhaps it must be completed too).

Also, I have not yet a fully functional radiant install to verify if my patched .map are ok.

But yes, it's probably doable too. I think about a tool to load the .map as a whole tree in python, and to be able to check "if this is a key", "if this is a value", which is safer than a fast and dirty sed script that works on characters only.

I think about doing the same for entities modification. Currently I dump entities, edit them outside then merge them, but my tool is already able to load them as a tree, so it's possible to write some functions like substitute_key("old name", "new name") or substitute_value("key name", "old value", "new value") or something like that.

[illwieckz]

Well, I have now a map parser already able to to parse and rewrite map. It's not complete yet (patches and brushes are handled verbatim for the moment) but the whole tree is now loaded and it's already possible to do something like that:

Code: Select all

qmap=Map()
qmap.read_file(file_name)

for entity in qmap.entity_list:
    for key in entity.key_dict.keys():
        if entity.key_dict[key] == value:
            entity.key_dict[key] = new_value

qmap.write_file(new_file_name)


[illwieckz]

Hi, I opened a new repository for my 's toolbox: github.com/illwieckz/grtoolbox.

At this time there is two tools:

• bsp_cutter.py
• map_cutter.py

As their name mean, they are tools to slice bsp or maps file.

I have many things to rewrite and improve in the bsp_cutter.py tool, many things are just almost working proof of concept and bad written, so the better thing you can do is to not read it at all and not try to use it yet.

The map_cutter.py tool will be more interesting for you, and it is far far cleaner.

You can already do that:

by default map_cutter.py write to stdout, but you can change the output file with -o/--out or replace the file with -i/--in-place

• Parse and rewrite the map in place: useful to sanitize a file modified by hand.
  Only one thing is not parsed yet: patch vertex matrices, they are just copied as-is (nothing is lost), all other things are parsed. It means, except the vertexes matrices, you can load an run through all the map tree from python.
  
  

  Code: Select all

  ./map_cutter.py --input-map plat23.map --output-map plat23.map
  
  
• Dump entities in bsp format.
  Produces a valid entities lump that can be merged later in a bsp file.
  

  Code: Select all

  ./map_cutter.py --input-map plat23.map --output-entities plat23.ent
  
  
• Fix deprecated entities in map
  Reads a csv for value_type/old_value/new_value this way:
  

  Code: Select all

  value, "target_speaker", "sfx_speaker"
value, "misc_teleporter_dest", "pos_target"
key, "team", "group"
key, "count", "amount"
  
  

  Code: Select all

  ./map_cutter.py --input-map plat23.map --substitute-entities fix.csv --output-map plat23.map

My current entity substitution csv is: samples/unvanquished/entity_substitution.csv

I can't check the substitution since I have no map compilation environment configured yet on my system, but you can try that (do a backup before or be sure your files are versionned):

Code: Select all

cd $devdir
git clone https://github.com/illwieckz/grtoolbox.git

cd $mapdir
for map in *.pk3dir/maps/*.map
do
    echo $map
    $devdir/grtoolbox/map_cutter.py --input-map $map --substitute-entities $devdir/grtoolbox/samples/unvanquished/entity_substitution.csv --output-map $map
done

I plan to add an option to fix entities key names with wrong case from a file, and an option to replace textures path from a csv file.
I will probably not do more things for the map_cutter.py tools, the next I will do is to clean the bsp_cutter.py (and use safe regexp for text parsing) and improve a lot the CLI to be able to merge patched entity/texture/lightmap lumps in one command for example (currently it's only a proof of concept). After that I will rewrite my map make tool from shell to python.

[kharnov]

You're doing an impressive job, illwieckz!

[GORE]

Whats the current status of this project? Would love to see it finished!