On the road to my Tremulous map pack

Ask questions about mapping in general, and show off your in-progress work.
User avatar
illwieckz
Project Head
Posts: 718
Joined: Sat Aug 11, 2012 7:22 pm UTC
Location: France
Contact:

On the road to my Tremulous map pack

Post by illwieckz »

I use this thread to track all I've done or other people have done and all must be done to allow me to release my Tremulous map pack.

TODO list

  • define package naming and versioning good practices: done by viech.

  • write a tool to be able to dump & merge bsp textures list lump to fix broken path: done.

  • write a tool to be able to dump & merge bsp entities lump to fix deprecated entities: done.

  • write a tool to be able to dump & merge internal lightmaps inside bsp entities lump to fix bad lighting: done, thanks viech and `ishq.

  • write a tool to be able to merge external layout file inside bsp entities lump to fix broken internal layout: work in progress.

  • determine better picture/sound format to use in release pk3: done, thanks to viech and kangz.

  • write a substitution rule for deprecated entities: done, thanks to viech.

  • fix gamma: done (use colorgrading).

  • fix windows-style path in textures and entities lump: done.

  • fix case sensitive path problem in textures and entities lump: done.

  • fix missing textures and sound paths: done.

  • deduplicate textures files in tremulous res package and give substitute shader for deduplicated path: done.

  • write a tool to build pk3 from source: done.

    • automatically extend the file containing build rules: done.

    • convert textures to crn (and webp for lightmaps, colorgrades and skyboxes): done.

    • convert sounds to opus: done.

    • build map: done, needs some patches in q3map2 (1, 2).

    • build minimap: done.

    • build navmeshes: done.

Last edited by illwieckz on Sun Aug 23, 2015 7:45 pm UTC, edited 17 times in total.
User avatar
illwieckz
Project Head
Posts: 718
Joined: Sat Aug 11, 2012 7:22 pm UTC
Location: France
Contact:

Re: On the road to my Tremulous map pack

Post by illwieckz »

Map status

Legend:

R package released
[V] confirmed deformVertexes bug (fixed, some remnants but does not prevent to play).
[E] confirmed entities bug (many other maps can have this bug, looking for this bug means testing all triggers, so only a few are reported here, but it's a blocking bug for many).
[T] unresolved shader bug (temple will be not released if I can't fix that, it's just a very good looking map but not a good playable map).
[W] weird bug on ethereal, difficult to explain, affects player moves, if I can't fix that, I will not release this map too.

Map list:

  • map 1984 b4

  • map ancient-remains r1 [V]

  • map arachnid2 r1.2 R

  • map atcshd r1.2 R

  • map atcszalpha b2

  • map bluedragon2 b3 [V]

  • map citadel r1 [V]

  • map cleanser r1

  • map compact r1

  • map cruz b6

  • map derelictb b4

  • map eden b3

  • map ethereal r1 [V] [W]

  • map flap r1

  • map gamma-core r1

  • map gloom2 b2

  • map groundsource r1 [V]

  • map hamunaptra r1 [V]

  • map karith r1.2 R

  • map medusa-ctm b1

  • map meep b1

  • map methane b1

  • map metro b1.2 [V]

  • map minetest2 r1

  • map nano r1

  • map nexus6 r1.2 [V] R

  • map niveus r1.2 R

  • map orion r1

  • map outpost-p4 b1

  • map powergrid b25

  • map procyon r1

  • map pulse b3 [V] [E]

  • map sirius r1

  • map sokolov b1.1 [V]

  • map temple r0.5.5 [T]

  • map thermal r1 [V]

  • map transit r1.2 [E] R

  • map tremor r1.2 R

  • map uncreation r1.2 R

  • map utcs r1

  • map veddak r1

  • map vega b1 [V] (Released by Ingar himself)

  • map wrecktify b2

Resource package:

  • res tremulous r1.2 R

Last edited by illwieckz on Mon Aug 17, 2015 2:54 am UTC, edited 10 times in total.
User avatar
illwieckz
Project Head
Posts: 718
Joined: Sat Aug 11, 2012 7:22 pm UTC
Location: France
Contact:

Re: On the road to my Tremulous map pack

Post by illwieckz »

Enhancement needed by this project.

Other suggested enhancement not blocking for this project but that could be good to do too.

Related

  • oga extension support for ogg audio files.

  • common ogg decoder able to discriminate opus and vorbis (currently .ogg extension is used for vorbis and .opus for opus).

  • external builtin layout (in pk3 in layouts/mapname alongside maps/mapname) and perhaps multiple built-in layouts: topic opened.

  • being able to force the engine to never end a game when we load an incomplete map (spawn entities missing) for development purpose: topic opened.

  • allow q3map2 to write the .minimap file when generating a minimap: issue opened.

Other bugs revealed by this project:

Functionalities that helped a lot this project:

  • -pakpath: allows to use source/test/release pkg dir for development, and allows to use one directory for unvanquished maps and another directory for tremulous maps on server.

Last edited by illwieckz on Mon Aug 17, 2015 3:03 am UTC, edited 5 times in total.
User avatar
Viech
Project Head
Posts: 2139
Joined: Fri Aug 03, 2012 11:50 pm UTC
Location: Berlin

Re: On the road to my Tremulous map pack

Post by Viech »

Good job so far! With regards to the deprecated/new entities, if a byproduct would be a script that replaces these inside .map files, too, we could actually do it on our new maps. :grin:

Responsible for: Arch Linux package & torrent distribution, Parpax (map), Chameleon (map texture editor), Sloth (material file generator), gameplay design & programming, artistic direction

User avatar
Tom
Dragoon
Posts: 253
Joined: Tue Jan 15, 2013 3:34 pm UTC
Location: France

Re: On the road to my Tremulous map pack

Post by Tom »

Excellent!

User avatar
illwieckz
Project Head
Posts: 718
Joined: Sat Aug 11, 2012 7:22 pm UTC
Location: France
Contact:

Re: On the road to my Tremulous map pack

Post by illwieckz »

Viech wrote:

Good job so far! With regards to the deprecated/new entities, if a byproduct would be a script that replaces these inside .map files, too, we could actually do it on our new maps. :grin:

To me .map parsing is a bit more complex than bsp. BSP lumps are just indexes/lists stored on different offsets, you can query textures without taking care of entities for example. MAP files are whole trees to parse, you can't for example only parse entities without parsing the whole file. But since it's a mostly a key-value file, perhaps this sed script works too (and perhaps it must be completed too).

Also, I have not yet a fully functional radiant install to verify if my patched .map are ok. :tongue:

But yes, it's probably doable too. I think about a tool to load the .map as a whole tree in python, and to be able to check "if this is a key", "if this is a value", which is safer than a fast and dirty sed script that works on characters only.

I think about doing the same for entities modification. Currently I dump entities, edit them outside then merge them, but my tool is already able to load them as a tree, so it's possible to write some functions like substitute_key("old name", "new name") or substitute_value("key name", "old value", "new value") or something like that.

User avatar
illwieckz
Project Head
Posts: 718
Joined: Sat Aug 11, 2012 7:22 pm UTC
Location: France
Contact:

Re: On the road to my Tremulous map pack

Post by illwieckz »

Well, I have now a map parser already able to to parse and rewrite map. It's not complete yet (patches and brushes are handled verbatim for the moment) but the whole tree is now loaded and it's already possible to do something like that:

Code: Select all

qmap=Map()
qmap.read_file(file_name)

for entity in qmap.entity_list:
    for key in entity.key_dict.keys():
        if entity.key_dict[key] == value:
            entity.key_dict[key] = new_value

qmap.write_file(new_file_name)

:grin:

User avatar
illwieckz
Project Head
Posts: 718
Joined: Sat Aug 11, 2012 7:22 pm UTC
Location: France
Contact:

Re: On the road to my Tremulous map pack

Post by illwieckz »

Hi, I opened a new repository for my :advgranger: 's toolbox: github.com/illwieckz/grtoolbox.

At this time there is two tools:

  • bsp_cutter.py

  • map_cutter.py :thumbup:

As their name mean, they are tools to slice bsp or maps file.

I have many things to rewrite and improve in the bsp_cutter.py tool, many things are just almost working proof of concept and bad written, so the better thing you can do is to not read it at all and not try to use it yet. :granger:

The map_cutter.py tool will be more interesting for you, and it is far far cleaner. :thumbup:

You can already do that:

by default map_cutter.py write to stdout, but you can change the output file with -o/--out or replace the file with -i/--in-place

  • Parse and rewrite the map in place: useful to sanitize a file modified by hand.
    Only one thing is not parsed yet: patch vertex matrices, they are just copied as-is (nothing is lost), all other things are parsed. It means, except the vertexes matrices, you can load an run through all the map tree from python.

    Code: Select all

    ./map_cutter.py --input-map plat23.map --output-map plat23.map
  • Dump entities in bsp format.
    Produces a valid entities lump that can be merged later in a bsp file.

    Code: Select all

    ./map_cutter.py --input-map plat23.map --output-entities plat23.ent
  • Fix deprecated entities in map
    Reads a csv for value_type/old_value/new_value this way:

    Code: Select all

    value, "target_speaker", "sfx_speaker"
    value, "misc_teleporter_dest", "pos_target"
    key, "team", "group"
    key, "count", "amount"

    Code: Select all

    ./map_cutter.py --input-map plat23.map --substitute-entities fix.csv --output-map plat23.map

My current entity substitution csv is: samples/unvanquished/entity_substitution.csv

I can't check the substitution since I have no map compilation environment configured yet on my system, but you can try that (do a backup before or be sure your files are versionned):

Code: Select all

cd $devdir
git clone https://github.com/illwieckz/grtoolbox.git

cd $mapdir
for map in *.pk3dir/maps/*.map
do
    echo $map
    $devdir/grtoolbox/map_cutter.py --input-map $map --substitute-entities $devdir/grtoolbox/samples/unvanquished/entity_substitution.csv --output-map $map
done

I plan to add an option to fix entities key names with wrong case from a file, and an option to replace textures path from a csv file.
I will probably not do more things for the map_cutter.py tools, the next I will do is to clean the bsp_cutter.py (and use safe regexp for text parsing) and improve a lot the CLI to be able to merge patched entity/texture/lightmap lumps in one command for example (currently it's only a proof of concept). After that I will rewrite my map make tool from shell to python.

:advgranger: :coffee: :granger:

Last edited by illwieckz on Sat Jul 18, 2015 8:54 pm UTC, edited 3 times in total.
User avatar
kharnov
Granger
Posts: 1851
Joined: Tue Mar 06, 2012 10:54 pm UTC
Clan: GT
Location: New York City

Re: On the road to my Tremulous map pack

Post by kharnov »

You're doing an impressive job, illwieckz!

User avatar
illwieckz
Project Head
Posts: 718
Joined: Sat Aug 11, 2012 7:22 pm UTC
Location: France
Contact:

Re: On the road to my Tremulous map pack

Post by illwieckz »

The current status is “work in progress”, in fact, I want this project to not only release maps but also fix unvanquished bugs, enhance unvanquished and q3map2 and define a usable-by-everyone mapping workflow.

Working on this big project is an excellent way to track bugs (because the more map you try to port, the more bugs you face) and to see what must be changed to make the mapping work easier (because the more map you try to port, the more by-hand per-map workarounds appear as annoying).

This is why the maps are not released yet, they will be released when they will be buildable by tools every mapper can reuse and when the fixes they need will be fixed for every player.

This is not only an map port effort, it's an effort to make unvanquished better, with or without these maps. :wink:

Last edited by illwieckz on Sat Jul 18, 2015 10:26 pm UTC, edited 1 time in total.
Post Reply