Page 1 of 3

About assets format

Posted: Wed Feb 25, 2015 5:17 pm UTC
by illwieckz

So, if I summarize some talks we had on #unvanquished-dev

1. Picture file formats

Picture format currently supported are:

  • jpg (lossy)
  • png (lossless)
  • tga (lossless)
  • webp (lossy or lossless)
  • crn (lossy)

1.1 webp will probably be droped in the future to reduce the dependency list. Right?

1.2 So whatever the source format, files in pk3 for release must use png for textures that must not be lossy, and crn for others. Right?

Some textures must NOT be lossy.

1.3 Color grading textures must be lossy, Right?
1.4 Is light maps can be lossy?
1.5 Is skybox can be lossy?
1.6 Is there other special textures that must not be lossy? (I don't know if there is masking textures or something like that).

2. Audio file formats

Audio formats supported are:

  • wav
  • vorbis (in ogg container)
  • opus (in ogg container)

2.1 It seems there is no sound that must not be shipped lossy, so whatever the source format, sounds in pk3 for release must be in opus. Right?

2.2 About opus files, the engine does not handle .oga extensions, only .ogg. Right ?

3. File names and file paths

3.1 Textures are shaders, and the engine loads each texture automatically as a shader, that's why the shader names in bsp texture lump does not known the file extension. Right?

3.2 Shaders described in shader-files can depends on picture files, these picture paths must contain the file extension. So, if I convert a texture used by a manually writen shader, I must rewrite the .shader file with the fixed path. Right?

3.3 Previously in the past some maps where ported from tremulous, and in pk3 textures are converted from jpg and tga to crn. But shader files were never fixed and describe original path, but it seems to work. How it works? Is there a special code in the daemon engine to load a crn file if a jpg, tga, png etc. is missing?

3.4 On my system (Ubuntu Linux), texture path are case sensitive, but many old maps assumed this was case insensitive, is it because the tremulous engine was case insensitive or because the mapper have done their job on Windows?

3.5 There is sound file paths in the entities lump of the bsp, is this the only place where sounds can be described ? Do I have to check another place to rewrite sound file paths if I convert a wav file to opus (and changing the file ext) for example ?

4. Entities

4.1 Is there a table that list old entities names alongside new entities names to use instead ?

5. Shaders

5.1 Shaders can only modify a picture, not a picture rendered by another shader, I can only use a shader name to map a picture file path, not use shader name to map another shader name. Right?

Thanks a lot!


Re: About assets format

Posted: Wed Feb 25, 2015 7:04 pm UTC
by kangz

1.1 I'm kind of against that
1.2 lossless webp or png, or crn or webp for some textures (normalmaps? need to ask a renderer guy)
1.3: colorgrade should be lossless
1.4: better lossless AFAIK as the feature size is small
1.5 can be lossy
1.6 textures with alpha channel might wnat to not have a lossy alpha channel near 0, if the shader is using alpha test ONE (not 100% sure about the name)

2.1 yes
2.2 not sure if oga support requires large changes to support it but the engine wouldn't know what to do with an oga extension for now.

3.1 Yes, more specifically 1) if a shader isn't found then the engine looks for an image with the same name 2) if no extension is specified for an image, it will try adding all the extensions it knows how to handle, to find an image.
3.2 and 3.3 see 3.1 2): if the texture name doesn't specify the extension, the engine will take any texture whose name without extension is the same name (starting from the pk3 higher up in the pk3 hierarchy)
3.4 not sure, but we want it to be case sensitive going forward
3.5 it is the only place where sound files should be specified. Maybe shaders too for the footstep sounds.

4.1 see g_entities.cpp I think

5.1 what.


Re: About assets format

Posted: Wed Feb 25, 2015 7:30 pm UTC
by Viech

1. Picture file formats

illwieckz wrote:

webp will probably be droped in the future to reduce the dependency list. Right?

I'm strongly in favor of this for a number of reason I'll provide when you ask for them.

illwieckz wrote:

So whatever the source format, files in pk3 for release must use png for textures that must not be lossy, and crn for others. Right?

You can use any format that works (see below). However, it is greatly recommended to use CRN for all texture maps (diffuse, normal, specular), though if you're porting old maps with low-res diffuse-only textures, JPG may work just as well given the neglible map load time and pk3 size…

illwieckz wrote:

Color grading textures must be lossy, Right?

Color grading textures must not be lossy. Every pixel is important.

illwieckz wrote:

light maps

Light maps shouldn't be lossy either, they already have a really low resolution compared to the surface they span. You'll get artifacts when you crunch them.

illwieckz wrote:

skybox

Special rules apply, I think CRN is not supported. Not sure about lossy vs. lossless here.

illwieckz wrote:

other special textures that must not be lossy

Not that I know of. UI elements, maybe (since they should look as sharp as possible).

3. File names and file paths

illwieckz wrote:

Textures are shaders, and the engine loads each texture automatically as a shader, that's why the shader names in bsp texture lump does not known the file extension. Right?

I don't know if the explaination is orrect but it should be true that file extensions don't need to be given even for non-shader textures. If multiple formats exist within the same pk3(dir), there is a priority list to select a format.

illwieckz wrote:

Shaders described in shader-files can depends on picture files, these picture paths must contain the file extension. So, if I convert a texture used by a manually writen shader, I must rewrite the .shader file with the fixed path. Right?

The extension given in shaders doesn't matter, it is stripped anyway. Thus there's no need to rewrite anything when the picture format changes. I don't even think you need to name an extension at all, anywhere.

illwieckz wrote:

Previously in the past some maps where ported from tremulous, and in pk3 textures are converted from jpg and tga to crn. But shader files were never fixed and describe original path, but it seems to work. How it works? Is there a special code in the daemon engine to load a crn file if a jpg, tga, png etc. is missing?

See above, extension given in shaders doesn't matter and never did as far as I know.

illwieckz wrote:

On my system (Ubuntu Linux), texture path are case sensitive, but many old maps assumed this was case insensitive, is it because the tremulous engine was case insensitive or because the mapper have done their job on Windows?

I wouldn't know, but obviously the solution is to use lowercase only for texture names. Shader names can be case sensitive as they aren't handled by the OS at all.

illwieckz wrote:

There is sound file paths in the entities lump of the bsp, is this the only place where sounds can be described ? Do I have to check another place to rewrite sound file paths if I convert a wav file to opus (and changing the file ext) for example ?

Speaker entities (among others) are used for map sounds, so some kind of path should be in there.

For any further question about shader files and the texture related file system hierarchy you can unpack the official texture packs and see how it's done there. It would be nice if maps used well-formed path and shader names so that they don't create junk when they are loaded by a map editor.

4. Entities

illwieckz wrote:

Is there a table that list old entities names alongside new entities names to use instead ?

Not yet, but you could parse it from this entities.yaml file (look for "deprecated: y"). Would be nice to have it on the wiki. Actually, a lot of people could profit from a whole page dedicated to porting maps from Tremulous to Unvanquished. :grin:

5. Shaders

illwieckz wrote:

Shaders can only modify a picture, not a picture rendered by another shader, I can only use a shader name to map a picture file path, not use shader name to map another shader name. Right?

Should be correct.

In case you want to generate .shader files, have a look at Sloth by the way.


Re: About assets format

Posted: Thu Feb 26, 2015 4:12 am UTC
by illwieckz

Thanks for all you answers!

The extension given in shaders doesn't matter, it is stripped anyway. Thus there's no need to rewrite anything when the picture format changes. I don't even think you need to name an extension at all, anywhere.

It's true that I can write tga to load a jpg or crn to load a tga, it works, but if the file is named .TGA (not .tga), the picture will not load for sure. So it's “almost” extension independant.

I have a new question, I want to create some “alias” for shader names. The idea is to deduplicates textures files and ship a shader file plenty of aliases to keep compatibility with unmodified third-party maps.

For example, these two textures are exactly the same, the second one is shipped by a tex-tremulous package as DEPS for the map-atcshd:

Code: Select all

textures/atcshd/rockground.jpg
textures/atcs/rockground.jpg

So I want to ship this texture only one time. If I delete “textures/atcshd/rockground.tga”, the engine will complain at map loading, as expected, with a “Couldn't find image file for shader textures/atcshd/rockground” message.

But if I write this shader, the “Couldn't find image file for shader textures/atcshd/rockground” message does not display so the texture is found, but the texture is not rendered:

Code: Select all

textures/atcshd/rockground
{
	map textures/atcs/rockground.jpg
}

There is probably one minor thing to do to have my shader fully functional. It seems the picture is loaded but not rendered… Do you have an idea, or do you know who can answer to this question?
:smile:


Re: About assets format

Posted: Thu Feb 26, 2015 6:56 pm UTC
by illwieckz

OK, my alias “almost” work if I do that:

Code: Select all

textures/atcshd/rockground
{
    {  
map textures/atcs/rockground.jpg }
}

I say “almost” because the texture is rendered wrong. In fact it seems there is a “per-default” transformation that is not done:

With file:

Image

With alias:

Image

And yes, they are the same:

Code: Select all

$ md5sum map-atcshd_*.pk3dir/textures/atcshd/rockground.jpg tex-tremulous_*.pk3dir/textures/atcs/rockground.jpg
f03873412474ca41e4299b5884189ddb  map-atcshd_r01+unv-a36-illwieckz.pk3dir/textures/atcshd/rockground.jpg
f03873412474ca41e4299b5884189ddb  tex-tremulous_r01.1.0+unv-a36-illwieckz.pk3dir/textures/atcs/rockground.jpg

Re: About assets format

Posted: Thu Feb 26, 2015 7:02 pm UTC
by Viech

Again, this is just a response to your first post.

illwieckz wrote:

I want to create some “alias” for shader names. The idea is to deduplicates textures files and ship a shader file plenty of aliases to keep compatibility with unmodified third-party maps.

For example, these two textures are exactly the same, the second one is shipped by a tex-tremulous package as DEPS for the map-atcshd:

Code: Select all

textures/atcshd/rockground.tga
textures/atcs/rockground.tga

You could create a tex-legacy package that ships all the trem texture files and is included by the individual trem maps. Note that if you do that you may need to play license detective for its readme file though. :grin: I'm actually in favor of not doing that and keeping the old maps self-contained.

illwieckz wrote:

if I write this shader, the “Couldn't find image file for shader textures/atcshd/rockground” message does not display so the texture is found, but the texture is not rendered:

Code: Select all

textures/atcshd/rockground
{
	map textures/atcs/rockground.tga
}

There is probably one minor thing to do to have my shader fully functional. It seems the picture is loaded but not rendered… Do you have an idea, or do you know who can answer to this question?

Depending on your setup, Unvanquished shouldn't be able to find the texture (e.g. if the pk3 that has it isn't loaded since it is no dependency of the map) or this shader should work. Try diffuseMap instead of map, it may be that the latter (legacy) keyword requires you to set a blend mode or stage or so…

If you want to create a shared texture package for the old maps note that this can require quite a bit of work depending on how organized you want it to be. As mentioned above I think it's best to just keep the maps self-contained, even if that means texture duplication. The textures are small anyway. I'd mostly make sure that the shader namespaces and file pathes are standard-conformant.


Re: About assets format

Posted: Thu Feb 26, 2015 7:03 pm UTC
by Viech

With regards to your second post: Try diffuseMap instead of map. Read the XreaL shader manual for more.


Re: About assets format

Posted: Thu Feb 26, 2015 7:27 pm UTC
by illwieckz

Yeaaaaaah! you rock!

Since it's a keyword for:

Code: Select all

{
  stage diffuseMap
  map <map>
}

I can just do that:

Code: Select all

textures/atcshd/rockground
{
	diffuseMap textures/atcs/rockground.tga
}

And it works!

I can now remove 1000+ duplicate files (81Mb).


Re: About assets format

Posted: Thu Feb 26, 2015 8:03 pm UTC
by illwieckz

In many tremulous maps I have these textures missing, do you know if it's something important?

Code: Select all

textures/common/black 
textures/common/caulk 
textures/common/clip 
textures/common/full_clip 
textures/common/nolightmap 
textures/common/slick 
textures/common/trigger 
textures/common/weapclip 
textures/NULL 

Edit: perhaps I can just create some placeholder?


Re: About assets format

Posted: Thu Feb 26, 2015 9:57 pm UTC
by Viech
illwieckz wrote:

In many tremulous maps I have these textures missing, do you know if it's something important?

Code: Select all

textures/common/black 
textures/common/caulk 
textures/common/clip 
textures/common/full_clip 
textures/common/nolightmap 
textures/common/slick 
textures/common/trigger 
textures/common/weapclip 
textures/NULL 

Edit: perhaps I can just create some placeholder?

Please look into the official tex-common package and re-read the map upgrade thread I posted earlier. (In short: Have maps load the tex-common package as a dependency but note that some shader names have changed, which means you'll have to edit the map (in your case, unfortunately the BSP) to point to the new shader name. Including a .shader file that uses the textures/common/ namespace in a map pk3 is technically a violation of namespaces even though our file system would prevent maps from breaking other maps through this violation. This is one reason why I encoraged you to just leave the maps as-is: They do violate this (and they would break each other in Tremulous, just like connecting to a modded server once could change your experience in a local game back then) but if you rework the map's shaders then you'd either violate these rules, too, or have a lot of work to do to do it properly…)

The map upgrade thread should explain this, but in case you missed on it: Every .shader file in the official texture packages has a header that explains how other maps should change the namespace used inside that file if they want to include a modified version of it in their own pk3. For example, if a map wants to use modified versions of the common shaders, they'd need to use the textures/mapname_common/ namespace for that. For example, Parpax has non-solid versions of some of the shaders in the ex set (which has the textures/shared_ex/ namespace) and those use the textures/parpax_ex/ namespace. It's important that maps behave like this because otherwise, even though our engine is resistant to breakage, the tetxure tree shown in map editors would be violated by those maps (if the editor loads them).