The effort to track all unvanquished assets in repositories

Post all of your progress here about official assets, and receive feedback from our team.
User avatar
illwieckz
Project Head
Posts: 718
Joined: Sat Aug 11, 2012 7:22 pm UTC
Location: France
Contact:

Re: The effort to track all unvanquished assets in repositories

Post by illwieckz »

So I did it. :bsuit:

Check github.com/UnvanquishedAssets/UnvanquishedAssets for the master repository. I edited my messages above to fit the current state. You can use them as sort-of documentation while waiting for something better on Wiki. :wink:

So, what's the story bro?

There is some design choices I'm not happy with but are needed because of pragmatism.

One of them is the res-emoticons_src.dpkdir repository, a repository for emoticons sounds too much, but the problem they are for many of them the old tremulous forum emoticons, I haven't managed to know who made them and I don't want to put them in main repository which this status. Once all the emoticons will be redrawed, we will be able to track them alongside others ui stuff in main repository and ditch that emoticons package. :defcomp: :tesla: :leech:

Another of them is the res-legacysounds_src.dpkdir and res-sounds_src.dpkdir repositories. Having two repository for the same purpose looks too much, but it's needed because we have to ditch the legacy one. The legacy one is the repository containing sounds from tremulous with unknown status. The idea is to never touch the legacy one, and override the sounds with new contributions in the res-sounds_src.dpkdir repository. When all sounds are overridden, we will be able to drop the legacysounds package from releases. :bomb:

The other controversial repository would be the soundtrack one. It can sounds pretty useless for 3 files. I made it separate too because we want to track original lossless assets and you can easily imagine the growing file size of a repository tracking lossless 44100 stereo tracks.

All the texture packages have their own repository, a mapper can fetch only the ones he needs, because no one has to get pk02 if tech looks enough for you. For example a Free Software Zealot™ would be able to only fetch DFSG-compatible texture packages to only build against them. :cyclops:

It's time to officially introduce Urcheon :announce: , this is the tool where I outsource all my knowledge. In fact, there is more work done in that tool than it that repository stuff. It's a now 2 years old long standing effort dealing with 3040 repositories from Unvanquished and other games. This tool set is designed to make asset management pleasant and efficient.

The repositories now uses the dpk/dpkdir format we talked about there. You can use this Dæmon branch (PR #14) and this netradiant/q3map2 branch (MR !19) to get the complete editing chain. The dpkdir support is already merged in Urcheon. :coffee:

You will also need two Sloth fixes (indexcolors, knownexts) to prepare texture packages correctly.

All the lossless textures in these repositories where heavily compacted with png format using the zopfli algorithm, ensuring acceptable compact size while keeping widely known format third party editing tools can read.

How-to

The only thing you have to do to build the repositories (once you got Urcheon and its dependencies) is that:

Code: Select all

git clone https://github.com/UnvanquishedAssets/UnvanquishedAssets.git
cd UnvanquishedAssets
make init
make package

Everything will be built in build/pkg/, you can use this as pakpath for Dæmon.

If you're just a mapper and want to build your maps against lossless original assets, you just have to do that:

Code: Select all

git clone https://github.com/UnvanquishedAssets/UnvanquishedAssets.git
cd UnvanquishedAssets
make init
make prepare

And just use src/ as pakpath for NetRadiant and q3map2.

:advgranger: :beer: :granger:

Last edited by illwieckz on Fri Apr 07, 2017 3:12 pm UTC, edited 5 times in total.

This comment is licensed under cc ​​by 4 and antecedent.

User avatar
illwieckz
Project Head
Posts: 718
Joined: Sat Aug 11, 2012 7:22 pm UTC
Location: France
Contact:

Re: The effort to track all unvanquished assets in repositories

Post by illwieckz »

Side note: every asset is now tracked except map packages and the tex-vega one (which is currently tracked in @Ingar's own repository), see this post to get their status and url for them when there is.

This comment is licensed under cc ​​by 4 and antecedent.

User avatar
Ishq
Project Head
Posts: 1145
Joined: Tue Mar 06, 2012 8:32 pm UTC

Re: The effort to track all unvanquished assets in repositories

Post by Ishq »

Wow thanks! This was a project that needed attention for a long time and I'm really glad you stepped up to take care of it!

User avatar
illwieckz
Project Head
Posts: 718
Joined: Sat Aug 11, 2012 7:22 pm UTC
Location: France
Contact:

Re: The effort to track all unvanquished assets in repositories

Post by illwieckz »

Some file size statistics:

(that scrolling pre is annoying)

$ du -shc src/*
140K src/res-ambient_src.dpkdir
375M src/res-buildables_src.dpkdir
3,2M src/res-feedback_src.dpkdir
16M src/res-legacy_src.dpkdir
261M src/res-players_src.dpkdir
13M src/res-soundtrack_src.dpkdir
174M src/res-weapons_src.dpkdir
20K src/tex-all_src.dpkdir
288K src/tex-common_src.dpkdir
19M src/tex-ej01_src.dpkdir
15M src/tex-exm_src.dpkdir
34M src/tex-ex_src.dpkdir
15M src/tex-pk01_src.dpkdir
78M src/tex-pk02_src.dpkdir
90M src/tex-space_src.dpkdir
97M src/tex-tech_src.dpkdir
33M src/tex-trak5_src.dpkdir
27M src/unvanquished_src.dpkdir
1,3G total
$ du -shc build/pkg/*
44K build/pkg/res-ambient_0+20170409-000707+4f43e57.dpk
80M build/pkg/res-buildables_0.47.0+20170409-000833+acc0496.dpk
2,6M build/pkg/res-feedback_0+20170408-231140+a69e7ac.dpk
2,7M build/pkg/res-legacy_0+20170408-144046+1d52239.dpk
73M build/pkg/res-players_0.21.0+20170409-001029+01f30b2.dpk
13M build/pkg/res-soundtrack_0+20170409-001127+a500934.dpk
30M build/pkg/res-weapons_0.49.0+20170409-000932+79e2dc8.dpk
4,0K build/pkg/tex-all_0+20170405-013918+2ac1767.dpk
36K build/pkg/tex-common_0+20170405-013625+fd7185e.dpk
6,4M build/pkg/tex-ej01_0+20170405-000010+152103b.dpk
12M build/pkg/tex-ex_0+20170405-000204+dbcf6cf.dpk
2,6M build/pkg/tex-exm_0+20170405-000113+fe965fc.dpk
5,4M build/pkg/tex-pk01_0+20170405-000245+74338ea.dpk
25M build/pkg/tex-pk02_0+20170405-000321+7084aeb.dpk
17M build/pkg/tex-space_0+20170405-000422+de2841d.dpk
21M build/pkg/tex-tech_0+20170404-234244+e9ea61f.dpk
12M build/pkg/tex-trak5_0+20170405-000453+a04c8ef.dpk
8,8M build/pkg/unvanquished_0.50.0+20170410-213046+791f212.dpk
307M total

These ugly versions starting with 0+ are there because I haven't set version tags in repositories yet. The Urcheon tool automatically writes version string based on git refs. If last tag is last commit, it just uses it (v0.50.0package_0.50.0.dpk), if there is commits since last tag, it adds +<timestamp>+<sha1> to the tag, as seen in 0.50.0+20170410-213046+791f212. By timestamp I mean the one of the latest commit. That's why I started a thread about version string: the version string is written by the tool, and must be predictable. Currently I use YYYYmmdd-HHMMSS (please, no more than one hyphen :confused:)

We need to define a version string format and stick on it because the tool relies on it for partial build. This is a computer task, not a human task. You can build a partial package giving an arbitrary git ref, in this case the tool adds the own package name as dependency with the computed versions string for that given past ref, but this magic can work only if the previous package was named using the exact same algorithm, even if someone else packaged the previous package. :wink:

This comment is licensed under cc ​​by 4 and antecedent.

User avatar
illwieckz
Project Head
Posts: 718
Joined: Sat Aug 11, 2012 7:22 pm UTC
Location: France
Contact:

Re: The effort to track all unvanquished assets in repositories

Post by illwieckz »

So I deleted the res-sounds repository (buildables sounds are tracked in res-buildables), merged res-emoticons and res-legacysounds as res-legacy, and merged res-voices and res-beacons as res-feedback (and imported announcements sounds). I also created a res-ambient repository for ambient stuff, it will be the right place to put generic sci-fi crates and humming background sounds™.

This comment is licensed under cc ​​by 4 and antecedent.

User avatar
Viech
Project Head
Posts: 2139
Joined: Fri Aug 03, 2012 11:50 pm UTC
Location: Berlin

Re: The effort to track all unvanquished assets in repositories

Post by Viech »

307MB for everything is crazy small! Thanks a lot for all the cleanup! :smile:

Responsible for: Arch Linux package & torrent distribution, Parpax (map), Chameleon (map texture editor), Sloth (material file generator), gameplay design & programming, artistic direction

User avatar
illwieckz
Project Head
Posts: 718
Joined: Sat Aug 11, 2012 7:22 pm UTC
Location: France
Contact:

Re: The effort to track all unvanquished assets in repositories

Post by illwieckz »

307Mb for everything except tex-vega and all map packages, but yes, there is a lot of clean-up. :-)

This comment is licensed under cc ​​by 4 and antecedent.

User avatar
illwieckz
Project Head
Posts: 718
Joined: Sat Aug 11, 2012 7:22 pm UTC
Location: France
Contact:

Re: The effort to track all unvanquished assets in repositories

Post by illwieckz »

For reference, the 0.50.0 release weights 717Mb.

tex-vega plus map packages weight 271Mb, but you can expect some weight loss too since there is currently 3 station15 paks and 2 antares paks, perseus ships tga textures and many maps ship tga lightmaps. To compensate, the unvanquished pak in my list does not ship game code, so you must add some Mb more for final pak size prediction. An average 150Mb saving can be expected for the whole.

This comment is licensed under cc ​​by 4 and antecedent.

Post Reply