New crunch format leading to 4 time faster compression and space saving but break compatibility

Talk about anything related to Unvanquished.
Post Reply
User avatar
illwieckz
Project Head
Posts: 717
Joined: Sat Aug 11, 2012 7:22 pm UTC
Location: France
Contact:

New crunch format leading to 4 time faster compression and space saving but break compatibility

Post by illwieckz »

Hi guys,

You probably know that we use the crn format for our ready-for-GPU textures. Packages just ship a lot of .crn files and the engine sends stuff to GPU drivers without decompression, which leads to short load time and acceptable quality.

We currently use an old fork of crunch, the one from BinomialLLC, plus some additions for the normal map use case. Since BinomialLLC re-uploaded a more recent version I submitted some PR to upstream our fixes and changes (making changes optional) but they never got merged. In the same time, Unity was also working on it and published their changes.

Alexander Suvorov wrote some very interesting articles about their improvements on the Unity Blog:

There was some news coverage on Phoronix by example:

So, it looks like their work is very good. And it's now shipped within Unity so plenty of people will use it massively.

They said in november that their modified crunch tool “can compress up to 2.5 times faster, while providing about 10% better compression ratio”. Well, I did a test on our own asset repository, re-crunching all the res-, tex- and unvanquished packages. This corpus produces 1797 .crn files. I benchmarked both our old crunch tool and the new unity one with our patches applied. I built myself the binaries so there is no bias on that part.

So, the numbers:

Unvanquished's crunch:

  • compression time: 115 minutes 8 seconds

  • produced file size: 269 megabytes

Unity's crunch:

  • compression time: 26 minutes 43 seconds

  • produced file size: 239 megabytes

So the unity crunch tool reduced compression time by 4.31 and reduced size by 11.15. They said “up to 2.5 time faster but I've seen some random textures being compressed 6 time faster and the average of the whole is 4.3 time faster, and yes the tool compress more than 10% more. The results are very very impressive and looks very good.

But there is a problem, a very big problem, to get these performances they had to broke the file format, and badly, they just swaped the old format with the new one and it looks like there is no way to handle both. There is only one code path in the new library, and it looks like there is no metadata to detect if a file is using the former format or the new one.

Compressing with Unity's crunch and decompressing with Unvanquished’s crunch:

Image

Compressing with Unvanquished's crunch and decompressing with Unity's crunch:

Image

This is how looks our game if we crunch our textures using Unity's tool and don't modify the engine (more weirdness can be seen there):

Image

So, what's the problem?

They have a very improved crunch library but they broke the format.

So what to do?

With that improved format, packaging unvanquished assets reduces time from two hours to less than half one. And files are smaller.

Unity guys ship their crunch library within Unity since months now, so a lot of people are producing .crn files and it gives big visibility to their format. Basically, they're probably just going to eclipse the old format. Since Unity is not just an small indie company we can expect some tools coming producing and reading only their format.

If we want to update the engine to the new format we must do it now or never: we already broke the package format (DPK) so everything already has to be re-shipped so it's the very right time to do other breaking changes.

If we update the engine to the new format we can rebuild all our packages, and I'm already doing it for the next release whatever the format changed or not. I also got permissions from guys like Matth to repackage and recompress their community maps for next release, so if we update the engine to the new format we lost nothing, even third-party stuff!

An updated crunch tree applying Unvanquished/Dæmon changes atop Unity improvements is findable there: github.com/illwieckz/crunch/tree/unity-daemon (use -rtopmip with -renormalize to get the behavior we need for normal maps).

ping @Ishq @Viech because some decision has to be taken.

Last edited by illwieckz on Mon Feb 12, 2018 2:08 am UTC, edited 1 time in total.

This comment is licensed under cc ​​by 4 and antecedent.

User avatar
illwieckz
Project Head
Posts: 717
Joined: Sat Aug 11, 2012 7:22 pm UTC
Location: France
Contact:

Re: New crunch format leading to 4 time faster compression and space saving but break compatibility

Post by illwieckz »

This a PR bringing Unity's crunch format to Dæmon:

https://github.com/DaemonEngine/Daemon/pull/70

I verified that if I use that modified engine to load stuff produced by Unity's crunch it works.

This comment is licensed under cc ​​by 4 and antecedent.

User avatar
illwieckz
Project Head
Posts: 717
Joined: Sat Aug 11, 2012 7:22 pm UTC
Location: France
Contact:

Re: New crunch format leading to 4 time faster compression and a bit of space saving but breaking compatibility

Post by illwieckz »

For reference, this is the total size of the complete 0.50.0 unvanquished set (and only this one): 717M

717M unvanquished_0.50.0/pkg

This is the total size of the complete for-0.51.0 unvanquished set: 479M

For 0.51.0 I will not recompile the map from repositories yet, so I'm just repackaging the 0.50.0 map pk3 using Urcheon, that new crunch etc.

All these packages were processed by Urcheon picking the right optimal format for each file, including that new crunch. Everything looks OK in Dæmon once that new crnlib is merged.

The total size of the complete for-0.51.0 unvanquished set plus the various third-party community maps is: 694M

And just for fun the tremulous-only community set alone is 72M. ;-)

Do not mind the version numbers, they are temporary ones to be sure they will be superseded by final ones.

40K UnvanquishedAssets/build/pkg/res-ambient_0+20170409-000707+4f43e57.dpk
70M UnvanquishedAssets/build/pkg/res-buildables_0.47.0+20180207-033230+cafd095.dpk
2,6M UnvanquishedAssets/build/pkg/res-legacy_0+20171031-180425+df85675.dpk
72M UnvanquishedAssets/build/pkg/res-players_0.21.0+20180204-194123+546923b.dpk
13M UnvanquishedAssets/build/pkg/res-soundtrack_0+20170409-001127+a500934.dpk
1,9M UnvanquishedAssets/build/pkg/res-voices_0+20171031-160610+0ab70a3.dpk
27M UnvanquishedAssets/build/pkg/res-weapons_0.49.0+20180207-031452+b43c90b.dpk
4,0K UnvanquishedAssets/build/pkg/tex-all_0+20170405-013918+2ac1767.dpk
36K UnvanquishedAssets/build/pkg/tex-common_0+20180123-010358+26a828a.dpk
5,9M UnvanquishedAssets/build/pkg/tex-ej01_0+20170423-010335+9617f08.dpk
11M UnvanquishedAssets/build/pkg/tex-ex_0+20171101-123541+c711067.dpk
2,3M UnvanquishedAssets/build/pkg/tex-exm_0+20171101-123541+e10a4ac.dpk
4,9M UnvanquishedAssets/build/pkg/tex-pk01_0+20171101-123541+b9969e9.dpk
23M UnvanquishedAssets/build/pkg/tex-pk02_0+20171101-123541+03a70f4.dpk
17M UnvanquishedAssets/build/pkg/tex-space_0+20170726-011950+2d03c88.dpk
22M UnvanquishedAssets/build/pkg/tex-tech_0+20170404-234244+e9ea61f.dpk
11M UnvanquishedAssets/build/pkg/tex-trak5_0+20171101-123541+20319df.dpk
19M UnvanquishedAssets/build/pkg/tex-vega_0+20171030-150559+b5e2d3c.dpk
8,6M UnvanquishedAssets/build/pkg/unvanquished_0.50.0+20180204-200353+72482c0.dpk
7,2M UnvanquishedMaps/build/pkg/map-antares_0+a04+1.dpk
11M UnvanquishedMaps/build/pkg/map-chasm_0+b2.dpk
10M UnvanquishedMaps/build/pkg/map-forlorn_0+b1.dpk
9,3M UnvanquishedMaps/build/pkg/map-parpax_0+d05.dpk
17M UnvanquishedMaps/build/pkg/map-perseus_0+b5a+1.dpk
3,5M UnvanquishedMaps/build/pkg/map-plat23_0+b13+1.dpk
20M UnvanquishedMaps/build/pkg/map-spacetracks_0+1.0+1.dpk
35M UnvanquishedMaps/build/pkg/map-station15_0+1.0+2.dpk
28M UnvanquishedMaps/build/pkg/map-thunder_0+b3+2.dpk
23M UnvanquishedMaps/build/pkg/map-vega_0+b4.dpk
14M UnvanquishedMaps/build/pkg/map-yocto_0+b2c+1.dpk
479M total

6,3M CommunityMaps/build/pkg/map-ctcs_0+1.1.dpk
21M CommunityMaps/build/pkg/map-fortification_0+1.1.dpk
31M CommunityMaps/build/pkg/map-hangar28_0+0.1b.dpk
35M CommunityMaps/build/pkg/map-jota_0+2.0.dpk
11M CommunityMaps/build/pkg/map-stahl_0+0.1.dpk
13M CommunityMaps/build/pkg/map-usstremor_0+1.0.dpk
15M InterstellarOasis/build/pkg/map-arachnid2_0+20180107-190845+d3864eb.dpk
12M InterstellarOasis/build/pkg/map-atcshd_0+20170728-131416+0689015.dpk
5,8M InterstellarOasis/build/pkg/map-karith_0+20170407-181457+58cb569.dpk
6,1M InterstellarOasis/build/pkg/map-nexus6_0+20170407-181716+b0a893b.dpk
3,4M InterstellarOasis/build/pkg/map-niveus_0+20170407-181734+7abf366.dpk
6,6M InterstellarOasis/build/pkg/map-rsmse_0+20170408-190818+4862e81.dpk
23M InterstellarOasis/build/pkg/map-terminus_0+20170408-191401+7af4e0e.dpk
9,1M InterstellarOasis/build/pkg/map-transit_0+20180107-231400+9bf1730.dpk
4,4M InterstellarOasis/build/pkg/map-tremor_0+20180107-214355+1a86638.dpk
3,3M InterstellarOasis/build/pkg/map-uncreation_0+20170407-181909+16e554c.dpk
13M InterstellarOasis/build/pkg/res-tremulous_0+20180206-052424+437556c.dpk
215M total

694M total

This comment is licensed under cc ​​by 4 and antecedent.

Post Reply