Hi guys,
You probably know that we use the crn format for our ready-for-GPU textures. Packages just ship a lot of .crn files and the engine sends stuff to GPU drivers without decompression, which leads to short load time and acceptable quality.
We currently use an old fork of crunch, the one from BinomialLLC, plus some additions for the normal map use case. Since BinomialLLC re-uploaded a more recent version I submitted some PR to upstream our fixes and changes (making changes optional) but they never got merged. In the same time, Unity was also working on it and published their changes.
Alexander Suvorov wrote some very interesting articles about their improvements on the Unity Blog:
There was some news coverage on Phoronix by example:
Crunch Texture Compression Showing Off Promising Results For Unity
Unity Continues Crunching More Out Of Crunch Texture Compression
Unity 2017.3 Released With Renderer Improvements, Better Particle System
So, it looks like their work is very good. And it's now shipped within Unity so plenty of people will use it massively.
They said in november that their modified crunch tool “can compress up to 2.5 times faster, while providing about 10% better compression ratio”. Well, I did a test on our own asset repository, re-crunching all the res-, tex- and unvanquished packages. This corpus produces 1797 .crn files. I benchmarked both our old crunch tool and the new unity one with our patches applied. I built myself the binaries so there is no bias on that part.
So, the numbers:
Unvanquished's crunch:
compression time: 115 minutes 8 seconds
produced file size: 269 megabytes
Unity's crunch:
compression time: 26 minutes 43 seconds
produced file size: 239 megabytes
So the unity crunch tool reduced compression time by 4.31 and reduced size by 11.15. They said “up to 2.5 time faster but I've seen some random textures being compressed 6 time faster and the average of the whole is 4.3 time faster, and yes the tool compress more than 10% more. The results are very very impressive and looks very good.
But there is a problem, a very big problem, to get these performances they had to broke the file format, and badly, they just swaped the old format with the new one and it looks like there is no way to handle both. There is only one code path in the new library, and it looks like there is no metadata to detect if a file is using the former format or the new one.
Compressing with Unity's crunch and decompressing with Unvanquished’s crunch:
Compressing with Unvanquished's crunch and decompressing with Unity's crunch:
This is how looks our game if we crunch our textures using Unity's tool and don't modify the engine (more weirdness can be seen there):
So, what's the problem?
They have a very improved crunch library but they broke the format.
So what to do?
With that improved format, packaging unvanquished assets reduces time from two hours to less than half one. And files are smaller.
Unity guys ship their crunch library within Unity since months now, so a lot of people are producing .crn
files and it gives big visibility to their format. Basically, they're probably just going to eclipse the old format. Since Unity is not just an small indie company we can expect some tools coming producing and reading only their format.
If we want to update the engine to the new format we must do it now or never: we already broke the package format (DPK) so everything already has to be re-shipped so it's the very right time to do other breaking changes.
If we update the engine to the new format we can rebuild all our packages, and I'm already doing it for the next release whatever the format changed or not. I also got permissions from guys like Matth to repackage and recompress their community maps for next release, so if we update the engine to the new format we lost nothing, even third-party stuff!
An updated crunch tree applying Unvanquished/Dæmon changes atop Unity improvements is findable there: github.com/illwieckz/crunch/tree/unity-daemon (use -rtopmip with -renormalize to get the behavior we need for normal maps).
ping @Ishq @Viech because some decision has to be taken.