The effort to track all unvanquished assets in repositories

Post all of your progress here about official assets, and receive feedback from our team.
User avatar
illwieckz
Project Head
Posts: 718
Joined: Sat Aug 11, 2012 7:22 pm UTC
Location: France
Contact:

The effort to track all unvanquished assets in repositories

Post by illwieckz »

The idea is to create a repository for each package, keeping sources and lossless files when possible.

The parent repository

The only one you would have to clone, getting everything else as submodules: github.com/UnvanquishedAssets/UnvanquishedAssets

All packages produced from repositories hosted at UnvanquishedAssets were published as drop-in replacements for 0.50.0 ones.

base pack

model stuff

textures

maps

other interesting repositories

Last edited by illwieckz on Mon Apr 24, 2017 2:46 pm UTC, edited 27 times in total.

This comment is licensed under cc ​​by 4 and antecedent.

User avatar
illwieckz
Project Head
Posts: 718
Joined: Sat Aug 11, 2012 7:22 pm UTC
Location: France
Contact:

Re: the effort to track all unvanquished assets in repositories

Post by illwieckz »

If you have png or lossless webp version of any tex- package, I will be happy to ask you for it. :wink:

This comment is licensed under cc ​​by 4 and antecedent.

User avatar
illwieckz
Project Head
Posts: 718
Joined: Sat Aug 11, 2012 7:22 pm UTC
Location: France
Contact:

Re: the effort to track all unvanquished assets in repositories

Post by illwieckz »

I'm currently pushing my repositories on some personal places (like https://github.com/interstellar-oasis/ or https://gitlab.com/illwieckz/ ), but in the end all this stuff must be moved in an official place. :announce:

I don't like the idea to push these assets repositories at https://github.com/Unvanquished/ it will pollute the repository list. An idea would be to create another github organisation like “Unvanquished-assets” (or something else you prefer), so “Unvanquished” is for code repositories. It will also help asset contributors to not be annoyed with non-asset repositories while browsing that repository list. What do you think about it? By the way, it's not an obligation. It's just my own taste. :beer: :coffee:

We need to decide promptly because it's annoying to me to put stuff on temporary places and creating issues on temporary places when it could be at the right place from the start. :wink:

This comment is licensed under cc ​​by 4 and antecedent.

User avatar
illwieckz
Project Head
Posts: 718
Joined: Sat Aug 11, 2012 7:22 pm UTC
Location: France
Contact:

Re: the effort to track all unvanquished assets in repositories

Post by illwieckz »

some thought from irc:

<illwieckz> What I think about is: a repository for every package, because it's better since git is not made for binary assets, and it's better since you don't mix everything, and because our awesome DEPS mechanism allows it (because xonotic don't have a DEPS mechanism they have a very big repo to track them all™ and it's just very annoying), then a parent git repository with each package as a submodule.
<illwieckz> An asset contributor can just clone the repository he needs (for example the sound repository if he is a sound designer, or the model one if he is a modeler/animator), or just clone the parent dir and fetch submodules, which is the typical scenario for a mapper.
<illwieckz> The mapper just have to use this parent dir as a pakpath in netradiant/q3map2 to build against original lossless assets.

This comment is licensed under cc ​​by 4 and antecedent.

User avatar
illwieckz
Project Head
Posts: 718
Joined: Sat Aug 11, 2012 7:22 pm UTC
Location: France
Contact:

Re: The effort to track all unvanquished assets in repositories

Post by illwieckz »

tex-common originals and sources are there:
http://dl.viech.name/tex-common-sources.tar.gz

tex-ej01 originals are there:
http://dl.viech.name/viechs-unsorted-as ... der.tar.gz /assets/tex-ej01/tex-ej01-ice_1.0.pk3
http://dl.viech.name/viechs-unsorted-as ... der.tar.gz /assets/tex-ej01/tex-ej01-clean_1.0.pk3
http://dl.viech.name/viechs-unsorted-as ... der.tar.gz /assets/tex-ej01/tex-ej01-common_1.0+crn.pk3
If you know the homepage or a source package for them, it could be good to reference. These are textures by well-known EmperorJack.

tex-space originals and sources are there (x4 resolution):
http://osirion.org/forum/index.php/topic,498.0.html
http://ingar.satgnu.net/files/unvanquis ... 02-src.zip

tex-pk01 and tex-pk02 originals and sources are there:
http://www.philipk.net/
http://www.philipk.net/portfolio/textures/pk02_tga.zip
http://www.philipk.net/portfolio/textur ... source.zip
http://www.philipk.net/portfolio/textures/pk01.zip
http://www.philipk.net/portfolio/textur ... source.zip

tex-tech originals are there:
http://cr4zyb4st4rd.co.uk/textures.html
http://www.mediafire.com/download/2ztnd ... shes_2.rar
http://www.mediafire.com/download/r7c9i ... Meshes.zip

tex-trak5 originals and sources are there:
http://arrrgh.fr/mapping/textures/trak/ ... xtures.zip
https://gitlab.com/xonotic/mediasource/ ... k5-src.zip

tex-ex originals are there:
http://evillair.net/v3/pages/textures/ex.php
http://evillair.net/v3/modules/download ... hp?file=24 eX256_textures.zip
http://evillair.net/v3/modules/download ... hp?file=25 ex_Textures.zip

The exm textures are eX-derivated textures by Viech (see the Alpha 27 map upgrade thread).

So, there is many stuff there, but with other names etc. I would be happy to get tga/png/webp ones with current names if someone have it. It will makes me save time.

Last edited by illwieckz on Sun Oct 09, 2016 3:35 am UTC, edited 2 times in total.

This comment is licensed under cc ​​by 4 and antecedent.

User avatar
illwieckz
Project Head
Posts: 718
Joined: Sat Aug 11, 2012 7:22 pm UTC
Location: France
Contact:

Re: The effort to track all unvanquished assets in repositories

Post by illwieckz »

<illwieckz> it looks like the normal maps from tga in trak5-textures.zip and from psd in trak5-src.zip are not exactly the one we use
<illwieckz> so, if someone has our tex-trak5 package in tga/png/webp format, it will makes me happy

Edit: it was an issue with crunch, the one from github.com/Unvanquished/crunch is mandatory, not the binary shipped in this repository, but the one you built from source.

Last edited by illwieckz on Mon Apr 24, 2017 2:49 pm UTC, edited 1 time in total.

This comment is licensed under cc ​​by 4 and antecedent.

User avatar
illwieckz
Project Head
Posts: 718
Joined: Sat Aug 11, 2012 7:22 pm UTC
Location: France
Contact:

Re: The effort to track all unvanquished assets in repositories

Post by illwieckz »

Viech sent me his texture build environment, all the tex- repos are now done. When possible, I made repositories with history (importing original files then applying changes to get what we currently have).

I have to double check them (comparing what they produces with what we produced before). After that I will be able to publish them. A decision is still to be made about where publishing them (see the dedicated post above).

The only texture set I haven't touched is the vega one since Ingar already manage a repo for it. Perhaps it could be good to extract that texture set from that map repository. The point of having source repositories for tex- packages is also to be able to build custom maps against original assets (and not against compressed ones, that can have issues or can be badly recognized by the toolchain).

This comment is licensed under cc ​​by 4 and antecedent.

User avatar
illwieckz
Project Head
Posts: 718
Joined: Sat Aug 11, 2012 7:22 pm UTC
Location: France
Contact:

Re: The effort to track all unvanquished assets in repositories

Post by illwieckz »

I do triple check:

  1. naming check (listing both package file list and comparing)
  2. content check (using dupeguru PE for textures)
  3. in-game check (checking daemon log for issues and playing a standard game)

An extra check must be done: compiling maps against the packages, but a good idea would be to make map repositories first. A test is good if it's reproduceable by someone else.

This comment is licensed under cc ​​by 4 and antecedent.

User avatar
Viech
Project Head
Posts: 2139
Joined: Fri Aug 03, 2012 11:50 pm UTC
Location: Berlin

Re: The effort to track all unvanquished assets in repositories

Post by Viech »

What do you mean, exactly? The tex-*.pk3 packages are already dedicated texture repositories for maps. (And in my opinion they should stay like they are and not be included in some bigger release package, if that's what you were planning…)

Responsible for: Arch Linux package & torrent distribution, Parpax (map), Chameleon (map texture editor), Sloth (material file generator), gameplay design & programming, artistic direction

User avatar
illwieckz
Project Head
Posts: 718
Joined: Sat Aug 11, 2012 7:22 pm UTC
Location: France
Contact:

Re: The effort to track all unvanquished assets in repositories

Post by illwieckz »

I want mappers being able to build against original files, not against converted one. This way it avoids all the format issues in the edition chain (netradiant), in the buildchain (q3map2) or specific bugs due to specific format (like shadowcasting issues with crunched textures). Uncommon compression formats (like webp, crn, etc.) are for internal use in the game engine itself. Also I want mappers being able to build Unvanquished-related stuff without having to rely on “.unvanquished/pkg” at all. Do you remember the time when Quake 3 modders were all building against “.q3a/baseq3” and how it was hard to prove the build was clean, so nobody cared for legal issues because it was hard to prove the build was clean because of the way the build chain was designed to mix Id Software “Intellectual Property” with user's contribution ? I don't want that for Unvanquished and Daemon based game makers. I want the Daemon engine to be an engine it's nice to make game with, and if something is annoying in Unvanquished build, it means it's annoying in Daemon-based game development and must be fixed. :tongue:

I think people must be able to create games with Daemon just the way we do Unvanquished, so if it's hard to me to rebuild Unvanquished the way Unvanquished is done, it means it's hard to create a game with Daemon. That's why I'm not only thinking about the engine itself, but I'm also thinking about the editing toolchain and the the way assets are managed.

That's why I remade all these tex packages a way they can be fetched independently, each one having his own build mechanism, done a way you can build multiple of them in a row. Because if we success at this point, it means everyone who wants to base his own game on Daemon can success at this point too. :wink:

The repositories I have on my computer will allow anyone to fetch and build one texture package:

Code: Select all

git clone …/tex-space_src.dpkdir
# fetching tex-space_src.dpkdir
cd tex-space_src.dpkdir
urcheon prepare
# preparing .
urcheon build
# building build/test/tex-space_test.dpkdir
urcheon package
# packaging build/pkg/tex-space_1.0.dpk

Or to fetch all texture packages:

Code: Select all

git clone …/UnvanquishedAssets
cd Unvanquished-assets
make init
# fetching tex-all_src.dpkdir
# fetching tex-common_src.dpkdir
# fetching tex-ej01_src.dpkdir
# fetching tex-exm_src.dpkdir
# fetching tex-ex_src.dpkdir
# fetching tex-pk01_src.dpkdir
# fetching tex-pk02_src.dpkdir
# fetching tex-space_src.dpkdir
# fetching tex-tech_src.dpkdir
# fetching tex-trak5_src.dpkdir
make prepare
# preparing src/tex-all_src.dpkdir
# preparing src/tex-common_src.dpkdir
# preparing src/tex-ej01_src.dpkdir
# preparing src/tex-exm_src.dpkdir
# preparing src/tex-ex_src.dpkdir
# preparing src/tex-pk01_src.dpkdir
# preparing src/tex-pk02_src.dpkdir
# preparing src/tex-space_src.dpkdir
# preparing src/tex-tech_src.dpkdir
# preparing src/tex-trak5_src.dpkdir
make build
# building build/test/tex-all_test.dpkdir
# building build/test/tex-common_test.dpkdir
# building build/test/tex-ej01_test.dpkdir
# building build/test/tex-exm_test.dpkdir
# building build/test/tex-ex_test.dpkdir
# building build/test/tex-pk01_test.dpkdir
# building build/test/tex-pk02_test.dpkdir
# building build/test/tex-space_test.dpkdir
# building build/test/tex-tech_test.dpkdir
# building build/test/tex-trak5_test.dpkdir
make package
# packaging build/pkg/tex-all_2.0.dpk
# packaging build/pkg/tex-common_2.0.dpk
# packaging build/pkg/tex-ej01_1.0.dpk
# packaging build/pkg/tex-exm_1.0.dpk
# packaging build/pkg/tex-ex_1.0.dpk
# packaging build/pkg/tex-pk01_1.0.dpk
# packaging build/pkg/tex-pk02_1.0.dpk
# packaging build/pkg/tex-space_1.0.dpk
# packaging build/pkg/tex-tech_1.0.dpk
# packaging build/pkg/tex-trak5_1.0.dpk

Of course, for mapping, there is no need to build these packages, and it's not recommended since it's better for the mapper to build his own map sources against texture dependencies sources. That way, mappers can build against “tex-space_src.dpkdir” if they only need this texture package, or building against “Unvanquished-assets/” if they need all tex packages.

That way open-source mappers who wants to make open source maps only can fetch open source tex-packages only. “Hello Xonotic mappers, Unvanquished will be a good target for you, I hope we will be able to work together, and I'm sure once you will have tasted the improvement we have made to the toolchain you will ask for them in Xonotic too, it's win-win”. :cool:

It's because of that I'm writing a toolchain and I'm modifying q3map2 like adding -fs_pathpath support to q3map2 the same way it works in the daemon engine. That way people can compile maps doing “q3map2 -fs_pakpath Unvanquished-assets" and can test maps doing “daemon -pakpath Unvanquished-assets/build/test".

Currently I'm trying to run Unvanquished only using the built packages from the repositories I made, running daemon from Unvanquished git and pakpath from Unvanquished-assets. It already works for model package (only faced one issue with flamer first person view, haven't yet tried to find the cause, everything else work as expected) and already works with tex packages (I still have to consciously check some names to be sure).

This way, this repositories also makes Unvanquished a fully reproducible game, for everything, from code to assets, by everyone (including Debian for example, since it's the example you gave for licensing). To me that is more important than having a completely free game, the reproducible build requisite comes before the open source build requisite, since open source stuff you can't build from source does not make so much sense. :cyclops:

I want anyone being able to do:

Code: Select all

git clone https://github.com/UnvanquishedAssets/UnvanquishedAssets
git clone https://github.com/Unvanquished/Unvanquished
cd UnvanquishedAssets
make
cd ..
cd Unvanquished
mkdir build ; cmake .. ; make
./daemon -pakpath ../../Unvanquished-assets/build/pkg

If I'm checking my repositories against yours it's because I redid the git history for each tex-package. For example for the tex-space package, I unpacked osirion files from env-osirion-r02.zip with original base names (like “osirion/osirion/sky01_back6”, converting them to png, and rotating bottom cw and top ccw, with a commit that says they are original files converted to png with bottom rotated wc and top ccw (following Ingar's instructions he left), then I made a commit renaming all that stuff to fit Unvanquished naming (renaming “osirion/sky02_left2.png” to “env/shared_space_src/sky02_rt.png” for example), then I made a commit to add shaders, etc.

Yes I like git history like I like to tell stories. Once upon a time, here finishes my wall of text. :coffee:

Last edited by illwieckz on Wed Jul 26, 2017 2:19 am UTC, edited 2 times in total.

This comment is licensed under cc ​​by 4 and antecedent.

Post Reply