[RFC] Fixing early design flaws (package format, package version]
What's happening?
In all projects some early decisions that looked good enough appear annoying later, when more stuff is done.
One of these early mistakes was to keep the pk3 file extension when the Dæmon VFS switched from classic id Tech 3 pak format to the newly dependency-based format. Well, because of this great change, our package format is not pk3 anymore. It raises a major issue: having to deal between standard pk3 and non-standard pk3 (ours) with ugly workarounds everywhere.
When TimePath added a cvar to enable third-party dæmon-based games loading legacy pk3, I suggested to add some metadata to pakpath directory, this way the engine would know if all the pk3 in one directory are standard pk3, and all pk3 in another directory are enhanced pk3. Well, even if the idea was not bad, it was wrong, it was wrong because it's wrong to flag the directory for the format of the files it contains instead of flagging the files themselves.
The PR was named “Support loading pak files which don't follow naming convention”, in fact, it would have been named “Support loading legacy pak format”.
I'm working on merging the old code by Neumond to enable pk3dir support in NetRadiant and GtkRadiant. The NetRadiant implementation is very complete, it can handles both standard pk3 and enhanced pk3, by the way, the code will probably never been merged as-is since it uses such an ugly workaround, it does that:
Code: Select all
# pseudo-gamepack:
pak_format = pk3
Code: Select all
# pseudo-code:
if pak_format == "pk3":
if game == "daemon":
findPk3WithDeps()
else:
findPk3()
That's bad , it means one day the NetRadiant code will look like that:
Code: Select all
# pseudo-code:
if pak_format == "pk3":
if game in [ "daemon", "unrealarena", "xonotic" ]:
findPk3WithDeps()
else:
findPk3()
You'll lose all the benefit of loadable gamepack, welcome back to GtkRadiant where you must hardcode everything. :cyclop: One of my earlier idea was to add a variable in the gamepack to flag the vfs, something like:
Code: Select all
# pseudo-gamepack:
pak_format = pk3
vfs_format = daemon
Code: Select all
# pseudo-code:
if pak_format == "pk3":
if vfs_format == "daemon":
findPk3WithDeps()
else:
findPk3()
This idea will work, but will still be wrong, the same way the idea to flag the directory was wrong. Every third-party program that have to deal with our paks and pakdirs will have to reproduces so many ugly workaround if they want to offer enhanced pk3 support: the Radiant editors, the Q3map2 compiler (currently loads every pk3dir so you don't know if your build is safe), the build tool I'm writing (Urcheon), and probably more.
We have to face the reality: our format is not PK3.
Let's introduce DPK.
Why not naming our package DPK? something like "Dæmon PacK" or "Deps PacK", also I thought it's a good idea to not increment the number (like PK5 or PK6) since other will have the same idea and the point is to not have to deal with different format with same extension. This way editors, compilers, builders, and even the game engine must be able to do:
Code: Select all
# pseudo-gamepack:
pak_format = dpk
Code: Select all
# pseudo-code:
if pak_format == "pk3":
findPk3()
elif pak_format == "dpk":
findDpk()
Clean, neat, awesome.
Also, an existing game full of legacy files (like Xonotic) switching to Dæmon engine would be able to do that:
Code: Select all
# pseudo-gamepack:
pak_format = [ dpk, pk3 ]
Currently in Daemon, enabling fs_legacypaks will load all versioned and non-versioned pak. If we switch to the DPK format, we will become able to load all legacy non-versioned PK3 but loads DPK the versioned way. We can also defines a priority: loads DPK first, PK3 later. This way, Xonotic Guys can switch to versioned package format for their own official packages, without having to load official packages they don't need, but being able to load legacy stuff, without having to fear them to overwrite their own stuff.
The first easy step would just be to add DPK as an extension name alternative first.
Adding DPK support in Dæmon ? Just add DPK extension as recognised one with PK3 first, then later you will be able to implement the format precedence.
Adding DPK support in GtkRadiant/NetRadiant/DarkRadiant ? just add the DPK extension as a recognized one alongside the PK3/PK4 one that are already there. Then later you can implement the package versionning for DPK.
Adding DPK support in XQF (server browser that parses some maps data) ? just add the DPK extension as a recognized one alongside the PK3 one. You want later save loading time just looking for "map-_.dpkdir" ? You can implement it later ! You will be able to do it !
Adding DPK support in q3map2 ? just add the DPK extension as a recognized one alongside the PK3 one that is already there, and later, if you have time, implement versioning.
The very first step will be to get the complete chain loading the DPK, it's just a matter of “if pak_format in ["pk3"] or ["dpk"]” if we have no time to polish it. Later, when the complete chain support DPK, when people are able to create content and play games using DPK, we can implement the versioning in software the knows nothing about it, later.
Extra benefit: fix early versioning mistakes
There is things that are not good in current package versions: many of them starts with a letter instead of a number: a5, b2, etc. It means 1.0 is smaller than alpha 1. Well, alpha/beta is a kind of quality flag, not a version, by the way, if someone wants to put the quality flag in the version string, he can safely put at the ends of the version string: 0.1a (alpha), 2.9b (beta), 6.8 (stable). The current workaround for packages that were badly versionned with a version string starting with a later is to start with r: r1.1.0 for example, by luck, r1.1.0 is smaller that src, but one day one guy will name his package map-castle_v2.pk3 and will not understand why the changes he does to map-castle_src.pk3dir does not show up. Also, there is plenty of old legacy maps with _trem versions on the web. When we will start to advertise them, people will found them and will put them on their servers and it will be very bad.
If we switch to DPK format, Unvanquished will not load the bad versionned PK3! It means every one get the opportunity to re-versionnize their package correctly! There is not many community mappers out there, they are still reachable so they can rename their package.
I also suggest to disable the DPK loading in engine and tools if the version starts with a letter instead of a number, logging a message so people will understand what is happening, with one or two exception: the src special version in build tools (for map-castle_src.dpkdir full of map source and lossless textures loaded by radiant and q3map2) and the test special version in engine (for map-castle_test.dpkdir full of bsp and crn files the engine load when a mapper tests what is he mapping before packaging).
Since we talk about version, I really don't like the current date policy. Why "2017-03-24-1524" and not "20170324-1524"? having so many hyphens is boring and my eyes have an hard time to read it fastly, it's the only place in the world where datetime is written in 4 chunks, it's not very comfortable, and it looks too verbose for a version string. Using the "one hyphen" format would be more compact and makes more sense: this way hthe hyĥen splits the version between date and time, it as a meaning, also, the hyphen will split between decimal numbers and sexagesimal numbers, which makes sense strongly. With hyphens everywhere it makes no sense at all. If we go the DPK way, we can redecides this kind of stuff, since there is no one DPK package out there.