New package format for upcoming 0.51.0 release (bye bye pk3)

Release your maps and discuss them, from alphas to finished releases.
User avatar
illwieckz
Project Head
Posts: 718
Joined: Sat Aug 11, 2012 7:22 pm UTC
Location: France
Contact:

New package format for upcoming 0.51.0 release (bye bye pk3)

Post by illwieckz »

:announce: Hi, upcoming 0.51.0 release will introduce a new package format to succeed to the good old pk3 one. Don't worry, it's just the same we already used…

TL;DR: .pk3 becomes .dpk

So, why the change ?

Over the years, Unvanquished improved a lot the good old pk3 format, adding version checking and dependency mechanism. The problem comes when you try to handle both the legacy pk3 and the pk3 plus addition formats. It's not only about the engine but also the editing chain, early patch for netradiant had to do ugly stuff like if pk3 format and game is unvanquished then do this if not unvanquished then do this and the more games want to use the Dæmon engine, the more ugly hardcoded exceptions would have to be made. That's not a good option, and these ugly workarounds even can't solve the issue of a Dæmon based games supporting both newer pk3 and legacy pk3, you see the point? Tools would have to do if pk3 format and game is Dæmon based and game is not Xonotic but it's not a legacy pk3 but perhaps it's an unvanquished game and I really don't know even the compiler itself would prefer to become a hermit to raise goats on a desert island.

So, we had to face the reality, our packages are not pk3 anymore, so the solution is simple: let's name them differently. :granger:

Let's welcome DPK

If you already published a map for Unvanquished the last years, you already use the DPK format: you just have to rename the pk3 extension to dpk extension and that's all.

You already know the naming scheme: map-<name>_<version>.dpk.

So just rename your map-castle_1.pk3 to map-castle_1.dpk and you're done!

Of course the uncompressed package directory is now using .dpkdir extension instead of the .pk3dir one, so you can rename your working directory this way: map-castle_src.dpkdir.

For more information about the DPK format (versioning, how to write a DEPS file to request other package to be loaded at map load etc.) you can see the DPK Format page on the Unvanquished wiki! :wink:

Redemption time

Well, I said you just have to rename your pk3 to dpk and that's all, by the way since no one DPK map is already used out there, it's the best time for you to fix naming or versioning issue. For example if you put a large version number and regret it, you can put a small one now: it's a fresh start for everyone of you.

Why do I say that? Basically many of us reproduced bad habits from tremulous, and also we were lacking solid building experience to discover and pick the best practices. During all these years I worked hard to build tools to assist the Unvanquished mapping process, and also patched a lot of others (including NetRadiant and Q3map2) so now I can tell you I discovered I made myself a lot of mistake in the past and I would invite you to not do them yourself. For example I versioned myself some early packages using the trem version number which is greater than the src version number I use in repositories, it means the released packages have priority over the work-in-progress one, that's painful!

So basically, it's a very good idea to reserve src version number for your work-in progress source package (for example: map-castle_src.dpkdir), it's a very good to reserver the test version number for your ready-to-test directory package with compressed assets etc (for example: map-castle_test.dpkdir). Using these version numbers is good because the Urcheon tool I wrote is able to handle them on an complete automated way. Urcheon is a tool that allows you to completely build entire libraries of packages (including maps and texture sets) handling the whole map compilation, asset compression and package compression process, and Urcheon is able to write the final package version number for you just reading your repository labels. :cool: :thumbup:

That's why for example using a _v2 version number is not good for you, you can, nothing will enforce you to not do it, it's a valid version number, but it can bite you hard in the future in the most unexpectable way. For example it means your previously released map-castle_v2.dpk package will have priority over your map-castle_src.dpkdir source package containing your upcoming v3 and radiant will loads the wrong assets. Also, if you want to use the Urcheon tool to handle your assets repositories and build your map and have it writing the version number automatically, you'll had to create a vv3 label to get your package automatically versionned map-castle_v3.dpk. :eek:

Like many other tools, Urcheon looks for v prefixed git labels as version numbers. And if someone wants to write another tool to do the same thing Urcheon does, he will probably does the same because that's a very good practice very commonly used in many development tools. That's also why version numbers like a1, b2 and r1 are not very good. They are very correct and will not bite you hard in your back, they are smaller than both src and _test version numbers, but it's will just makes you using ugly version labels in your git repositories: va1, vb2 and vr1. :dazed:

Well, it would probably be a good idea to enforce the starts with a number rule but I'm not that strong: you're currently free to do things that will annoy you one day. Basically it also means the build tools can't distinguish the valued and the v0.1 git labels, first one will produce a package named map-castle_alued.dpk and second one will produce a package named map-castle_0.1.dpk which looks weird, and with a always starts with a number rule the tools would be able to discriminate those and ignore the valued[/u] label to pick another label formed the v### way in history.

Basically, if you really want to flag your package as alpha or beta or things like that it's better for you to put the letter at the end of your version number, more like a flag or a tag than a version number: map-castle_0.1a.dpk, map-castle_0.2b.dpk, and map-castle_1.0.dpk. No one will enforce you to do things the other way, but doing this looks to be the very best practice to not get bad and unpleasant surprises later. :eek:

Examples and suggestions

This is a list of official and contrib maps I know, and what can be a best version (you're free to do whatever you want though):

Official maps (I can do the job for you at 0.51.0 release time)

  • map-antaresa04+1.pk3map-antares_0.4a.dpk

  • map-chasm_b2.pk3map-chasm_0.2b.dpk

  • map-forlornb1.pk3map-forlorn_0.1b.dpk

  • map-parpax_d05.pk3map-parpax_0.5d.dpk

  • map-perseusb5a+1.pk3map-perseus_0.5b.dpk

  • map-plat23_b13+1.pk3map-plat23_0.13b.dpk

  • map-spacetracks_1.0+1.pk3map-spacetracks_1.0.dpk

  • map-station15_1.0+2.pk3map-station15_1.0.dpk

  • map-thunderb3+2.pk3map-thunder_0.3b.dpk

  • map-vega_b4.pk3map-vega_0.4b.dpk

  • map-yocto_b2c+1.pk3map-yocto_0.2b.dpk



You can of course suggest other versions number you prefer, for example you can ask to rename map-chasm_b2.pk3 to map-chasm_2.dpk or map-chasm_2.0b.dpk, or to rename map-plat23_b13+1.pk3 to map-plat23_1.3b.dpk or map-plat23_13.dpk or map-plat23_13.0b.dpk or… tell us what you prefer and what you want. :grin:

Preview map (I can do the job for you at 0.51.0 release time)

  • map-freeway_1.0map-freeway_1.0.dpk



Maps I maintain (I will do the job myself)

  • map-arachnid2r1.2+55d0babe.pk3map-arachnid2_1.2.dpk

  • map-atcshd_r1.2+55d0babe.pk3map-atcshd_1.2.dpk

  • map-karithr1.2+55d0babe.pk3map-karith_1.2.dpk

  • map-nexus6_r1.2+55d0babe.pk3map-nexus6_1.2.dpk

  • map-niveusr1.2+55d0babe.pk3map-niveus_1.2.dpk

  • map-transit_r1.2+55d0babe.pk3map-transit_1.2.dpk

  • map-tremorr1.2+55d0babe.pk3map-tremor_1.2.dpk

  • map-uncreation_r1.2+55d0babe.pk3map-uncreation_1.2.dpk

  • map-rsmse_0+unv-a37-illwieckz+2.pk3map-rsmse_0.1.dpk

  • map-terminus_b22.pk3map-terminus_0.2b.dpk



Maps from various contributors (I can do it for you if you're OK)

  • map-combat-t3_1.3.pk3map-combat-t3_1.3.dpk

  • map-ctcs_1.1.pk3map-ctcs_1.1.dpk

  • map-fortification_1.1.pk3map-fortification_1.1.dpk

  • map-hangar28b1.pk3map-hangar28_0.1b.dpk

  • map-mms-stahl_v01.pk3map-mms-stahl_0.1.dpk

  • map-jotav2.0.pk3map-jota_2.0.dpk

  • map-usstremor_v1.0.pk3map-usstremor_1.0.dpk

  • map-stalkyard_b4.pk3map-stalkyard_0.4b.dpk



The same way you can ask for another version, like renaming map-hangar28_b1.pk3 to map-hangar28_1.dpk or map-hangar28_1.0b.dpk for example.

Let's rock!

Ping @Ingar, @Viech, @SuperTanker, @Pevel, @EmperorJack, @tvezet, @Matth, @FreeSlave etc.

Just remind that renaming .pk3 will work and will be enough, but do notice that if you want to change something you've did in the past to make something better instead, it's time ! :coffee: :hammer: :wink:

Note that if you want and if you give me permission, I can not only rename your maps package for you but also optimise them (recompress them using crunch, building missing navmeshes if needed etc.). That's a special question for @Matth since in the past you already asked me to make navmeshes for you: do you allow me to optimize and/or complete your previously released maps? :smile:

It would be very cool to have all your contrib maps properly renamed at 0.51.0 release time. :drink:

Let's rock, and let's pave the way to Unvanquished 0.51.0! :bsuit:

Image
Fig 1. People react to Unvanquished 0.51.0 rumor

Last edited by illwieckz on Wed Feb 07, 2018 2:58 am UTC, edited 2 times in total.

This comment is licensed under cc ​​by 4 and antecedent.

User avatar
illwieckz
Project Head
Posts: 718
Joined: Sat Aug 11, 2012 7:22 pm UTC
Location: France
Contact:

Re: New package format for upcoming 0.51.0 release (bye bye pk3)

Post by illwieckz »

:announce: Sorry, I forgot to say 0.51.0 release will merge tex-ej01-clean, tex-ej01-ice and tex-ej01-common into a lone tex-ej01 package, so maps relying on them will need a DEPS file rewrite.

The 0.51.0 release will also introduces a res-ambient packages for cool crates and ambient sounds and stuff like that. Currently only some ambient sounds are shipped there but if you use it, please add this package in DEPS file of your map. Currently it ships 30-60HzHum, drone1 and drone2 but basically, if you use any sound from sound/ambient or sound/movers directory please add the res-ambient package in DEPS file of your map even if Unvanquished still relies on tremulous ones since replacements will take places in this package and be removed from the tremulous compatibility archive.

So, if you use sound files from the sound/ambient path or sound/movers path please add the res-ambient package as DEPS to avoid missing sounds in the future, or you can ask me to do this for you if you're a contrib map author. :granger:

Last edited by illwieckz on Sun Feb 04, 2018 7:57 pm UTC, edited 3 times in total.

This comment is licensed under cc ​​by 4 and antecedent.

User avatar
Viech
Project Head
Posts: 2139
Joined: Fri Aug 03, 2012 11:50 pm UTC
Location: Berlin

Re: New package format for upcoming 0.51.0 release (bye bye pk3)

Post by Viech »

I'm happy about the .dpk format, but I dislike your version proposals.

I'd stick with

  • a01a99 for alpha releases,

  • b01b99 for beta releases and

  • r01r99 for (final) releases.

Also one can use d01d99 for "delta" releases when in the process of porting maps from Tremulous without falsly calling them an alpha or a beta, as I did with Parpax. All of these strings compare as they should against the src special version string.

I'd use this scheme even with Unvanquished itself starting with the beta cycle, b03 is just nicer than 0.89.0

Note that your proposal of e.g. map-plat23_0.13b.dpk for the 13th beta version is semantically wrong, because this way you get the 13th release cycle with minor release b(eta) instead of the b(eta) release cycle with minor release number 13.

Responsible for: Arch Linux package & torrent distribution, Parpax (map), Chameleon (map texture editor), Sloth (material file generator), gameplay design & programming, artistic direction

User avatar
illwieckz
Project Head
Posts: 718
Joined: Sat Aug 11, 2012 7:22 pm UTC
Location: France
Contact:

Re: New package format for upcoming 0.51.0 release (bye bye pk3)

Post by illwieckz »

In fact I'm not sure if it's a good idea to put the alpha/beta/whatever semantic in version number at all, it's looks to be another metadata to me. And if it were me I would probably just use one number without any dot so nobody would complain about the meaning of minor or major or whatever! :tongue:

By the way I don't want to do anything against someone. Except the 'v.*' versions that can clearly lead to some troubles in the future, we can live with numbers like 'a01' the same way we live with mosquitoes and hurricanes. :tongue:

This comment is licensed under cc ​​by 4 and antecedent.

User avatar
Matth
Dretch
Posts: 54
Joined: Sat Jul 19, 2014 8:57 am UTC
Location: Germany

Re: New package format for upcoming 0.51.0 release (bye bye pk3)

Post by Matth »

Note that if you want and if you give me permission, I can not only rename your maps package for you but also optimise them (recompress them using crunch, building missing navmeshes if needed etc.). That's a special question for @Matth since in the past you already asked me to make navmeshes for you: do you allow me to optimize and/or complete your previously released maps? :smile:

Sure thing! :beer:

User avatar
illwieckz
Project Head
Posts: 718
Joined: Sat Aug 11, 2012 7:22 pm UTC
Location: France
Contact:

Re: New package format for upcoming 0.51.0 release (bye bye pk3)

Post by illwieckz »

Thanks!! :beer:

This comment is licensed under cc ​​by 4 and antecedent.

User avatar
tvezet
Posts: 5
Joined: Mon Mar 06, 2017 2:44 pm UTC

Re: New package format for upcoming 0.51.0 release (bye bye pk3)

Post by tvezet »

map-hangar28_b1.pk3 → map-hangar28_0.1b.dpk

sounds nice to me, go ahead :granger: :thumbup:

User avatar
Viech
Project Head
Posts: 2139
Joined: Fri Aug 03, 2012 11:50 pm UTC
Location: Berlin

Re: New package format for upcoming 0.51.0 release (bye bye pk3)

Post by Viech »

illwieckz wrote:

And if it were me I would probably just use one number without any dot so nobody would complain about the meaning of minor or major or whatever! :tongue:

The modern approach is to start with 0.10 and then grow exponentially to Firefox version 42 or so.

Responsible for: Arch Linux package & torrent distribution, Parpax (map), Chameleon (map texture editor), Sloth (material file generator), gameplay design & programming, artistic direction

User avatar
Viech
Project Head
Posts: 2139
Joined: Fri Aug 03, 2012 11:50 pm UTC
Location: Berlin

Re: New package format for upcoming 0.51.0 release (bye bye pk3)

Post by Viech »

tvezet wrote:

map-hangar28_b1.pk3 → map-hangar28_0.1b.dpk

sounds nice to me, go ahead :granger: :thumbup:

This is factually wrong though. It means that the beta release cycle is merely a sub-release of Version 0.1. Don't do this to our maps illwieckz! :frown:

Responsible for: Arch Linux package & torrent distribution, Parpax (map), Chameleon (map texture editor), Sloth (material file generator), gameplay design & programming, artistic direction

User avatar
tvezet
Posts: 5
Joined: Mon Mar 06, 2017 2:44 pm UTC

Re: New package format for upcoming 0.51.0 release (bye bye pk3)

Post by tvezet »

Viech wrote:
tvezet wrote:

map-hangar28_b1.pk3 → map-hangar28_0.1b.dpk

sounds nice to me, go ahead :granger: :thumbup:

This is factually wrong though. It means that the beta release cycle is merely a sub-release of Version 0.1. Don't do this to our maps illwieckz! :frown:

I see your point, but for me it's just a package name which should contain a version number, so one can easily find the latest package. The sole reason why i included the beta state in the package name was the fact that it seemed common for Unvanquished maps and i'd like them to be versioned as uniform as possible. We can't stop people from inventing new crazy ways for that, but we could agree on standard syntax and semantics and write them down as such.
Make an offer, if it works with Unvanquished and everyone is fine with that, i'm fine too.

Post Reply