Adventures in the admin system from the X servers.

Talk about anything related to Unvanquished.
Little*Butterfly
Posts: 14
Joined: Sun Mar 03, 2013 4:47 pm UTC

Adventures in the admin system from the X servers.

Post by Little*Butterfly »

Hello Unvanquished team,

Some of you may know me from the team of people who run the X Group of Tremulous 1.1 servers, currently servers W, X, Z and F.

After an interesting exchange with Ishq about collaboration with the Unvanquished project, we decided I should share some of the staff's experience when running multiple servers and the things that help us keep things running.

THE X GROUP GLOBAL SYSTEM
Before we had globals, we were using the plain old !ban command and in the reason for the ban, we would direct players to our internationalized ban information page at http://xserverx.com/ban/ so they could understand in their language, the reason for the ban and details on how to appeal the ban.

At some point in time, our developers gave us a mysql-based mechanism to further extend the functionality provided by the legacy !ban command. We are able to apply different restrictions to players depending on the IP/subnet they connect from.

The restrictions are: denybuild, handicap, mute, forcespec and ban
In addition to the restriction, we're able to whitelist players by their GUID or their static IP address, overriding any other restriction that might be effective for their subnet.

Example: to denybuild a newbie builder/deconner named DeconMeister from IP address 192.0.43.10, our admins do

!gdenybuild 192.0.43.10 [Description here]

We have other commands to list globals, delete entries and add/delete players to the whitelist.

Functionality of the global system is limited. We're presently only able to specify single IP addresses (/32) and /24 and /16 subnets. It was desirable to have full CIDR support (/12, /20 etc.) support with our globals but we're out of active developers to implement this functionality. We also lack a mechanism to automatically purge global entries that have not been 'triggered' in a certain amount of time. Our globals database is public and can be viewed at http://xserverx.com/globals/

I believe it would be great if Unvanquished could do something as versatile for the admin system.

Ideally, I would be able to have commands like these for managing player restrictions

!gadd [B|H|W|D|M|S] [IP|NET|SLOT] [Description]

Example to Denybuild Noobmeister, I would do

!gadd D 192.0.43.10 Deconmeister-deconner

Or if she connects with a broad range of dynamic IP addresses in a /12 netblock

!gadd D 192.0.43.0/12 Deconmeister-deconner

Other features I would have loved to have if Tremulous was still being actively developed, are:

1. AUTOMATIC LOG ROTATION
At the time of writing, our server W games.log is 583 MB with 9.5 million lines, starting 15 september 2012. I'd like a setting in server.cfg to automagically rotate logs by date or size, storing a specified number of old logs in the form games.log.0, games.log.1

2. EXTENDED TIMESTAMPING
Presently, to find which date an incident occured, I have to find the "Realtime: " entry in the log file and then start from there. I have plenty of disk space so I have no problems in having an extended timestamp where each line of the log file would have the full date before the time

For instance, rather than have
23:27 ClientConnect: 13 [192.0.43.10] (ABCDEF12345ABCD1235) "DeconMeister"

I would have
2013-02-04 23:27 ClientConnect: 13 [192.0.43.10] (ABCDEF12345ABCD1235) "DeconMeister"

Of course the timeformat is selectable in the server cfg to cater for people who like to use different formats such as YY-MM-DD or DD-MM-YYYY

3. CLIENT-SIDE MESSAGE FILTERS
When moderating a match, I may want to filter out messages like who was killed by whom on a server without FF. I am more interested to see build events to catch those pesky deconners.

I'll amend my post when I remember of other things.

User avatar
Ishq
Project Head
Posts: 1145
Joined: Tue Mar 06, 2012 8:32 pm UTC

Re: Adventures in the admin system from the X servers.

Post by Ishq »

Thanks for summary of the comprehensive admin system. Your feature requests are quite sensible as well, we can include them, in some form or another. One modification I'd make is to store server logs by day or month instead of by size.

Black Hole
Posts: 2
Joined: Tue Apr 02, 2013 8:05 pm UTC

Re: Adventures in the admin system from the X servers.

Post by Black Hole »

Hi there, I am Black Hole, one of High Administrators of the X Server Group, long-time admin, and assistant to Little*Butterfly.

I would like to make a note of a couple additional features that would be greatly beneficial to the everyday server activities of our servers. Please excuse my brevity compared to LB. If you are at all confused with what I am describing please mention it and I will give a further explanation.

1. Inactive/Active Globals

It would greatly help if there was a way to mark a certain global as inactive, which would cause it to remain in the list, but enter a disabled state.

2. Automatic Expire Times

Additionally, a way to specify a time after which a global would enter an inactive state as above, or if the above cannot be implemented, be deleted. This could apply a special Expired state to the global allowing review by administrators of Expired globals periodically.

3. Whitelist

Similar to globals, we have a whitelist system which can whitelist players by IP or GUID who may be caught in a global subnet. This is extremely important in some cases and it would be nice to have a !wadd command similar to the !gadd to help with this. Additionally, a feature that could be very helpful would be the ability to specify a specific global to be whitelisted from, or, of course, all globals.

4. Verify changes to data files

Would be nice for some double checking to be done when attempting to modify globals or admin.dat to make sure the changes remain persistent. May seem trivial, but we have had some rather annoying problems with this

5. Web Viewing/Sorting Capabilities of Globals

The fact that we can view globals through a web interface is very beneficial. This feature is NOT something that would need to be implemented directly into the game, however, when choosing how to store the global data, it would be best to consider a method which would allow a way to do this.

I'm sure I will think of some more smaller things, but there are the big things that come to the top of my mind.

Thanks for your consideration of these matters. We will be watching this project.

User avatar
Ishq
Project Head
Posts: 1145
Joined: Tue Mar 06, 2012 8:32 pm UTC

Re: Adventures in the admin system from the X servers.

Post by Ishq »

This is interesting because this is an area that Unvanquished has not really explored too deeply: server administration. We've added some useful commands here and there, but we've never seriously developed with server administration in mind. Please keep these ideas coming.

In response to your ideas:
First of all, in Unvanquished, your guid will not change from server to server ( unless the server does something funky :-/ ), so that'll help with global servers from the start. Unvanquished uses pubkey authentication so that even if your GUID is stolen, one needs the actual "pubkey" file to actually authenticate to get admin. Without the file, it will show up on /listplayers that this player has a guid but no pubkey alerting other admins to potential funny business.

As for the real web integration, this is an interesting feature that has not been explored in depth. Unvanquished may perhaps support MySQL or SQLite in the future depending on large community interest and how well the game picks up.

I'm uncertain what you mean by verifying changes to data files. Are you asking to verify that the data is actually written and not merely buffered? (I think trem had a cvar for this, but I think it only applies to logfiles. I haven't looked too closely at when data is actually written for admin.dat. Anyways, see if setting g_logfilesync 1 helps any).

Little*Butterfly
Posts: 14
Joined: Sun Mar 03, 2013 4:47 pm UTC

Re: Adventures in the admin system from the X servers.

Post by Little*Butterfly »

Can you give me a description of the logic of the pubkey system coupled with a GUID, in plain english?

User avatar
Ishq
Project Head
Posts: 1145
Joined: Tue Mar 06, 2012 8:32 pm UTC

Re: Adventures in the admin system from the X servers.

Post by Ishq »

User connects to server.
User sends pubkey to server.
Server generates serverside GUID from pubkey. (MD5, eg, as long as no server messes with this code, GUID will be the same from server to server).
Server sends randomly generated key to user to decrypt.
User decrypts key and sends back decrypted key.
Server checks to see whether decrypted keys match.
If they match, server grants user admin.

Some things to note:
There is no longer a "qkey" file which stores the GUID. There is, in fact, no clientside GUID. Only a pubkey.

chanjyj
Posts: 1
Joined: Wed Apr 03, 2013 12:57 am UTC

Re: Adventures in the admin system from the X servers.

Post by chanjyj »

Hi all, Chan here as the head admin of W. Not in rather than flying a desk I actually fly the plane compared to Black Hole and Little*Butterfly (pffft!).

Ok down to business. A quick background of the W population. W is also the server I believe will be used as a test base for Unvanquished if it comes to it; as this is the only "regular" server besides AA which has the population base to do full-scale evaluation. I have excluded F server as I believe the modifications needed to change it to a "bot server" are too much.

The W population is unique. It has a large number of new players as well as regular players. This is due to easier gameplay, where free credits are given, there is a larger number of build points etc - which makes it easier for the new players to learn the game. This has also resulted in a situation where the new players make noobish mistakes and create chaos. This is where we have stepped in, via the system to try and rectify issues.

  1. Players with no GUID will not be able to decon the OM or RC, the 2 most critical structures.

  2. Players with no GUID will not be able to /callvote kick.

As you can see, this hinges on the GUID system.

If possible, I would like to see this implemented into Unvanquished as an option. Alternatively, force all players to have a pubkey, and build a multi-tier admin system:

Level 1: Not able to decon OM/RC. Not able to /callvote kick.
Level 2: Full rights of regular players
Level 3: Team Managers (Current Level 2)
Level 4: Junior Admins (Current Level 3)
Level 5: Senior Admins (Current Level 4)
Level 6: Senior Admins who are PAs to the Head Admin (Current Level 5)
Level 7: Head Admin

Perhaps I'm going a little off tangent. But could we build a semi-automated system that tracks players by GUID and does the following:

  1. Identifies all Level 1 admins upon connection, and sends a !cp message to them informing them of W gameplay.

  2. Auto-promotes Level 1 admins to Level 2 admins after 3 months of play (they can of course, be promoted faster manually if deemed fit).

In addition, I would like to see an in-built system that will alert an in-game admin if there are excessive decons by a single player. Eg:

  1. Player ABC decons more than 5 structures within X number of minutes - !cp message is printed to all in-game admins warning them of possible malicious behaviour

  2. Player ABC decons more than 5 structures within Y number of minutes after the initial stage 1 trigger, the game will auto !pause, !cp the admins in-game and get them to take action. This will be contingent on the fact that admins are in-game. Should admins not be in-game, the server will simply print a special message in logs with player alias, GUID, IP and partial !buildlog. This will make reviewing appeals / complaints much easier.

Regards
Chan

Black Hole
Posts: 2
Joined: Tue Apr 02, 2013 8:05 pm UTC

Re: Adventures in the admin system from the X servers.

Post by Black Hole »

I think some of the automatic promotion stuff shouldn't be in the main game and should instead be left to QVM modifications based individual server needs. Things like automatic promotions may work for W, but wouldn't work for X.

Forcing all clients to have a GUID will help a lot. In the end, I think in game situational actions should be a priority over the more critical back end things. I personally think my associate Chan's suggestions would defiantly be helpful but effort should primarily be dedicated to the more core back end functions as opposed to in game procedures.

As regards my post about verify data after writing. Simply I mean a way to check to make sure the data was actually written to the disk after it was asked to be. Specifically concerning admin.dat since that is where we have had some issues.

Little*Butterfly
Posts: 14
Joined: Sun Mar 03, 2013 4:47 pm UTC

Re: Adventures in the admin system from the X servers.

Post by Little*Butterfly »

Black Hole, Chan. From what I understand from Ishq's explanation of the revamped UID system, Unvanquished will use the pubkey authentication by default unlike the default tremulous 1.1 which does not and had to be 'upgraded' with backport or tremfusion. Question to Ishq, have your devs considered that MD5 is no longer considered a mathematically secure hashing algorithm because of collision vulnerabilities? Is it too far-fetched to consider this vulnerability non-acceptable for a computer game and consider using SHA-256 instead?

Also, how does an Unvanquished server behave if I use a hacked client to send NO pubkey at all. Am I denied the right to join the server completely? Can't join a team? Also please think of a mechanism to prevent this kind of abuse:

My understanding is that the server-side GUID is created and stored when a player connects. I assume there is no Tremulous-like !register process to create this server-side GUID and my next line of thought follows this assumption and outlines a theoretical attack against the system:

Skilled programmer who is also an ass decides to attack my server by creating a hacked client which does this:

Connect to my server with a pubkey (server creates my server-side guid)
Disconnect.
Client deletes the pubkey and generates a new one which we call pubkey2
Connect to my server again with pubkey2 (server creates my serverside guid2)
Disconnect
Client deletes the pubkey2 and generates a new one which we call pubkey3
Disconnect

Repeat process from multiple shell accounts with a tremulous-tty-style client. Done in quick succession, this has the potential of spiking server load as the server computes hashes, encrypts/decrypts data and filling up the file where the GUIDs are stored with crap.

User avatar
Ishq
Project Head
Posts: 1145
Joined: Tue Mar 06, 2012 8:32 pm UTC

Re: Adventures in the admin system from the X servers.

Post by Ishq »

Yes. The idea is that all clients will have pubkey by default and it cannot be disabled. Connecting without a valid pubkey will mean you will be unable to connect. Despite MD5 not being mathematically sound, it is extremely quick to calculate and the actual chance of there being a collision is quite low. Even if there were a collision, the wrong person would not get admin because said person would not be able to decrypt the key the server sends. Lastly in response to your abuse example, as I previously mentioned, MD5 is very quick to calculate and the fact that the user will have to generate a new 512bit pubkey everytime he starts his client (I have tested this: rapid connection to the same server with a tty client. It takes a couple seconds to actually connect), which means the server will have a couple seconds of break each time and not really feel the load.

In response to chan and Black Hole: Unvanquished will aim to provide a QVM that is the lowest common denominator for all the servers. Anything extra, you will need to add yourself.

Post Reply