Firstly, I'd like to apologise for not making a blog post sooner. I've been struggling for topics as of recent, so if you've got anything that you particularly want to see me then please drop me a Forum PM. Now let's talk about what we've been getting up to as of recent on the HR side of things!

Due to the nature of the position, it's hard to talk about everything I do because not everything I do can be spoken about openly which is difficult when you're trying to keep the community looped into what the human resources team is up to, especially when we often deal with a load of issues on a daily basis. However, both the admin team and HR have still been making observations to how specific processes could be made to work better, one observation we've made recently is how forum warnings are issued and how we punish people who break them using the warning system.

Before the revamp earlier today happened, the warning system was quite tedious to use as you could never be too sure what rule violation fell under which point. This could lead to people getting warned under the wrong warning type, meaning points may not be issued properly for the violation that happened, though it's worth noting that we didn't see this happen in practice.

So because we saw an issue, we went off to work! Our wonderful Doctor Internet started to write up a generalised list of all of our forum rules, combining them into simple, short descriptors for each section. For example, posting rules 2, 3 and 4 could all be shortened by referencing them as disrespect. We ended up doing this for every rule there is, combining them and then once done, admins had the opportunity to comment and suggest changes to what the list was.

After a few days and admins having agreed that the newly proposed list was reasonable, we had our final list! Then it goes over to me and Overlewd so we can decide the appropriate amount of points to assign, as well as how long it should take for those points to expire. Two more days later, we finally came up with a nice list of everything, as well as times and points. From there, it's just a matter of making sure the forum system is updated to include them.

This did take admittedly longer than I'd like, as I had no idea how to set that sort of thing up, nor did I know whether I had access to change that but after a little bit of tinkering around on the forums for a while, I realised I did have access so after a quick message out to the staff saying to avoid warning someone for the next 20 minutes, I started changing it up! Thankfully it didn't take that long, the system for sorting out warning types is actually quite straightforward as it turns out!

You might be seeing the list of warning types and thinking to yourself, "Jeez Night, 5 points for unsafe content, threatening and lying in the courthouse? That's a bit extreme isn't it?" and in normal circumstances I'd agree, but when we say unsafe content, we're talking about harmful or malicious software, hacks, warez, spyware and IP loggers, that sort of stuff. The stuff that you obviously shouldn't be posting in the first place.

With threatening, staff have been briefed that warnings under this type shouldn't occur unless the comment is a clear, serious and credible threat. In some cases, it may be more appropriate to warn someone under disrespect/flaming, trolling/useless/irrelevant content or perhaps not at all, but this of course will be up to staff discretion.

Hopefully with these changes, punishments on the forum will become a bit more standardised and uniform, making it fairer for everyone. Before, we only had 4 or 5 different options to warn someone under. Now look at how it is!

If you find you get warned for double of what's listed here, it may be that your permanently suspended and in this case, all point values are doubled. If you don't think your warning level makes sense, of course feel free to speak to either the issuing staff member or HR as normal.

Thank you all, and until next time!

In time, their duties increased with the implementation of newer systems and also with access to new abilities. For example, assisting in the Public Discord by answering questions or assisting in the Team Discord by collecting information, giving opinions and helping test new features.

While their primary objective is to assist players in-game, Teachers also do a lot of work on the forums, discord and other areas for the community. We especially value the assistance of Teachers who are experienced, meaning who have been a Teacher for a long time as their experience can guide newer Teachers and help them ease into the team faster.

Teacher Activity

Teacher Activity has been raised as a discussion point recently, and we are working on resolving the issue by bringing new guidelines to how the team works. With the new guidelines, we wanted to ensure that Activity is not only in-game, but also on the forums. We also wanted to encourage Teachers to host more events and mini-events by bringing a new event requirement.

In-Game Hours

Teachers are required to have at least 10 hours a month, which might increase in the future however we did not want to make the requirement high, forcing Teachers on the server as that would take the fun away and if you are volunteering, the payment is the fun you get from assisting the community and enjoying the server.

We wanted to ensure, while we still had Teachers go in-game, their experience wasn't forced, and they were having fun.

Forum Activity

We could view forum assistance of Teachers as one of the secondary objectives of sorts. They assist in the Help & Support section of the forums, with moderation abilities. They keep the forum section organized, and provide support to questions raised. Teachers also have a Teacher Lounge in which they can suggest new Teacher systems, organize events, give opinions in matters.

Hosting Events

Encouraging roleplay is another secondary objective of the Teachers, this can be done by them hosting mini-events on the live server, hosting events on the event server or rewarding a roleplay they believe deserves recognition with REPs.

To improve on this front, we wanted to bring a new requirement, in which Teachers are required to host at least 1 event in 3 months. These events could be a mini-event on live or an event on the event server. We know that hosting an event is difficult, that's why we encourage Teachers to work together and for mini-events provide a list of ideas they can use.

Strike System

Alright, this is not as evil as it sounds.

To keep track of Teacher mis-steps, the previous liaisons Night & Daley introduced the strike system for the teachers.

In this system if an incident was deemed strike worthy, the Teacher in question would receive one. If a teacher received 3-strikes, their position would be taken from them.

Strikes do expire, however they stay on record.

Thankfully, no one received 3-strikes since the implementation of the system.

This system currently ties into the activity requirements.

If a Teacher doesn't fulfill the activity requirement of that month without a valid reason of absence (exams, real life occurences) they will receive a strike.

This way, we provide the Teachers with a chance.

Meetings

We want to be informed about our Teachers, and ask how they are doing, what can be improved and their opinions on subjects regularly.

That is why, we decided to have interviews with every Teacher individually every month. We also decided to have bi-monthly Teacher Team meetings to get an overall discussion going, addressing general Teacher Team concerns.

Application Process

The application process has not changed much in foundation for years, some elements of the application forms have changed but in general, it is the same. However, we did have concerns to address and issues to resolve regarding the procedure.

For example, while voting for a player internally, we can't be sure about some elements brought up, which can be minor and/or resolved via a trial run.

That is why we decided to introduce a brief trial period to new Teachers. This would ensure the team gave a chance to players with minor concerns, and we would try to resolve those concerns during the trial run.

This was mainly introduced so we made sure we keep getting new blood into the team.

With the Trial Period, we also introduced an application interview in which we talked with the applicant, asking questions brought up internally or just to get their opinion. This is so we can get to know the applicant better and address concerns raised on their thread.

In short, we are working to improve the Teacher Team, the procedures and better the experience of the Teachers and the community.

In 2018, two years after our last huge upgrade, we noticed that some of our software packages, tools and even the operating systems on both of our servers would enter EOL (End-Of-Life) state in spring 2019. This lead to extensive planning and discussions in a joint working group for months, mainly so that the next upgrade would provide future stability including improved workflows for our developers. I remember back in 2016 when upgrading both servers was quickly done because our backups contained data of just one year (August 2015 - August 2016). But after 3 years and 3 months (08/16 - 11/19) you can imagine how many backups and how much data would need to be taken care of.

Meanwhile members of the joint upgrade taskforce suggested that we could also upgrade our development workflows, using the latest PHP Version, supporting Docker and moving our VCS (Version Control System) from SVN to git. Not only would this improve our overall performance on development, but also help us reduce bugs, and offer new modern services as well.

The Limelight SCM Application

And this is where the upgrade journey started in April 2019.
Searching for a suitable git hosting provider was a no-brainer and we went with Github.com. Our HoS (Head of Security) Faustie set up a GitHub organization and I started working on a source control management application, Limelight SCM, that would provide continuous deployment (CD) for our web services. Doctor Internet then migrated these services to git and we started using them in production.
To aid our developers in upgrading their PHP Code to support PHP 7.x+, we added a tool to our SCM that would perform sanity- and compatibility-checks.

Our SCM application is unique in such a way that it offers:
- Namespace Management (folder routing to subdomains)
- Branch Management (production, development, feature, fix)
- Deployment Statuses / Reports
- Dealing with unstaged changes
- Support for external tools (Composer, PHP7 compat, ..)
- Protection of deployments (.htaccess, webserver, ..)
- Integrated namespace blacklist
- Permission management for deployments
- Adapters for extending/configuring the SCM

For instance, this is our .scmconfig adapter configuration:

Upgrade Day(s)

Before we were able to start upgrading, we had to carefully make backups.
Those backups contain the databases, game servers, teamspeak and other system utilities.

The Beta Server
Until Nov 11th, the beta server lived in its own SVN repository, and a couple of changes had to be made for supporting git. Instead of working on two repositories (Live and Beta), we decided that with git we wanted just one main repository with separate branches:

Master Branch = the production-ready code (live server)
Development Branch= represents the beta server

Goodbye Beta Server.....for now
Those changes ultimately resulted in archiving the beta server to its own git repository, as we were unable to simply merge the beta code to the new development branch of the main repository. Both repositories had different commit histories.

Hello again, Beta Server!
Instead, the new beta server was created from the development branch of the main repository, and we started backporting code from the beta repository to the development branch of the main repository.

Upgrading the DS1 server
Now we were able to start upgrading our server DS1(EU) that is primarily responsible for running the gameservers and its databases. The installation was quickly performed until we ran into some difficulties when booting our secondary NIC (Networking Interface Controller).
After booting into rescue mode - investigating, we quickly found the problem. Our secondary NIC was using an identifier that was not yet configured on the new freshly installed OS.
So we did that, brought the secondary NIC up and it was just fine.

Game servers back online!
After retrieving our backups, installing the database server and additional dev services, we focused on getting the game servers running.
This included testing and tackling some issues on the beta server.

Yes, we  spent the whole day and night (from 11th Nov 11am to 12th Nov 7 am) on upgrading the DS1 server.

Configuring our new development environment
We kept monitoring the game servers until we started setting up our dev environment. For that we configured Githubs CI/CD tooling, and as we were ready to test it, suddenly the whole thing was unoperative. Luckily many other companies/developers reported the same issue so that Github was pressured to fix it.

Four hours later, our CD pipelines (deployments) started deploying code.
On Nov 13th at 2 am, our developers were able to use our new development environment, thus pushing updates.
By 1 pm, Doctor implemented a proper git configuration (.gitignore, Code permissions & Co.) and added his gamemode linter to our automated workflows.

At that time the linter reported over 8200 warnings...not bad actually.
This helps us reduce error prone code in the future.

Commit Dispatcher & DB Processor
Now it was time to somehow get our commit messages into the changelogs database, on which our changelogs service relies on. To keep things organized and in one place, I decided to re-write my old changelogger NodeJS application, this time using a scripting language called Bash-Scripting.

Meanwhile Temar enrolled for some lessons in linting.

Later I realized how using database-insertions in Bash Scripting was not safe enough, so I had to choose another route. I decided to write a PHP CLI (Command-Line-Interface) Script, and split the application into parts -
the Commit Dispatcher and DBI (Database Insertion Processor).

Forums -> Github issues integration

At the same time Doctor was working on migrating our gamemode issues boards to Github. To make things even prettier, he proposed writing an integration that would link our forum bug-reports directly to our Github issue boards, making it easier for developers to work on fixing bugs. He started working on a bbcode-markdown-converter to convert forum tags (bold, italic, etc) to Markdown format.

And one hour later he presented first results:

Bambo lives in the future
Bambo and Temar were one of the first devs to publish an update using our new dev environment. It was not long before Night noticed a bug in the changelogs viewer/in-game loading screen:

With git we started using a more accurate time tracking that includes timezones.
Our old SVN timestamps were one hour behind to accurately match the timezone settings of our DS1 server. With Doctors addition to support timezones in the changelogs-viewer, we also went with using the deployments execution timestamp + 3600 seconds (1 hour) to get the most accurate time for the changelogs.

Upgrading the VPS1 server

Now that the DS1 upgrade was completed, we focused on getting it done on VPS1 - our web-server.

On Nov 15th around 1 pm, we created some last-minute backups and started the procedure. The setup was done 3 hours later, and this is where the journey starts.
Switching to the latest PHP 7 version was much harder than expected.
Even though we had been using using our SCM compatibility checker for months, many services did not successfully pass the switch.
It took us 7 hours (11 pm) to manually search and fix these services.
By Nov 16th at 2 am we completed upgrading/testing our forums and started backporting our ACP (Admin Control Panel) and Gamepanel to support our latest PHP version.

By 4:30 am in the morning the nightmare fun was over and we sent the entire web-server back online.

During the day, Doctor completed his Forums -> Github integration while also fixing the Limelight API.

Since then we....

• we fixed some broken deployments
• improved our development environment
• added documentation scripts to our CI/CD workflows
• brought back our in-game update announcer on Dec 5th
• published the bbcode-markdown-converter as an open source project
• published the DB processor as an open source project
• and more....

Special thanks to all involved and you for sticking with us!

Stay tuned on development regarding our services:

Our Open-Source Projects: https://limelightgaming.info/
Our Github Organization:  https://github.com/limelight-development

CityRP2 Updates:
Changelogs: https://limelightgaming.net/services/changelogs/
Discord Channel:   https://discord.gg/t9NAJBW

Web Updates:
Discord Channel:   https://discord.gg/6XdkFGv

API Updates:
Discord Channel:   https://discord.gg/4vxZhdg

Twitter: https://twitter.com/L2_Gaming
Youtube: https://www.youtube.com/channel/UCPwbrL76xojKB59fau6O-vw

Step 1: Suggestions

The first step is that people post their suggestions, in the suggestions forum. Surprising, I know.

These stick around until:
A. They have been open for about a week. AND
B. They have had like 10 votes.

If they have enough support (50% I think), it gets moved into the "Staff Review" section, if it doesn't, it gets denied.

Step 2: Waiting for Staff Review

Here suggestions sit until staff get around to posting review threads.

From the review manager, we can send polls to development, moderation, administration and business. We need to fill in a quick description for the suggestion, and any development advice we have.

This is also the place where we check for if a suggestion is feasible and possible. If it is, a staff review is posted internally, if not, it's denied.

Step 3: Staff Review

Every votes. And leaves reasons. Except when they don't :<

The default length of these polls are three days, but can be extended.

If it passes, the internal thread is moved for development, and the external thread is moved to "approved. If it doesn't pass, the reasons are posted publicly and the thread is moved to denied. The internal thread is then moved to the archive.

Step 4: Additional Planning

Here, suggestions are mixed with ongoing plans (MESKA, new skills, optimisation etc) to form full plans. This was done by the content team, but is generally handled by development. This can be simply adding suggestions together to make larger updates, or more in-depth planning through working groups.

In this step, new updates may make old suggestions irrelevant, and so suggestions are sometimes denied months after because they're no longer relevant.

Step 5: Development

Then, developers are matched to updates. Some developers work on local servers (such as Burnett). Others work on beta directly (myself). The developers work to build the update, sometimes referring back to update plans or working groups, speaking to the community input, developers or administrators to get input into the new plans.

Now, why does this take so long to do? We don't have many developers, some developers are better at particular things. Some prefer other things. Some are looking to expand skill sets. Some need more support. Overall, we get more suggestions than we can implement, so the backlog increases.

Step 6: Debugging

Debugging takes several phases. Stage 1 is merging everything onto beta. Beta has several differences to live. Seperate databases, often has newer code being tested, but generally if code works on beta, it should work on live.

Stage 2 comes after that, where code is merged live and tested again on the staging server. This is a copy of live, same DB, same code, it even logs to the same files. It's as close as being live as we can get, without rebooting the live server.

Stage 3 is when everything is live. We can't catch everything, so we rely on bug reports from the community, you catch what we miss, and do things we don't think of.

Secret Step 7: Balance

While we strive to keep things as balanced as we can on release, we might get things too good or too bad, so we change values. Contraband gets buffed and nerfed. Timers changed from 5 minutes per cycle to 1 minute and is now on 2 minutes. Wood values have been changed maybe 6 times since they were made sellable. Weapon damage and fire rates are changed to improve the meta and make each weapon viable. This can also be prompted through suggestions. Sometimes data says stuff is balanced, but the community thinks otherwise, so it can be easier to just rebalance stuff than to explain the data.

So hopefully that explains how we do suggestions, why they can take so long and how they're chosen to be worked on.

The future, it's a scary, it's an unknown; we need to make sure we're ready for it. So, we make plans, roadmaps and try to figure out what we're doing, both for the short and the long term.

So, what are the issues that we face in development which we can plan to resolve. After all, any issue we face can be broken down into smaller ones, right?

#1 - Content

Content keeps people engaged. People like doing stuff. Whether that's replacing things with improved things, or making new things, new content brings people back and keeps people here. For this, we're looking to do a few things.

#1.1 - MESKA

MESKA, if you haven't seen it, is our metrics, skills and achievements engine. It's designed as a tool for longer term player progression, making it so "money" isn't the only endgame for players. Developing skills takes time, it can't just be crammed into a weekend.

For development though, we're planning to build new skills to link in, developing the skills that are already there. This can be tying old systems into MESKA (as with Fishing) or developing new systems which link into MESKA (as with Faustie's upcoming work).

#1.2 - Events

To help both staff and players with their events, we're working on improving our event tools. Pre-made sets of content, easier modification of game rules / jobs / whatever else, easier management of maps, sets of tools to make it easier to run events.

All of this links into just "making things easier" for players running events, staff hosting events and developers who make sure everything's in place for these events.

#2 - Improving Technology

One of the things we've tried to stay at the forefront of is keeping our systems and security up to date.

#2.1 - Forums

One of our plans is to continue ensuring the forums remain secure, up to date and quick. This can be through updating plugins to improve speed, writing new plugins to keep users secure and more.

#2.2 - The full stack.

With the infrastructure upgrade comes upgrades to many core sets of software and services. This lets us use new features, improve code readability and more.

#3 - Paying Back Debt

As I said back in Dev Blog #7, we don't want to wait for the bailiffs to call. Removing old functions, rewriting old code, improving the code base and optimising where we can are key improvements which need to be made. It makes it faster to develop, prevents rewriting stuff we already have, and much more to let us focus on getting stuff done, not working around arcane systems to get it done.

What is the procedure?

Since the community's inception in 2015, we've always relied on the same system that we use today, this being how administrators are able to "recommend" a player for staff, then other administrators also vote as to whether they find the candidate suitable for the position or not. If the vote is successful, then great! A mentor is assigned to that person and they're promoted to trial mod. If not, feedback may be shared at HR's discretion with the candidate and hopefully they'll stand a better chance when they're next raised as a potential candidate.

Why do we use this system?

Seeing how administrators have to interact with our players on a pretty much daily basis (across forums, Discord and in-game), they should know our players quite well. Of course, we can't expect everyone on our admin team to know every player that joins our community - after all, over 46 thousand people have joined our servers since they've went live, that's a lot of people! Where an administrator is unable to form an opinion, they're able to "abstain" from voting whilst they try to gain an opinion, if they're unable to do this, they may be asked by HR to choose either to vote in support of the candidate, or vote against the candidate dependent on how tight the vote is.

Why do we continue to use this system?

This system has presented very few, if any issues for us over the years, and well, there's no need to fix what isn't broken. This system has been used to onboard/progress all current staff on the team (amongst either LL or pre-split) and has reliably worked for us since we've implemented it.

Would we consider allowing staff applications?

This is a difficult question to answer. There's many advantages and disadvantages to using a system like that, so let's quickly cover some.

Advantages:

• Those who would like the position can easily make themselves known, instead of hoping that an administrator picks up on them.
• It could possibly allow us to onboard more staff.
• Less paperwork for the admin team.

Disadvantages:

• It's very easy to make yourself look good on paper - people may not be as genuine as they make out.
• It may lead to a decrease in staff quality, as the bar for entering the staff team could be lowered due to this.
• A candidate may be able to persuade others to vote in their favour, when they should be solely relying on the opinions of others rather than trying to sway it in their favour. Keeping the process blind to the candidate would overcome this.

And those are just a few off the top of my head. Would we be willing to consider it? Maybe, who knows? Personally I stand somewhere in the middle. I see the advantages to it, but the disadvantages put me off. If the admin team wishes to change this, of course this will be discussed then but until that time comes, we believe we have a fair system that works well for all parties involved.

To conclude...

Do we want more staff? Yes, definitely. We're always on the lookout for new members of staff that might be able to join us and make a difference, but we would expect these candidates to prove themselves to us before a vote is made. If you'd like staff, chuck our tags on, try to be helpful to others, keep your record in check (after all, how can you be expected to enforce the rules if you break them yourself?) and keep a cool head, if you're lucky then you might just find yourself on top of a roof being offered the position!

We'd love to welcome more people onto the staff team, but we (or so I'd like to think!) have a high expectation of our staff and the candidates that we consider for staff. Whilst this blog post should hopefully help to clear up some questions and concerns, you can find a bit more detail on our staff process in the thread I posted a while back. If you've got any questions, you're more than welcome to ask either myself or Overlewd and we'd be happy to try and answer them.

Until next time!

I've been wondering about what to write, because for the most part my position includes a lot of work that I'm either not able to talk about at all, or stuff that I'm not able to talk about too much but I figured I can talk about a change that we made a little while ago in how our moderation team works - mentorship and trial periods. Hopefully by the end of this blog post, you'll learn why we implemented these two systems, what the key problems were and learn a thing or two about how progression is handled internally amongst the team. Here goes! Let's start off with...

Identifying the problem.

Back before the change, we were severely restricted in our ability to take in new members of staff because there was a lot of risk attached. The way that the "stage one" role was designed gave a lot of powers, which made it difficult for us to hand out to an array of people and if there was any hint of risk being attached to that person, it would most likely result in us denying that candidate. We risked potentially exposing sensitive staff tools and internal information to someone who may have ended up being a poor fit for the role. This meant that we needed to be extremely careful when it came to voting a candidate into moderator, because we needed to be fairly certain as to how they'd behave on trial. This could lead to significantly long periods of time where promotion threads weren't touched or a poor voting outcome for the candidate. All of this whilst our server desperately needed more moderators to help manage the player count when other members of staff weren't able to. We needed to think of a solution, and fast.

The Solution.

And so became the trial moderator system we have today! The idea of the trial moderator system was to allow us to take more of a risk with those we let on to the team whilst still maintaining a high quality of staffing that you see on the server. Trial moderators do not have much access internally and have limited access to the more advanced tools, but are still able to handle a lot of the common problems that happen on the server. Trial moderators are quite limited in the amount of time they're able to punish someone for, however are able to ask for extensions first or feedback into cases if they're ever unsure about a case. This is where the mentorship system ties in, as it greatly helps our trial moderators with understanding how to handle situations where they may not be entirely sure on how to do so.

The mentorship system is our way of ensuring that trial moderators didn't feel like they were just abandoned and left on their own to fight off whatever may come their way. When a promotion is approved amongst the admin team, we also decide who is going to work alongside the trial moderator and make sure that they feel comfortable and are assisted to do the best that they can and (hopefully) progress further up the chain! A mentor is someone who a trial moderator could turn to if they have questions of any sort, and they would also act as a way for administrators to communicate their feedback (both positive and negative) to the trial moderator so that they're able to improve. After all, how can you be expected to improve if you aren't told what you're doing well and what needs a bit of work? We highlighted this recently amongst the entire team with the wave of staff feedback that we gathered, which we aim to conduct on a regular basis.

The introduction of both these systems really helped us introduce a lot more staff members to the team and we hope that it continues to do so. I personally feel like the addition of both of these systems has really come to help us in regards to staffing the server, and after seeing (and even myself, having mentored a few of the staff members on the team today) the effects of the mentorship and trial moderator programs alike, I really believe that it's a success and I hope our player base sees it that way too.

Conclusion

I hope this blog post has provided a decent insight into our progression system and how it's handled, with some luck you'll now see how things were from our perspective and why the change needed to be made. As always though, if you've got further questions then you're welcome to ask myself or Overlewd and we'll do our best.

Till next time!

What does Wikipedia say?

Technical debt (also known as design debt^[1] or code debt) is a concept in software development that reflects the implied cost of additional rework caused by choosing an easy (limited) solution now instead of using a better approach that would take longer.^[2]

Wikipedia, licensed under CC Attribution Share Alike.

That doesn't really help much, does it?

In simple terms.

Technical Debt is just like real debt. You borrow from the future, to the benefit of now. Whether that be building systems which "just work", without considering how that can be expanded in the future, or not documenting complex systems. It works now, but takes more time and effort to fix later.

Is this a bad thing?

In the short term? No. We all write code first, get it working, then make changes, optimise, write documentation afterwards. So long as it stays limited, and gets fixed quickly, before the system goes live? That's just how development works.

So, how does this effect Limelight?

Well, it takes a few forms, affecting different developers differently.

Documentation

This is a big, big issue, affecting both gamemode and web developers. People have been annoyed by my documentation updates, but they're needed.

Documentation, to keep it simple, is written words, used to pass information between developers. Most documentation is written as a "func-doc", or function document. It's a block of text explaining the function, what it does, its arguments and returns.  

This is an example from L2 Plates. Not the best example, but it gives a broad overview of what the function does. It draws a license plate on a vehicle. Each argument is explained. Each argument type is noted. At a glance, you know how to use the function.

But, there's another layer, the function and its arguments are self-documenting. The function name is readable and describes what it does (it draws a plate). You can guess what each argument is. The entity being drawn on, the plate data, the license plate text, the plate title, the material and colour being used, and a set of trace data. It's easy to understand and quick to read.

Let's find a bad example.

Here, the name of the function (player has free laptop) doesn't match what the comment says it does (enforcing a player's laptop REP restriction), it takes me longer, as I have to check the code to see which is right.

This sort of thing makes it slower to develop, and easier to get things wrong.

Testing

Not so much a problem in CityRP, but more with web development. Anyone who's paid attention to my discord's open source library channel has noted a lot of commits relating to testing. This is automated unit testing, to make sure the libraries works as intended, and that changes I make don't break stuff.

For example in the PHP version of InterDB, there's currently 5 test sets, running 21 tests with 31 checks, and that only covers 54% of the code.

Compare that to Limelight services, where, as of writing, none of our code is tested automatically. This means that changes can accidently break stuff, and if nobody notices, it might get pushed live.

Yikes.

Old Code

While old code isn't bad in itself, it's often a product of the time it was written. Code can be slower, use features that have been / are being removed, or just extremely unclean. Again, as I keep saying, it makes development today slower, in exchange for getting it working in the past.

What are we doing?

Quite a bit. New Style Guides, requirements to document code, testing requirements. Training developers to recognise poor practices, and rectify them.

As you've probably noticed, there's a few staff on the team who aren't really around as much. This isn't necessarily their own fault, as this can be because they might be dealing with personal issues, burn out, college, school work or they might even be on holiday. Up until now, we've been quite generous with members of staff and their activity, but it's time to change. We want to have a staff team that actively engages with our players, that knows what happens on the servers as well as what happens within the community. If you're only visiting the forums once a week for 5 minutes, that doesn't always tell you what's going on, and with administrators especially who are expected to vote on topics that directly influence the shape of the server, it's difficult to vote without being around to know what shape the server is in to start with.

Before this policy change came into effect, we only had one requirement to activity. As long as you met the quota of hours per month that we laid out, you would be fine however we didn't have many ways to enforce this properly and as a result, a lot of staff members slipped under the radar when they went inactive, so eventually it just became a waiting game for that member of staff to become too inactive that they were eventually demoted after warnings, and whilst that wasn't a horribly bad system, it definitely needed work, so when I joined the SA team, amongst advice from other administrators telling me that this was a problem that needed a solution, I discussed this with the other SA-HR, @Overlewd, and after a while of planning it out, we made the policies we have today!

But Night, what are these policies?!

Good question! These policies differ for administrators and moderators. We'll start with the policy for moderators since that's the easiest one there is.

Following the three strike system, a moderator should aim to get 20 hours per month on the server. Failing this, a strike will be issued. If three strikes are issued, then the moderator will be immediately demoted to veteran (assuming they're senior moderator). A strike can expire assuming the moderator is able to meet activity requirements for 2 consecutive months.

If a moderator warns us ahead of time that they aren't able to meet the activity requirement, assuming that the reason is fair and isn't repetitive, the strike for that month will not be issued however this won't count as a consecutive month towards expiring a strike.

For administrators, the policy is the same wording as above, except with their additional duties kept in mind as well:

An administrator should aim to achieve 60% of all monthly staff review polls voted on. This follows the same three strike rule, however instead of immediate demotion to veteran, the administrator would be demoted to senior moderator. These strikes are separate from activity strikes, and expire in the same fashion as well. . Exceptions to this policy can be made by HR, however since it's not difficult to vote on staff review polls, this will be kept in mind when determining whether a reason is valid or not to count as an exception. Any member of staff demoted by this policy is able to reapply to their position after a cooldown. If they're successful in reapplying to the role, they'll return with 2 strikes (either activity or staff review, whichever they were demoted for) to prevent the situation from repeating itself.

But Night, what about the administrators who have other positions in the community?

If an administrator is spending a lot of time outside of the game in their role, this would naturally be taken into account when deciding whether they've done enough work in that role to compensate for the missing hours. For example, if I've spent 4 hours or so organising events, doing graphical design and writing threads, in order for me not to receive a strike for that month, I would need 16 hours in-game.

What about the superadmins and even the owner?

Seeing how our responsibilities, for the most part, lie outside of the game, we're not included in the scope of either policy. We'll aim to meet the requirements where we can but we can't gain strikes through this system.

Are developers included?

Yes and no. Developers who are only developers aren't included in the scope of this as it would be down to Dev Management for them to handle, but Developers who also hold either moderator/administrator privileges are included in that yes, but would be placed to developer only.

How do HR keep track of this? Surely it must be difficult...

We already post end of month recaps to both the moderators and the administrators, with a breakdown of which staff member has done the most player reports, hours in-game, posts on the forum and all sorts. Using these statistics (as well as a wonderful little table made by @Doctor Internet), we're able to keep track of staff members who don't meet the requirement. Look how nice it is!

What if someone's really struggling? Can HR help them at all?

Of course we can! We have complete discretion over the two policies. It's in our interest to work with the staff member rather than trying to shove them out because staff members can be difficult to replace, especially as you end up higher in the team. We can grant exceptions to people if the reason is good enough, although this exception will almost always be an exception for the current month rather than on a continuous basis unless there's extenuating circumstances.

Is it just superadmins and the owner who are immune to this policy?

No. There's currently two other people who are immune to this policy, @Burnett and @Doctor Internet because these two are critical to Limelight both as a business and a community. Burnett helps to maintain our security infrastructure and ensure that services are secure, as a result he needs the access to look after them and that means he's best placed as an administrator. Doctor Internet, as you're likely aware by his development blog posts and his dedication to his role in developer management, helps coordinate our development efforts alongside @Faustie, as well as helping to maintain critical Limelight services and infrastructure as well. Whilst they're immune from this policy, they're both expected to try and achieve the requirements where they can but they won't accumulate strikes either.

What do HR aim to achieve by putting these policies in place? Is it not just words?

You're right. It is just words for now, but these words give us strength to address the issue of activity much better than how it was before when this policy didn't exist. You'll see these words slowly turn into actions in a few months time, where we're then given the means to address the staff members who aren't meeting what's expected of them. This should in turn help to increase the staff presence on both the server and the forum, engaging with our community and being aware of what's happening.

Conclusion

So I hope that's provided a nice insight into how our new activity policies work. I guess only time will tell whether this move was good or bad, but we'll cross that bridge when we come to it. Here's to the first HR blog of (hopefully) many. As always, if you've got any questions, feel free to shoot Overlewd or myself a message and we'll try our best to answer them. Peace out, take care all.

What are Item IDs?

It's simple, item IDs (technically know as "unique IDs") are a way for the gamemode to say which item is which.
Why not just use names? Names can overlap (Think how both the Undercover and Civilian Dodge Charger have the same name), names can have spaces, names can be super long and hard to type.
Item IDs, on the other hand, are fairly short, simple, have no spaces, and are to the point.

For example, Closed Suit 'Charles Bishop' with Overcoat? That's a long name. It's ID? trenchcoat_08. Nice and simple.
It's used by the inventory command, the item use functions, the F1 menu. It's how the game talks about items.
That's why generated commands from the keybind generator use those IDs.

So, now we know what they are and why they're important, what are the issues?

Well, there's one big issue. They're really hard to change.
You make a mistake, and set a Burger to have the Unique ID "Bruger"? Unless you catch it before it goes live, you've not got a chance.
The moment someone buys a burger and has it in their inventory, it's saved in the database, you can't just rename it.

You could issue refunds, but that means turning the servers off, and going through each person's items.
You could make a new item, and quietly turn off the old one, but then people lose stuff.
Or, you could make a new item, and set the old one to be replaced by the new one. When people join, the game quietly turns the old item into the new one. But people don't always join, so that old ID is still there.

And, if you want a new item, that ID can't be used, as people still have it. It's difficult.

Combine this with the fact that some functions are case insensitive (an issue that caused the market to not show a type of ammo).
Well, it's all pretty yikes. Lots of small issues that combine into bigger ones.

Corrections

So, how do we do corrections? Glad you asked.

It's pretty easy, let's grab the code that defines a hat.

BASE:Clone()
    :SetName("Grey Cs Baseball Hat")
    :SetSkin(2, true)
    :Register("baseballhat2_3")
ITEM:Replacement("baseballhat2_3", "baseballhat07_3")

BASE is the baseball cap item, which has the stuff that's the same. The model, the description, the price.
ITEM is a table which has all our functions in it.

So, part 1, we make the real item. baseballhat2_3 is a real item that works.
Then, ITEM:Replacement() clones that item, making an identical copy (same price, same description).
It marks it as replaced by the real item, makes it unbuyable and makes it so using it gives you the real item (back in the day, it wouldn't auto-replace, so that's so people could get the new item))
Finally, it takes that copy, and says to the game "this is the old item".

Two items, one old, one new, and when people log in, it replaces.
Pretty cool.

But, that only works when people connect. If people have old items and don't join for a while, they keep them. We'll solve that later.

Replacements

Honestly, this is the same as corrections, but we use a different item. (Also the example uses older item definitions, but it does the same thing.)

local ITEM = {}
ITEM.name = "Toyota Supra"
ITEM.cost = 190000
ITEM.model = "models/supra.mdl"
ITEM.plural = "Toyota Supra"
ITEM.uniqueID = "supra"
ITEM.description = "A tuned car with a fast nitro gas system. This item is 0 big"
ITEM.vehiclelist = "Supra"
ITEM.category = "Vehicles"
ITEM.replaced = true
ITEM.replacedid = "corvette_c7_stingray_2014"

cityrp.item.register(ITEM, "base_vehicle")

Refunds

Uh, refunds. Well.
Three ways.

Set item.outdated (I think it's outdated anyway), then that adds the sell button for a full refund.
We take the server down, find anyone with the item, and give them money.
Or, we create a "refund cheque" item, that's worth however much, and use the replacement technique to give everyone cheques (useful if there's multiple items).

#1 and #3 have the same issues, they're manual.
#2 can take a long time. But, we have a tool for it.

The Mass Refund Service.

In a secure section of the website, is the mass refund service.
It runs a script in PHP, gets every player, their inventory, scans through, checks against a list, and issues refunds.
But, it's running on a website, with limited resources, and makes a huge webpage. It's slow, and last time I used it, it crashed my Chrome.
Two hours of downtime, and no way of checking what actually happened? That's not good enough.

The Mass Refund Tool.

So, instead of that, I wrote another PHP tool.
A CLI tool, thats quick and versatile, and does a bunch more stuff than just refunds.

You want a list of all items on beta, where less than 30 are owned, but have a total value of more than $100k?

Boom, there it is.

Now, when I say quick, how quick?
To give you an idea, the service took 2 hours for the last set of DB changes we ran. The tool did it in 10 minutes.

But, more than that, the service was very simple. It took a list of IDs and a money amount, and would take any of those IDs and give money back.
It worked, but only really for refunds.

What can the new one do? It can, refund items to money, convert items to other items, convert items to multiple items, or do combinations of all of that.
People had an old car, and you want to give them a new model? Easy.

{
    "ferrari": "spider",
    "mclaren_p1": "mclaren_2014_p1",
    "mustang": "1970_ford_mustang_boss_429",
    "volvo": "volvos60"
}

Want to refund anyone who had charged taser cell schematics?

{
    "sch_charged_cell": 30000
}

You want to give anyone who had a GMC Savana News Van 5k and a regular GMC Savana? Not a problem.

{
    "gmcsavana_news": {
        "money": 5000,
        "gmcsavana": 1
    }
}

Now, what else is nice. Imagine you wanted to refund everyone who had a GMC Savana.

You could run both the refunds for the News van first, then a refund for the Savana, but the tool handles that for you.

{
    "gmcsavana_news": {
        "money": 5000,
        "gmcsavana": 1
    },
    "gmcsavana": 60000
}

Anyone who had a Savana gets 60k, anyone who had a Savana News gets 65k.

So, what did we do this morning?

Well, I went over the list of replaced items, more than 100 of them, and set the refund tool to process them.
And it took 5 minutes.

Specifically, we:

• Fixed some non-existent beanies.
• Removed accessories which never existed.
• Replaced the BMW GTR with the Nissan GTR.
• Replaced the old Ferrari, Volvo, P1, Kamaz, Mustang, Police Charger, Tow Truck with their current IDs.
• Replaced all the old fire engines with the current one.
• Fixed the old GMC Savanah.
• Replaced the old GMC Savanah News with the GMC Savana & a 5k refund (as the News van was more expensive).
• Replaced the old Secret Service Suburban and Fire Suburban.
• Replaced the 3 old sign IDs with the current one.
• Removed a bunch of beta items.
• Replaced the HL2 axe with the woodcutters axe.
• Merged Quad Mag / BetaC Mods, both are replaced by the extended mag kit.
• Fixed there being like 3 versions of each ammotype.
• Replaced all goldfish with salmon, all fish with rotten food.
• Fixed the Burger unique ID.
• Replaced the old fishing rod with the new one.
• Changed the turkey dinner unique ID (as it was still christmas dinner).
• Old Gov/Police caps.
• Some really really old suits.
• Processed fully the response clothing, dodge monaco police, pepper spray, package, christmas cracker, 5.45x39mm and charged cell refunds.

Then, deleted all the replacements / old items.
Saving DB space, as we don't have items in inventories that can't be used.
Saving game code space, as we don't have items there.
Freeing up new IDs if we ever need them.
And finally, letting people use items that were stored in their inventory.

So finally.

Hopefully this has been insightful, if a bit rambly, on item IDs, our refund tools, item files and more.
I hope y'all have a good day.

Part 1 - Graphs

Graphs are nice and easy. Nibble sized chunks of data, for getting small overviews of large data sets.

So let's break this down nicely.

Numero uno. The roadmap. It's good, most people know about it.
Dos. REP Rewards. Less people knew, but still the vast majority.
Tres. Dev Blogs. Again, less people knew, but still a majority.
Four. Changelogs. People know. Not many read. :(

Now, the big hitter. Communication. Obviously, we need to do better here, and I can see why. It can seem like the only way to talk to dev is through suggestions. Whilst we are bound to suggestions, both myself and Faustie are willing to talk to people, take in viewpoints and explain why certain choices were made. If you have an idea about how to open up new channels of communication, please let us know.

Part 2 - Long Form

As well as graphs, we had some longer form questions, so it's time to look through some of the responses there.

2a - Are you satisfied with development blog content?

The developers can sometimes seem lazy, only releasing (to us) useless document updates in the changelogs, or meaningless blabbers of code. Limelight needs real updates and for the developers to be working on the game itself rather than making everything look pretty.

Whilst this isn't anything to do with dev blog content, it's interesting. Let's look into it a bit more.

As I've said before, the changelogs are there to record the work we do, and let the community know what we're doing. The added documentation is there to pay back some of the technical debt from over 10 years of amateur coding, repeated over-patching and lack of any standards. If that's what I'm doing, then that's what gets logged.

That being said, I can see why it would be annoying, seeing documentation updates instead of anything you can physically do. You can check the roadmap, see what's also being worked on (as it's not always worked on in public). We don't want to totally hide everything we do.

Not really, seems like another place LL can promise updates but never deliver, reminds me of Theresa May. What happened to the Evobank? - BREXIT

Nice meme.

Everything we've released a dev blog on has been released. I see no undelivered promises there?
And where's EvoBank? The developer who was working on it resigned. We can't really control resignations.

No, Its massive lack of updates even though the developer team is big. the suggestions being approved should be a priority for the develepors.

Again, not about dev blogs...
But still, we are working on approved suggestions. Nearly 37 pages of them.

No, lack of updates. That big development team but only 1 guy doing the work. What a shame

Again, this isn't to do with the dev blogs. Seems that people misunderstood the question, which is a shame.
To answer it anyway, lots of people code and do work. @Bambo works his ass off with the forums. Other people work behind the scenes, on larger projects. I tend to work on smaller ones. Different people doing different things is pretty fair, I think.

Fuck u

ur house or mine bb?

2b - Further Questions/Comments

More devblogs would be great.

Pushing out devblogs is hard. They have to be written, not just another place to "promise updates but never deliver", but also be on topics people are interested to learn about.

instead of making an useless Rep Tree and thins like that better improve the server or add atleast useful content on the Server which actually changes the Gameplay but better just Improve it so you have no ping lag and atleast 30-40FPS constantly, don't say "Oh no, that's impossible" Many other and BIGGER servers with more/same amount of content somehow can do it.

Woah Cowboy, hold up the rodeo.

Firstly, the REP tree was made by a pure web developer. Someone who doesn't code in lua, and doesn't have the access to make lua changes. The alternative is them sitting there, with their thumbs up their butt.

Secondly, ping lag was basically nullified at higher player-counts by our networking module, and we're working on optimisation through our optimisation project. Take a look at dev blog 2 for more info about that.

Why does it take so long to implement such a small suggestion that has been out there since 2016, like adding a few props or 1 swep.

It can be that it's just a low priority. For the vast majority of suggestions, this is the reality.
Some require changes to content, which take days to do.
Some don't fall within current planning.
Sometimes, we just miss them. There are a lot of suggestions.

There are many reasons, and those are just some of them.

No, instead its a statement. The Development Team work hard, and there hard work is greatly appreciated by the community. And I am 99% sure everyone else will agree with this, although there may not be a "public update" released we all know you are working on making the community better. Keep up the good work.

Thank you.

When a suggestion goes into staff review, would it be possible we see developer's thoughts about the suggestion?

It depends. If it gets denied, you see them anyway. If it gets approved, it doesn't really matter.

Over the past year it may seem like we have lost more features then we gained. Cruical elements like Atmos or Vcmod (understandibly) got removed. I do understand that there was pretty much no choice in doing so. But therefore we also lost imortant elements of rp imersion. Back in our early days Limelights selling point (so to speak) was its variety of imersive ingame content [atmos, photon, those "new" tdm cars, evo city radio etc. Its more then understandable that some of those things have been taken out but i am hoping there is some way to replace them in the nearer feature. Since we cant pump out cars endlessly other content will have to catch peoples attention. Therefore the ehnaced crafting system as well as the skill system will be a cruical step forward. Putting out a roadmap is also a well thought out decision as this way our people can get excited for whats to come.
What i personaly would love to see (also i dont deem it to be too nescessary), would be more instruments, a return of the rt screen, a cooking system (as a part of the crafting and skills system) and perhaps some more immersive misc content as this never stopped to amaze people. But i understand that we are pretty much going that direction already.

For sure, I agree with a lot of this.

That being said, it's a difficult balancing game, new content vs improving connectivity between content vs fixing issues in old content vs updating old content.
Really heavy systems that people want, ie Photon, VCMod, Stormfox vs Optimisation.
New flashy systems, vs having detailed systems with solid interconnections between them.

Doctor Internet Head of anything and everything fucking demote the twat

I think I've seen this before somewhere.
Though, let's be fair. I only run the contributor program, so I'm not really the "head of anything and everything". I report to Faustie for my contributor/development duties, to HR for my administration and to Enzyme/Faustie for Business and Data Protection. I don't run everything myself.
Nice that you think so though <3

From the outside LL appears as though it has very little updates, the little updates that are released are made to be Huge updates when really all that's been done is added a few Cars....

I hope this year has made a different for the updates released. We've done a lot of work, not just releasing a few cars.

Doctor seems to be the only person whom has been truly active in development, while the voting system when down the pot. I only remember seeing two votes being voted upon, I believe its a good way of finding out the opinion of the whole community, including those whom don't use the forums.

For myself being the only active one, that's not really true, plenty of developers are active, even if less so than myself. Some are active in other areas, such as Bambo, Burnett and Faustie.

For the voting system, it's a pain to set up, so is generally only used for larger votes. We could use it more, but it's a pain for us to use.

Get a tutorial of every updates. (ex. Meth cooking since we don't have a gravity gun)

Teachers and the guides sections.

Pollux > Doctor Internet at Development <3

gonna drop a big fat :doubt: on that one.

It would have been nice if Dr. Internet did not remove me from his friends list after I have reported many bugs and did my best to help the development in giving feedback and ideas.

I'm going to brutally honest. Anyone who sends me bug reports via steam or discord gets told the same thing. Post a bug report. If you kept doing that, that's most likely why you got removed.
If not, just drop me another invite. I clean out my steam friends every few months for people I don't talk to, I probs deleted you then.

Aside from that, I'm always up for more feedback and discussion. It's good for everyone.

The theme in-game (health bar etc) should be changed back to normal

What is "to normal"?

Yes, what happend to the GTA 4 and GTAv limelight props packs? Can gamers ever see those prop packs come back?

They're still there, in our content packs?

In the end, I hope this has answered some questions, and given some more insight into what we do, and why we do it.

I was bored and checking various automated systems, and comparing them against stored logs. The system had flagged up some suspect accounts, where there was a difference between the items spawned with / picked up / bought, and the items that they had in their inventory on the next spawn. Looking into this further, I saw masses of weapon drops, a single weapon being used, followed by an inventory manager transaction, a weapon /drop, and the weapons being picked back up. So myself and Nightmare hopped on staging, started testing, and I started looking through the inventory manager, item drop and weapon drop codes.

How did the exploit work?

The exploit worked by using the interaction between two unrelated systems, the inventory manager and the /drop command. When dropping weapons from the inventory, if you drop all the weapons in your inventory, you automatically de-equip the weapon you're holding too. The inventory manager doesn't do that, however. This meant you could have no weapons in your inventory, but still have one equipped in your hand. /drop didn't care about that, it would always create a weapon item anyway. So, as long as an exploiter had a friend they could get their other weapon back from, they could repeatedly do this (albeit only with single weapons, and each round could take up to 60 seconds).

However, the inventory manager was only a single method of being able to exploit this. Administrators could have used it by spawning weapons from the Q menu (though I found no evidence of this), or if players have had items removed with the item take command, this condition could also have been present.

Has it been fixed?

No, I'm telling everyone how to do it because I specifically didn't patch it. /s
Yeah, it's been patched. And the fix was Q/A'd. I'm shocked.

How was it fixed?

/drop was patched. If the player doesn't have any of the selected weapons in their inventory, /drop acts as /holster.

What have we learnt?

Well,
1. Old code doesn't always do what you expect it to do when you integrate it with new systems.
2. We should always check for edge cases, even when other systems should prevent those edge cases.