Case study
Rockstar Games, FiveM and RedM
From putting out fires to a solid platform · Lead / Senior Software Engineer
FiveM and RedM let players run their own multiplayer servers for GTA V and Red Dead Redemption 2. The platform is open source: a Rockstar team of four, and growing, works on it alongside contributors from the community.
Where it started
Incidents came in daily or weekly, and many could not be acted on, so they ended up somewhere on the backlog. The team shipped about one regression a week. A good build could take months to reach most players, if it reached them at all. Whole platforms, including the Linux server and RedM, had no automated checks.
How the team worked
- Design came before code. Work was designed and thought through before anyone built it. That did more than anything else to cut regressions and hotfixes.
- I planned the work and wrote the tasks, clearly enough that people could start right away.
- Guidelines and documentation: I set up code guidelines, and organized and wrote documentation on our systems and processes.
- Review and testing became a rule. Every change was reviewed and tested thoroughly, including the features next to it, and reviews spread knowledge across the team.
- The team took ownership of their areas.
How it shipped
- Deploys went out in pairs, and only when people were on hand to watch them and fix problems quickly.
- Releases went out in stages. Builds moved from latest to beta to stable, and each step was monitored weekly.
- CI covered every platform and every pull request. I moved it to GitHub's runners, added the platforms it had been missing, and ran every contribution through it, the team's and the community's alike.
- Contributions arrived ready to review. Checks made sure each pull request was well formed, followed the guidelines, and said whether it had been tested.
- Community reports became actionable. Better reports meant we could triage quickly and act on them.
What changed
- Incidents: from daily or weekly to monthly.
- Regressions: from one a week to one a month, caught in controlled environments before they reached players.
- Downtime on the latest builds: down to single-digit minutes. Players on beta and stable were not affected. (This excludes the login service, which another team owned.)
- Release time: a good build reached players within a month at most, instead of months.
- The questions changed: questions about what a task meant stopped entirely, and questions about design took their place.
- The team grew more capable than I expected, and developers began running the CI workflows on their own work, for the early feedback.
- The trade-off: reviews took longer, but they caught errors early, reduced frustration and made the team more collaborative.
What it made possible
With a stable platform underneath, the team could take on foundational work. One example is a new C# scripting runtime (Mono v2), brought in under a live platform without breaking existing servers' scripts, and later extended to RedM. Calls into the game API became 160× faster on Linux servers and 10× on Windows, events and exports 84× faster, and the overhead of entering and leaving the runtime went from 0.04–0.06 ms to zero.