What issues are the FPS team facing?

The tl;dr is that we feel the current build doesn’t feel like it lives up to the standards we’ve want to achieve with Star Citizen. There are several issues that will need additional time in order to deliver the first iteration of the gameplay we want you to experience. The challenges facing the FPS launch are a mix of technical blockers and gameplay issues. The most significant technical hurdle faced today is the networking backend. After attempting to work with the legacy code, we decided that we needed to drop some of the legacy technology. That meant developing what we’re calling a Generic Instance Manager (GIM) and rewriting both the Matchmaker and (for the larger project) the game Launcher from scratch. Those efforts are all going well, but they’ve all taken additional time for our engineers.

Going into further detail on the technical side of these systems, one of our big hurdles was, as noted, the creation of the GIM. This new system will be responsible for all the game servers that make up Star Citizen, and we’ve built it to have far more direct control over the internal state and operation of each game server than was available before. The GIM not only manages Arena Commander and Star Marine instances, but also provides a solid framework for instanced multi-player Hangars as well as the instanced Universe game servers that will form the persistent universe. The GIM allocates and recycles game servers at a much faster rate and in a more reliable way than before, helping to get player sinto the action more quickly and keeping them in their games with less incident. The development of this system, which has been ongoing for some time, has been a group effort involving engineers from around the company. Once it’s integrated, it will not only improve the Star Marine experience but also chart the ‘behind the screens’ course for Star Citizen’s future. We’re looking forward to testing it in action internally next week!

The new GIM isn’t the only ‘home grown’ system we’ve come to need for Star Marine. A second challenge has been the need to rewrite the game’s Matchmaking system from scratch, taking an entirely different approach to the process that will eliminate long waits during the Match search process. Situations that use to result in “Match Not Found” no longer exist and every player/group is guaranteed a match in a match and in shorter time than they have seen before. The Matchmaker now keeps friends together such that players in a public group will always be matched to the same team as expected. I’m happy to report that, as of this week, the new matchmaking software has been integrated and is undergoing testing.

The third process currently in progress for improving Star Citizen’s backend netcode is what we’re calling the “Phoenix” dynamic environments system. Each time the team kick off a new build of Star Citizen all the data that the servers need is automatically copied out to hard disks in Google; this is a snapshot of our game data. These disks are broken into two to three conceptual parts: Base Image (the OS plus a few other things), Logs, and Server Data (Code and Assets). When we build an environment, we mount duplicates of these disks to each Virtual Machine (VM) that we bring up. Duplicates of the snapshot are created very rapidly, around 45 seconds for 200 gigs of data.

We’ve written some automation code to automatically run commands on the VM to configure it appropriately for what type of server it will become (Game, Matchmaking, Party, etc.) During this process, a new DNS entry is assigned to server based off the version of the data uploaded. When a new build is created, and we need to push it to an environment, we trigger a command that automatically shuts down all VM’s, unmounts the duplicated disks of the Base Image and Server Data disk (Log disks are always kept for troubleshooting), and then restarts the server with the new duplicates based of the new snapshot and the environment is running and ready on the new version.

This entire process takes about 8 minutes. When we want to take a QA environment that is built this way, and extend it to become a PTU environment, we send a command to our Provisioning layer and it goes out to Google, requests more VMs, builds more disk duplicates, mounts those snapshots, runs Chef commands to configure it, adds their DNS entries, and connects them to the existing infrastructure to be used. At that point we have a PTU environment. We repeat this process to build Production. Each time we expand an environment it takes about 8 to 10 minutes depending on the type of environment and the configurations we need.

The benefit of this dynamic creation and the environment expansion is threefold. First, any changed configs, misplaced settings, or broken processes are completely removed when the VMs are rebuilt using the new disk duplicates. Any configuration changes that need to persist should be made at the Chef level. Second, we can make absolutely sure that PTU and Production is the exact same environment that QA tested on, so there will be no strange differences we failed to catch in QA when we go live. The third benefit is simply speed. It is much faster to dynamically recreate environments on the fly than to recopy data each time. Those last two points are a pretty big deal.

If our experience has taught us one thing it is that having a consistent test environment that can be rolled out quickly, and this new system is pretty quick. It’s a huge force multiplier for our ability to rapidly iterate our test versions, which means QA and ultimately our backers will be able to do more varied testing more quickly. The more accurate we can get versions to our QA and to our backers the better we can ultimately make the game.

These new systems and processes were initiated to replace very serious limitations in what had come before. We’re taking additional time to develop them properly and will in order to get them right we will ultimately need more for proper integration (testing, bug triage and the like.) But we’re already seeing a great improvement: the new system is far more reliable and handles more concurrent players due to improved networking protocol and a streamlined back-end service architecture. In short, doing it ourselves makes for a better game today and sets the stage for even bigger things to come!

On the gameplay side, we are dedicated to making sure the game represents what we want for first person action in the Star Citizen world. This is where things are a little less technical and more about the ‘feel’ of the experience. One of the biggest issues on this front is getting the visuals right. If you read our last design post on the FPS, you will remember that one of the ways we want this experience to stand apart is that we aren’t ‘faking’ animations: anything your character does in the first person needs to look correct when viewed in the third person by another player without duplicate ‘fake’ animations that look different to each person. Making this look right is something that’s taking more R&D time than we had anticipated.

It’s a challenge that we will meet…but it’s going to require careful work. We’ve tapped the new Frankfurt studio, which is staffed by Crytek veterans who know every in and out of the engine and some key ex-Crytek leads from LA and Austin to help the team in Denver make this work.

As we continue to tackle these challenges, the FPS team is continuing to improve other areas beyond the initial spec. New characters and weapons and so on, already scheduled, are being developed while newly recorded mocap animations are going in on a regular basis…and other resources are working on more subtle map changes. For instance, artists have been conducting additional lighting and detail passes on the Gold Horizon map with an eye towards quick silhouette reads and making it easier to understand where you are in the level at any time. These kinds of passes aren’t as sexy as building a new spaceship or firing a new weapon…but they’re essential to providing the kind of detail and gameplay we want out of Star Marine.

Arena Commander, for instance, “shipped” with what we thought would be a very early version of the control system, and we’ve certainly heard no end of the debate since! Like it or not, we know that with Star Marine we need to release a build that at the very least shows people where we want to go and not just what we were able to do before a clock ran out.

What do the FPS issues mean for the status of the rest of the project?

Star Citizen’s development is distributed across several different modules or sub-projects, with development happening on all of them simultaneously. By the numbers, only 15% of the team has been working on Star Marine; it’s just been the major focus because it was the next public release. This means that development of other areas, such as Squadron 42, multicrew and the persistent universe, have continued while issues with FPS have stalled development there (though even in that case, development continues in other areas: while network engineers battle back end code, weapons artists and level designers continue to work towards future FPS milestones).

I don’t want to say that there is no impact: integrating the FPS properly will help move every part of Star Citizen forward, as the tech will help form the blood and sinews of the whole game…but I can’t stress enough that two additional months spent on Star Marine are not the same thing as two months of a delay for Star Citizen. The persistent universe team in Austin is still building brilliant new worlds, the ship team in Santa Monica is coming up with great concepts and integrating existing ships in preparation for future Arena Commander updates…and of course the Squadron 42 team in the UK is full speed ahead on the single player adventure. The biggest issue we have faced is that all the recent Arena Commander work, including new flyable ships has been done on the Star Marine branch of the game’s build. We expected to have 1.2 launched and wanted to take advantage of the great new tech Star Marine’s integration provides.
 

What’s next?

To that end, we are going to investigate releasing a build with Star Marine disabled that would allow you to experience some of the changes and updates we’ve made over the last few months to the core code base. There are some technical challenges in doing this, and it won’t happen overnight…but I feel that it’s incredibly important to do because we need to test with the public, we need to collect your feedback and frankly we need to continue proving that we’re working on what you care about.

When will we see Star Marine? Tonight, I don’t have an absolute answer for you. What I will tell you is that we know exactly what we have to do, and we’re already well on our way to doing it. With allocation of additional resources and increased cross-studio focus on the FPS portion of the game we are on our way… we’re just not there quite yet. I’m confident that with the significant updates and changes to the backend architecture discussed above that we will have an experience worthy of the Star Citizen name; it’s just going to take some additional integration and testing. On the public side, I know that it’s time to open up our communications on the Star Marine rollout process: starting with this message and continuing each week, we will provide a high level update on the challenges just as we did for Arena Commander.

We ended the 2012 pledge campaign with ‘The Pledge,’ in which I outlined our new company’s goals to be open about our process. Today, I want to rededicate ourselves to this: I can’t promise you we’ll meet every internal deadline or that every decision we make is something you’ll agree with. There will be challenges that we struggle to overcome, and we will never be able to predict all of these with certainty…but I can promise you we will keep you informed and that we will not stop working until the game is done right. After all, that’s why we’re here in the first place. Your support is letting us create the game we want to make before anything else. Because of you, we have the freedom to make sure things work the way we want, even if it takes more time and more effort. We won’t let you down!

- Chris Roberts