Hello airport CEO and welcome to the 101st development blog! Last time around we celebrated a staggering 100 development blogs with pomp and circumstance, writing up a looong text about the past, current and future, had an interview with Mitchell, our artist all while blasting away several pictures of various contents and origins. All in all, it was an exhaustive piece of work which is why we skipped posting last week and instead focused on development (well, to be fair we couldn’t really keep us from updating you and made a short Forum post instead). So, in those two weeks we’ve climbed a few hills and it’s time to report on our progress. If the 100th devlog was a visually appealing and fun devlog, the 101st will be quite the opposite as we’re getting down and dirty with the nature of this devlog’s main topic, performance, which means a lot of tech talk and less everything else. Now, we’ve already gone on way too long without any real content so off we go…
Yes, yes it does, at least when you build large airports. When you build large airports with a few thousand moving agents, the CPU struggles with crunching all of the data that is thrown at it by the game engine, which in turn results in a slower processing cycle and a consequently lower frames per second. As we’ve mentioned before, Airport CEO is currently mainly criticized for its lacking UI and for its poor performance on larger airports, which is why we’ve divided those two areas into two separate sprints. We’re hearing your feedback and have decided that performance comes first, which is why we’re currently focusing on the performance sprint as we have several different ongoing tickets and are attacking the issue on several different fronts. Airport CEO is a performance critical game with a large code base that wants to crunch a lot of data each frame and we’ve had to re-evaluate several aspects of our code as both you and we have realized that it’s lacking and that we can do better. Performance is an ongoing struggle and it’s impossible for us to say exactly how well these initial improvements will perform as, although our initial tests show promising results, they are nothing without being thoroughly vetted on the experimental branch. Hopefully these weeks efforts will result in an improved FPS and, if not, serve as additional guidance to what we need to do to support larger airports.
We have been working very hard with the development, and ever so slightly started with implementing, a new path finding system which in theory should be able to calculate paths much faster than what we’ve currently got deployed in the game’s code base. We have mentioned the new path finding system in many previous devlogs and once it is implemented we promise to talk a bit more in-depth about how the system was actually implemented, and how it works, but for now we simply want to provide you with a short, but a bit more detailed than usual, update on where we’re currently at. The main foundation of the system is fully developed and it’s core principles work well, as we’ve ran a lot of tests in our (newly built) test environments but also a few basic tests in the real game worlds and player saves. While the system works well, we still have some challenges to deal with in terms of handling different zone type permissions as well as some minor path accuracy and path smoothing problems. As the new system is a lot more centered around pre-calculated paths we have in our initial tests seen great reduction in the overall time it takes to calculate a path, especially when it comes to longer and more complex paths which is exactly what we want to achieve with this change. It is hard to give a general figure as the measured time vary depending on what kind if path we are trying to calculate but we in general we have seen anywhere from only five percent faster calculations on shorter paths to an outstanding 32 times faster on longer paths, compared to the old system. However, these tests have so far mostly been in an isolated test environment so it is still not exactly clear how much better the new system will perform in a real, player built, airport save with thousands of active persons. As mentione, despite how many tests we run it is nothing compared to when you, the real airport CEO, test the system which is why we’re always hesitant on promising any more exact figures without having a new improvement or feature deployed on the experimental branch first. We will use this week to iron out the final issues and hope to deploy the new system on the experimental branch later this week or during next. This is considered a major change and it will most likely introduce new path finding related bugs, therefore we are spending extra time to test it before we deploy. The system will probably also require an additional few weeks of fine-tuning after the deployment before we can really say if it has been successful or not, as a large part of the implementation process is not only developing it but also fine-tuning it and its related calculation variables.
Another performance enhancing aspect we are working on is taking a good look at all of our data structures, meaning how we store data at run-time. As most of you would know, hot data (data that you need to access frequently) is stored in the computer’s RAM and from a code perspective it quickly becomes obvious that different data structures and collections can have different impact on performance related to accessing and interacting with that data. In short, we are currently fully invested in optimizing this aspect with the purpose of hopefully gaining a few additional frames per second. Are you interested in hearing more about the code side of this? If not, scroll down, otherwise stick around. When we started off creating the game’s architectural foundation and code base we primarily built it around a data set known as (C#) Dictionaries, essentially a key-value pair data structure for storing and retrieving data. This makes a lot of sense for Airport CEO and its underlying systems as each structure, item, person or other agent are equipped with a unique identifier. By using the unique identifier as the key in the dictionaries, we can do really fast look-ups and it also makes it easy to both add and remove objects on the run. However, the downside to using dictionaries become apparent when airport CEOs start building really huge airports as we need to iterate through increasingly larger dictionaries. Dictionaries are hash-table based which means that it is slower to iterate over than iterating over a traditional array, even including a conversion to, say for example, a list. Since our ambition is to support airport CEOs in building large airports we are in the process of going through these dictionary constructions and are complimenting them by adding iteration friendly arrays. The plan is to keep the dictionary for fast lookup methods via the unique key, and keep a second dynamic array solely for iteration purposes. We are also in the process of dividing the data into different categories for even faster iteration through various types of items and structures. This will make complex iterations such as “finding the nearest object of type X” faster as we only need to iterate over the very specific object type, without doing any pre-processing or preparing of the data structure. What we essentially are doing is trying to re-think parts of the performance critical code base in a more data oriented fashion, as opposed to traditional MVC patterns (which is what we are used to coming from an IT systems consultancy background), by laying out the data in the memory to reduce performance hits like cache misses. Note that with this change we do not expect drastical performance improvements as the methods we have are already quite fast and usually not invoked each frame, but as you know, every FPS counts.
Another performance aspect we’re dealing with is the mitigation of Unity related bottle necks. We’ve learnt a lot over the years developing Airport CEO, not only in regards to C# performance but also in regards to under what conditions Unity as a game engine performs its best. One large issue we’ve discovered, and that has triggered a redesign of the generic person game object (passenger, employees, contractors), is its sheer amount of game object children. ACEO is as you know a game that doesn’t shy away from the details but that also has its price when it comes to simulation performance. Moving an object with several attached child objects can become an increasingly performance costly task for each new agent that is spawned, which is why we’ve been focusing a lot on redesigning ACEO’s agent objects to fit the same kind of detail but on a smaller simulation footprint. For each child we remove from an agent we gain additional performance, something that mainly becomes notable the more agents we simulate.
There are many more Unity related performance bottlenecks we are looking into to, not only in relation to simulation but also in relation to the UI. That however, will have to wait until we really start tackling the UI improvement sprint.
So what have our performance efforts so far really amounted to? Again, we won’t say for sure since we’ve yet to complete the performance sprint but here are two (short, trying to think about this website’s performance too (lol)) .gifs comparing Alpha 25.2-2 to Alpha 25.3.
As you can see, the difference is somewhat noticeable. However, please note that these .gifs are captured in an isolated test environment and only show the simulation of large amounts of passengers moving from one point to another. As we move ahead with fully implementing the new path finding system and wrap up the other performance improvement tasks we’ll expand our tests into player submitted airports to see what actual performance differences we can see between the two builds. Have you built a large airport that you want us to test? Please send it in via the bug reporting sheet!
So that’s it for this week… as mentioned in the Forum post we’ve also been looking a lot on the issue with stuck vehicles and have implemented several fixes for this that will make its way out with Alpha 25.3, which is expected to be deployed (updated!) Wednesday 14th of February 2018 (will not contain the new path finding system). If you get any stuck vehicles in 25.3, we really urge you to send in the save in which it occurs so that we can jump on the problem immediately. For those who are airline design contest winners we would like to mention that we during this week will look at shipping solutions for your prize, more information on this later.
Thanks for reading, you’ve made it all the way to the end! See you throughout the week and stay tuned for the experimental deployments! Fly safe.
Innovative rotor configurations, sleek cockpit designs, and formidable thrust! These are just a handful of features that define the helicopters in Airport CEO, a new type of...
Read moreUnique engine placements, see through nose cones and raw power! Those are just a few of the components that summarize the eastern aircraft, birds rarely seen flying in the west...
Read more
