Hello airport CEO!

Welcome to another development blog, the 122nd in order, as we're painfully aware of having spent several hours converting the previous blog posts from the old website to this shiny new one (well, OK, not us, but someone helping us did it and it looked awfully boring).

We're back again with a more regular length update (well maybe not) and as usual have exciting news and updates to share with you... so without further ado; let's get into it!

‍

Introducing Alexander!

We're beyond delighted to finally introduce you to Alexander, the newest Apoapsis Studios member whom we've hinted about for several weeks now, and who's finally started developing Airport CEO together with the rest of the team. But let us not occupy this text anymore with our gabble and instead let Alexander do the talking. The below text is pulled from Alexander's introductory forum post and tells us a little bit about who he is.

‍

Hello all airport CEO’s out there!

My name is Alexander and as some of you might already know I started working with Olof and Fredrik at Apoapsis Studios on Monday this week. I wanted to take the opportunity to shortly present myself and tell you about how I ended up as a game developer of Airport CEO.

I have a master's degree in engineering and I’ve always been interested in technology. Everything from computers, airplanes, space-exploration to biotech and medicine get my attention. Ever since my dad brought a Windows 95-machine to our home, I’ve been interested in not just playing around in games, but also explore how the software worked. It always gave me some level of satisfaction opening an .ini-file located in a game’s folder and see that those lines actually made sense! Altering a parameter in said file altered the conditions of the game. Editing a .png-file in another folder changed the appearance. Experiences like this quickly developed into an interest for programming and game developing and I’ve through my years nourished the idea of someday releasing a title.

I got to know Olof and Fredrik a couple of years back through University. They told me they were working on a computer game and as I’ve been working on my own projects during my spare time, I immediately started pestered them with questions. On top of that, they are some of the most devoted and nicest people I’ve ever met! So when the application for the game developer position was posted, we met for a long interview and it was pretty clear I would thrive and fit at Apoapsis. It seems like they thought the same.

A week has now (quickly) passed and I don’t regret my decision boarding Apoapsis Studios one bit. A lot to digest now in the beginning, but so fulfilling working with something I love! Also, what has been most evident is the devotion from the Airport CEO-community! Without you the game wouldn’t be as good and promising as it is! I will (and I know Olof and Fredrik also will) work my actual ass off making Airport CEO one of the best management-games ever created!

‍

Thank you Alex, and once again a really warm welcome! This is indeed going to be an interesting time and experience for all of us. There will be challenges ahead as the Airport CEO code base is nothing short of daunting with its sheer size and complexity, but the fact that Alexander has already implemented and deployed a few items during his first week (such as the new medium information desk and small boarding desk and upcoming flood lights) is very, very promising for the future expected increased development pace. You can as always follow Alexander's work, or anyone of us, via the public Trello board to see what we're currently working on and as we get the new and expanded Apoapsis Studios team up and running with proper workflows and routines we'll be sure to report back here with our organizational operational progression.

If you want to give Alexander your own special welcome, don't hesitate to pop over to his introduction forum thread! You can also check out our instagram account and peek into our new Dev Hub HQ.

‍

Solving the CPU utilization problem (Alpha 29.9-0)

Let's talk tech. As of the latest default and experimental deployments of the Alpha 29.X-X updates we've been alerted of, and consequently discovered, that the Airport CEO executable has been struggling a lot both in terms of dropping frame rates but perhaps most noticeably a very high CPU utilization. As of one recent deployment, we got several reports in stating that the ACEO executable immediately on launch consumed between 80 to 100 percent of the total available CPU processing power which caused CPU core temps to go up and fans to come on. We immediately started running tests but unfortunately quite quickly realized that this was a multi-faceted issue to which there was no immediate quick-fix and so we've spent the past two weeks analyzing incoming bug reports, analyzing the code, performing mitigations and drawing conclusions. As of the latest experimental deployment we have finally fixed this problem and in the process we actually ended up gaining a noticeable performance improvement. Lets go over what we've been working on:

While the main menu may look it has no computational performance challenges featured within it, that's not entirely true. The main menu background consist of several aircraft and cloud agents that travel across the sky background (this is not a video and is simulated in real time). While the clouds do not have any particular performance impact, the rendering of the aircraft contrails certainly do. While most PCs have no immediate issue rendering those particle systems, they always try to do so at a specified frame rate and if there is no such specification, the highest possible FPS count which in turn drove the CPU to work as hard as possible to deliver those frames. Since this is "only" a start menu, there's no point in rendering it faster than at least 60 and at max 144 FPS and so what we did was simply to limit the number of frames we'd push to those levels which immediately solved that problem.

However, it naturally did not impact the fact that you'd still launch into an empty lot and immediately have the CPU at 100 percent utilization, core temps going up and fans coming on. As we investigated that issue, we discovered a texture related problem that could on lower end GPUs result in a performance bottleneck. The ground textures were being incorrectly rendered which in turn caused the GPU to work overtime trying to account for that incorrect rendering. When the GPU is "moving at a slower pace than the CPU", the CPU needs to cycle a wait operation in order to match the overall game execution pace and as it turns out that CPU wait cycle is simply running a bunch of random calculations with the purpose of keep itself busy - causing a huge overhead and slowing down the game. Once the incorrect texture rendering was sorted, the bottleneck disappear and we once again shifted the game to be CPU bound rather than GPU bound.

However, this problem was not in any way really related to the core problem of the huge CPU utilization issue. But with both of these issues sorted we could really begin to dive deep and rule stuff out. We continued our investigation and with the aid of our community, especially the people in the internal test group, we were finally able to make a series of assumptions that led us in the right direction. We realized that ACEO has always have a pretty high CPU utilization but it was after a specific Alpha 29.X-X update that we started seeing a pattern of reports, the update where we launched a performance improvement consisting of multi-threading the auto flight planning alforithm. This additional threading was what pushed many peoples' devices over the top and that's when we realized that it contained a fundamental flaw which affected all other multi-threaded systems (which are quite a few). This fundamental flaw, for each system, created a lot of hidden overhead which in turn was what was impacting the CPU and driving up those super high utilization percentages.

After a quick fix the issue was resolved and as we verified the results, that the CPU did in fact drop down to an expected five to ten percent on a new world, we also quickly saw that in terms of overall performance there was an unexpected significant boost on all platforms. On a mid-range PCs we saw that there were now longer no issues of pushing around 4000 passengers in a player built airport at around 60 FPS, while our high end machine could push 12700 passengers in another player built airport at around 30 FPS (which we've intentionally modded to spawn insane amounts of passengers for test cases exactly like this). This is very, very exciting and we're hoping that our experienced performance improvements will be passed on to you as of Alpha 29.9. We've not applied any frame rate limit to the game world as of this update, you can either run the game at max FPS with a slightly higher CPU utilization or lock the rate to 60 FPS using VSync for a guaranteed lower CPU utilization. As an example, on a high end machine you can either run a new world at around 1200 FPS with 30 percent CPU load or at 60 FPS with one to two percent CPU load (results may vary...).

Below is a .gif of those 12700 passengers, all sharing a terminal waiting for a flight that will never show up... imagine the horror.

Or check out this high defenition imgur video!

With the inexplicable high CPU utilization problem fixed and gone forever we'll no revert our resources back to remote stand bug fixing, general bug fixing, multiple floor, new asset implementation (handled by Alexander) and starting the implementation of the finished soundtrack.

Alpha 29.9 is currently deployed on the experimental branch and will get switched over to the default branch as soon as possible.

‍

In other news

The soundtrack is completed! Not only do we have the finished music with us ready for implementation, we've also together with Sinephony produced a short behind the scenes video showcasing the recording process of three songs. We'll drop this video a few days prior to the soundtrack update being released and while no exact date will be given it shouldn't be in a too distant future. We, and Sinephony, are extremely exited to see what you think about it. Oh, and another Embraer aircraft model is in its final production stage as we're preparing for the next batch or aircraft to be created with the closing of the mentioned polls on the 1st of February.

‍

Other than that we're, as you might now, building a new website for Apoapsis Studios, holding a large aircraft vote and a GA vote and have had quite a few productive studio meetings planning the development work ahead in accordance with your bug reports, the feature requests and all other input we take to guide the development of Airport CEO.

That's it for this week! See you in the next development blog and... fly safe!