Developer Insights #19: Try, Fail, Try Again...and Again

Who are you??

My name is Darrin, and I’m the new Director of QA here at Intercept Games. I’m new to the Intercept team (I joined a week before the KSP2 launch…) but I’m not new to QA. I’ve been doing software testing since the early 90s and I worked at Visio/Microsoft for 17 years and then at Amazon for the last several years (most recently as a founding member of the Luna game streaming service). I’m an old-school hardcore gamer - I’ve been playing since I had Bards Tale on my Commodore 64, and my Xbox gamer score is 150,000+ and I don’t pad that with those silly games to just farm achievements! I’m doing my best to learn KSP2 as fast as I can, and just so happen to sit directly next to Nate, and I’ve received the rarely given “high five” that I will cherish always.

Please contact Private Division Customer Support for any game breaking issues such as hard crashes: Submit a request – Private Division Support. But, if you are posting about an issue on the Discord, the Forums, or even on Reddit, on the more data we have the better (aka in the Specs discussion in Discord, you’ll want to post your detailed specs info!)

Bugs and Stuff

Running into an issue is frustrating; I feel your pain. Trust me when I say, bugs drive us just as crazy as they do you (honestly probably more, because we need to retry them over, and over, and over!) But when you find an issue that is important enough to post about – you can do it one of 2 ways:

A. “This game doesn’t work! I can’t play!!” (end of message)

Or

B. “This game doesn’t work, here is some info and specific steps of what I was doing – please fix it!!” (adds a bunch of information we can work with)

We will obviously do as much as we can about B, but there is literally nothing we can do about A. I’ve chatted with Dakota and the others on the Community team… and they don’t ban people for hitting us with hard criticism. You bought the game, and you are VERY entitled go full rage-mode on the boards when you run into an issue. But, at the same time: we don’t have to read it. So, if you are over the top and just trolling the boards… we probably aren’t reading that. But, if you are giving harsh but fair critiques – we take that to heart and bring that voice back to the greater team and do everything we can to ensure it gets taken care of.

So, help us help you and please give us all the info we might need. This helps us move issues up the chain faster and keeps others from having to ask you for the same information over and over:

Title: (a sentence that summarizes the issue)

Specs: (see further below for how to give us all the info we need here)

Severity: (high/med/low). This is your opinion – but it typically goes (High = crash, Med = feature not working as expected, Low = there is an issue, but has a work around)

Frequency: (high/med/low). Does this happen a lot? Can you reproduce it consistently? Or was it a one off? (still want to know about it, but we’ll categorize it as such)

Description: Tell us what you were doing, what you expected to happen and what actually happened, etc. More information here the better, but we want specifics not generalizations. Screenshots are good, videos are good, save files, etc. HOW you were making something is very important here. Giving us a fully built rocket for us to load and try…. might not have the same results has giving us the steps of how you built that rocket. How you created it and the order you made it in is often far more important than the end result.

Helping us Test

I can hear some of you now: “Hey…! Why do we have to do the work for you???” I’ll be 100% honest – you don’t. Very few bugs that come from the community are not already found by a fantastic QA team here. But sometimes bugs can seem to be a “one off” or very difficult to put together consistent repro steps.

The more information we have and the more we know people hit it, the less time it will take to turn-around a meaningful fix. We watch the forums… I personally spend a lot of time looking over the Discord and I know a Senior Engineer who keeps a close eye on the official Forum here. And no, Dakota didn’t mute me as punishment… I just don’t trust myself to chat in most channels (but I do read and react) - but for some important areas/issues, I’ll get directly involved. Most of us don’t respond a lot, but we see the conversations, take notes, and use that information when we can.

Why did you fix bug A and not bug B?

“Wait… if the QA team is finding all these bugs, why aren’t they all fixed?” The process of deciding what bugs to fix is quite complex and is based on many factors. We do not always go after the easiest bugs to fix, but rather bugs we feel will have the largest customer impact at any given time. But at the same time, we must consider the impact of any bug that while fixed could have other implications (aka regressions). We factor in the severity of the bug combined with the frequency as well as a ton of insider knowledge from people here on the team and we work very closely across all roles and make these decisions together as a team.

We care a LOT about bug regressions… so if you see a new bug that didn’t exist previously, then please pass along the information to us and we’ll look into it asap. But, remember… a change to how something works from one build to another is not necessarily a bug. We’re in Early Access and we’ll make changes to how things work at times to see how that goes and iterate on that over time.

How long does fixing a bug take?

Investigating a bug takes time to get a precise scenario for the engineer to address (that’s why the steps and info above can help). They will then do their own investigation on the fix itself (what it would take, what parts of the code would be affected, how risky of a fix it is, could it break other parts of the title, etc). Once that’s approved there is a code review of the changes that were made. Then the change is put into a specific branch/build of KSP2 where testers will do full investigation of the original bug and do halo testing around the areas that the code affected. Then, finally it goes into a release candidate build along with a bunch of other fixes and it gets tested again in a very deep regression suite of tests to ensure that other (supposedly unrelated areas of the product) still function as expected. There’s actually even quite a bit more to it than that - but it’s a good summary of how we do things here.

How do you test a game as complex as KSP2?

Kerbal Space Program is….. huge. I’ve owned the testing of products that ship to hundreds of millions of users that had a simpler test matrix than KSP2 does. There are a LOT of areas, processes and stages of testing a product, and I’m not going to go into all of them here (it would be a doc 20x longer than this). So, this blog is mostly focused on end-user reported issues and how we deal with those.

But with regards to complexity our KSP2 Test Lead Josh who’s been playing KSP since 2013 & testing it for the past 5 years, put some information together:

"Talking about the complexity of construction, and variables for potential issues, there is a consideration on why something may not be known, tracked, or reproduced before.

How unique is your vessel? It’s straightforward to test parts in a vacuum (the “by itself” kind, not specifically the “no air” kind) and confirm that all the bits and pieces are functioning as designed. However, interactions between various parts are where we often see things go sideways. There is a hierarchical interaction that can cause problems, and compounded by where it was placed relative other parts, when it was placed (order of construction), which is further exacerbated by what was done before the vessel was made and how it’s been flown.

Let’s break down possible iteration numbers to give an example as to the potential complexities available in KSP 2:

• If you were to use every stack attach part that has a top and bottom stack attach node in every possible permutation without duplicates, it would be factorial 245, which translates to a number 481 digits long.

  • For reference, the estimated number of atoms in the known universe is only about 10^80, or an 81 digit number.

  • So… The above is interesting, but not realistic. In normal circumstances, you are not likely to build a vessel using EVERY stack attach part.

    • If you assume people are only using about 30 stack parts in a single vessel stack (so a non-repetitive permutation of 30 out of 245 possible stack parts) you still end up with 7.429002947 E+70, or a number about 71 digits long.

    • That’s permutations, however. What about just using specific parts together in general in no particular order (combinations)? That has to be a significantly smaller number right? Well, even dropping this down to only combinations using 15 of the 245 potential stack attach parts with double (top/bottom) attach points still gives us a total of 3.396886498 E+23 possible combinations (or a 24 digit number).For comparison, the number of Stars in the sky is estimated only about 200 Sextillion (or a 2 followed by twenty-three 0’s, also 24 digits long in total).

  • Keep in mind, this is not counting variations using...

    • Parts with a single stack attach node

    • Stack parts that are attached radially (and the surface attach only parts)

    • Subassemblies that can be inside fairings/cargo bays

    • Parts stack attached to other radially attached parts connected to the center stack of the vessel

    • Adapters that allow stack attachments in other directions

    • Adapters that allow a single stack to diverge or converge (such as bi/tri adapters)

    • Translate/rotate transform parts (which also add new orientations to branch off from)

This isn’t to say we cannot infer certain configuration details. For example, it’s likely after an engine you probably are not placing another engine, or even a fuel tank (usually it’s a decoupler or separator). But what if, buried in your vessel somewhere, there were 2 engines stacked atop one another, and then that was vertically translated into a fuel tank and then hidden? A picture is worth a lot, but it wouldn’t help reproduce your issue. A save also would also not immediately make clear how this was built.

An example of how configurations can have impact, we recently fixed an issue with a handful of small parts that would cause the entire vessel to fly apart at a certain point in flight, only when used in conjunction with other specific (similar) small parts in a specific hierarchy order. Not all combinations of a vessel using those parts caused this, only in specific permutation orders.

One of the best ways to ensure a bug you encounter in flight is tracked is to give us as much information as possible. A log file and save help a lot, but if you have a workspace file for your vessel, in some circumstances, that can be even better.

There is more to this game than parts, but the above is a math heavy example of the complexity we are working with in test. This is by no means the only game to have this many potential variations to consider, and it won’t be the last, I am sure. The one take away is this, the more information that can be provided with the circumstances something went wrong, the quicker we can identify the issue. Attempting to reproduce issues with just a list of the 20 parts used to create a vessel? That could take a VERY long time."

All that being said - someone on the team did the math and even with our entire product team testing on a particular build of KSP2… our total user base will pass that time spent testing within hours of release.

We use a matrix method that combines every feature with frequency of use across all hardware combinations and operating systems. This testing effort is what really adds to the time between patches, and then if we combine bug fixing alongside of new feature development… it’s big. Automation helps of course, but as users of the product you can understand that the majority of testing needs to be hands on.

It’s not all crazy matrixes and slogs of testing we have to do. We also have fun playtest events with themes (quite a few are ones we do just before the community team sends them out as challenges):

From a recent playtest where a Designer on our team Chris is trying to branch out from rockets.

How big is the QA team?

I’ve worked at both Microsoft and Amazon as well as a few smaller companies, and our QA to Dev ratio is on par with just about any team I’ve ever been on. We have a dedicated QA team here at Intercept (Seattle) who follows the feature crews as new features are developed and a larger QA team within Private Division (Las Vegas) who owns regression passes and overall validation. We supplement needs (hardware testing, OS’s, etc) with some vendor testing from time-to-time. We even have a large amount of previous hardcore Kerbal players that joined our team and have been helping us out, no matter their role (and I don’t mean just Nate!). We’ve got a ton of experience on the team going back to the early KSP1 days as well as others who have come during KSP2 and are now some of the best creators on the team.

Here are a couple pics from Lo - who is a member of the PD QA team in Las Vegas.

What are you currently working on?

You can pretty much look at the roadmap and see all the stuff that’s coming. And… we’re involved in ALL of that. Lately we’ve been deeply involved in Patch 1 and Patch 2, while at the same time getting early looks at features that are still quite a ways out - something like Multiplayer (below) is something we’ll be involved in for a very long time and looking at it while we test all the other new features that are coming.

Min/Recommended Hardware questions

Every company handles minimum requirements a bit differently, but we use them as our bar of initial support. (aka we don’t officially support or test anything below minimum requirements.) We set that bar based on the performance checks we’ve done in-house during our testing & evaluation process. Some users may choose to try to run KSP2 on a below the minimum bar computer even though we don’t officially support it and we allow that (aka we don’t do a check at launch to determine hardware and force a hard stop at that point). This choice is a tough one, because we want to allow users who might have some very specific or unique hardware to try to run KSP2 and this allows some specs clearly below the min bar to try to run even though that will likely cause issues.

And that’s where things can get crazy – because we’ll see someone post on the boards that the game is unplayable and that they have a good computer to play it on, but after drilling in – the majority of the time the computer is below the minimum requirements bar (and the telemetry we currently have tells us the same story).

Somewhere between our recommended settings and our minimum settings is where we do the majority of our testing. We have a large variety of hardware in-house and we supplement that with vendor testing, but there will always be cases where users will try hardware that we haven’t and don’t have access to. In these cases, we greatly value the input from the user base and will acquire hardware from time-to-time ourselves, when needed, based on that info.

Constructive feedback can provide positive results: There was a very large thread on our Discord and a subset of users who all had specs below the min bar - they all had integrated graphics cards and were having very similar results (KSP2 wouldn’t load - at all). I checked in with our Senior Engineer, Mortoc, on this topic and he suggested a potential workaround by having them set their Windows resolution to 1024x768 - and now most of them are now able to play. No promises of course since it’s below the minimum requirements, but it’s worth a shot if you are also having that issue.

Our goal is to slowly decrease the minimum requirements over time, but we can’t guarantee where it will go to, or what those requirements will be. We’ll use the data that comes in from users, our automated benchmarks that we run as well as in-house testing to monitor performance over time.

Building functional complex ships is a goal on the team as well. Marc, a tester on the team since early KSP1 days (aka Technicalfool on the Discord) sent these to me when I asked for a ship to use for testing.

Best way to get your Specs info:

The more spec information we can get, the better that we can help you solve issues. Typing it in is great, but even the best of us can have a memory lapse and think we have 16GB when maybe we only have 12GB. Same with video cards…. It can get super confusing - and if someone on the team is investigating an issue for you only to find out the specs you listed were incorrect… it’s wasted time. So, a screenshot of your specs is best and here’s how you can get it:

IF you are running via Steam: Click “Big Picture Mode” in the upper right corner of the Steam window. Then bring up the menu from the lower left and choose: Settings → System. Scroll down to the Hardware section and then use the Snipping Tool to get the info needed. (To exit this mode - click in the bottom left of the window and then click Menu - Power - Exit Big Picture mode)

Alternatively, if you don’t have Steam or want a lot more info:

In Windows – hit the Winkey (⊞ Win) to bring up the start menu and type in: dxdiag

This will bring up the DirectX Diagnostic Tool. The first tab (System) will show your system model, processor, and memory. Now, just click the “Save All Information” button at the bottom, and it’ll put everything into a text file.

Nothing in that file should be harmful to send, but if you don’t want to share everything else on there, just do a screenshot of the important areas (the snipping tool works great for this):

The other tabs will give you information about your displays and video cards (which is typically more important if you are having performance issues). If you have multiple monitors or a laptop connecting to a monitor – then you might have to click on the last display tab to see the correct video card information. Again, if you don’t want to share all your info – just get a screenshot of the important stuff:

We’re here for you.

The good, the bad and even the ugly. We like to hear what you have to say. Our fantastic community team keeps us in the loop (and coordinates us doing stuff like this doc and the AMAs) and we love to meet and chat with you all in person as well. We had the opportunity to chat with some of you at GDC (had quite a few Mun and Minmus landings) and we’re looking forward to more in-person events in the future.

How can I join the QA team or other roles at Intercept Games?

All our open roles are listed at Intercept Games

Most of the KSP team are previous players and there are more than a few former mod designers (and a writer) who are now a part of our Kerbal team. We love the game and for many it’s the dream job… I know it is for me. So, if you love KSP and want to be a part of the team- check out our reqs listed above. But no matter what is currently listed – we are always on the lookout for great people. Even without an open req, we keep our eyes open here on the boards and other places that we interact with the users. And for QA specifically… keep an eye out, because one (or 2) might show up on that list above very soon…

Thanks everyone from the KSP2 QA team!

This isn’t the entire QA team, but I did get a chance to meet with a lot of them last week. I’m the super tired looking one on the far left (it’s Vegas - what can I say??). Mike is the professional wrestler in the middle (that’s not a joke btw).

Safety Note:

Public Postings

If you share personal information, post an image or video, or provide other content in public forums such as on a message board, chat room, comment field, or profile page, other people can view, collect, and use that information. If your user name or ID contains your name, your name will be publicly available on leaderboards and elsewhere as described in this Privacy Policy. Users of such public forums may be able to identify you, use the information to send you messages, or copy any of the images, video, or content you have shared. There is no expectation of privacy or confidentiality on any of these public forums. Please do not share your personal information in public forums or in your user name or ID. You are responsible for any information or content you publicly post using our Services.