Newsletter #6 - January 2019 (Patreon)

Hello my loyal patrons!

Today we will break the existing newsletter mold a little.

Instead of focusing on upcoming WIP projects (of which there are a few), we will go back in time a little bit and revisit the CleanROM 2.0.X.X release(s) and one of the most notorious XD+ bugs, an in-depth explanation of it, and how it was finally overcome in this release.

I understand that this post will favor GPD XD+ patrons over the rest, but given how long XD+ users had to suffer through what was known as the "refresh bug", I figured that we needed a technical summary post of sorts to bring closure to a nearly year-long period of suffering and bring attention to the real cause of the issue, so that GPD themselves can pick up on it eventually (and hopefully merge it into their stock firmware as well at some point).

Now, let us unravel the most notorious bug the GPD XD+ has ever seen, together.

What is the "refresh bug"?

For those uninitiated, or people who simply don't follow the GPD discord server or dingoonity (or any other retro gaming community out there), here's a small explanation of the issue:

On stock firmware the GPD XD+ exhibits a very peculiar issue.

Every 10-15 seconds it suffers from a random framedrop, which users could notice as a 1-second micro-stutter in the video & audio output of the system.

The constant reoccurrence of this issue pretty much ruined the experience for most people and made some go as far as shunning the GPD XD+ in favor of its older model, the GPD XD (which didn't suffer from this issue).

The bug's name (aka. barking up the wrong tree)

This issue had most of us (developers) puzzled for the longest time.

Big names over at dingoonity & the GPD discord server all had their theories about where this bug might be coming from, amongst said people were skelton (the author of LegacyROM for the original GPD XD) as well as myself.

The initial theory was that this could be bound to the device's LCD timings stored in the DTB portion of the kernel, in layman's terms this would mean that the LCD panel is mis-configured (aka. the hardware doesn't match the configuration in the software).

As such, the issue was christened as "refresh bug", and a long goose-chase through LCD related kernel code and configurations began, all under the pretense that the issue was in one way or another linked to the LCD panel.

Oh how wrong we were... and here we are, 6 months later.

The bug's cause (aka. the breadcrumbs & the gingerbread house)

When I first started working on CleanROM 2.0.X.X, a full re-code of the operating system from scratch (source) I had set myself a few key goals.

One of them was to merge back together the split kernels caused by the different GPD XD+ revisions out on the market (see "Headphone jack polarity" setting). Another one was to deal with the "refresh bug" (see "CPU hotplug support" setting) once and for all.

Of course there were several other key points on my list, but those two were bold and marked in red at the very top of my list.

As such, I spent a lot of my time testing a variety of things and procuring countless logs from my GPD XD+ in the hopes of finding a possible cause for the issue.

A few dead ends / red herrings later I had finally pinpointed the most likely suspect:

For some reason CPU core 4 kept toggling off and back on again whenever the 10-15 second stutter occured. I didn't have much to go on yet but a hunch that this might be the cause of all our woes.

To test my theory, I went diving into kernel code once again, working my way up the call stack (starting at the CPU core toggle logging message print code) until I finally ended up inside the mt_hotplug kernel module, the same module responsible for toggling CPU core 4 off and on again.

I disabled the mt_hotplug kernel module to test the theory, and lo and behold, the issue was gone. The day was saved and Mario got his princess. Or not...?

The fallout (aka. life without mt_hotplug)

Disabling the mt_hotplug driver did fix the issue, but it came at a (albeit small) cost.

You see, mt_hotplug itself isn't bad, the driver has its purpose and there's a reason it's there in the first place.

Mobile devices (like our GPD XD+) are all about battery conservation and getting the most runtime out of the limited amount of juice it has.

There's two main ways of going about doing this:

• Reducing / Scaling the CPU frequency when the system is idle (CPU governors / cpufreq kernel module)
• Disabling unused CPU cores (CPU hotplug support / mt_hotplug kernel module)

By disabling mt_hotplug, we essentially turned off half of all CPU-related powersaving features. While this is true, it is hardly visible during normal daily usage and (in my personal tests) shaved off less than 20 minutes total battery runtime over a full day of use, making it, in my opinion, a worthy trade-off for fixing the "refresh bug".

Other people's opinion's might differ however, and as such, I've made this setting a toggle inside CleanROM's settings menu (with the default setting being off of course).

After further inspection, I also found that the interactive governor on the GPD XD+ (which happened to be the default) was broken and didn't upclock the CPU properly.

I fixed that issue by making the ondemand governor the new default governor for our GPD XD+, which unlocked quite a visible performance gain in the Dolphin emulator (just to name a example).

The final conclusion (aka. where to go from here)

The "refresh bug" is dead... but is it really?

In summary: The implemented fix for the issue is to disable a battery optimization, and as a side-effect we reap the benefit of fixing a bug that has been haunting us forever.

But the real question here is: Why is the mt_hotplug driver acting up in the first place?

The very same driver works perfectly fine in the Xiaomi Mi Pad 3, a device that is pretty much an identical twin to our GPD XD+.

My best guess is that something in the GPD XD+'s thermal framework is reporting faux values, forcing mt_hotplug to take CPU cores offline when in reality there is no need to.

But this is a story for another day. (Or rather, a bugfix for a future CleanROM revision...)

Thanks & Greetings

I wanted to take the time to thank each and every one of my patron's for making all of this possible in the first place.

Without you guys there would be no CleanROM, or any of my other WIP projects that are en-route. I wish all of you a good day and, as always, keep an eye out for future posts or drop me a comment.

Until next time!

- Black-Seraph