No idea why, but when I switched back to the old output stage, it worked perfectly fine. Some loose connections somewhere, bad solder joints or something maybe.
However, this means that untill the problem comes back again, I will leave the circuit board in the box. This also means that I won't replace C6 and C7 (changing from 1uF to 10uF would have improved the bass responce as the 1uF filters away anything below 140Hz, greatly impacting the bass drum when tapping the signal from the stereo mix jacks).
Now, if I could only get that Hihat working again. Tried swapping U42 for a new, known good opamp, but no luck.
Also, I will try moving the headphone amp away from the box to see if I can get rid of some noise.