So I have a short update on the matter.
I have tried debugging this the best I could and here are the results:
I recalibrated the printer a few times and ran the flow calibration about 5 times. The thing is, just going by the first layer, the -20 one looks almost perfect. No bubbles, nothing, just straight lines. However, since I have to evaluate the end result, 0 always comes out the best, albeit having a horrendous first layer.
You can see, the bottom left shows the -20 on the calibration test. The top one in the middle is the 0. I ran the test with 0.98 flow rate and in the end it stayed at 0.98 (maybe with a tendency to 0.9702). No matter how often I did the test, the result for the first layer is always bad, but the top most layer turns out fine and really bad for the -20 square.
You can see after the second calibration, on the top layer, 0 looks pretty ok, the other ones don’t.
I was reading up a little more and saw, that the 0.6 nozzle doesn’t seem to have a good profile. So I switched back to the 0.4 nozzle today and tried calibrating it again. Unfortunately, the results were the same.
I then saw, that an Overture PLA profile was added to the official list and tried that out. It turned out to be worse than my custom one (derived from the Bambu Matte PLA), although it does seem better, if I reduce the flow to 0.9702.
I keep thinking it might have to do with speed. The first layer is printed really slowly and creates those ridges on the sides, while the other layers are printed fairly fast. Is that something, that can be fixed with pressure advance? The Overture PLA preset seems to have a flow ratio of 1 out of the box, meaning that the preset must have been good enough.
@Lexi My seam settings are the default ones:

Cheers,
AnotherGuyver