Multicolor print - weird slicer defects

I’m having an issue that I can’t figure out a solution for… I have a multicolor print that has inset images and text with a uniform shape for the first 1 mm of the print, then it transitions to just the background color with sparse infill. The problem is that defects start to show in the 2nd layer and get worse until it goes completely haywire for the last two multicolor layers. After this when it switches to solid color with infill it goes back to normal.

The part is multiple parts imported as one, where the color parts are separate from the white.

The pictures are of each layer as it progressively gets worse.

Any ideas?