Bambu Lab Customer service has shipped me a new drive gear, which is the feeding gear in the AMS manifold.
When replacing this gear I noticed a lot of plastic grindings in the housing for this gear.
I could not notice any major difference between my “old” gear and the new one but I decided to replace it anyway.
After cleaning this and replacing the gear unit I was able to print a 3 day model with over 1300 color changes without issues!
I am not sure if it was the gear that caused issues or the plastic residue that was in there but my issue is solved!