I think I got it figured out. Hopefully it helps someone down the road. I had the AMS heaters running. It limited one to 50*c (the one the first filament came out of) and let me set the second to 65c. When it would swap from AMS A to AMS B it would pause until the temperature dropped before it would dispense the next filament. I was printing in PETG, but I guess it was worried about the softening temp. I turned off the heaters and it stopped pausing for so long and went back to a reasonable filament change time.