You should be able to turn off (most) of the purge by setting the flushing volume multiplier to 0 (or something close); but I would not turn off the prime tower if you are doing multiple colors or filaments (unless you are doing a full bed print and just want to yolo it). This could negatively impact the quality and strength of your print (and it the worst cases, cause a print failure)
When a nozzle is at or near temperature, doing nothing, even for a second, it has the potential to ooze out a little bit of filament. This can pose problems such as having an “air gap” where it anticipates filament, and/or a tiny “blob” that it ends up sticking to your part somewhere. Priming is intended to solve this. You could reduce the prime tower size and volume (the defaults are overkill) but take great care if you decide to turn it off.