I can see now why a simplistic xy hole compensation setting in the slicer is going to have problems after discovering that a guy on printables did a bunch of measurements and found that small holes really do have a greater percentage of shrinkage:
https://www.printables.com/model/449520-hole-diameter-shrinkage-test
so no matter what xy hole shrinkage compensation you pick, it will be wrong for the holes that are a different size.