Fit issues with X1C

I use bambu pla. I find that with some models the fit between objects can be marginally too big. I’m trying to build the A1 mini phone charger and it has turned into a real nightmare with holes too small, not lining up, not able to slot together etc etc.

Am i doing something wrong? Or is this how it is? Are there any simple ways of adjusting the fit?? Help!!