I can’t disagree with the “I can” statement, but making the pocket for the magnet slightly deeper and even make a tiny overhang at the top prevents them from touching each other.
You might want to read here for more ideas: Perfect Flush Magnet Fit - #7 by KYZ_Design