From my understanding Brim is generated from the model and not from the support so turning it off won’t change the support size.
Have you tried Tree style supports?
You may also try playing with the “Normal support expansion” setting in Support/Advanced (1.7.4.52) Using a negative number “should” shrink support size.