You're doing something wrong with the dremeling of the rear tower. In more than 2 years of X-5 abuse I changed mine only once, and it was because of a ballstud failure.
The tranny TBH needs some mods to work on astro. Mine is perfect with a 17t Losi pulley and a belt roller.
If the motor is rotating even when fully screwed on
, you need to shave some plastic where the long screw goes into the chassis, to allow it to clamp harder.
You can strenghten the towers with braces. I use the Fiberlyte front one and it's ace, never a failure [knock on wood] !
Give this car a chance; if you liked the XX4, you should love the X-5.