Were you using a GTB? They are supposed to thermal if the motor or ESC gets too hot. I'm not sure if other controllers use the temperature sensor in the motor though.
Sounds like you fried the windings, definitely worth measuing the temp when exploring gearing options. Even without the thermometer though I'm guessing if you can smell burning that's too hot
.