I think you know the answer - stop being lazy!
Whip them out, you've got to tap them square too afterwards, so will be much easier to do a good job with them out. You've got the pillar drill, - use it would be my advice, don't take any chances when you don't have to.
Once you're done, post up a pic. I did mine a while back drilling standard blocks, bulkheads and not forgetting the spacer shims. Do schumacher include screws and shims with the blocks now? I'm also wondering if there's additional material around the M4 hole.