I do have one. Doesn't bother me about being tracable, flying is not illegal. If the worst was to happen e.g you had a flyaway or such, I have public liability cover in case it lands on somebody's head! But to be fair it would be very difficult for somebody to prove you were liable in the event of a malfunction.
I have flown over houses, don't really do that anymore but I've not got anything against anybody who does (unless their peeving in my house window).
Flying in a field or open space could still be a concern, if you lost control of your craft (flyaway) what's to say it wouldn't fly over to the nearest housing estate and land on someone's head? Or a dog walker or fly up and hit a low flying real size craft.
It's like anything in life, just use a bit of common sense.