I have a smith's 'fang tooth' blenny, supposed to be venomous, but he's certainly not dangerous. Same with my orange spot rabbitfish. I figure, don't try and catch them with my hand, I'll be fine.
I do have one small colony of Palythoa, but they're not really dangerous either... as long as you know the risks, it's not that hard to keep from getting yourself poisoned by them.
I'm probably much more likely to be electrocuted by my tank than I am to have one of the animals in it injure me. I suspect the same is true for many of us.