Hmmm. Sounds like a good idea, but again, code changes are required. It would be easy enough to turn off voting all together, but I don't think that's what you want...
I'll add it to the to do list, and in the mean time, the best thing to do is to vote with a rating closest to the current score...