Right now when something is reviewed, it's reviewed by ONE person only, then it's done.
Instead that number should be increased to something like THREE.
Example: User1234 posts his first answer where he writes "I like shoes".
Experienced user WeirdPhilosopher reviews the post and tells the user "you need a reference to you liking shoes". He then clicks done! Review 1 done
Experienced user Fabulous reviews the post and agrees with WeirdPhilosopher, and adds a link to the [FAQ] and [help]. He clicks done! Review 2 done
Experienced user Cskbinnx reviews the post and agrees with the two first reviews but also flags the post as not an answer, and tells User1234 that he did not answer the question. He clicks done. Review 3 done.
Now the post disappears for requiring review.
My example was a pretty stupid situation, but WeirdPhilosopher and Fabulous missed things during their reviews so having someone more review as well doesn't really hurt. And while this post was easy, more difficult situations might happen, especially when it comes to edits. This would also mean no one can do an edit without being peer reviewed first, which I think is a really good system.
This would ensure things stay at a high quality, even if it's not a full peer review board.