I feel like I may regret asking, but, enlighten me...
If you watch any UFC and see Mark Hunt, the Samoan, in the heavyweight division you wonder whether he may have never stopped eating other humans!
You've hit the nail on the head: the small gene pool is the key.
Your points about slavery are valid too, slaves were cherry picked based on their physique and stature. Not natural selection, but unatural selection.
As for Polynesia, it's more than just cultural why New Zealand have such a strong rugby team: Maoris (indegenous Polynesians) are another example of genetically blessed peoples.