"Shouldn't they only be allowed to use their own physical gifts?"
You mean the ones that some have hundreds of thousands of dollars to train while others don't have that luxury? Like THAT isn't an unfair advantage?
And what about genetics? Should two abnormally tall individuals get to mate to create an abormally tall competitor in basketball? Isn't that unfair to the normal-sized people trying to play basketball?
And how about clutzes? Some people just don't have athletic ability - isn't it unfair to THEM to not be able to compete with others who do?
At what point do you draw the line?
Oh, that's right - you draw the line at the "feet" of the guy who doesn't have any.