It's because the whole point is you've been taken from your own story, and have to find your way back so the story can continue. The characters are supposed to come together and find their way back to their friends, so having more than one person per fandom would make it less interesting.