Based on the discussion of race and Game of Thrones (most recently in the context of casting the Dornish) I decided to take a look at how Westerosi (and Essosi) geography line up with real-life geography, to get a feel for what the phenotype of the various peoples might be realistically. Take note that this is, of course, something of a simplification, as migration and other specific factors can impact the color of a person's skin and hair beyond mere geography. That being said there is an identifiable correlation between latitude and skin color / hair color.
As a baseline, I chose to use the Land of Endless Winter, whose name (as well as the general description of the lands beyond the Wall) gave me the impression it should correspond to Lappland or Iceland, both of which border the Arctic Circle. From there I move southward by comparing a map of Westeros with one of Western Eurasia and North Africa, accounting for scale and using the seemingly reasonable assumption that the planet upon which the series takes place has a similar diameter and distance from its sun to Terra's. So here we go.
- First off, we have the lands beyond the Wall, which per my baseline, correspond roughly with northern Scandinavia. As such, the Wildlings would probably possess very pale skin and an unusually high rate of blonds and redheads (although brown or black hair would still be more common), like the Sami.* However, this isn't necessarily guaranteed, as the Inuit (the lovely lady on the left) in North America and the Yakut in Siberia both live about as far north as the Lapps but have noticeably darker skin, probably because of their diet, which contains unusually high levels of Vitamin D, thereby reducing the evolutionary advantage of pale skin in subarctic latitudes. Using the former assumption of pale skin (which is more typical), it seems the show isn't far off, although realistically, the Wildlings might actually be slightly whiter than they are (but not by much).
- Heading south to the so-called North, we find ourselves in an area which corresponds pretty closely to the Scandinavian heartland, the Baltic republics, or Scotland. This means once more pale skin and relatively light-colored hair. Richard Madden is a Scot and Sean Bean is Yorkish, so this is pretty close to what the show gave us and in any case we're still dealing in only very small variations and the difference between an average person from group 1 and one from group 2 would likely be small and indistinguishable to those with a significantly different phenotype.
- The northern Riverlands, the Vale, and the Iron Islands are all at about the same latitude, which is pretty close to that in England, the Low Countries, northern Germany, and Poland. Again, white skin, though noticeably paler on average from that in group 1. Blond and red hair are also a lot less common now, though still not exceptionally rare. Interestingly, this also corresponds pretty closely to Braavos, which means Syrio Forel may actually have been too dark, though it's worth noting that it's certainly possible that he is not ethnically Braavosi.
- The southern Riverlands, the Westerlands, and the Crownlands are to the south of group 3 and it is at this point we begin running into problems. This area corresponds basically to most of France, southern Germany, northern Italy, and the Ukraine. People from this region would be white, but blond or red hair would be pretty uncommon, though that doesn't necessarily rule out the golden curls of Jaime or Cersei (it would just be noteworthy, which it is in-universe). And, interestingly enough, King's Landing – which on the show looks like it's in Sicily or Andalusia (or Malta, where it was filmed IIRC) – seems wildly off-location now, since it's actually closer to the latitude of Bordeaux or Venice. This is also roughly the location of Pentos, which like King's Landing is probably too warm in the show.
- Beneath the lands of the Lannisters and the Tullys, we have the Reach and the Stormlands, homes of House Tyrell and House Baratheon. This is where we first run into some real issues with the casting, as this area corresponds basically to northern Spain, Italy, and the Balkan peninsula, along with Turkey. Logically, Robert Baratheon and his whole family (including grumpy Stannis and poor Gendry) should have an olive hue to them or at the very least look tanned. The same goes for the Margaery Tyrell who, as lovely as Natalie Dormer is, looks far too pale in consideration of her house's geography. This also corresponds to Myr and the Dothraki Sea, which means that while Jason Momoa was probably still a shade too dark to play Khal Drogo, he's not that far off. Varys and Thoros should be darker, though (although I'd be sad to recast Conleth Hill, who's just so excellent in the role).
- Finally we come upon Dorne, the land of much past and future controversy. I haven't read all of A Feast for Crows yet but it seems obvious to me that Dorne is supposed to be something akin to Moorish Spain, which matches the geography I came up with pretty closely. Specifically, Dorne corresponds to southern Spain (Andalusia) and most of North Africa (with the exception of Libya and Egypt, which are just a bit too far south). At this point we're talking a solid olive hue, trending towards a light brown on the darker side and a heavy tan on the fair end. This means that Pedro Pascal is (unfortunately) too pale. But we all suspected that anyway. This is also the same latitude as Volantis.
- I won't go into too much detail here, but lands further south yet of Dorne include Slaver's Bay, the Red Waste, Qarth, Sothoryos, and (interestingly enough) Valyria. Which makes you wonder why the Targaryens are all so pale...
In any case, I thought this might be helpful and/or interesting. Of course, it's a fantasy world, so one can always say that the characters can have whatever skin color you want, be it white, black, brown, or fluorescent green, but clearly Westeros and its accompanying countries are intended to represent a fairly realistic world (crazy seasons aside) with some important and natural parallels to our own, so I don't think my speculation is all that far-fetched.
In any case, feel free to make use of this in your head canon or to discuss below.
*My conversation with artiofab reminds me that while Lapp is/was the common term for the inhabitants of Lappland, the people themselves prefer the name Sami, much as the Inuit prefer that name over Eskimos. Sami also shares a common root with Suomi, which is the Finnish word for Finns, which makes sense considering that the Finns are basically Swedenized/Russianized Sami, with a very similar language and phenotype.