Perhaps it's because I saw the 2005 version first, but aesthetically, I find
most of the people cast in that film closer to how they appear in my head when reading the book. The only thing that tends to bother me in casting is when filmmakers go against the few physical characteristics Austen described--Darcy needs to be tall, attractive and look 27/28. Mr. Collins is in his mid twenties and of a larger scale, Jane is beautiful, Mary is plainer than her sisters, Elizabeth has dark eyes...
While the BBC mini series is the closest to the canon story, I find the casting disconcerting for some of the reasons you mentioned. However, the biggest distraction for me is the actress cast as Jane. Every time I watch that version I can't help thinking that she, as styled anyway, just
isn't the most beautiful sister and certainly isn't the most beautiful girl in the neighborhood! For me, Rosamund Pike is a near perfect Jane. Greer Carson and Laurence Olivier are a creditable Elizabeth and Darcy, but they are and look older than their canon counterparts. I find the other sisters in that version fit Austen's descriptions fairly well, but Charlotte and Anne are too pretty and Collins too old. It makes things easier if I try to remember that filmmakers are engaging in their own form of fanfiction!