Oh, now that's interesting. I love the way each of us can read the same line and have different interpretations of it. I've always read Darcy's statement, "You cannot always have been at Longbourn," as him trying to reconcile a woman of whom he thinks highly (Elizabeth with her extensive reading, witty debating, and good manners) coming from a household and community of which he views very meanly. It's a little like that line in his letter in which he notes that Elizabeth and Jane always conducted themselves very correctly, even when their mother, younger sisters, and even occasionally their father, did not.
Darcy assumes that Elizabeth can't possibly have gained these traits by staying with the Gardeners (not only are they in trade, but relatives of Mrs. Bennet!), and posits that she must have spent time elsewhere-- perhaps a few months every year with godparents or some gently-born branch of the Bennet family. He has built up a little fantasy in which Elizabeth feels the same as he does toward her (without bothering to talk to her very much); by making this a statement rather than a question, Austen emphasizes this as a tendency of his character, perhaps. He is still battling himself over whether or not to offer for her, and would really like to hear that she has some other connections that his family would find more acceptable.
Anyways, my two cents, for what it's worth!
-Jean
