His description as a "...heavy-looking young man..." doesn't scream physically attractive to me, but I wasn't suggesting that he was ugly or that an attractive man cannot have an unpleasant personality.
I was just thinking of how having an
attractive, pompus man preaching at you in your drawing room would be slightly less pleasant than the alternative. If he'd been as you say, "a man of excessive beauty," they'd have had someone pleasant to look at while he was being an annoyance! There is also the general human tendency of tolerating pleasant-looking people a little longer than less attractive ones. Though, with Collins' personality a handsome face would have only propelled him so far.