This is a pretty dumb discussion because the "toughest" sport isn't necessarily the best sport.

But more importantly, on top of that, rugby really isn't a game focused on contact, in the same way that hockey is not a contact-focused sport. Football, on the other hand, requires it. Football players are specialized for their roles; an NFL linebacker doesn't have the build to play rugby, nor is there a single professional rugby player who could be effective as a linebacker (nevermind offensive and defensive line, ha.)

Rugby also has considerably less down-hill running, and therefore considerably less down-hill tackling. That's the reason for a helmet and pads; any idiot can check, make shoestring tackles, or make arm tackles. But line up 245 pounds on either side and let them run straight at eachother and if you aren't wearing pads, you're both going to die.