Dark Matter can't be 'ordinary' matter

I have a vague recollection that there are arguments from basic principles that say that Dark Matter can't be ordinary matter. In other words, we know without looking, that it can't comprise loads of brown dwarfs floating around in interstellar space. I thought I remembered the argument being developed in a description of primordial nucleosynthesis, but I might be imagining that.

Have I got any of that correct?

via International Skeptics Forum http://ift.tt/1E3tfWg