The butterfly weed also does not have enough toxins to protect them. The leaves are a bit rough and hairy, too. You are right about caterpillars having no choice eating what is there, but I have heard of caterpillars refusing some species of milkweed and dying.
Herndon, VA (USA)
Take care of the small things....