As Farfalla said, it could be so many things. Sometimes it is a matter of trial and error, but can be heartbreaking and discouraging when you first start out.
Something did come to mind as I reread your postings in an effort to figure out what might be going on.
Several years ago, in my search for milkweed, in the same place I harvested common milkweed, I came across a plant that looks similiar to swamp milkweed and has milky sap when you break a stem or leaf. Thinking I had come across another form of milkweed just as I was running low on common milkweed I decided to give it a try. To my chagrin, the cats died after a few days, so I stopped using it. Just recently I came across this plant again and finally was able to identify it.
It is called Dogbane or Indian Hemp (Apocynum cannabinum) and is considered poisonous. If ingested it may cause cardiac arrest, as it contains resins and cardiac glycosides. I am not sure if a caterpillar can have cardiac arrest or not, but do know that it did kill a few of my little guys before I realized it wasn't good for them. The type that grows here in PA and NY looks very similar to young swamp milkweed. Especially when it first comes up and hasn't branched out yet. One clue that helps me identify it that it has red stems, and that it has a small cluster of droopy white bell shaped flowers instead of large upright flower cluster found on most types of milkweed.
Here is link with two pictures if you would like to check it out.
http://www.ces.ncsu.edu/depts/hort/cons ... pocyca.htm
Hope this helps