Well, from my rather unfortunate extensive experience with red spider mites, I'd say it sure sounds like them. I've also seen a reddish rust coloration on my milkweed leaves that doesn't rub off and doesn't leave a residue on my finger. I think that's just got to do with the plant's growth etc., it's not an "outside problem" so to say.
But the residue on your fingers? Spider mites. Also, if there are enough on there, you'll start to see very fine webbing that they weave.