Disturbed caterpillars of all butterfly species frequently wander off. In some other species (like checkerspots), leaving the plant is a normal resting behavior. The parasite hypothesis is very unlikely, but some full-grown parasitized larvae do in fact act strangely.
I would gather up however many of the larvae are left and try to rear them indoors in some sort of container with fresh plant sprigs.
You... all... stare... but you'll never see, there's something inside me