I have had the same problem in my butterfly garden this year.
The plants didn't look bad in the spring but as summer came on they began to look really raggy. No blooms either. I wasn't sure if it was the drought we had been having here in central Fl or if the plants just need to be replaced. Maybe the plants can also get a disease. My plants are 3 years old and bloomed great the first 2 years.
Hopefully somebody out there may know what the problem is and can answer our questions.