'Invasive' is so relative to whether you 'want' more plants. In the seven years we have been here, there have been no new plants coming up from the privit, seed or otherwise. I understand you can propagate it by cuttings, so I will try that. I would not mind more in other areas of the yard. I may try 'planting' some of the seed. I'll also keep an eye out for the golden variety.
I also wish the milkweed in our yard would live up to its reputation of being invasive and spread faster, so we would not have to collect plants elsewhere for feeding the caterpillars!
We have another weedy shrub in our yard, commonly called wild elderberry by locals who warned us it is invasive and should be pulled out. It has nice fern-like leaves, much like sumac (which we also like) and in the spring it too has nice white lilac-like blooms which have now turned into red berries. Today we were rewarded by seeing a Baltimore Oriole feeding its young at the berries.
The outer edges of our yard have a lot of trees and shrubs, most of which are native. It can be a little 'wild' (natural), but the shelter provided supports many interesting birds, just as the garden attracts the butterflies.