You should leave it until late May early June. If the seeds haven't matured, it will give them more time. Also, it provides shelter for other animals and there may be chrysalis of butterflies who winter over in the area hiding on it. I never clean up my yard until late spring.
Many moths also are wrapped in the leaves that fall from trees.