Can you tell what kind of worm or bug they actually are? Are they caterpillars, or are they bag worms, or something else? Are there webs in the tree? Have you gotten a good close look at the worms?
Here is the newsletter about bagworms.Caterpillars There are other kids of worms and caterpillars, you may want to look through the
Natural Organic Library.
How large is the tree? If it's small you can put some Bt in a backpack pump or use a hose end sprayer and spray the entire tree. If you have bagworms, you may need to hand-pick the casings (again, if it is a small tree and you can reach them).
If it is a larger tree you'll have to just treat what you can reach with the Bt. Next year put out some tricagramma wasps, and over the winter you may need to spray an essential oil if it is something like bagworms. Don't spray neem this time of year, you'll kill the tree outright. Neem needs to be used in the coolest part of the day and preferably in cooler weather.
This is a smattering of information - your first assignment is to get close to the worms and get a good photo of the worms and the damage. Close up photos are very helpful if they're in focus. This thread can accept photos, so if you can get photos and size them to fit (about 1000px wide and/or tall), add them to the thread. Loading to the forum requires fairly small photos. If you use a computer and can load them into Flickr or Photobucket, or if you manage photos with Picasa, any of those will let you either load the photos here or paste in the link from the place where they are hosted.