Worm castings is simply dirt that red worms lived in. The worms break down organic matter into compost. Worm castings are low in Nitrogen and not very rich in anything else. Worms or worm castings alone will not nearly supply the necessary amount of food to sustain the plant. Mostly, castings are used to increase drainage in the soil.
The only other benefit I can see of adding worms directly to your soil is that they will aerate the soil. Also, worms need food too. So unless your adding organic waste to your soil they will not survive very long. Thats why the worms belong in the compost pile, not in your potted plants. In any case, they won't hurt the plant.
