Well my house is very clean and I vaccum very often so its not crumbs and what not they are after. They haven't shown up in weeks so they are gone as far as I know. What I think I was seeing was a "scout", its the ant that checks out new turf to find food and if/once he does the workers will follow the sent of that scout to the location of the food, the are intelligent as hell!
Sugar ants on a plant can also be a sign of scale or aphids. The ants seek out these pests and feed on the nectar exuded by the pests, so are an early warning sign of a problem. There are passive ant baits that you can use which have no impact on your plants. The worker ants collect the bait (which often contains Borax as the active ingredient) and carry it back to the nest and the queen, killing the whole colony. So yes, sugar ants are an early warning of a problem with pests.