Weird ... if it were me I would have kept them for good seeds. Its too late to remove them now anyway. You would have to had to have found the actual pollen sacs to make a difference.
I personally cut them down to curtail further Sex change in other plants. Dirty clothes can carry pollen. Shower and change Everytime before getting close to plants. It's a must do if possible