I'm going to take a wild guess here. I think somewhere along the line when you were cross breeding & back crossing you had a hermie plant that pollinated the plant you were cross breeding. Pollen from a hermie will sometimes (often) produce a hermie plant. I haven't done any breeding & not sure if things work that way as far as some branches producing male & some female. But I do know that pollen from a hermie will produce hermies.