Yeah you'll want to pick off those dead calyxes, they can actually have the potential to germinate mold spores on them, which could potentially lead to stem rot. So I always pick them off (they come off easily), they have nothing to do with the developing bud other than sex indication