So I kept getting a grey fungus that turned into a nasty thick cobweb, it turned the stem brown and the leaves/buds grey.
I realized it was spreading through the plant quickly by growing into the stem and spreading along it. I figured this out after cutting off some buds and it kept spreading...