It is normal for the bottom leaves to yellow and fall off. I usually trim them off even if they don't yellow because being that low on the plant they don't contribute much to photosynthesis. Brown sevearly damaged leaves are always a detriment and the plant stuggles to support and repair them. This energy could be used to greater effect forming new growth and buds. Thats why it's better to cut them off.
Use a sharp and sterile scissors. Cut the branch right where it meets the main stem. Do this after watering in the morning. This gives the plant a whole day of light to recuperate and form a scab. By using sterile instruments and cutting cleanly you will avoid any type of diseases.