Should I worry about finasteride side effects even after stopping the drug?

Some men who took finasteride for urinary problems (Proscar) or balding (Propecia) have reported permanent sexual side effects such as low sex drive or ejaculation problems. It is not established for certain that the side effects are caused by the drug.