Ok, Ive managed to remove it.
Basically, the reason it did not uninstall is because I wasnt using Windows default Add/Remove. I use a third party Add/Remove as it gives more options and does better clean up. Strange that the Ask Toolbar only adds remove options from Windows own Add/Remove and not others?
Anyway, thank you for your time.