Seoul is pretty safe, safe enough to travel alone for a lady – pickpocketing is not very common and violent crime is minimal if almost unheard of. As long as you have common sense, you will be fine! (I mean, you should avoid drunk men at night but that’s just common sense!)
Seoul has one of the finest public transportation systems in the world – buses and subway trains(Seoul Metro) are cheap, efficient and safe. I have to say the public transit is safer than taxis.

You cannot go to Jeju island from Seoul by taking public transportation! It takes 45 minutes by air plane. (Jeju is an island located south of peninsula.) While South Korea in general is a remarkably safe country, the crime rate on Jeju is even lower. Violent crime is almost non-existent, although just like in all tourist hubs, there are a number of overcharging taxi drivers, rip-off merchants and pickpockets.

For the budget accommodations in Jeju:
http://wikitravel.org/en/Jeju#Sleep