Was just there this weekend...
Downtown is great for wandering about on foot. Avoid W. Baltimore and areas to the E. of Fells Point. Apparently there are some bad spots N. of Inner Harbor as well (not a local - this is anecdotal).
We took a stroll from Inner Harbor, up Light St. to Federal Hill (lots of interesting restaurants and bars - although not quite the density as Fells Point). We then took a left on Fort St. and walked about 2 miles straight down to Ft. McHenry, which is definitely worth a visit. Fort passes through residential areas. There's a free bus loop with a stop just outside the fort, and will take you back downtown. You could also wander back via the waterfront if you want to complete the loop on foot.
The water taxis are fun, routes from Inner Harbor to Fells Point, Fort McH, and other points of interest (I think they start running April 1).
Somewhat related suggestion - if you haven't seen "The Wire" on HBO, I would recommend it under any circumstances (my favorite HBO series ever), but it might be a good time to watch it, before going. Of course, it is a gritty, fictional crime series and doesn't reflect the tourist experience (hopefully), but it does sort of give you a flavor for the city (shot in Balt, lots of local references).