The Old City, which is adjacent to Banglamphu on the north and Thonburi across the Chao Phraya River on the west, is the historic heart of Bangkok where you can find most of the ancient buildings and the major tourist attractions. Because of the city's conscious decision to preserve this historic area, it's the largest part of Bangkok to escape constant transformation. Much of the residential sections look run down, but the whole area is safe, and it's one of the best in the city for a stroll.
The Grand Palace and other major sights are within a short distance of the river and close to where express boats stop. There is a Skytrain station next to a boat pier downriver from the Old City as well, making the Chao Phraya convenient no matter where you're staying.
Of course, the magnificence of the sights, and the ease in reaching them, make them rather crowded. During rainstorms is about the only time you find few people at the sights. Arrive at the Grand Palace early in the morning, when the weather is cooler and the crowds are more manageable.