Indoor localization solutions are constrained by infrastructure requirements, privacy concerns, or accumulating drift. We present GraphWalker, a system that aligns commodity smartphone IMU trajectories to semantic floor plan graphs through constrained beam search. GraphWalker operates on coarse topological building representations where nodes encode semantic locations and edges represent traversable connections, requiring no WiFi fingerprinting, Bluetooth beacons, or dense sensor networks. The system transforms IMU-derived trajectories through coordinate alignment, then employs iterative beam search to maintain and score multiple path hypotheses based on topological constraints and heading consistency. Evaluation demonstrates 45% reduction in endpoint localization error (from 11.5 to 6.3 meters) and 35% reduction in graph distance compared to direct IMU projection, with median along-trajectory error of 1.4 meters. These results establish semantic graph-constrained localization as a scalable solution bridging infrastructure-free inertial tracking and high-precision infrastructure-dependent systems