Inertial indoor localization is attractive because it requires no infrastructure and reveals no identifying signals, yet unbounded drift and the absence of a global coordinate frame prevent it from serving as a standalone solution. Existing map-constrained approaches mitigate drift using floor plans, but typically rely on dense geometric maps, calibrated probabilistic models, or offline global inference. We present GraphWalker, a real-time indoor localization system that aligns IMU-derived trajectories with coarse semantic floor plan graphs through online trajectory-to-graph alignment. GraphWalker maintains multiple candidate path hypotheses through bounded beam search and independently corrects drift for each hypothesis, requiring only a coarse semantic graph, a known starting pose, and commodity inertial sensing. On 152 real-world trajectories totaling 9 km of walking across three floor plans, GraphWalker improves localization accuracy by up to 50% over map-free and map-constrained baselines and reduces path errors by 42%, while maintaining real-time operation.