Direct Pose Estimation and Refinement