Theoretical Advances in Reinforcement Learning: Online Average-Reward and Offline Constrained Settings