Advancing Bandit Optimization for Generalized Structures: Algorithms and Theory