Perpetuating Inequality in Pursuit of Diversity