Need help in understanding PPO Lag

[jyao97]

Hi folks,

I am researching PPO Lag and am trying to understand the Lagrange relaxation on the constraints. I think the original optimization goal of PPO is to maximize the advantage function, but when considering cost control, the cumulative sum of the constraints seems similar to state values or action values. i.e. $max A(s, a) \quad s.t. \sum c < d$. The direct relaxation should be like $A(s, a) - \lambda (\sum c -d)$, which seems meaningless. How can I move the penalty on costs to the advantage item?

Thanks a lot!