[Question] Is the length of trajectory (episode) controlled by the done in step() function?

[YimengZhang94]

Question

In RL, a trajectory (or episode) is a sequence of states and actions in the world, see the following link for more explanations:
https://spinningup.openai.com/en/latest/spinningup/rl_intro.html
In SB3, is the length of the trajectory (episode) controlled by the done in step() function?
If I have an infinite horizon, then the done is always False during all total_timesteps in the model.learn() function, right? If I have a finite horizon, I just need to set the done = True when one trajectory (episode) ends and the model.learn() function will identify it automatically, right? Thanks in advance.

Checklist

• I have read the documentation (required)
• I have checked that there is no similar issue in the repo (required)

[Miffyli]

Yes, exactly! SB3 follows the definitions of Gym, where indeed done=True means end of an episode. Note that the rollouts (collecting samples) does not depend on episode lengths, unless you explicitly set it so in some of the algorithms.

Closing as resolved. Please refer to the docs for further information :)

[araffin]

Hello,

For infinite horizon, I recommend you taking a look at #284 and #633 .
One way to deal with that is having a timeout (setting done=True) but telling the agent to ignore that termination and treat the problem as infinite horizon by providing info["TimeLimit.truncated"] = True (done automatically by the TimeLimit wrapper).