cannot run GTrXL demo since v0.5.0

[klcheungaj]

• I have marked all applicable categories:
  • exception-raising bug
  • RL algorithm bug
  • system worker bug
  • system utils bug
  • code design/refactor
  • documentation request
  • new feature request
• I have visited the readme and doc
• I have searched through the issue tracker and pr tracker
• I have mentioned version numbers, operating system and environment, where applicable:

they worked fine in v0.4.8 and v0.4.9 but not since v0.5.0

cartpole:

/home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/treevalue/tree/integration/torch.py:21: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  register_for_torch(TreeValue)
/home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/treevalue/tree/integration/torch.py:22: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  register_for_torch(FastTreeValue)
Traceback (most recent call last):
  File "/home/kenny/repo/DI-engine/dizoo/classic_control/cartpole/config/./cartpole_r2d2_gtrxl_config.py", line 78, in <module>
    serial_pipeline((main_config, create_config), seed=0)
  File "/home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/ding/entry/serial_entry.py", line 61, in serial_pipeline
    policy = create_policy(cfg.policy, model=model, enable_field=['learn', 'collect', 'eval', 'command'])
  File "/home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/ding/policy/base_policy.py", line 844, in create_policy
    return POLICY_REGISTRY.build(cfg.type, cfg=cfg, **kwargs)
  File "/home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/ding/utils/registry.py", line 96, in build
    raise e
  File "/home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/ding/utils/registry.py", line 82, in build
    return build_fn(*obj_args, **obj_kwargs)
  File "/home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/ding/policy/base_policy.py", line 134, in __init__
    model = self._create_model(cfg, model)
  File "/home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/ding/policy/base_policy.py", line 219, in _create_model
    return create_model(model_cfg)
  File "/home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/ding/model/common/utils.py", line 31, in create_model
    return MODEL_REGISTRY.build(cfg.pop("type"), **cfg)
  File "/home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/ding/utils/registry.py", line 96, in build
    raise e
  File "/home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/ding/utils/registry.py", line 82, in build
    return build_fn(*obj_args, **obj_kwargs)
  File "/home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/ding/model/template/q_learning.py", line 1117, in __init__
    raise NotImplementedError("not support obs_shape for pre-defined encoder: {}".format(obs_shape))
NotImplementedError: not support obs_shape for pre-defined encoder: 4

Pong

/home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/treevalue/tree/integration/torch.py:21: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  register_for_torch(TreeValue)
/home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/treevalue/tree/integration/torch.py:22: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  register_for_torch(FastTreeValue)
A.L.E: Arcade Learning Environment (version 0.7.5+db37282)
[Powered by Stella]
/home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/gym/core.py:329: DeprecationWarning: WARN: Initializing wrapper in old step API which returns one bool instead of two. It is recommended to set `new_step_api=True` to use new step API. This will be the default behaviour in future.
  deprecation(
/home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/gym/wrappers/step_api_compatibility.py:39: DeprecationWarning: WARN: Initializing environment in old step API which returns one bool instead of two. It is recommended to set `new_step_api=True` to use new step API. This will be the default behaviour in future.
  deprecation(
/home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/gym/utils/passive_env_checker.py:227: DeprecationWarning: WARN: Core environment is written in old step API which returns one bool instead of two. It is recommended to rewrite the environment with new step API. 
  logger.deprecation(
/home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/gym/utils/passive_env_checker.py:233: DeprecationWarning: `np.bool8` is a deprecated alias for `np.bool_`.  (Deprecated NumPy 1.24)
  if not isinstance(done, (bool, np.bool8)):
Traceback (most recent call last):
  File "/home/kenny/repo/DI-engine/dizoo/atari/config/serial/pong/pong_r2d2_gtrxl_config.py", line 89, in <module>
    serial_pipeline([main_config, create_config], seed=0)
  File "/home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/ding/entry/serial_entry.py", line 61, in serial_pipeline
    policy = create_policy(cfg.policy, model=model, enable_field=['learn', 'collect', 'eval', 'command'])
  File "/home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/ding/policy/base_policy.py", line 844, in create_policy
    return POLICY_REGISTRY.build(cfg.type, cfg=cfg, **kwargs)
  File "/home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/ding/utils/registry.py", line 96, in build
    raise e
  File "/home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/ding/utils/registry.py", line 82, in build
    return build_fn(*obj_args, **obj_kwargs)
  File "/home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/ding/policy/base_policy.py", line 161, in __init__
    getattr(self, '_init_' + field)()
  File "/home/kenny/anaconda3/envs/dienginev0.5.0/lib/python3.10/site-packages/ding/policy/r2d2_gtrxl.py", line 355, in _init_collect
    assert 'unroll_len' not in self._cfg.collect, "Use default unroll_len"
AssertionError: Use default unroll_len

[PaParaZz1]

We have fixed this problem in the above commit.

Here is a screenshot example of the final converged agent: