Multi-agent NMF

[FoxYouFR]

Copy of a Slack discussion with @sibocw, so everyone can participate and be referred to

Conceptualizing the modifications to make in order to have multiple flies in the same simulation

• Florian

Hi Sibo! I'm working with Barbara (team social) and one of our first goals would be to be able to have two flies in the simulation interacting (probably not touching to avoid weird physics issues). I read a bit the code, and it seems like the fly and the environment are being treated as one single "object". I'm thinking it would be great if I try to separate the actors (fly) from the environment, and thus allowing the use of as many flies as wished by the user. Would you agree that this is a good first step to get to a multi-agent environment? Also, this brings the question of how the end user will create such an environment: would she create the flies herself, or simply ask the environment to create X flies (i.e. what will the end user API look like)? What are your thoughts on this strategy?

• Sibo

Hi Florian,I think you are totally on the right track to think about how to add multiple flies to the same environment. Here’s my thought on how to go about this: in the end everything has to live in one simulated world — there’s only one physics (and therefore only one mujoco Physics object). The fly and objects in the environment are all part of this world. In fact, the fly is attached to the root world object (worldbody) using a free joint (all 6 DoFs are free). You can see how this is done here. I would use this as a starting point and change the API so that the user can add more flies.What are your thoughts about the API for having multiple flies? I can imagine something like this would be the most intuitive:

arena = FlatArena(...)
fly_1 = NeuroMechFly(arena=arena, spawn_pos=spawn_pos_1)
fly_2 = NeuroMechFly(arena=arena, spawn_pos=spawn_pos_2)

but then it’s unclear what the user should call .step(), .reset(), .render() etc on, so maybe it’s a bad idea. Alternatively we can do something like this:

arena = FlatArena(...)
sim = MultiAnimalNeuroMechFly(
    arena=arena,
    spawn_positions=[spawn_pos_1, spawn_pos_2],
    ...
)

I’d be curious to hear your thoughts.

• Florian

I think the first proposition is great, as it is easy to add as many flies as wished like that. With the second proposition, we would need to return the simulation as an all-knowing object with access to all agents/flies, or directly all inside of those as a tuple:
env, arena, flies = MultiAnimalNeuroMechFly(...)

I'd however add the arena as a property of the environment, since it is common to all flies (right?), along with the list of flies, something like that:

arena = FlatArena(...)
fly_1 = NeuroMechFly(spawn_pos=spawn_pos_1, ...)
fly_2 = NeuroMechFly(spawn_pos=spawn_pos_2, ...)
sim = NMFSimulation(arena=arena, flies=[fly_1, fly_2], ...)

It also allows easier manipulation of flies during the simulation, e.g. allowing vision at a certain point. Also, flies can have different capabilities (vision/olfaction/...).

• Sibo

sounds great, go ahead!

• Florian

Hi, I'm coming back to you with an update on what I've been doing these last few days. I've been trying to separate the agent/model from the environment. I split the "core" module in two: agent and simulation. The agent is a class (NMFAgent) that should contain everything related to the model (that is, the object that is used by the spawn_entity method). This class contains functions such as _add_body_sensors, _add_adhesion_actuators and _set_compliant_tarsus and all attributes that are specific to a single fly. All flies are instanciated by the simulation (a class that I called NMFSimulation, extending gym.Env), not by the user. The user creates flies using the NeuroMechFly class (for the moment, a dataclass for simplicity). I designed it so that the simulation can easily give general parameters to the flies, such as the self.physics and self.sim_params. The simulation is responsible for the camera and the MuJoCo functions (step, reset, render, ...), among others. This is where it's getting complicated:

• The camera for the moment is an attribute of the model, but this would probably not be the case anymore in multi-agent scenarios: how should that be handled?
• The floor contact points depend on the fly friction, which could be different for all flies (unless we specify it for the whole simulation)
• Overall, I think the collision handling is going to be quite hard, I'm not sure on how the simulation is going to know which fly is impacted by each collision
• What should the observation/action spaces look like for multi-agent?

We will talk with the professor (and you?) on Monday on the scenario we'd like to create. It might not be necessary for us to have a multi-agent (one fly and one fake model could be fine), but it's definitively more interesting if the two flies are actually simulated (and cooler), but also more complicated and risky. Let me know your thoughts, and if I'm heading to the right direction or if I'm making things more complicated than what it should be. Thanks!

• Sibo

Yes I think you will get more answers from Victor on Monday regarding these.

• Re cameras: There are two types of cameras: tracking cameras that follow a fly (which I guess should be part of the model/fly), and global cameras that are just fixed to a certain place in the arena (which I guess should be a part of the simulation). Victor has thought about this a bit more so I would encourage you to talk with him on Monday.
• Re friction: the floor and the fly body parts all have their own friction parameters, so in principle it should be possible to have different friction parameters in different flies. While I can’t really think of a reason why we would want different friction parameters across flies for now, it could be useful later this year when MuJoCo releases their GPU-accelerated version — we might want to run many flies with different physics parameters in parallel to do a parameter sweep,
• Re multi-agent observation/action space: I can imagine that it will be a dictionary mapping the fly id to its action as if there were just one fly. Please talk to Pavan and Alfred about this on Monday.

End of transcript