-
Notifications
You must be signed in to change notification settings - Fork 898
Media Driver Operation
This section assumes familiarity with Aeron Terminology.
The mediadriver is a separate process that provides a means to demux and provide buffers of data for Aeron to process from various transmission media. It decouples the means of data transmission from protocol processing. There can be many different mediadrivers for various media, to take advantage of different optimizations (such as Linux epoll and sendmmsg/recvmmsg).
The API/stack (or just API) is the Aeron library that interacts with the application and provides the API as well as the bulk of the protocol processing. The mediadriver and the API/stack communicate via a set of shared memory buffers.
Each Term (which is unique to a Session and unique to a Channel) is associated with a buffer. This is purely for data.
For receiving data, the mediadriver recvs off a socket and must look up Session ID, Channel ID, and Term ID and send that data to a specific buffer.
For sending data, the buffer contains data to be sent to a given socket or communication endpoint. Thus the mediadriver picks up the data and just sends it as there is an easy/direct association between buffer and socket or communication endpoint.
There are two control buffers between the API/stack and the mediadriver. One is for the API/stack to send control Frames and instructions to the mediadriver. The other is for received control Frames and notifications to the mediadriver. These buffers imply no ordering. Events are events and order is not assumed. Loss recovery is not performed on these buffers either.
This means that at any one time, the number of shared memory buffers is:
- 1 per Term ID for recv (incoming data)
- 1 per Term ID for send (outgoing data)
- 1 per mediadriver for control (incoming to API/stack)
- 1 per mediadriver for control (outgoing from API/stack)
How do send buffers get created?
The API creates a new buffer for a new Term ID. This is regardless of old Session and Channel or new Session and Channel.
How do recv buffers get created?
The API explicitly desires data on a given Channel ID. At this point, it does not know about what sources (Session IDs) and Term IDs will appear. But it does know that the API is interested in a specific Channel ID. The API creates a new buffer for the initial Term ID it will begin with. When data arrives on that Channel ID, the Session ID and Term ID are filled in for that buffer. The mediadriver will create new buffers for Terms as needed on this Channel ID. If a new Session ID is seen, it is handled as a new Term would be and the mediadriver creates it.
When a new send or recv buffer is created, a control message for mediadriver <—> API/stack must be sent to let the other side know of the new channel and act appropriately.
A typical mediadriver has 3 threads.
Conductor Thread:
- This reads from the incoming control buffer setting up and tearing down new sessions, channels and receivers. This thread also deals with detecting loss, handling NAK state, and retransmitting based on received NAKs.
- Manages the allocation of buffers.
- Sends errors or acknowledgement messages on the outgoing control buffer.
- Maintains internal mapping of term buffers, and sends information about new term buffers and their mapping to the receiver and sender threads.
- Looks for loss in all Term buffers on the receive side
- Looks for NAKs it needs to retransmit for, grab data and send retransmit
- Tracks recent retransmits and ignores new NAKs.
Receiver Thread:
- Reads data and control messages from the sockets.
- Forwards along data onto the receiver buffer that corresponds to the Channel ID/Session ID/Term ID triple.
- Receives SMs for Senders, needs to map SessionID/ChannelID/TermID to a sender thread Term buffer and update flow control state for Sender Thread.
- Receives NAKs for Senders, needs to map SessionID/ChannelID/TermID to a sender thread Term buffer and update NAK handling state for Admin Thread.
Sender Thread:
- Reads data buffers from producers and sends data out on sockets.
- If it can see data to send it maps from Term buffer to send socket. This is setup by AdminThread.
- It is informed of new Term buffers by Admin Thread.
- Tracks per ChannelID flow control state, updated by Receiver Thread when it receives an SM.
Communication between threads is in the form of MPSC ring buffers. These are the same data structures used for the control and data buffers between the core and the mediadriver but use in process bytebuffers for memory rather than shared memory IPC.
Suggestion of having the end of a Term have a flag point to the next Term. This can work in both API -> mediadriver and mediadriver -> API directions for Term rollover.
The mediadriver and API will need to know about new Term buffers. New Session IDs, Channel IDs, etc. are, in essence, just new Term buffers.
Data reception is probably the more complex case. New Terms need to be created when new Sources (i.e. new Session IDs) are seen with interesting Channel IDs. I think we have to have the ability for the mediadriver to create new Term buffers and the API to notice them.
Operation: mediadriver controls buffer creation for data reception
- mediadriver is instructed by the API what it is interested in. This includes Channel IDs and Transmission Media specifics (such as IP addresses and ports to listen on for UDP).
- mediadriver discards any data that does not match. e.g. an uninteresting Channel ID.
- mediadriver creates new Term buffers as needed.
- mediadriver signals new Terms to the API via Term rollover flags, Term index buffer, etc.
Data sending is easier. The API creates static Terms for each Channel it will send on. When they have to roll to new Terms, the mediadriver will know via the rollover flag.
The API and the mediadriver communicate with one another via the control buffer. Here is a list of commands that need to be exchangeable between the API and mediadriver.
| Message | Parameters | Description |
|---|---|---|
| Add Receiver | Destination String, List of Channel IDs | Have mediadriver set up state and listen to a destination and a given set of Channel IDs. Generates a Location Response by the mediadriver. |
| Add Channel | Destination String, Session ID, Channel ID | Have mediadriver set up state to be able to send on a Channel within a Session. Generates a Location Response by the mediadriver. |
| Remove Receiver | Destination String, List of Channel IDs | Have mediadriver stop listening to a given destination string for the API. Removes all Channel IDs and Terms as well. |
| Remove Source | Destination String, Session ID | Have mediadriver tear down state and forget about sending on a given Session, its Channels, and its Terms totally |
| Remove Channel | Session ID, Channel ID | Have mediadriver tear down state and forget about sending on a given Channel and its Terms totally. But keep Session viable. |
| Remove Term | Session ID, Channel ID, Term ID | Have mediadriver stop allowing a given Term ID to be recoverable. Once created by the API, Terms stay around until they are removed by the API. |
| Request Term | Session ID, Channel ID, Term ID, Destination String | Media driver should setup state for the next Term Buffer |
| Message | Parameters | Description |
|---|---|---|
| Error Response | Code (int), String | Error Response to previous command from API. Indicates command can't be fulfilled. |
| Error Notification | Code (int), String | Notification of an error condition in the mediadriver to the API |
| New Receive Buffer Notification | Session ID, Channel ID, Term ID, Destination String | Notifications of new buffer(s) due to new Sessions/Sources for receiving data |
| New Sender Buffer Notification | Session ID, Channel ID, Term ID, Destination String | Notifications of new buffer(s) due to new Sessions/Sources for receiving data |
The Core Stack also spins up an admin thread which communicates with the admin thread in the mediadriver via the control protocol. The Receiver and Sender integrate into the existing application's threading model so don't run on their own threads. The receiver and sender communicate with the admin thread by sending messages to its command buffer.
Conductor Thread:
- Receives information about how fast data is being written into channels via the command buffer. It then calculates the rate and remaining time before a term buffer is exhausted. When the term buffer is expected to be exhausted soon the admin thread sends out a message to request a new term buffer across its outgoing control buffer.
- Receives responses when new term buffers are mapped by the mediadriver. These are then communicated to the receiver and sender.
We can rendezvous the new term ids to the receiver and sender by hashing into an array of term buffers which are updated.