In-Depth: RGMII Ethernet Buffering

[DemoniacMilk]

I am pretty amazed by the buffering system of the RGMII Ethernet interface and would like to implement a similar system for another interface, but do not understand a part of the buffering.

What I (mis)understand so far:

The Buffering:

• There is just one big buffer providing memory to store a couple of ethernet packets
• A pointer is pointing at the starting location of each packet
• If the packets location is free/unused, the pointer is put on a stack that holds all pointers to free/unused packets (at program start: all of the pointers are on the stack)
• The buffer_manager (BM from here on) grabs a pointer from the stack and passes a it to the low level driver (lld)
• the lld fills the buffer and notifies the BM when a packet has been received by sending the pointer back over a channel
• BM reacts by trying to grab a free buffer from the stack
  • if there is no free buffer available, the current buffer is sent back to the lld resulting in the packet to be overwritten/dropped
  • If a free buffer is available, this buffer is grabbed and sent to the lld, processing continues
• The BM checks the packets destination MAC address
  • If not for any of our client threads, put the buffer on the stack for free buffers
  • Else add the buffer pointer to a list with pointers of buffers in use

This is where I get a bit confused. I do not understand how the buffer for packets in use works. It seems like the buffer is a ring/fifo type buffer?

There is an additional function doing more filtering (e.g. ethertype). This function grabs one of the pointers to a buffer in use and if the packet does not pass the additional filtering the packet pointer is simply added to the stack with unused buffers. In this case, using a fifo/ring buffer is fine to me.

On passing the additional filtering, there are n Clients that want a packet. The packets pointer is added to the respective clients FIFOs. If the FIFO is full, the packet is dropped for just the client.

The clients get a notification, so they can (at some point) grab the packet. Maybe Client 1 and 3 want the current packet and client 2 wants the packet received next. We don't know when client 1 and 3 have grabbed their Packet, so it is possible for Client 2 to grab its packet earlier? In this case a fifo/ring buffer would end up with a hole, so I am wondering, what the actual buffering mechanism is.

[peter]

RGMII:

Looks like a good summary of the RGMII buffering layer. Well done.

The buffers used by the RGMII layer are optimised for speed and therefore comprise a number of independent buffers for RX (32 by default) and TX (8 high priority, 8 low priority by default). They do not need to be contiguous in memory. The ethernet MAC manages the pointers which always point to a buffer able to receive a maximum sized ethernet packet.

As a result, a buffer can be filled with a packet and then sent to one or more clients. Only once all clients have finished with the buffer will it be added back to the pool of buffers. The pool of available buffers is simply a stack of pointers to the packet buffers. The stack is either empty or contains one or more available packet buffers. The top buffer pointer is popped and returned when a buffer is requested. A pointer is pushed back onto the stack if a buffer is freed.

As a result, the actual location of the packet buffer in memory doesn't matter. If one particular client is particularly slow at consuming packets then the buffer pointer will simply not be in the free buffer pointer stack for longer periods of time.

Hope that helps understand how the RGMII buffering works. Basically, fragmentation is not an issue because of the use of a fixed number of pointers to maximum sized packet buffers.

MII:

The buffering used by the MII ethernet layer is different and optimised for size instead. In the MII case there is one large memory buffer that is carved up into packets as they are received. The advantage is that a small packet will only consume a small amount of the buffer. The disadvantage is that you get fragmentation and the buffer will be seen as completely once the allocation pointer wraps around to the last buffer which has yet to be freed.

[DemoniacMilk]

Thank you for your reply!
The info on the MII buffering is interesting too, I actually did not take a closer look at that.

My mistake was thinking that a buffer pointer is part of of the fifo holding 'used' or the stack holding 'free' pointers at any time.
But there is a third possibility, with the packet not being in any of those two:

• Else add the buffer pointer to a list with pointers of buffers in use (from opening post)
  • This buffer is a FIFO
  • The FIFO is used only to process the data, as in: Determine what client would like the packet
• After processing the packet:
  • No client wanted the packet! Dont use any of the data received and just put the pointer to the packet back onto the stack for free buffers
  • If clients want the packet:
    1. The packets pointer is copied to all client queues (if queue not full)
    2. Information on how many clients got the pointer are stored in a counter within the packet
    3. The pointer is removed from the FIFO
    4. When a client grabs the packet, the counter is decremented
    5. On reaching 0, the packet data is no longer needed and the pointer is put back on the stack for free pointers

Did i get that correctly?
Also, what exactly is the hardware lock used for?

[peter]

I think you have correctly understood the functionality of the buffering.

The hardware locks are used as a semaphore in order to prevent two logical cores (threads) simultaneously modifying state that then becomes inconsistent. This is only necessary when there are multiple parallel contexts that could modify the same state. So, in the case of the RX it needs to be locked to be safe. The TX is done by a single core managing the queue and so no locks are required.

[DemoniacMilk]

I really liked the idea of using a stack holding free buffers and re-wrote a communication interface to use a buffering system inspired by the ethernet buffering.
I did not add any memory locks and sometimes my stack counter value is jumping significantly. I'll add memory locks and see if the problem still occurs.

EDIT: I added memory locks and fixed another bug. the random value changes are fixed now.