RGMII Buffering Bug

Technical questions regarding the XTC tools and programming with XMOS.
Post Reply
maxwinkel
Member
Posts: 9
Joined: Wed May 03, 2017 8:40 am

RGMII Buffering Bug

Post by maxwinkel »

Hi all,

I'm working with the RGMII interface of the XCore200 with the lib_ethernet library. So far, my application is now working pretty well. But...

I've found, that if I send 64 byte packets at full line rate to the interface, suddenly my user application does not get any packets from the lib_ethernet. It gets the rx_ready notification for every packet, but once it tries to pick them up via get_packet() it gets ETH_NO_DATA. The reason is, that the buffer in the clients queue gets freed by drop_lp_packets() after sending the notification, but before the packet can be picked up:

When running with only one LP client (no HP clients) as I do, you have three buffer "pools":
- A stack of free buffers, available to store packets in.
- A queue of used buffers where packets have already been stored to.
- A client queue.

The rgmii_buffer_manager() process takes the buffers from the "free stack" and inserts them to the "used queue" after the lld process has written a packet to it. Then, the handle_incoming_packet() function, called within the rgmii_ethernet_rx_server() takes the packets from the "used queue", inserts them to the "client queue" and sends out the "rx_ready" notification. Since the handle_incoming_packet() function only handles one packet per call and its called within the rgmii_ethernet_rx_server() main loop only once per iteration, it's quite slow. The rgmii_buffer_manager() in contrast is very fast - able to operate at full line rate. So, virtually immediately you'll end up with an empty "free stack", a full "used queue" and none or only one packet in the "client queue". After calling handle_incoming_packet(), which transfers one packet from the used queue to the client queue and notifies the client, the rx_server checks if sufficient buffers are available in the "free stack". If not, it starts freeing packets from the client queue(s):

at the end of rgmii_ethernet_rx_server():

Code: Select all

    if (buffers_free_available(free_buffers) <= RGMII_RX_BUFFERS_THRESHOLD) {
      drop_lp_packets(client_state_lp, n_rx_lp, used_buffers_rx_lp, free_buffers);
    }
This will remove the one packet from the client queue making it unavailble for the client to pick it up and the "get_packet()" call fails.

My workaround was to remove these three lines, which works for me, because I'm not using a HP client which needs access to available buffers. So far, I was not able to come up with a general purpose solution working with more than one LP and possible a HP client. For me, the problem is solved now, but perhaps somebody else will have the same problem, so I thought I'll share my investigations with you.

Best regards,

Max


peter
XCore Addict
Posts: 230
Joined: Wed Mar 10, 2010 12:46 pm

Post by peter »

Hi Max,

The intention is to preserve high priority traffic over the low priority traffic by ensuring there is buffer space available for high priority traffic at all times (assuming the high priority traffic processing can be done in time). The intention is that once a low priority client has been notified, if the packet is dropped before it is collected then the rgmii_ethernet_rx_server should simply return ETH_NO_DATA:

Code: Select all

      case i_rx_lp[int i].get_packet(ethernet_packet_info_t &desc, char data[n], unsigned n):
        ...
        else {
          desc.type = ETH_NO_DATA;
        }
        break;
You are right that these three lines should probably only be active when there is a high priority queue active, otherwise this code is problematic. It should probably look like:

Code: Select all

 if (!isnull(c_rx_hp)) {
   if (buffers_free_available(free_buffers) <= RGMII_RX_BUFFERS_THRESHOLD) {
      drop_lp_packets(client_state_lp, n_rx_lp, used_buffers_rx_lp, free_buffers);
    }
  }
Regards,

Peter
peter
XCore Addict
Posts: 230
Joined: Wed Mar 10, 2010 12:46 pm

Post by peter »

I have added an issue against lib_ethernet (https://github.com/xmos/lib_ethernet/issues/17)
maxwinkel
Member
Posts: 9
Joined: Wed May 03, 2017 8:40 am

Post by maxwinkel »

Hi Peter,

I agree, your patch will work if the HP queue is not used. But I'm afraid in the case, that the HP queue is used, the problem will persist if the LP client is too slow picking up the packets. It should be assured, that the client queue does not run empty. If free packet descriptors are needed for a HP client, they should be picked from the used queue instead of the client queues, if possible.

Best regards,

Max
Post Reply