"transparent" ethernet bridge

[DemoniacMilk]

The idea is to connect a PC to a router with a setup similar to
(internet)-Router --(eth)-- xcoreA --(serial bus)-- xcoreB --(eth)-- PC
where the xcores are invisible for router and pc (they just forward all the packets without any modification).

A thread connects ethernet and serial bus by simply receiving the packet of one bus to transmit it on the other.

The serial bus is self written and tested. It allows a maximum data rate of about 19,5 MBit/s (full-duplex, 20 MHz clock) with a packet error rate of <0,01%.

Using wireshark, I can confirm that packets artificially generated on xcoreA are received correctly on the PC, so the bridge seems to work as intended. The PC is able to establish an internet connection.
However, when running a speedtest or trying to open a web page, I end up with a high amount of packet retransmits. Most websites fail to load, just lightweight sites like the google start page are okay to load.

Running a speed test using the setup shown above results in 5 MBit/s download and 2,5 MBit/s updload (t-online) or 1 MBit/s download and 0,2 MBit/s upload (Ookla). In both cases, Wireshark displays a ton of packet retransmits.
(Internet connection: 50 Mbits/s down 20 Mbit/s up on both tests).

I wonder, what might cause all the retransmits?

[peter]

It could well be the bursty nature of the traffic being sent from the internet. It is probably able to create very large amounts of data in a short time which need to be buffered somewhere. The PC will probably be advertising a very large TCP window that the server providing the website will therefore believe can be buffered. However, depending on which ethernet MAC you are using in software on the xCORE there might not be very much buffering allocated.

If you try increasing the buffering of the ethernet MAC you are using on both ends of the serial link then do you see better performance?

Which ethernet MAC are you using? You can define your own ethernet_conf.h file in you application to override the defaults that are in default_ethernet_conf.h

[DemoniacMilk]

that is a nice hint, i will check that out.
Im using the gig-ethernet library. I will try to increase the buffer size tomorrow.

[DemoniacMilk]

I increased the RX buffer size from 32 to 120 (close to max memory) and the client qeue size from 8 to 32. Cannot feel or measure a difference.
I added prints (xscope) as well to see when a packet is dropped due no buffer being available in a) the client qeue or b) the general buffer system.
I tried to download xTIMEcomposer over the bridge, but the download fails (gets stuck at 0..100 KByte downloaded)

[peter]

I guess the next thing I would try is to add xscope probes to determine where packets are getting to and being dropped.

In order to do this I would create a config.xscope file in your project (in the src/ folder or with the Makefile).

Code: Select all

<?xml version="1.0" encoding="UTF-8"?>
<xSCOPEconfig ioMode="basic" enabled="true">
    <Probe name="packet rx" type="CONTINUOUS" datatype="INT" units="Value" enabled="true"/>
    <Probe name="mac filter" type="CONTINUOUS" datatype="INT" units="Value" enabled="true"/>
    <Probe name="no free buffers" type="CONTINUOUS" datatype="INT" units="Value" enabled="true"/>
    <Probe name="client queue full" type="CONTINUOUS" datatype="INT" units="Value" enabled="true"/>
</xSCOPEconfig>

and then annotate RGMII source with (note you'll need to include xscope.h) and not the probe names are simply the capitalisation of the name in the config.xscope with spaces changed to

rgmii_buffering.xc:213

Code: Select all

      case c_rx :> uintptr_t buffer :
        xscope_int(PACKET_RX, buffer);  // Packet received from MAC
        // Get the next available buffer
        uintptr_t next_buffer = (uintptr_t)buffers_free_take(free_buffers, 1);

rgmii_buffering.xc:237

Code: Select all

          else {
            // Drop the packet
            buffers_free_add(free_buffers, (mii_packet_t *)buffer, 1);
            xscope_int(MAC_FILTER, buffer); // Filtered by MAC lookup
          }

rgmii_buffering.xc:241

Code: Select all

        else {
          // There are no buffers available. Drop this packet and reuse buffer.
          c_rx <: buffer;
          xscope_int(NO_FREE_BUFFERS, buffer);
        }

rgmii_buffering.xc:309

Code: Select all

        } else {
          xscope_int(CLIENT_QUEUE_FULL, buffer);
          client_state.dropped_pkt_cnt += 1;
        }

And so on for the cases where the packet is dropped:
- line 318: no client wanted the packet
- line 336: run out of HP buffers

If you then run your application with --xscope then it will create an xscope.xmt file which you can load into the xTIMEcomposer -> Trace -> Offline xSCOPE viewer to see where the packets are going. Whether they are being correctly received by the MAC layer at least, and if so which point do they get to after that.

[DemoniacMilk]

I added xscope outputs for

• packet received in buffer manager
• no buffer available in buffer manager
• packet filtered by mac filter (the mac filter is customized and should forward all packets to the bridge)
• client queue full
• no hp buffers available
• lp packets dropped
• bridge received eth packet

On loading the generated xscope file, I can only see the traces for "packet received in buffer manager" and "bridge received eth packet", the other traces are not shown (so i guess they contain no data). This is the case for both xcores used.

Example record:

[peter]

So, it looks like there is no dropping in the ethernet layer - which means the application is keeping up and simply dropping all the data over the serial link.

Thinking about it though, I'm not sure whether what you are trying to do can work at all. You are effectively trying to make a wire that should be carrying 1GB of data go over a 20MB link. This equates to a wire which is 98% lossy, when all the systems and protocols are designed to work in systems where the data error rate is very small.

The following link has the equation on page 15 and diagram on page 16 of the data rates you will get with links down to 98% reliable (you should be able to extrapolate down to 2% reliable): https://www.mef.net/Assets/White_Papers ... _-_v14.pdf

I think that explains it,

Peter

[DemoniacMilk]

Throughtput calculations with 40 ms RTT and 98% loss result in a theoretical throughput of 0,29 MBit/s if im not mistaken. The question is: can you really look at it as a 98% lossy wire? Or is it 60%, as the internet speed is 50 Mbit/s? (doesnt really matter if 98% or 60% loss rate, both horrible). How do the speed tests measure data rates of 1 Mbit/s and 5 Mbit/s (ping ~20ms), what is more than calculated.

When changing the RTT to 1 ms on 98% loss rate, theoretical throughput increases to 11 MBit/s, so on diretly connecting 2 PCs using the bridge, I should be able to achieve a throughput similar to this I think? Measurements with iperf still result in a data throughput of about 500 kbit/s only.

The GBit Eth Library didnt show any signs of buffer overflows, meaning the ethernet library was able to receive all packets and the eth-to-serial bridge thread was able to grab and send the packets before any buffers were full.
In case all buffers are fully filled (in total ~130 packets can be saved in serial TX and eth RX buffers) with 1500 byte packets, transmitting all of them would take 78 ms. I am not sure what the time to live usually is set to and if the buffer delay might be a problem.

[DemoniacMilk]

Okay, I am really confused now.

I adjusted the xscope outputs to output counter values for received packets, dropped packets etc.
This is the result (PACKET_RX: ethernet packets received in buffer manager, bridge rx: number of packets that have been received from the ethernet to serial bridge, bridge tx: number of packets forwarded by the bridge - same value as bridge rx):

ETH_RX_BRIDGE_RX_COUNTS_XSCOPE.PNG

That is .. pretty interesting. I do not understand how the bridge receives twice as many packets as the ethernet library.

The bridge simply listens to "packet_ready()" notifications set by the ethernet library, then grabs the packet and sends it.