Runtime Negotiation of Xlinks

Technical questions regarding the XTC tools and programming with XMOS.
User avatar
Paolomio
Experienced Member
Posts: 64
Joined: Tue Oct 05, 2010 7:33 pm

Runtime Negotiation of Xlinks

Post by Paolomio »

The XMOS development boards, example code, XC compiler, and documentation all assume that Xlinks connect between devices that are all running in a single universe. That is, there is a single XN file, and all the devices are tied together in a configuration that is unchangeable. In this case, every core on each device (or at least the ones that are connected via links) “knows” about the other in advance, and the compiler takes care of all configuration needed to make communication over these links easy and transparent.

But how to build a system in which devices do not know about each other in advance, but must still be able to communicate over Xlinks? An example of such a system might have numerous plug-in boards, each with one or more XMOS devices, interconnected via Xlinks. Since the system configuration could be different every time the system boots (2 wire or 5 wire links, different firmware versions and possibly different functionality on boards, etc.), there is no way the compiler can know how to configure the links, and in fact there is no built-in facility to have XC do this.

Luckily, the XMOS devices do have a mechanism to do exactly this sort of thing, but it takes a bit of programming to make it all work. Since this is the sort of functionality I needed for the product I am working on, and since I chose the XMOS L2 part as the basis of this product because it could do this, I’ve managed—with the infinite patience of Ali Dixon and others at XMOS—to make this work. I’d like to pass this knowledge on to the xcore community so others can use these parts in a run-time configurable setting.

Hardware

The first thing to resolve is the hardware connection. The X1LA link shares some of its pins with the SPI boot flash, if your part is configured to boot from flash. In particular, XS1_PORT_1B is shared between XLA4out and the SPI flash chip select (CS) signal. This means that using a 5-wire link on X1LA is normally not possible…the CS pin of the flash will be toggling madly when there is link traffic, and it will likely start trying to respond via it’s MISO pin. This signal happens to be shared with XLA4in, and thus the link will fail in 5-wire mode.

There is a solution, however. Use a 74HC157 or similar (I use NC7SZ157 for this) as a gate to disable the CS input to the flash. Connect any available single-bit pin on Core0 to the select input of the ‘157, tie the CS input to the Z output, tie I1 high, and tie I0 to XLA4out. By setting the select signal high (I call it ENABLE_XLINK in the code), the flash is cutoff, and the link can be run in 5-wire mode. This same signal can be used to enable external LVDS buffers, if you’re using them to extend the xlink bus to the outside world or over long traces.

Static Forwarding Mode

Normally, Xlinks are not very tolerant of bad connections, incorrect configuration, and garbage on the lines. They’re intended to connect from device to device, and the internal switches provide a huge amount of functionality—routing, packetizing and handling headers, etc. In their normal, fully functional configuration they can’t cope well with, for example, changing configuration from 5 wire to 2 wire mode.

But in their infinite wisdom, XMOS has designed into the switches a mode that is intended for just this sort of noisy, uncertain environment. It’s called Static Forwarding Mode.

In static forwarding mode, the internal routing features of the switch are turned off, and it is configured to connect exactly one channel between two xlinks. At first this doesn’t seem like a very good thing…only one channel? I have 32 channels available, and I can only use one? As you’ll see below, this isn’t the limitation it first seems, but you’ll have to do some programming to get all those channels back.

Before trying to connect two devices, configure them in static forwarding mode using the following function call:

Code: Select all

void ConfigStatic(streaming chanend ch, unsigned int link);
This routine, and all the others referenced here, is in the xlink.xc module I’ve posted on xcore—look in the projects section for one titled “Runtime Negotiation of Xlinks”. Note that the exact same code, in the same way, should be run on the devices on both sides of a link. There is no inherent master/slave relationship here; all devices are equal as far as this code is concerned.

Use a streaming channel to communicate with the outside world. Normal, non-streaming channels are synchronized. That is, one side sends a control token and the other side is expected to receive it. Once that happens, both thread are in sync, and the data can be transferred. At the end of the transfer another control token is used to release the link.

The problem with that in our case is wrapped up in the words “one side…the other side.” Normal channels have a kind of polarity. One of them is always the first to send, the other is required to receive first.

Streaming channel ends don’t do any of this. They are, in a way, raw channels, and it’s up to the two ends to keep track of who’s sending and who’s receiving. They’re perfect for the case where the system doesn’t know anything about the other end of the link until boot time. But it means we have to keep track of who is the sender and who is the receiver.

Link, in the function above, refers to which Xlink to use. 0 means X1LA, and 1 means X1LB. Note that these routines only work on L1 and L2 devices. They will not work at all on a G-series part. If you want to use X1LC or X1LD on a part that has those available, you will need to modify the routines to account for the strange register ordering that is used for the various links (for example, 0x82 is the config register for X1LA, 0x82 for X1LB, but X1LC is 0x80, and X1LD is 0x81.)

Negotiation

Once the links are configured for static forwarding, start the negotiation of a connection between the two sides. Do this using NegotiateXlink():

Code: Select all

int NegotiateXlink(streaming chanend ch, unsigned int link, int fivewire, unsigned int link_delay);
Pass a streaming chanend that will be used to send and receive data from the link (the exact same one used in the call to ConfigStatic() above), the number of the link to use, the width of the link—1 for 5-wire, 0 for 2-wire, and the link_delay in number of cycles between symbols. link_delay can range from 1 (fastest) to 2047 (very slow). NegotiateXlink will return 1 if the link is connected, and 0 if not.

NegotiateXlink() first enables the switch (which is different from setting it to static forwarding—it still hasn’t been enabled yet) and configures it to the speed and width specified. It then tries to send HELLO over and over again until the other side responds with a CREDIT token. For more information on how the link protocol works at this level, see the latest copy of the “XS1-L System Specification”, which you should read anyway if you are working with xlinks in this hands-on way.

Once both sides have sent HELLO and received CREDIT back, they are connected. It is worth testing this connection to make sure everything is working as expected. The NegotiateXlink() function does this by sending and receiving a series of ints over the link and comparing the received data with the expected values. If this works you can be sure the links and channels are configured correctly.

There is one detail, though. If you look at the code you’ll see a call to ClearSwitch(). This function does what its name implies; it clears any garbage that may be in the switch’s fifo. Since there is no way to know if bytes in the fifo are control tokens or data, a select handler is used to select on either data or control tokens. A timer is used to put an end to the process once the fifo is empty.

Normally you wouldn’t need to do this last step when configuring a link. But if you have previously tried to configure the link in, say, 5-wire mode and failed, then you try again in 2-wire mode, the switch may be left with garbage in its fifo. Without a way to clear this out there would be no way to use a connected link (yes, they will connect with garbage in the switch) to send data over a channel.

Note that, since there is no way to know when the other device will be ready to start negotiating an xlink, or (perhaps) what mode or speed it can support, you will need to call NegotiateXlink() in a loop until it is connected to the link you wish to use. Depending on your system you may wish to try 5-wire connections first, then 2-wire--if your hardware may be connected either way—or you may wish to try different speeds for the negotiation.

Using the Link

Once the links are connected you are ready to send and receive data with the other side. This is a “raw” physical-layer connection, and you may want to put a protocol on top to support more functionality. The routines here are already set up to support two simultaneous links (logically “left” and “right”), since the direction maps and channels are set up to do so. So, communicating with two different devices over two links is possible. One use of this is to implement a “soft switch” that emulates some of the direction handling that the hardware switches have when not in static forwarding mode. For example, if each device is given a different ID number, and each packet send starts with this ID, the receiving code can tell if the packet if for it, or should be sent back out the other link.

Additionally, multiple channels can be simulated on top of this implementation by adding a “channel” ID to the packet header. The receiver can then map that ID to a particular channel that is in use internally to send and receive with any channels in any thread. In fact, this could provide more than 32 logical channels for communication across these xlinks; a single byte channel ID would handle 256 logical channels.

A Few Caveats

This all looks easy, and in fact it looked easy (on paper) when I started this work many months ago. But the XMOS Xlinks are a bit like quantum mechanics; if you think you understand them, you probably don’t.

More seriously, it is important to make sure that both devices on either side of a link are ready to start negotiation. Depending on the requirements of your implementation you may need to synchronize both negotiation threads (do this by manually configuring the links using ConfigStatic() and ConfigSSwitch(), then calling HELLO() in a loop until it returns 1) before doing the final negotiation. For example, this may be needed if your system supports 2-wire or 5-wire connections, and either one is possible on any given link, only known at run time.

Also, note that delays inserted either intentionally or accidentally may cause the xlink negotiation to fail (or succeed, and fail without the delays--a personal favorite!) This can happen when printing debug strings before or during the negotiation process. Note also that running via XTAG2, without any debug or console output, may still introduce delays that affect the process. In such cases, synchronizing the two thread before starting negotiation, as outlined above, may solve the problem.

I hope this is helpful to others, and please let me know if you find any problems in the code (full disclosure: this is a simplified version of what I am using, and I may have made errors in the process of making it generally useful.) Additionally, I’d love to see any feedback on this, including ways to improve the code or to correct my explanation.

Paul Messick
Avermetrics


Heater
Respected Member
Posts: 296
Joined: Thu Dec 10, 2009 10:33 pm

Post by Heater »

Paul,

That is one great write up. It's going to need some serious study together with the documentation.

What I'm interested in doing is connecting to a non-xcore device over 2 wire links. The "alien" device could be an FPGA, for example, or perhaps some micro-controller bit banging the lines.

Specifically I have code at the moment that can bit-bang (Tx and Rx) the 2 wire tokens from two cores of the 8-core Propeller MCU from Parallax Inc. This at 5MHz.

Given that initially these MCU/FPGA's will be on different boards and different power supplies it sounds like the techniques you have developed are exactly what I will need. If I remember correctly it is possible to slow the xlink bit rate down enough to match the slow bit-banging device. At least for L devices.

The motivation here is to make use of the link hardware to make this connection rather that dedicating a thread to creating a UART or such like.

I would be grateful to hear if you have any ideas or suggestions to help with these "alien" connections.
User avatar
Folknology
XCore Legend
Posts: 1274
Joined: Thu Dec 10, 2009 10:20 pm
Contact:

Post by Folknology »

Very cool thread, with a great deal of hard work contributed, maybe you should put this on the wiki?

That way it can be added to, amended and grown.

I will definitely take a closer look when I have finished my current work as this would definitely be interesting for the intermodule XS1 interconnects I am working on.


Update you could use the project wiki page for this.
regards
Al
User avatar
Folknology
XCore Legend
Posts: 1274
Joined: Thu Dec 10, 2009 10:20 pm
Contact:

Post by Folknology »

Hmm what is this?

Code: Select all

#pragma select handler
regards
Al
User avatar
Paolomio
Experienced Member
Posts: 64
Joined: Tue Oct 05, 2010 7:33 pm

Post by Paolomio »

Folknology wrote:Hmm what is this?

Code: Select all

#pragma select handler
It's a pragma that tells the compiler the following function should be set up such that it can be used in a select case, just like timers, ports, channels, etc. The first parameter must be a channel or port (or maybe timer as well, can't remember). This is documented in the Tools manual.

Paul
User avatar
Paolomio
Experienced Member
Posts: 64
Joined: Tue Oct 05, 2010 7:33 pm

Post by Paolomio »

Folknology wrote:... maybe you should put this on the wiki?
Update you could use the project wiki page for this.
Good idea...I'll do that later today.
User avatar
Paolomio
Experienced Member
Posts: 64
Joined: Tue Oct 05, 2010 7:33 pm

Post by Paolomio »

Heater wrote: What I'm interested in doing is connecting to a non-xcore device over 2 wire links. The "alien" device could be an FPGA, for example, or perhaps some micro-controller bit banging the lines.
You should be able to do this using some variant of this code. The advantage here over the lightly documented "toaster oven" scheme in XN files is that this code doesn't need to know anything about the outside device, or even if it exists. You will need to fully support the HELLO/CREDIT protocol (this is why this scheme only works on L devices--the G devices do not use the HELLO and CREDIT tokens).

One suggestion: the first thing I would try to do is make a protocol sniffer using your external processor. Use 4 port pins connected to the 2-wire link, all of them inputs. Connect them to a pair of L1's that are connecting over normal xlinks and decode the traffic and output it to a console or other human-readable location. That way you can see what the traffic is for a HELLO/CREDIT, and for channel activity across a link. Note that you'll need to emulate the CREDIT token protocol, and this is also rather lightly documented. When you get a good handle on this it would be a great thing to add to the wiki...

You can indeed really slow the links down. Setting them to a link_delay of 2047 is a symbol delay of more than 20uS, so the 10 symbol byte transfer will take on the order of 400uS/byte, nearly Hayes Modem speeds.

Paul
Heater
Respected Member
Posts: 296
Joined: Thu Dec 10, 2009 10:33 pm

Post by Heater »

Paolomio,

This is most encouraging. This sniffer idea will for sure help and going slowly is always good for initial experiments.

Now I just have to get some L devices....
User avatar
Folknology
XCore Legend
Posts: 1274
Joined: Thu Dec 10, 2009 10:20 pm
Contact:

Post by Folknology »

Thanks, I was aware of select functions but I must have missed select handlers clearly not paying enough attention!

Was it added with the 11.x tools?

Could be useful in a number of cases - oops sorry bad joke ;-)

regards
Al
User avatar
Paolomio
Experienced Member
Posts: 64
Joined: Tue Oct 05, 2010 7:33 pm

Post by Paolomio »

Folknology wrote:Was it added with the 11.x tools?
It's been there for a while (it maybe showed up in 10.4), but it's--as I say--lightly documented. I only found out about it when someone from XMOS sent me a code snippet that did this.
Post Reply