But how to build a system in which devices do not know about each other in advance, but must still be able to communicate over Xlinks? An example of such a system might have numerous plug-in boards, each with one or more XMOS devices, interconnected via Xlinks. Since the system configuration could be different every time the system boots (2 wire or 5 wire links, different firmware versions and possibly different functionality on boards, etc.), there is no way the compiler can know how to configure the links, and in fact there is no built-in facility to have XC do this.
Luckily, the XMOS devices do have a mechanism to do exactly this sort of thing, but it takes a bit of programming to make it all work. Since this is the sort of functionality I needed for the product I am working on, and since I chose the XMOS L2 part as the basis of this product because it could do this, I’ve managed—with the infinite patience of Ali Dixon and others at XMOS—to make this work. I’d like to pass this knowledge on to the xcore community so others can use these parts in a run-time configurable setting.
Hardware
The first thing to resolve is the hardware connection. The X1LA link shares some of its pins with the SPI boot flash, if your part is configured to boot from flash. In particular, XS1_PORT_1B is shared between XLA4out and the SPI flash chip select (CS) signal. This means that using a 5-wire link on X1LA is normally not possible…the CS pin of the flash will be toggling madly when there is link traffic, and it will likely start trying to respond via it’s MISO pin. This signal happens to be shared with XLA4in, and thus the link will fail in 5-wire mode.
There is a solution, however. Use a 74HC157 or similar (I use NC7SZ157 for this) as a gate to disable the CS input to the flash. Connect any available single-bit pin on Core0 to the select input of the ‘157, tie the CS input to the Z output, tie I1 high, and tie I0 to XLA4out. By setting the select signal high (I call it ENABLE_XLINK in the code), the flash is cutoff, and the link can be run in 5-wire mode. This same signal can be used to enable external LVDS buffers, if you’re using them to extend the xlink bus to the outside world or over long traces.
Static Forwarding Mode
Normally, Xlinks are not very tolerant of bad connections, incorrect configuration, and garbage on the lines. They’re intended to connect from device to device, and the internal switches provide a huge amount of functionality—routing, packetizing and handling headers, etc. In their normal, fully functional configuration they can’t cope well with, for example, changing configuration from 5 wire to 2 wire mode.
But in their infinite wisdom, XMOS has designed into the switches a mode that is intended for just this sort of noisy, uncertain environment. It’s called Static Forwarding Mode.
In static forwarding mode, the internal routing features of the switch are turned off, and it is configured to connect exactly one channel between two xlinks. At first this doesn’t seem like a very good thing…only one channel? I have 32 channels available, and I can only use one? As you’ll see below, this isn’t the limitation it first seems, but you’ll have to do some programming to get all those channels back.
Before trying to connect two devices, configure them in static forwarding mode using the following function call:
Code: Select all
void ConfigStatic(streaming chanend ch, unsigned int link);
Use a streaming channel to communicate with the outside world. Normal, non-streaming channels are synchronized. That is, one side sends a control token and the other side is expected to receive it. Once that happens, both thread are in sync, and the data can be transferred. At the end of the transfer another control token is used to release the link.
The problem with that in our case is wrapped up in the words “one side…the other side.” Normal channels have a kind of polarity. One of them is always the first to send, the other is required to receive first.
Streaming channel ends don’t do any of this. They are, in a way, raw channels, and it’s up to the two ends to keep track of who’s sending and who’s receiving. They’re perfect for the case where the system doesn’t know anything about the other end of the link until boot time. But it means we have to keep track of who is the sender and who is the receiver.
Link, in the function above, refers to which Xlink to use. 0 means X1LA, and 1 means X1LB. Note that these routines only work on L1 and L2 devices. They will not work at all on a G-series part. If you want to use X1LC or X1LD on a part that has those available, you will need to modify the routines to account for the strange register ordering that is used for the various links (for example, 0x82 is the config register for X1LA, 0x82 for X1LB, but X1LC is 0x80, and X1LD is 0x81.)
Negotiation
Once the links are configured for static forwarding, start the negotiation of a connection between the two sides. Do this using NegotiateXlink():
Code: Select all
int NegotiateXlink(streaming chanend ch, unsigned int link, int fivewire, unsigned int link_delay);
NegotiateXlink() first enables the switch (which is different from setting it to static forwarding—it still hasn’t been enabled yet) and configures it to the speed and width specified. It then tries to send HELLO over and over again until the other side responds with a CREDIT token. For more information on how the link protocol works at this level, see the latest copy of the “XS1-L System Specification”, which you should read anyway if you are working with xlinks in this hands-on way.
Once both sides have sent HELLO and received CREDIT back, they are connected. It is worth testing this connection to make sure everything is working as expected. The NegotiateXlink() function does this by sending and receiving a series of ints over the link and comparing the received data with the expected values. If this works you can be sure the links and channels are configured correctly.
There is one detail, though. If you look at the code you’ll see a call to ClearSwitch(). This function does what its name implies; it clears any garbage that may be in the switch’s fifo. Since there is no way to know if bytes in the fifo are control tokens or data, a select handler is used to select on either data or control tokens. A timer is used to put an end to the process once the fifo is empty.
Normally you wouldn’t need to do this last step when configuring a link. But if you have previously tried to configure the link in, say, 5-wire mode and failed, then you try again in 2-wire mode, the switch may be left with garbage in its fifo. Without a way to clear this out there would be no way to use a connected link (yes, they will connect with garbage in the switch) to send data over a channel.
Note that, since there is no way to know when the other device will be ready to start negotiating an xlink, or (perhaps) what mode or speed it can support, you will need to call NegotiateXlink() in a loop until it is connected to the link you wish to use. Depending on your system you may wish to try 5-wire connections first, then 2-wire--if your hardware may be connected either way—or you may wish to try different speeds for the negotiation.
Using the Link
Once the links are connected you are ready to send and receive data with the other side. This is a “raw” physical-layer connection, and you may want to put a protocol on top to support more functionality. The routines here are already set up to support two simultaneous links (logically “left” and “right”), since the direction maps and channels are set up to do so. So, communicating with two different devices over two links is possible. One use of this is to implement a “soft switch” that emulates some of the direction handling that the hardware switches have when not in static forwarding mode. For example, if each device is given a different ID number, and each packet send starts with this ID, the receiving code can tell if the packet if for it, or should be sent back out the other link.
Additionally, multiple channels can be simulated on top of this implementation by adding a “channel” ID to the packet header. The receiver can then map that ID to a particular channel that is in use internally to send and receive with any channels in any thread. In fact, this could provide more than 32 logical channels for communication across these xlinks; a single byte channel ID would handle 256 logical channels.
A Few Caveats
This all looks easy, and in fact it looked easy (on paper) when I started this work many months ago. But the XMOS Xlinks are a bit like quantum mechanics; if you think you understand them, you probably don’t.
More seriously, it is important to make sure that both devices on either side of a link are ready to start negotiation. Depending on the requirements of your implementation you may need to synchronize both negotiation threads (do this by manually configuring the links using ConfigStatic() and ConfigSSwitch(), then calling HELLO() in a loop until it returns 1) before doing the final negotiation. For example, this may be needed if your system supports 2-wire or 5-wire connections, and either one is possible on any given link, only known at run time.
Also, note that delays inserted either intentionally or accidentally may cause the xlink negotiation to fail (or succeed, and fail without the delays--a personal favorite!) This can happen when printing debug strings before or during the negotiation process. Note also that running via XTAG2, without any debug or console output, may still introduce delays that affect the process. In such cases, synchronizing the two thread before starting negotiation, as outlined above, may solve the problem.
I hope this is helpful to others, and please let me know if you find any problems in the code (full disclosure: this is a simplified version of what I am using, and I may have made errors in the process of making it generally useful.) Additionally, I’d love to see any feedback on this, including ways to improve the code or to correct my explanation.
Paul Messick
Avermetrics