Is XC a realistic choice for real world applications?

Technical questions regarding the XTC tools and programming with XMOS.
User avatar
Folknology
XCore Legend
Posts: 1274
Joined: Thu Dec 10, 2009 10:20 pm
Contact:

Is XC a realistic choice for real world applications?

Post by Folknology »

You may have noticed my XC advocacy over choices such as C or C++ on Xmos for problem solving, I love the idea of CSP and an event driven cores, I like the cleanness and the safety it provides. Yet recently I have begun to experience creeping doubts, this has culminated in a personal crisis of belief in XC this week. Let me tell you a story..

As some of you may know part of the Amino project and thus hardware projects related to that such as NetStamp requires a TCP/IP networking stack. Thus over recent weeks I have been getting my head into Xmos's TCP/IP stack (version 1.3). This isn't the first time I have looked at it, but I hadn't taken a deep look at it since an early version. This stack is based around the opensource uIP written by Adam Dunkels which differs considerably from the earlier version based on the opensource LwIP which is an uIP derivative. Although I do not have the details why the switch was made I believe it's to enable a smaller footprint stack (uIP has lower code/data overhead ). I won't focus too much on that decision here as that's worthy of another thread entirely.

Xmos has added x_tcp ontop of uIP to enable XC usage of uIP, it also enables multicore support by adding channel based communication. Unfortunately this process makes the simple uIP interface much more difficult to understand, to experience this compare Xmos tcp/ip application examples to those provided with uIP. It is clear that the uIP examples are very simple to grasp and get started with. Of course we are ignoring the fact here that Xtcp enables XC's power to bear down on writing tcp/ip applications. However I have been singularly unable to write a 'pure' XC tcp/ip application using xtcp! Its not that I am unfamiliar with writing TCP/IP apps, I have been doing that for years. NO it seems that its not possible to write a TCP/IP app in XC without dropping down into 'C'.

1) So am I being stupid or is it possible to write pure XC based TCP/IP applications?
2) Why has Xtcp been written in such a way that makes writting TCP/IP apps more complex than the uIP examples?

Both 1 and 2 are exibited from the examples provided in the stack (1v3).

Of course Xtcp is only the only the top layer of Xmos's networking stack, I was gasping at the bit to get down low level to the Ethernet/Mii layers to see XC and Xmos strutting their stuff. My first insight of the Ethernet side of things came a while back when reading Xmos's XC book, it uses Ethernet Mii transmit and receive protocols as examples of clocked/strobed I/O interfaces, a chapter I particularly enjoyed! Here I thought would be Xmos playing on their home ground, Mii is a protocol/interface built for the XS1/XC combination right? How wrong could I have been, gone were the clean XC event structures and streaming channel employment, replaced instead by the most confusing files of code I have had the displeasure to try to fathom in the last decade! I have been through these undocumented lines of code many times, I am still unsure as to which language style they are written. It is clearly not XC and CSP based that's for sure, it also does not appear to match familiar 'c' based patterns that I can understand, it switches between C and XC and completely and utterly defies understanding, despite hours, no days of analysis.

3) Thus I believe a real world application like Mii Ethernet interfacing (quite common) cannot be solved with Xmos finest XS1 and XC. rather one has to resort to some unintelligible gobbledegook code as illustrated in the current Ethernet part of the networking stack. I feel quite dirty having spent significant time inside that code..

QED XC is not a realistic choice for real world applications, courtesy of Xmos's own large body of networking code..

or is there an alternative explanation, I eagerly await superior knowledge from above.

I am sure others have also tried to understand this code, please add your feedback here.

(This is an unmasked cry for help)

regards
Al


User avatar
leon_heller
XCore Expert
Posts: 546
Joined: Thu Dec 10, 2009 10:41 pm
Location: St. Leonards-on-Sea, E. Sussex, UK.
Contact:

Post by leon_heller »

It might just be down to the person who implemented it. I had a similar experience where I used to work - I had to design the hardware and write the software for a large embedded application that had been prototyped on a PC. I simply couldn't understand the code I was given, it was so bad that I couldn't even create a Jackson diagram showing how the code was structured. I refused to do any more work on the software until the code was rewritten, which took about a month. I then had no problems porting it to my hardware. Perhaps XMOS needs to do the same with that application.
User avatar
Berni
Respected Member
Posts: 363
Joined: Thu Dec 10, 2009 10:17 pm
Contact:

Post by Berni »

Yes it is a lot of work to port some TCP/IP stack in to XC (Lack of pointers can be quite a pain)

I was fed up with xtcp being really clumsy and hard to use so i riped it out and wrote my own until i found out that the uIP port keeps locking up and gave up on it since i didn't feal like digging trough the entire uIP code to find why it locks up.It made me quite angry since i already had my own XC layer up to a working state.

Also the FAT library thing i don't find all that great aether so i had to port my own.(But its fine for simple stuff)
User avatar
Andy
Respected Member
Posts: 279
Joined: Fri Dec 11, 2009 1:34 pm

Post by Andy »

Folknology wrote: QED XC is not a realistic choice for real world applications, courtesy of Xmos's own large body of networking code..
I don't think this is fair. The Ethernet MAC code was clearly written some time ago when the tools were in their infancy and uses some XC language features that are no longer supported. This obviously doesn't help the case for the quality of the software module but I don't think it should reflect on XC itself.

If anything, the reason for the unorthodox code (shared memory etc) is to squeeze as much performance out of the hardware as possible... after all, it's a high performance protocol. I suspect it would be quite possible to write a MAC in very clean XC which is easier to read but a bit slower / uses more memory. Maybe this is something XMOS should do.

It's clear that uIP has shortcomings. It was written for 8bit micros and uses its own horrible multi-tasking, which makes it quite unsuitable for direct use with XC. Perhaps what we need is an initiative to start writing a purely XC TCP/IP stack.
User avatar
Berni
Respected Member
Posts: 363
Joined: Thu Dec 10, 2009 10:17 pm
Contact:

Post by Berni »

Well sometimes you need to use code written for 8bit MCUs because its designed to be memory efficient. The lvIP stack just barely fits in to one core (MAC code is on another core) and something like a mp3 decoder also (I think they needed to get rid of some features to make it fit)

So it comes mainly down to those 64KB of RAM. Most of the memory is used to store the actual program and lots of other high performance micros have more than 128KB of flash for program memory to keep the code out of the small 1 to 16KB of ram. But i guess the xmos runs way too fast to keep reading code from flash and would need some prefetch thing that would break the deterministic timing.

What i found is that xmos is really great for interfacing stuff but when you want to do ethernet, large displays, serious number crunching etc its best to just take an ARM.
User avatar
Folknology
XCore Legend
Posts: 1274
Joined: Thu Dec 10, 2009 10:20 pm
Contact:

Post by Folknology »

Perhaps what we need is an initiative to start writing a purely XC TCP/IP stack.
Well perhaps this is the initiative for you to start a new project on Xcore - XC/TCP/IP.

But there are some basic constraints, right now the network stack will run on a single core on an L1 and you can squeese a basic tcp/ip app into the tcp/ip thread assuming low memory app like say for instance telnetd to query the XS1 status. This is should be the basic benchmark bench mark to achieve with the project on a single L1. So the question is can an complete Mii/Ethernet/TCP/IP+Telnetd written in XC fit into a single core L1? If it doesn't then its usefulness to customers and community is severely constrained. If however you think it might be possible to achieve, count me in as the first project contributor to your project.

If XC isn't up to it then the other alternative to consider is writing the entire stack in a more coherent C, perhaps using LwIP rather than uIP which would have performance advantages as well as better reliability. the memory issue could perhaps be overcome by using a driver approach and not copying the buffer data.

This also brings to mind another idea, maybe Xmos should take a leaf out of Google's book and apply the 20% time rule. They could use the 20% time to start community based projects (here on Xcore) to solve problem areas like these. That would also bring the good folk at Xmos closer to the community by working with them directly.

regards
Al
vanjast
Member++
Posts: 30
Joined: Sat Jan 16, 2010 9:57 pm

Post by vanjast »

Where Xmos fails (as did Inmos with the transputer) is the lack of hardware specific info, as you get with other embedded processor databooks. This puts the developer into a trial and error mode of development which a lot of people usually don't have time for.

They must get their 'datasheet information' in order, otherwise things will go the same way as the transputer. This cpu (and tranpsuter) are brilliant concepts, but to make it a success, there must be the nitty gritty info.... tons of it.

At the moment I feel like a headless chicken going through reams of PDF's to find a simple hardware related piece of info, without this knowledge would break my idea.

This same idea applies to firmware, which I agree, there is tons more info, but it does seem 'all over the place', and needs some serious restructuring.
;)
User avatar
Andy
Respected Member
Posts: 279
Joined: Fri Dec 11, 2009 1:34 pm

Post by Andy »

Berni wrote:Well sometimes you need to use code written for 8bit MCUs because its designed to be memory efficient.
It's true that the software stacks written for 8 bit micros are going to be smaller because they have less memory to work with in the first place. However, this doesn't mean that an equivalent stack written for a 32 bit architecture has to be much bigger. The fact that a lot of stacks written for 32 bit micros are bigger is a consequence of them having more memory and the programmer having to worry less about optimisation.

Many of the techniques used to decrease memory on 8 bit micros perform poorly on a 32 bit architecture such as an XCore. For example, uIP tries to be "memory efficient" by using lots of chars and by doing a 32 bit add using 4 separate 8 bit adds... these sorts of things are only to going to cause the code to run slower and increase in size.
Berni wrote:What i found is that xmos is really great for interfacing stuff but when you want to do ethernet, large displays, serious number crunching etc its best to just take an ARM.
Best for who? Would you care to elaborate? I find this statement nonsensical.
User avatar
russf
XCore Addict
Posts: 146
Joined: Thu Dec 10, 2009 10:17 pm
Contact:

Post by russf »

Andy wrote:
Berni wrote:What i found is that xmos is really great for interfacing stuff but when you want to do ethernet, large displays, serious number crunching etc its best to just take an ARM.
Best for who? Would you care to elaborate? I find this statement nonsensical.
Sorry, Andy. I'm jumping in to support Berni, whose statement does not appear nonsensical to me, at all. I have just spent several hours comparing ARM boards from KwikByte (one of which I own), Beagleboard, Technologic TS-7800. etc. Why? Simply because I agree that there is a class of application that although theoretically possible in XMOS, cannot be achieved in a reasonable time. I don't have unlimited time, so taking an ARM may be best for me!

(BTW, interested to hear suggestions/experiences or ARM boards - russf AT topia.com.)

I was wrestling with the same code as Al, all of last weekend, trying to prototype my application. All I was trying to do was repeatedly send a buffer out from the xtcp stack. Nothing exotic. I started with SimpleWebserver, because I could not have faced pulling it all together from scratch. And I replaced the body of the server itself. My code is clean. I have done a lot of research, copy-and-paste, and testing. Yet, I dropped the whole thing early this week with a strong sense that I may not pick it back up. The main reason was that I suspect race conditions somewhere in all those layers and I have no confidence that any single person is going to solve them. (One symptom here is that removing some prints causes a freeze, or a resource error related to a channel.) And if I get my simple test code running, I will still need to strip down the server stuff, tidy it up, and then write my real application. Not sure I have the stomach for it. Not sure anyone at XMOS has the stomach for it either.

BTW, if you want to take a look at my code, please, be my guest: http://github.com/topiaruss/xmostcptests Try knocking out the body of the debug function, which will stop most printing, and kill the thing dead.

This experience was part of the motivation for me asking on another thread for an Open repository and ticket system, where we can start to build a test suite, and bring many brains to the problem. Develop a sense of competition, test our mettle, etc. Let a hero emerge Berni? Folknology? ;) . I have no interest in poking around in a stack, then lobbing the odd patch over the wall, in the hope that people I have no contact with, will take a look. This is the Internet. This is 2011 (almost). This is survival. We need to make progress together now, or move on.

I want Berni to be able to commit a change, issue

Code: Select all

$ svn up
test his change, and come right back with comments and support. I want Al to turn my 5 lines of code into 2, and be able to pick up the benefits instantly, and test. I want to correct Al's spelling :P

Squeezing a TCP/IP stack into an exotic device like this, using, a "charismatic" language like XC, with its idiosyncrasies, is an unnatural act. IOW, it's hard. I was dreaming of merging a high value sensor and a controller into a tiny package (thumb sized) to remove the need for an SBC and hang an ethernet cable directly out of the back of the sensor... into a router -- Bam!

At this stage, I'm about to capitulate, and get real.

The reality is that neither I, nor Al, nor Berni, all smart people in one way or another, is going to solve this alone and come up with a clean sustainable solution. XMOS's repository and ticket system place a divide between inside and outside that prevent the three of us being able to make contributions that stick. We have no idea if the internal project will suddenly switch stack again. We have no idea of the level of competence of the people involved. But we have doubts.

I feel that I'm on the East, and XMOS is on the West. That tall building looming over us is not a watchtower it's an Ivory Tower. The wall needs to come down!

So, yes, I agree with Berni on his point, and with Folknology on the broader question.

--r.

(Just to provide some context, I was an early adopter with the Transputer, after working up to that point with about 7 processor architectures, in a range of languages from Macro-11, to Oregon Pascal, to PLM/86, to Fortran, to C. Pushed the Transputer and Occam into the then-famous-but-now-dead California network company called Vitalink, for a 15 port ethernet switch. Took my colleagues to Transputing 91, which the old school around XMOS may remember well as, erm, a time of change. And I still have a bag full of Txxxs.
So, please don't see me as the enemy.)
Last edited by russf on Fri Oct 29, 2010 7:56 pm, edited 2 times in total.
Reason: fixed quoting
User avatar
Andy
Respected Member
Posts: 279
Joined: Fri Dec 11, 2009 1:34 pm

Post by Andy »

Russ - I think the points you raise are at a slight tangent to the crux of Berni's comments. I'm more interested in hearing why he thinks the processor in general is not suited to the application areas he mentions. I'd rather not turn this into another discussion about the deficiencies of xtcp.
Simply because I agree that there is a class of application that although theoretically possible in XMOS, cannot be achieved in a reasonable time.
Of course, there are such classes of application, but I don't believe the specific applications he raises are the type that "cannot be achieved in a reasonable time".

The part that didn't make sense was the rather sweeping comment that an ARM is better for "ethernet, large displays, serious number crunching". He didn't seem to imply that this was just because of the lack or quality of available software.

I can think of several examples in these categories where an XCore would have an advantage over other micros such as ARM.
Post Reply