Making XMOS really Open will build community, reduce delays

Technical discussions related to any XMOS development kit or reference design. Eg XK-1A, sliceKIT, etc.
User avatar
russf
XCore Addict
Posts: 146
Joined: Thu Dec 10, 2009 10:17 pm
Contact:

Post by russf »

tautic wrote:That's a valid observation. Joerg mentioned that this post is getting traction, and that they are "carefully listening". Hopefully more people will speak their minds around this - there is certainly a ton of potential.
Many on here have more of a hardware point of view, and I can understand that they may not have come across what's happening in software development, where big companies such as facebook are collaborating massively, and much of their dirty laundry is in the open with open ticketing.
tautic wrote:Perhaps there is some concern around additional investment of time which might be required to maintain these code projects if a repository were to be built.
Time will need to be invested. Some companies make repositories, and those are managed largely by the community. They take the view that nothing can really go wrong - if a repository is defaced, it can be fixed - but that does not usually happen. If there are some wrong turns, well, talk about it, and move on.
tautic wrote:If we do get such a repository, it wouldn't surprise me if the vast majority of code is member submitted and maintained. I.e. let XMOS focus on the technology, and it's users focus on the implementation.
I agree. If XMOS can learn to "grasp the sword lightly" they will learn what a might blade they wield.

Best wishes,

--r.


User avatar
lilltroll
XCore Expert
Posts: 956
Joined: Fri Dec 11, 2009 3:53 am
Location: Sweden, Eskilstuna

Post by lilltroll »

I have been thinking around this for a long time. I guess I had the lucky path with Audio, and that much of XMOS internal SW development seems to have been focusing on Audio the last year, with many updates of the code. In difference with the TCP/IP 1v3, I have just received more that 10 ? new versions during the last year fixing old problems and adding new functionality both on the OS-driver side and the XMOS-code side.
I am the admin. of the USB-Audio group with more and more members for every week, and with users that seems to build products aimed for "mass"-production.

On the other hand I'm wondering how you would make this Open "thing" to "take of"
Take a look at http://www.xcore.com/wiki/index.php/Special:AllPages, it hasn't expanded in the rate I was hoping for.
This Open "thing" would need a lot of uploading with improved code to get running, not only downloading.
Probably not the most confused programmer anymore on the XCORE forum.
User avatar
russf
XCore Addict
Posts: 146
Joined: Thu Dec 10, 2009 10:17 pm
Contact:

Post by russf »

lilltroll wrote:I have been thinking around this for a long time. I guess I had the lucky path with Audio, and that much of XMOS internal SW development seems to have been focusing on Audio the last year, with many updates of the code. In difference with the TCP/IP 1v3, I have just received more that 10 ? new versions during the last year fixing old problems and adding new functionality both on the OS-driver side and the XMOS-code side.
I am the admin. of the USB-Audio group with more and more members for every week, and with users that seems to build products aimed for "mass"-production.
Great to hear that the group is moving ahead, like that.

Perhaps the relative success is due to the ease with which you can hear (or not) the success of the audio stack.

I have the feeling that I was the first person in the word to try to *send data* with the xtcp stack at anything more than IRC kind of transfer rates! And have still not heard what testing was involved before that code was lobbed over the wall at us.

Do you think that if the project you mention were placed in a controlled repository, that you and others could work on improving testing, and adding small fixes to start building a culture of cooperation? This kind of thing really starts to move rapidly when a client asks YOU to make an extension that they are happy to have added to the open source repository. At that stage, you could work with XMOS to agree on how that should be done, create a branch project, and get moving. All the time, others could be testing, or XMOS could be making comment, or coming in with suggestions. Your client gets some glory, and also the benefit of the community coming in to expand and improve the code.

If you were careful to build some automated test setups, you could all be working on changes and improvements and achieve a lot in a short time. Before each check-in - run the tests!

lilltroll wrote:On the other hand I'm wondering how you would make this Open "thing" to "take of"
Take a look at http://www.xcore.com/wiki/index.php/Special:AllPages, it hasn't expanded in the rate I was hoping for.
This Open "thing" would need a lot of uploading with improved code to get running, not only downloading.
The main thing is to make the decision, Git, or SVN. Then pick GitHub, or Google Code.

SVN has some benefits due to the app/module structure, which would benefit from the EXTERN reference, to knit alternative groupings of different versions of code together for testing.

But with some simple Shell scripts, you can also do all you need with GitHub.

Then XMOS would simply unzip, manage .ignores and publish all the important files (sources only, not built).

For SVN, it's important that people can sign up and be approved for repository write access after they have signed an agreement, BEFORE they try to get a writable repository.

With Git, it's less important to have the agreement up front. You can wait for a pull request. If it looks worthwhile, get a signed agreement, before merging to the main repository.

Keep zip files for those who have no desire to contribute, and for released versions. Tag the repository carefully before cutting a zip release.

Finally, I repeat, that XMOS should get out in front, and 'own' the repositories. Not in a heavy-handed way, but with the sense of offering a flag for everyone to rally around.

Best,

--r.
User avatar
Interactive_Matter
XCore Addict
Posts: 216
Joined: Wed Feb 10, 2010 10:26 am
Contact:

Post by Interactive_Matter »

I think it is a good point for me to chime in. Two topics are really interesting:

The most important on is the the cultural - or non technical:
lilltroll wrote:I have been thinking around this for a long time. I guess I had the lucky path with Audio, and that much of XMOS internal SW development seems to have been focusing on Audio the last year, with many updates of the code. In difference with the TCP/IP 1v3, I have just received more that 10 ? new versions during the last year fixing old problems and adding new functionality both on the OS-driver side and the XMOS-code side.
I am the admin. of the USB-Audio group with more and more members for every week, and with users that seems to build products aimed for "mass"-production.
I think XMOS is agressivley advertising itself as an audio solution. Which is not bad at all - I like it and I really want to realize a project in that area. But if you try to gather a bunch auf audio experts - don't be astounished that the biggest movement is in the audio part.
On the other hand networking is important an there are alot folks out there that want to improve the TCP/IP libraries. But they can't. Private or spin off repositories are no good way - they hould be managed and backed by XMOS. If e.g. XMOS has no time to imprve the TCP/IP stuff - why not add folknology or some other active members as commiters?
I think it is the cheapest and easiest way for XMOS to get this fixed. According to my impression XMOS is not able to support this broad application areas of XMOS chips since there are myriads of options (UAV, Audio, TCP/IP, DSP, whatever).
But XMOS is able to support and maintain an ecosystem which supports on the one hand a diversified base of projects (which is more or less the state of xcore.com right now) and a growing code base for supporting more application areas (which is one of the topics we are discussing right now.
lilltroll wrote:On the other hand I'm wondering how you would make this Open "thing" to "take of"
Take a look at http://www.xcore.com/wiki/index.php/Special:AllPages, it hasn't expanded in the rate I was hoping for.
This Open "thing" would need a lot of uploading with improved code to get running, not only downloading.
I have a wiki of my own, which tried to maintain and get up running. Getting a community supporting a wiki is one of the hardest things to start – especially if your users are engineers. Since they love to code and design. But one of the biggest complain in technology industry (Electrical Engineering & Computer Science) is documentation. It is never enough it is allways one of the lesser loved jobs.
In comparison it is much easier to just file a ticket or submit a merge request of a piece of code.
But a community needs some care. An xcore.com is on a good way. Do not point at areas that are not really good working currently – look at the areas which show very promising results (the projects, the forum, this thread!).

The easiest to achieve is the technical question:
russf wrote:The main thing is to make the decision, Git, or SVN. Then pick GitHub, or Google Code.

SVN has some benefits due to the app/module structure, which would benefit from the EXTERN reference, to knit alternative groupings of different versions of code together for testing.

But with some simple Shell scripts, you can also do all you need with GitHub.

Then XMOS would simply unzip, manage .ignores and publish all the important files (sources only, not built).

For SVN, it's important that people can sign up and be approved for repository write access after they have signed an agreement, BEFORE they try to get a writable repository.

With Git, it's less important to have the agreement up front. You can wait for a pull request. If it looks worthwhile, get a signed agreement, before merging to the main repository.

Keep zip files for those who have no desire to contribute, and for released versions. Tag the repository carefully before cutting a zip release.

Finally, I repeat, that XMOS should get out in front, and 'own' the repositories. Not in a heavy-handed way, but with the sense of offering a flag for everyone to rally around.
I personally vote strongly for git. I have used CVS, SVN and git and many other version control systems in the past and git is a bit more complex than the rest, but the best system:

GIT and especially github makes it easy to branch off from the official release, try one or two things to improve it. Get it working and if you are confident with your work you can easily file a merge request to get it back to the official repository. This is hell with SVN. The ease of git to manage this will make it more likely that the main code will improve over time.

The ease of git to distinguish between a comit and sharing the results (uploading/pushing) encorages many small commits, which make it easy to understand code changes. SVN does not have this separation. Therefore I have to code until i got something I can share with others. This leads to very big and hardly understandable commits.

Git is also starting to maitain sub modules - as SVN has it right now. I think it is technically as sound as SVN.

With GIT and github or gitorious you do not have to wait for some kind of agreement to enable someon as committer. People can branch their private versions of libraries without asking, fix some things there and request merges to stream their changes back in the main repository. This is much more flexible and easier as with SVN.

But this is very special technical detail. But from this discussion you can easily deduct

A strategy for XMOS to directly realize some quick wins in less than a month:

XMOS has allready some software with Open Sorce licenses. Make a list which libraries are de facto open sorce right now.

Those libraries are probably managed in a version control system at XMOS. As a first guess I assume it is SVN.

XMOS should simply sign up at github and create repositories and publish the libraries there. After that a lot of bug request and possible fixes should emerge in this eco system and it will be clear which user can be added as a a committer.

What will this cost for XMOS?
Something like 2 to 3 weeks of labour: Creating the list, reading the license agreement on github and decide if you can live with it, alternatively install gitorious. Read the manual how to create a public git repository from the internal repositories, upload to git. Write a mail to all engineers wheer the open source repositories can now be found. Help the internal developers to deal with git.

What will XMOS gain?
Unfortunaltely the discussion in this thread will slow down. XMOS will get a large marketing attention, since they can claim to 'go open source'. XMOS can then shift their resource from maintaining libraries to maintaining an eco system and by that gain very much momentum for a broader and better tested library support.

What are the risks?
For some managers 'open source' sounds like 'free beer'. If you provide open source software you provide free beer. Wrong. Open Source enables others to use your code, check it and make suggestions for improvements. You get better code for a much lower price.
For some manager 'open up' sounds like 'giving away intellectual property'. If you publosh the libraries e.g. on github you enable all your competitors to steal your intellectual properties and see your weak spots. Wrong. If you proudly publish your code in a public repository (remember it was open source by the license before) you invite others to improve it. So you save money. You have to deal with criticism but get better results in the end. You outsource (yes open sourcing is a kind of out sourcing) your libraries to get back precious time and money to concentrate on your main assets: Chips and you special libraries (e.g. USB2.0 Audio). Instead of having stressed developers which cannot maintain all libraries and are by tht unable to fix bugs in a old or not often used library you pay one or two 'comunity managing developers' which coordinate bug tickets and patches to enable the community to improve your product (this is like 'free beer') while maintaining control over your source code even if it is 'open source'.

I think if you are a small company and if you have a great product going source is a great way to gather attention and progress to even outpace bigger competitors which are working closed source. The people in this thread really want to help to improve XMOS - is there a reason to not let them?

Thats my 5 cents ;)

But don't get me worng. XMOS is on a very good way and you are allready mainting a good eco system with xcore.com. I think now XMOS is facing the challenge that the users are pushing XMOS to go the whole way.
User avatar
jonathan
Respected Member
Posts: 377
Joined: Thu Dec 10, 2009 6:07 pm
Contact:

Post by jonathan »

Been a while...

OK, let's separate issues a bit. Break the problem down into smaller chunks that can be ticked off one by one, implemented, proven.

Firstly, I start with an observation. There are complaints that XMOS just "chucks stuff over the wall". This is true and a valid complaint. But... what we need to be careful NOT to do here is to gather up all those things that have been chucked over the wall, strap a bomb to them, and chuck them back over the wall into XMOS.

XMOS is in the transitional period between an exciting startup to a global competitor. Note: this has happened in spite of their 1990s approach to open-source. Demands on time at this point, when the company is by other metrics succeeding are likely to be met with at best a little reticence and certainly with a lower priority than serving the growing customer base.

Manifesto

XMOS has shown a clear commitment to open source.

What needs to be done is relatively simple. As a group interested in open-source, we need to create a manifesto for the solutions we want to see. This manifesto will then drive discussion, frame ideas, and enable those within XMOS with the power to change business practices to see solutions, rapidly, without having to wade through pages of debate.

This document needs to be curated by one of our community who feels strongly about these issues. I would instinctively suggest Russ, but do not want to volunteer him. Russ?

The Three XMOS Prongs

The way I see it, XMOS has three areas that can be tackled with a new open agenda.

Ticketing

I deal with this first, because it is the simplest, in my view. Support tickets should, by default, be public. A simple checkbox should enable the user to request a private support ticket. This was suggested earlier by - I believe - Interactive Matter.

This would produce a far more workable, obvious and intuitive support experience than the current "support board" and "closed ticket system" that co-exist extremely awkwardly on the main XMOS website.

Most importantly, visibility over previous tickets will massively reduce the support burden.

Open Source Applications

This is the second-simplest issue, and covers the application code that XMOS creates for customer use - including for example the network stacks, memory interfaces, audio, and other peripheral interfaces.

This code belongs in a co-developed open repository (or set of repositories) on an externally-hosted system such as github. Stable releases should (as they must currently) be curated and periodically frozen by XMOS or a very small number of external experts, but with a large body of contributors.

XMOS must be happy to develop on these repositories, and see themselves as working with the customers and the users to develop an even better set of libraries and application code examples. Fears over code quality will disappear in the first month or so; the bugs, poor commenting, hacked implementations etc will get fixed rapidly.

This will massively improve the quality and coverage of the reference (and demo) applications.

Open Source Tools

This is a more complex issue, as a small number of the components of the tools might be commercially sensitive, or might have a commercial or strategic impact on XMOS' business. I believe there are only two parts of the tools to which this applies and I also believe these are not critical parts of the tools that the community badly needs right now.

Moreover, if 1000 users use the application repository, perhaps 10 will use the Tools repository. The most important thing to do with tools is to release everything that your customers might need to fix bugs. It is unacceptable for a customer to get stuck against a tools bug at 7pm on a Friday (UK time - that could be working day in the US!) and be unable to dive in to fix it themselves (XMOS does not even run support at weekends, let alone bug fixes!).

It is a bonus if they can eventually contribute these bug fixes back to the main branch. But the process can be far more supervised: whilst the main development repositories should be public, commits should only be possible from XMOS or from a tiny number of core trusted tools developers in the community. This will enable people (mainly, think students/academics) to spin off their own experimental repositories, but these will not affect the main branches.

By opening up tools code, the main benefit is that your customers can become familiar with how to fix problems that might arise and understand the structure of how the tools work. This enables them to reduce the support burden on XMOS and also ensures that bugs are discovered and fixed far faster.

Conclusion

The next step is for someone to take the ideas in this thread (and others) and curate them into a simple manifesto that covers a roughly agreeable subset of the ideas and concepts that have been expressed here in a way that represents the interests of the community and customers.

I know that there is a lot of support within XMOS for open-source. I believe that offering simple solutions to well-defined problems, and taking on some - or most - of the time burden for delivering these solutions, will secure the backing of this company we all support.
Image
User avatar
XMatt
XCore Addict
Posts: 147
Joined: Tue Feb 23, 2010 6:55 pm

Post by XMatt »

I am curious as to this comment with respect to the tools.
jonathan wrote:This is a more complex issue, as a small number of the components of the tools might be commercially sensitive, or might have a commercial or strategic impact on XMOS' business. I believe there are only two parts of the tools to which this applies and I also believe these are not critical parts of the tools that the community badly needs right now.
Can you clarify the tools the community badly needs right now and what this actually means? I understand the arguments for the applications but the tools seem to have been now tagged onto the end of this discussion?
User avatar
jonathan
Respected Member
Posts: 377
Joined: Thu Dec 10, 2009 6:07 pm
Contact:

Post by jonathan »

Sure, I can explain.

Firstly, this is perhaps the wrong way to look at the problem. The question should not be "why do you need it?" but more "why do we need to keep it secret?". However...

In my experience, xrun and xflash (for example) are quite buggy. The bugs are relatively easily fixed, but at the point they are discovered, XMOS cannot possibly fix these bugs quickly. These are critical components that very often delay or stop development entirely. Until XMOS fixes them, no progress can be made. This has stalled several (commercial) projects of mine at fairly critical points.

Not being able to see how they work not only prevents debug, but also prevents the user exploring the code to remove their own misunderstandings about the usage. The documentation for these tools is also relatively poor.

These are two of the components that I believe the community (and customers) badly need to see in source form, for the reasons above.

There is no commercial risk to the release of these tools; only good things can come by exposing this source.

On the other hand, xburn and the simulator might contain source code that is commercially sensitive - xburn for the programming of the OTP and the simulator for the cycle by cycle operation of the processor. These tools therefore have reasons not to release them - though I would suggest the release of the source code for a fast behavioural simulator instead!

The point of dividing into sections was to get over the non-controversial sections first, and get those agreed. :-)
Image
User avatar
russf
XCore Addict
Posts: 146
Joined: Thu Dec 10, 2009 10:17 pm
Contact:

Post by russf »

EDIT: Please see this message:

https://www.xcore.com/forum/viewtopic.php?p=6191#p6191

Where renaming from XMOS Foundation to XCommons Foundation is discussed. The remainder of this post is unchanged.

--r.


Preamble

Each day sees larger fraction of our society's technology infrastructure and commerce becoming influenced by Open Source development. Some of the biggest corporations use and contribute to Open Source software and recommend it to their top-tier clients. The benefits of the Open Source development model to large and small companies, and society as a whole, are well documented [1].

Predictably, entities in the XMOS community now wish to collaborate to improve the acceptance of XMOS technologies in their various markets. There is a large potential benefit to XMOS from supporting a vibrant developer community, and there is a similar benefit to community members from being able to use well tested code from the community.

The nature of XMOS technology makes it a natural nexus for a diverse range of hardware and software technologies. Metcalfe's Law [2] is a useful metaphor to describe this space.

When Open Source development is fostered across the boundary between a predominant Company and a community, fears arise on both sides. Contributors from the community want to ensure that the hardware and software designs, code, concepts and implementations that they contribute remain open and free for use by the community. On the other hand, the Company needs to keep its options open, and protect itself from liability and claims related to community contributions that become incorporated in its products.

These are valid concerns, and there exist several very tangible examples of how to solve them. Without quoting specific documents, one can point at large Open Source collaborations sponsored by Yahoo, Facebook, and Google, and smaller projects such as Zope, and the related Plone, both of which have very lively and successful communities, while still on a scale that mere mortals can still comprehend.

A recent thread on xcore.com [3] extolled the need for an official open code repository, to capture many small contributions from individuals, and foster the emergence of task groups to improve specific areas of source code already opened by XMOS. The discussion touched on xtcp, and usb code, specifically, mentioning quality and performance concerns, and it was clear that several external potential contributors would rally around an open repository, where they have not rallied in the past around unofficial ones.

All of the well known and successful Open Source projects center their existence around functional and tested code in shared repositories. While it is simple to create shared repositories in the public space, for the benefits laid out above to accrue, the network effect that is created by an official repository is crucial.

The same thread touched on an open ticketing system[4] as an important way to exchange information within the community, and help developers outside XMOS understand what issues will or will not be fixed internally, giving them the option of collaborating to produce community contributions, and also preventing duplication of effort. Open ticketing is rife across the Internet, and companies such as Facebook depend on it for rapid feedback and to engage developers.

Recognizing the enormous shift in perspective required on the part of XMOS to successfully engage with a growing community, this MANIFESTO is designed to bootstrap discussion. In itself this may never be perfect document, but that is not the intent. The success of this document will be measured in the extent to which it brings together a community with a shared purpose, and to the degree that a formal organization emerges to meet the needs of all concerned. This document is just the start...

Call to Action

So, now, formally, the members of the XMOS community, including all users or potential users of XMOS technologies, the company XMOS Ltd itself, and its employees and business partners, are hereby invited to participate in the institution of an XMOS Foundation.

Membership

All members of the the wider XMOS community, including all users or potential users of XMOS technologies, employees and business partners of XMOS, are invited to participate in the discussion, the empowerment of a pro tem board, the agreement of articles of incorporation, and the holding of an election of officers.

Goals

The XMOS Foundation shall represent the interests of the community of users and producers of XMOS technologies, and promote the development of hardware and software designs that help broaden the scope of the XMOS platform, and grow the market for XMOS related products.

In particular to:
  • Provide clear, neutral, and sustainable ownership of contributed code.
  • Provide a decision-making structure for essential community activities.
  • Interface with XMOS, the company, to represent the needs of the Foundation membership.
  • Assist XMOS in the transition to greater transparency and openness for the benefit of Foundation members.
Priority Tasks

As a high-priority, the initial board of directors will take steps to develop a simple yet robust legal framework for ownership of hardware and software source code contributions that protects the rights of all contributors, and encourages collaboration.

The board will immediately foster productive and professional engagement with XMOS Ltd, to determine what role the Foundation can play in the transition to openness.

To take part in the Foundation Discussion, please join the group here: http://groups.google.com/group/xmos-foundation?hl=en


Notes

The point of the first goal-bullet is to ensure that, as the XMOS eco-system grows, Foundation members can contribute improvements in the knowledge that they will remain usable by the whole community. This counteracts the fear that would exist both inside and outside the community that if members use code or hardware designs in a product, there may be a legal challenge that would harm their business, by causing expensive lawsuits. Anyone who contributed under this agreement would sign their contributions over in perpetuity to the Foundation, and the foundation would protect the rights of foundation members (and non-members) to use that code without fear of legal issues. This is a lot like the role of the Apache Foundation, Plone Foundation, etc.

The expectation is that the foundation would attempt wherever possible to use existing legal patterns, such as those exemplified by the Apache Foundation, the Plone foundation, etc, to reduce duplication f effort, and maximize effort on such projects as improving the technologies, support for collaboration, supporting new technology ports, improving the standing of this technology in the marketplace, etc.

References

[1] This wikipedia article gives a useful history: http://en.wikipedia.org/wiki/Open-source_software

[2] http://en.wikipedia.org/wiki/Metcalfe's_law

[3] http://www.xcore.com/forum/viewtopic.php?f=8&t=820
User avatar
russf
XCore Addict
Posts: 146
Joined: Thu Dec 10, 2009 10:17 pm
Contact:

Post by russf »

Our google group is now here:

http://groups.google.com/group/xcommons-foundation

Our source repository is here:

https://github.com/organizations/XCommons-Foundation/

And the foundation is now

XCommons Foundation

Things are moving fast. If you would like to help grow the Foundation, please Come over and read the original post on the group.

Here's an extract from the manifesto:
The XCommons Foundation shall represent the interests of the community of users and producers of XMOS technologies, and promote the development of hardware and software designs that help broaden the scope of the XMOS platform, and grow the market for XMOS related products.
In particular to:

Provide clear, neutral, and sustainable ownership of contributed code.

Provide a decision-making structure for essential community activities.

Interface with XMOS, the company, to represent the needs of the Foundation membership.

Assist XMOS in the transition to greater transparency and openness for the benefit of Foundation members.
We are very friendly to everyone with an interest in helping XMOS technologies reach a wider audience.

--r.
User avatar
lilltroll
XCore Expert
Posts: 956
Joined: Fri Dec 11, 2009 3:53 am
Location: Sweden, Eskilstuna

Post by lilltroll »

Is http://git-scm.com/ the way on (my client) side ?
Probably not the most confused programmer anymore on the XCORE forum.
Post Reply