XDE 11.2 #1842 - symlinked project folders

Technical questions regarding the XTC tools and programming with XMOS.
User avatar
russf
XCore Addict
Posts: 146
Joined: Thu Dec 10, 2009 10:17 pm

XDE 11.2 #1842 - symlinked project folders

Post by russf »

Hi all,

Background
I'm managing several projects.
Organization is important for the project.
I want to be able to 'compose' super-projects from a tried and tested library of projects kept in a github tree.
My requirement seems reasonable, and is comparable to methods I've used on other projects. (wave to svn:externals, etc.)

What I'm doing
I have a workspace containing module_xmos_common.1v3, and ProjX that builds and runs as expected

When i do:

Code: Select all

 cd
mv workspace/ProjX/ dev/lib
ln -s dev/lib/ProjX/ workspace/ProjX
And rebuild, I would expect the build to succeed, again. But, it fails. with
Makefile:33 /build/Makefile/common: No such file or directory
xmake: *** No rule to make target `/build/Makefile.common'. Stop.

When I look at line 33 in the XDE, where the error marker is set, I see:

Code: Select all

#include $(lastword $(sort( $(wildcard ../module_common.1v*)))/build/Makefile.common.
The problem is the traversal from the working directory using the '..'. I can verify this by replacing '..' with /home/russ/workspace/. With that simple change, the build proceeds as before.

So...
For now, I can achieve what I want by making this simple change, but I'd rather not, for several good reasons. I'd prefer the include line not to care whether I'm including from within a real directory, or a symlinked one, (Bear in mind that I cannot hardlink a directory).

Has anyone found a nice way around this? If so, could we have that incorporated in a future release?

I'm also interested to know if others are looking for similar means for modular building.

Thanks for your patience.

--r.


User avatar
davelacey
Experienced Member
Posts: 104
Joined: Fri Dec 11, 2009 8:29 pm

Post by davelacey »

I think that over time we will need:

1) An (optional) environment variable that can be set for the location of the common Makefile. This would mean
having to change every application Makefile

2) Either with the above, this wont let you pick source components from several locations (which I think you are trying to do). So, ideally, there should be an environment variable for a component search path which can be optionally set.

I've raised issue trackers on these here:

https://github.com/xcore/xcore_template/issues/3
https://github.com/xcore/xcommon/issues/5

However, the second one could be a bit awkward and may take a bit of time to implement due to the vagaries of make and other things the Makefiles support.

In the meantime, you could try copying all your modules etc into /dev/lib *including* module_xmos_common and then symlink the things you need back to your workspace (including module_xmos_common). The build should work then.

Regards,
Dave
User avatar
russf
XCore Addict
Posts: 146
Joined: Thu Dec 10, 2009 10:17 pm

Post by russf »

davelacey, I *really* appreciate you getting back with such a coherent response.

A couple of responses, in return.
davelacey wrote:I think that over time we will need:

1) An (optional) environment variable that can be set for the location of the common Makefile. This would mean
having to change every application Makefile
This sounds like a good solution, and would probably overcome the issue I'm having.

Another approach (just a thought) is to investigate how the traversal '..' could be interpreted in the context of the top level Makefile,or xmake, instead of the lower level Makefile. That would ensure that the target of .. became the workspace, rather than the parent to the component, which you correctly realise is in this case /dev/lib.
davelacey wrote:
2) Either with the above, this wont let you pick source components from several locations (which I think you are trying to do). So, ideally, there should be an environment variable for a component search path which can be optionally set.
You are exactly correct that I want to be able to pick source components from several locations. This is going to arise as we (internally and in the community) have different repositories (different authors/project) containing multiple components, and we want to compose across them.
I would prefer a symlinking approach (or equivalent -- I appreciate that symlinking at the OS level leaves out the windoze folks) above any search path approach.
First of all, the symlink approach is 100% explicit. It says what I want, and it says where to find it. It can also be versioned, since the references (links but not targets of links) are stored within git, for instance. If you use a search path approach, you still need a way to say what you want to compose, and you are left with the problem (in the common path semantics, at least) of priority by path order, which is a build-time thing, and not easy to reconstruct when a third party is trying to rebuild a project.

Secondly, as we get more complexity and history, we will need to modify dependencies from project A to project B, (imagine support for Atmel, vs Winbond as an example) so we might need a separate path for each project. We also would need to store the PATH as part of the project, so we can repeat a build) but that still would not tell us what would be found on the path at build time. That way lies madness! Consider Java CLASSPATH :( Ugh!
davelacey wrote:
I've raised issue trackers on these here:

https://github.com/xcore/xcore_template/issues/3
https://github.com/xcore/xcommon/issues/5
I have not had chance to look at these, yet. Will do so.
davelacey wrote:
However, the second one could be a bit awkward and may take a bit of time to implement due to the vagaries of make and other things the Makefiles support.
My sympathy.
Make has given me a rash, since about 1985. You might be entertained by zope's buildout framework, that I have used a lot on python projects. It's a dream for repeatable web app development and deployment, where repeatability and meeting version dependencies must be guaranteed. But it has its own learning curve, and does not introspect C dependencies.
davelacey wrote:
In the meantime, you could try copying all your modules etc into /dev/lib *including* module_xmos_common and then symlink the things you need back to your workspace (including module_xmos_common). The build should work then.
This seems about right, and it's probably the best solution for the moment. I'll be giving this a shot, today (Business day just getting serious in CA, while you XMOS guys are already down-the-pub ;) ).

Thanks again for the prompt and thoughtful response.

Best regards,

--r.
davelacey wrote:
Regards,
Dave
User avatar
russf
XCore Addict
Posts: 146
Joined: Thu Dec 10, 2009 10:17 pm

Post by russf »

Just wanted to say, that, yes, I am now doing the following:

Code: Select all

cd ~/dev/lib
ln -s /home/russ/workspace/module_xmos_common.1v3 .
So that I'm putting the module_xmos_common file in the component directly, but in the form of a symlink back to the One Ring that Binds them All, which gives the whole thing a nice flavo(u)r again.

I still need a bit more experience with this new component system, then I need to get over and look at the sdram module. I hope to send a pull request to Dan before the end of the week.

Best wishes,

--r.
User avatar
davelacey
Experienced Member
Posts: 104
Joined: Fri Dec 11, 2009 8:29 pm

Post by davelacey »

Just to note that the latest version of Xcommon supports setting a list of paths in the environment variable XMOS_MODULE_PATH to search through when building.
User avatar
russf
XCore Addict
Posts: 146
Joined: Thu Dec 10, 2009 10:17 pm

Post by russf »

Thanks Dave,

Great news. I'll check it out when I'm next in xmos land.

--r.