Having once again spent a reasonably large amount of time screwing with various bits of the IMP configure scripts in order to get things to work on a mac, I would really like us to simplify things. I suggest the configure options simply be extra_lib_paths="a//b/lib c/d/lib" extra_include_paths="a/b/include c/d/include" python_path="a/b/lib/python-2.5/..." which get parsed and prepended to the scon environment before any tests are performed and get added to all build environments. (I am not sure if space is the best delimiter, but that can be solved later, perhaps by using python syntax "['a/b/include', 'c/d/include']").
The cgal, boost, em, modeller etc arguments can then be removed and configure can then simply test if it can build against modeller, em, cgal, boost etc and not worry about making sure all the paths are added in the right places. We then only have to worry about adding the correct library dependencies to link commands and otherwise ignore dependencies in the build scripts.
This also makes it clear that all libraries are searched for in all locations. Keren, Javi and I have all at various points been confused and frustrated by imp picking up copies of libimp in the diva modeller build when neither the imp config arguments nor the directory name (/ diva1/home/modeller-SVN) gave us any clue that such could happen.
This also removes the distinction between setups which we happen to have encoded into our config system (which work magically) and setups which differ in some way from what is encoded (which can't be easily made to work). There is currently no way for a user to tell whether things will work, what to do to fix them if they don't and no simple way to document what will work. And we can't ship prebuilt mac binaries for C++ programmers since the boost versions have to match, so we have to support the large number of different ways things can be set up on macs at the very least.
Any better suggestions?
Daniel Russel wrote: ... > I suggest the configure options simply be > extra_lib_paths="a//b/lib c/d/lib" > extra_include_paths="a/b/include c/d/include" > python_path="a/b/lib/python-2.5/..." > which get parsed and prepended to the scon environment before any > tests are performed and get added to all build environments. (I am not > sure if space is the best delimiter, but that can be solved later, > perhaps by using python syntax "['a/b/include', 'c/d/include']").
There is already a perfectly good delimiter for paths. It is ':' on Unix systems and ';' on Windows. Using space obviously prevents filenames containing spaces from being used (obviously you cannot use : or ; with regular paths, but everybody is familiar with this restriction already, since no other tools would work either). The existing IMP scons paths use standard delimiters. I don't know which delimiters you use in your recent patch but there is no reason not to use the standard ones.
Otherwise, your patch sounds reasonable to me - except perhaps for the rpath stuff on non-Mac systems, which is rather controversial on Linux systems and will not work at all on Windows or many other Unixes - and I will commit it once a) the nightly builds are fixed and b) I have time to review it and check out the rpath stuff. (Unless others have comments, of course.)
> The cgal, boost, em, modeller etc arguments can then be removed and > configure can then simply test if it can build against modeller, em, > cgal, boost etc and not worry about making sure all the paths are > added in the right places.
I agree. Most of these tests could be boolean rather than "package" options. I guess they are modeled after the autoconf-style --with options where you can specify a path for such packages, but most people use package managers these days and so don't specify a path. Like you say, it leads to all sorts of problems when dynamic linking is used.
> Keren, Javi and I have all at various points been confused > and frustrated by imp picking up copies of libimp in the diva modeller > build when neither the imp config arguments nor the directory name (/ > diva1/home/modeller-SVN) gave us any clue that such could happen.
I'm not sure how your solution would avoid that since the Modeller run scripts obviously have to add the Modeller library directories to the dynamic library path (and just like IMP, they try to do that by prepending rather than appending). I generally recommend installing Modeller locally rather than using the build in /diva1 if you want to build with Modeller anyway.
Ben
On Nov 26, 2008, at 12:42 PM, Ben Webb wrote:
> Daniel Russel wrote: > There is already a perfectly good delimiter for paths. It is ':' on > Unix > systems and ';' on Windows. Agreed. Colon used to be the MacOS directory delimiter, so I was unsure of its status on macs. I That was my only concern. But it looks moot. > > Otherwise, your patch sounds reasonable to me - except perhaps for the > rpath stuff on non-Mac systems, which is rather controversial on Linux > systems and will not work at all on Windows or many other Unixes - > and I > will commit it once a) the nightly builds are fixed and b) I have time > to review it and check out the rpath stuff. (Unless others have > comments, of course.) I agree that it is a bit controversial, but everything I have read prefers it to using LD_LIBRARY_PATH (since that, among other ill effects, changes the behavior of 3rd party libraries). I don't know how dynamic link paths are handled on Windows and no one seems to use it around here. There used to be some equivalent of rpath, I seem to recall, but that was years ago that I last did windows programming. How do things currently work on windows anyway (since we can't count on csh being around)?
> I'm not sure how your solution would avoid that since the Modeller run > scripts obviously have to add the Modeller library directories to the > dynamic library path (and just like IMP, they try to do that by > prepending rather than appending). Yeah, it is still I mess. I retract my supposition :-)
Daniel Russel wrote: > I agree that it is a bit controversial, but everything I have read > prefers it to using LD_LIBRARY_PATH (since that, among other ill > effects, changes the behavior of 3rd party libraries).
I will look at the patch. But rpath will probably have to be optional - defaulting to true on OS X. The Modeller binary packages on OS X and Linux are built with rpath for precisely the reason you mention - to avoid the bundled copies of Fortran runtime libraries, HDF5, libpython etc. from conflicting with system copies. But the Unix and Windows builds do not use rpath.
> I don't know > how dynamic link paths are handled on Windows and no one seems to use > it around here. There used to be some equivalent of rpath, I seem to > recall, but that was years ago that I last did windows programming. > How do things currently work on windows anyway (since we can't count > on csh being around)?
Windows looks for dynamic libraries in the same directory as the parent binary (executable, or Python extension) and then falls back to looking in PATH. DLL and their paths can also be registered in some random place in the registry, IIRC.
Ben
On Nov 26, 2008, at 1:38 PM, Ben Webb wrote:
> Daniel Russel wrote: >> I agree that it is a bit controversial, but everything I have read >> prefers it to using LD_LIBRARY_PATH (since that, among other ill >> effects, changes the behavior of 3rd party libraries). > > I will look at the patch. But rpath will probably have to be > optional - > defaulting to true on OS X. The Modeller binary packages on OS X and > Linux are built with rpath for precisely the reason you mention - to > avoid the bundled copies of Fortran runtime libraries, HDF5, libpython > etc. from conflicting with system copies. But the Unix and Windows > builds do not use rpath. From what I understand, one shouldn't be using rpath on mac os and should be using install_name_tool and the linker to make sure the correct paths are saved in the library itself. Unfortunately, scons doesn't seem to set library ids automatically for some reason, so we will, at some point, have to patch installation ourselves.
>> > > Windows looks for dynamic libraries in the same directory as the > parent > binary (executable, or Python extension) and then falls back to > looking > in PATH. DLL and their paths can also be registered in some random > place > in the registry, IIRC. Maybe that was what I was thinking of. Anyway, the current setup doesn't seem to do much for most windows users anyway, so no great loss :-)
> a) the nightly builds are fixed and The windows problem looks trivial but it looks like boost is broken on the suns. Did something change there? I am not too inclined to worry about suns and four year old compiler versions.
I haven't bothered with the sporadic failures of the transform test case as that class will change once Keren gets her transform code in.
Daniel Russel wrote: >> a) the nightly builds are fixed and > The windows problem looks trivial but it looks like boost is broken on > the suns. Did something change there? I am not too inclined to worry > about suns and four year old compiler versions.
No, boost has never worked on the Suns in combination with Python 2.5, only with older Pythons. I also don't worry about this for the same reasons.
> I haven't bothered with the sporadic failures of the transform test > case as that class will change once Keren gets her transform code in.
Unless Keren is planning to do this very soon, one of us will have to back out that code in the meantime. There has not been a successful nightly build for over 2 weeks, and that is a problem for anybody in the lab that relies on the builds.
Ben
>> I haven't bothered with the sporadic failures of the transform test >> case as that class will change once Keren gets her transform code in. > > Unless Keren is planning to do this very soon, one of us will have to > back out that code in the meantime. There has not been a successful > nightly build for over 2 weeks, and that is a problem for anybody in > the > lab that relies on the builds. From the looks if it, that is due to the windows DLL exports thing on Vector (and perhaps the sun failures) as the transform passes most times you run and passes most nights. Not great, but not to blame. Anyway, we can always comment out that test.
Previously, there were failures due to us not being able to make changes to examples. That broke builds until just before your return.
Ben/Daniel, got back to the IMP coding today. I can have it all in by Friday - hope it is not too late. On Nov 26, 2008, at 4:41 PM, Ben Webb wrote:
> Daniel Russel wrote: >>> a) the nightly builds are fixed and >> The windows problem looks trivial but it looks like boost is broken >> on >> the suns. Did something change there? I am not too inclined to worry >> about suns and four year old compiler versions. > > No, boost has never worked on the Suns in combination with Python 2.5, > only with older Pythons. I also don't worry about this for the same > reasons. > >> I haven't bothered with the sporadic failures of the transform test >> case as that class will change once Keren gets her transform code in. > > Unless Keren is planning to do this very soon, one of us will have to > back out that code in the meantime. There has not been a successful > nightly build for over 2 weeks, and that is a problem for anybody in > the > lab that relies on the builds. > > Ben > -- > ben@salilab.org http://salilab.org/~ben/ > "It is a capital mistake to theorize before one has data." > - Sir Arthur Conan Doyle > _______________________________________________ > IMP-dev mailing list > IMP-dev@salilab.org > https://salilab.org/mailman/listinfo/imp-dev
-
Ben Webb -
Daniel Russel -
Keren Lasker