Request for Comments, as regards `common_deps`

[fgeorgatos]

Hellos to all,

I have been bugging fellows for a while yet this should not wait any longer,
we discussed this in yesterday's EB hangout:

• we need to specify (as far as feasible) a preferred list of common dependencies;
  It does not have to be final/perfect/ideal, but there has to be some guideline today;
  if not we will pay the high price of effort duplication in the next weeks/months.

This issue intends to supersede #384, so I copy-paste from that:

easyblock = 'Bundle'

name = 'common_deps'
version = '2.0'

homepage = 'http://hpcbios.readthedocs.org/en/latest/HPCBIOS_2013-01.html'
description = """The purpose of this collection is to provide common dependencies in a bundle,
 so that software/modules can be mixed and matched easily for composite pipelines 
 in Life Sciences and other fields"""

toolchain = {'name': 'goolf', 'version': '1.7.20'}

dependencies = [
    ('bzip2', '1.0.6'),
    ('zlib', '1.2.8'),
    ('libreadline', '6.3'),
    ('ncurses', '5.9'),
# **the ones beyond this line are also known as biodeps**
    ('Boost', '1.58.0', '-Python-2.7.10'),
    ('SAMtools', '0.1.19'),
    ('Perl', '5.20.0', '-bare'),
    ('Java', '1.7.0_80', '', True),
    ('libpng', '1.6.17'),
]

moduleclass = 'bundle'

This is intentionally a conservative upgrade over biodeps/1.6, to minimize issues.

I am a little bit inclined to serve life scientists' needs (because applications are many and painful),
however your arguments should be shot from all directions, regardless of my priorities.

Request For Comments, as regards effects of version advancement/creep for:

• SAMtools/1.2 is known to be better than others;
  however @rjeschmi reports TopHat compatibility issues, so 0.1.19 might be more prudent choice
• Perl/5.22 was released last week;
  @rjeschmi reports that 5.20 has greater compatibility and should play conservative
• ncurses/6.0 pre-release is out since last week; it should be backwards compatible with 5.9 line;
  my guess is that we will assume "do-not-fix-if-not-broken" for now, although 6.0 would be nice
• Boost is known to be incompatible with itself. For now we make the assumption 1.58 is OK
• Java/1.8 is supposed to be better and compatible; not all might agree, fi. look for GATK here:
  http://stackoverflow.com/questions/21402136/java-1-8-versus-java-1-7-compatibility-issue

In short, the above proposition is a somewhat conservative upgrade,
with the outlook of making a more advanced common_deps release in the future.

your feedback?!

[fgeorgatos]

and also, ever important:

Python/2.7.10 is assumed to be the latest and greatest.
This will inevitably leave behind things like QIIME which require 2.7.3, but we cannot let everybody wait in the name of intricacies of a single package...

[rjeschmi]

The goal is a reasonable starting point.

Oh you might want to pick a version of cmake, particularly one that would
be compatible with bcl2fastq. It was something I hat tried to get working
in the merge sprint but didn't have luck and redists a bunch of deps that
we would probably build with easybuild

On Fri, Jun 12, 2015, 6:52 AM Fotis Georgatos notifications@github.com
wrote:

and also, ever important:

Python/2.7.10 is assumed to be the latest and greatest.
This will inevitably leave behind things like QIIME which require 2.7.3,
but we cannot let everybody wait in the name of intricacies of a single
package...

—
Reply to this email directly or view it on GitHub
#1689 (comment)
.

[fgeorgatos]

@rjeschmi: I can already smell a dependency hell approaching ;-) but let's do our best...

[pescobar]

I would not add samtools as dependency for all my bioinformatics tools. Neither perl, python or java. IMHO, I would define as biodeps just the latest versions of

bzip2
zlib
libreadline
ncurses
libpng

I think the version of Perl/Python/Java should not be binded to bionfo tools. That should be a site-specific decision.....my two cents....

[fgeorgatos]

@pescobar: OK, thanks.

As I see it, we converge as of today (ncurses/6.0 is not released yet).
In the meeting we thought of renaming biodeps to common_deps, to make it more appealing for other fields; ie. it is going to include: bzip2, zlib, libreadline, ncurses, libpng

N.B. Not having a PR for common_deps today does not block us from now on to advance:

• as long as people stick on these versions, many new PRs are going to be aligned:

dependencies = [
    ('bzip2', '1.0.6'),
    ('zlib', '1.2.8'),
    ('libreadline', '6.3'),
    ('ncurses', '5.9'),
    ('libpng', '1.6.17'),
]

Thanks for pruning a good part of the ongoing dependency hell tree!

[fgeorgatos]

IMHO, cmake is invariably a builddep, and as such there is no need to
specify version (anything goes).
Unless, there is a counter-example, I'd suggest to keep options open here!
F.

On 12 June 2015 at 12:07, Robert Schmidt notifications@github.com wrote:

The goal is a reasonable starting point.

Oh you might want to pick a version of cmake, particularly one that would
be compatible with bcl2fastq. It was something I hat tried to get working
in the merge sprint but didn't have luck and redists a bunch of deps that
we would probably build with easybuild

On Fri, Jun 12, 2015, 6:52 AM Fotis Georgatos notifications@github.com
wrote:

and also, ever important:

Python/2.7.10 is assumed to be the latest and greatest.
This will inevitably leave behind things like QIIME which require 2.7.3,
but we cannot let everybody wait in the name of intricacies of a single
package...

—
Reply to this email directly or view it on GitHub
<
#1689 (comment)

.

—
Reply to this email directly or view it on GitHub
#1689 (comment)
.

echo "sysadmin know better bash than english"|sed s/min/mins/
| sed 's/better bash/bash better/' # signal detected in a CERN forum

[fgeorgatos]

@npklein: are you satisfied with the above versions? do they make sense in your context?

[npklein]

@fgeorgatos looking through our custom .eb files we don't use those dependencies very often, and most dependencies I only see back in the .eb files for the different version of 2 or 3 different programs, so I don't think I can give a meaningful comment about common dependencies.

[pescobar]

I have just hit this problem rebuilding my python-2.7.5 installation
pyenv/pyenv#149

That bug report is for ubuntu 14.04 but I found the same issue in centos6. By now, I will stay in libreadline version 6.2

[fgeorgatos]

@pescobar: only now I notice your comment on libreadline/6.2, thanks! I can't recall such issues on rh6.
I am not changing course yet, but if this bug bugs more people, this should certainly be revised.
Ideally, we should have automation to catch such issues, as early as possible.