Notes from my first attempt to work with PDL source

[falsifian]

Today I tried building PDL from source for the first time. I'm recording things I ran into; fixing them might help future newcomers.

I'm not sure if a Github issue is the best format, but arguably there are bugs...

What went well

PDL::FAQ pointed me to this GitHub project*, and the instructions in INSTALL worked: perl Makefile.PL, make, make test. (I didn't try make install or make doctest).

*I already knew about it, but I like it when the official docs are correct.

Problem/suggestion 1: document how to use it without installing

Summary: I needed to run perl with -Iblib/arch -Iblib/lib to use the PDL I just built with make.

The first thing I tried to do after make was perl -Iblib/lib -MPDL -E 'say $PDL::VERSION' because I was curious how the version number of git head is kept. This failed with:

PDL::Core object version 2.081 does not match bootstrap parameter 2.082 at /usr/libdata/perl5/amd64-openbsd/DynaLoader.pm line 223.
Compilation failed in require at (eval 1) line 4.
BEGIN failed--compilation aborted at (eval 1) line 4.
BEGIN failed--compilation aborted.

After some fiddling I got it working by passing -Iblib/arch in addition to -Iblib/lib.

Might be worth documenting this for those who don't want to make install for whatever reason.

Problem 2: make test TEST_FILES=t/core.t (or whatever) is slow and also doesn't work

According to the ExtUtils::MakeMaker docs, I should be able to run make test TEST_FILES=t/core.t to test just t/core.t.

First, it took a while for the command to run. I saw a lot of lines like Manifying 1 pod document. Manually commenting out the prerequisites for the pure_all target in Makefile sped it up significantly. Could the Makefile be fixed so that it realizes it already did whatever it's doing?

Second, it didn't actually work: I see

PERL_DL_NONLAZY=1 "/usr/bin/perl" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, '../../blib/lib', '../../blib/arch')" t/core.t
Cannot detect source of 't/core.t'! at /usr/libdata/perl5/TAP/Parser/IteratorFactory.pm line 256.
        TAP::Parser::IteratorFactory::detect_source(TAP::Parser::IteratorFactory=HASH(0xd92ec844be0), TAP::Parser::Source=HASH(0xd93750ef0a0)) called at /usr/libdata/perl5/TAP/Parser/IteratorFactory.pm line 211
        TAP::Parser::IteratorFactory::make_iterator(TAP::Parser::IteratorFactory=HASH(0xd92ec844be0), TAP::Parser::Source=HASH(0xd93750ef0a0)) called at /usr/libdata/perl5/TAP/Parser.pm line 472
        TAP::Parser::_initialize(TAP::Parser=HASH(0xd93750d6fa0), HASH(0xd9380f50b20)) called at /usr/libdata/perl5/TAP/Object.pm line 55
        TAP::Object::new("TAP::Parser", HASH(0xd9380f50b20)) called at /usr/libdata/perl5/TAP/Object.pm line 130
        TAP::Object::_construct(TAP::Harness=HASH(0xd92dc373928), "TAP::Parser", HASH(0xd9380f50b20)) called at /usr/libdata/perl5/TAP/Harness.pm line 865
        TAP::Harness::make_parser(TAP::Harness=HASH(0xd92dc373928), TAP::Parser::Scheduler::Job=HASH(0xd93750f51a8)) called at /usr/libdata/perl5/TAP/Harness.pm line 664
        TAP::Harness::_aggregate_single(TAP::Harness=HASH(0xd92dc373928), TAP::Parser::Aggregator=HASH(0xd92c0976910), TAP::Parser::Scheduler=HASH(0xd93750f51f0)) called at /usr/libdata/perl5/TAP/Harness.pm line 756
        TAP::Harness::aggregate_tests(TAP::Harness=HASH(0xd92dc373928), TAP::Parser::Aggregator=HASH(0xd92c0976910), "t/core.t") called at /usr/libdata/perl5/Test/Harness.pm line 136
        Test::Harness::_aggregate_tests(TAP::Harness=HASH(0xd92dc373928), TAP::Parser::Aggregator=HASH(0xd92c0976910), "t/core.t") called at /usr/libdata/perl5/Test/Harness.pm line 118
        Test::Harness::_aggregate(TAP::Harness=HASH(0xd92dc373928), TAP::Parser::Aggregator=HASH(0xd92c0976910), "t/core.t") called at /usr/libdata/perl5/Test/Harness.pm line 151
        Test::Harness::runtests("t/core.t") called at /usr/libdata/perl5/ExtUtils/Command/MM.pm line 72
        ExtUtils::Command::MM::test_harness(0, "../../blib/lib", "../../blib/arch") called at -e line 1

The ../.. in the two paths passed to test_harness look suspicious there.

I was able to work around this by just running perl -MExtUtils::Command::MM -e 'test_harness(0,"blib/lib","blib/arch")' t/core.t

Problem 3: What is Known_problems? Two READMEs?

README and INSTALL refer to something called Known_problems. Is that a file that used to exist?

Also it's a little odd that both README and README.md exist. I guess the latter is there so the project looks nice on GitHub. Maybe README.md should explain that, and point to README for more information?

[falsifian]

Oh, and I should mention: I did accomplish what I set out to do, namely adding test cases for some issues I'd recently filed: #431

[shawnlaffan]

After some fiddling I got it working by passing -Iblib/arch in addition to -Iblib/lib.

FWIW, you can also use the blib module.

perl -Mblib -MPDL -E 'say $PDL::VERSION'

perl -MExtUtils::Command::MM -e 'test_harness(0,"blib/lib","blib/arch")' t/core.t can be simplified using prove. e.g.:

prove -vbm t/core.t

The -b flag does the same as the blib module. The -v flag makes the output verbose and might not be needed, while the -m flag helps synchronise STDOUT and STDERR so warnings and the like are closer to the main output in the log text.

[mohawk2]

For developing the "core" (Core, Bad, Slices, Ops, Primitive, Ufunc, Math, MatrixOps), I some time ago added a core target for building, and a coretest to build then test it. This fits nicely in with fixing things in core that broke a non-core (but still in main PDL) module, e.g.:

make core && (cd Libtmp/Transform && make test)

Or for iterating something in core:

make core && prove -b t/ops.t

@falsifian How do you feel about updating the FAQ to reflect the above, and also merging the README into README.md (and then deleting it)? Obviously it would be desirable to include @shawnlaffan's excellent points (for which thank you).

[falsifian]

@falsifian How do you feel about updating the FAQ to reflect the above, and also merging the README into README.md (and then deleting it)? Obviously it would be desirable to include @shawnlaffan's excellent points (for which thank you).

I hope to do this but may be a couple of weeks (which is what I said on IRC two weeks ago...)

[mohawk2]

Nudge. @falsifian Feel like having a go at this?

[falsifian]

Nudge. @falsifian Feel like having a go at this?

Thanks for the reminder! Yes, but as before might be a couple of weeks ;-) Seriously, I estimate about a 50% chance I'll actually get to it within 2w this time; I do want to make PDL developer docs better.

[mohawk2]

Great to hear! Please take a look at #393, which captures various things I believe should go into developer docs. It would be really valuable to get someone with a bit more distance than me to ask questions about things that don't make sense.