SourceXtractor Python API

[ayllon]

I have almost the full pipeline callable from Python, except for the Prefetcher. Measurement can be done multithreaded, and the output is a numpy array (although it could be directly a FITS file)

We need to agree on what the API would look like, though.

Abstraction leaks

Users may want to keep hold of the sources even after sourcextractor is done. i.e. for pickling, or to do something with them after the full pipeline is done (plotting?)

This is risky business, as some properties will be pointing to internal details (as a pointer to the detection frame).
I think the best option (partially implemented) is to add a detach() method that gives a source that exists outside of the pipeline, with whatever properties had been already computed. No on-demand properties removes the trickiness of dealing with internal properties.
However, we need to make sure we have no lazy property evaluation (i.e. stamp must hold a "materialized" stamp, and not be lazily evaluated via the measurement/detection frame when requested)

class MyListener:
    def __call__(obj):
        self.objs.append(obj.detach())

...
pickle.dump(my_listener.objs, open('whatever.pickle'))

Building the pipeline

Right now it is an almost 1-1 translation

segmentation = sx.Segmentation(context)
partition = sx.Partition(context)
grouping = sx.Grouping(context)
deblending = sx.Deblending(context)
output = sx.Output(context)

segmentation.set_next_stage(my_filter)
partition.set_next_stage(live)
grouping.set_next_stage(deblending)
deblending.set_next_stage(output)

segmentation()

Kind of verbose. An option would be to do it at construction time

segmentation = sx.Segmentation(context)
partition = sx.Partition(context, segmentation)
grouping = sx.Grouping(context, partition)
deblending = sx.Deblending(context, grouping)
output = sx.Output(context, deblending)

segmentation() # Not very intuitive?

Or a-la keras

segmentation = sx.Segmentation(context) # Inconsistent!
partition = sx.Partition(context)(segmentation)
grouping = sx.Grouping(context)(partition)
deblending = sx.Deblending(context)(grouping)
output = sx.Output(context)(deblending)

segmentation() # Not very intuitive?

Maybe we can get rid of the context everywhere with something like

with sx.Context(measurement, config):
    segmentation = sx.Segmentation() # Inconsistent!
    partition = sx.Partition()(segmentation)
    grouping = sx.Grouping()(partition)
    deblending = sx.Deblending()(grouping)
    output = sx.Output()(deblending)

Maybe if the context initialization is done in two steps

with sx.Context(config):
    measurement = MeasurementConfig()
    segmentation = sx.Segmentation()(measurement)
    partition = sx.Partition()(segmentation)
    grouping = sx.Grouping()(partition)
    deblending = sx.Deblending()(grouping)
    output = sx.Output()(deblending)

Any other ideas about what the API could look like?

[ebertin]

This is amazing @ayllon! As far as I am concerned I have a slight preference for the last proposal. Right now are there other sequences one can reasonably think of for the source extraction pipeline?

[ayllon]

Would something like this make sense?

with sx.Context(config):
    pipeline = sx.Pipeline([
        sx.Segmentation(),
        sx.Partition(),
        CustomFilter(snr=10),
        sx.Grouping(), 
        sx.Deblending(),
        sx.Measurement(),
        sx.FitsOutput()
    ])
pipeline()

then, we can leave the set_next_stage calls up to Pipeline.

So we can keep the API for the custom components as:

class MyFilter:
    def __call__(self, obj):
         if something_with(source):
             self.__next(source)

I think it makes more sense to have custom stages as functors.

We can also have

pipeline = sx.DefaultPipeline(config)

[HannoSpreeuw]

This is very interesting for me, since I would like to control SourceXtractor++ executions throught Python, possibly analogous to sep.
So I stumbled upon astromatic-wrapper, but it seems outdated.

Do you have a complete working example of the steps above? Something that I can run myself or reproduce?
Perhaps it is enough if you tell me how sx is imported.

[HannoSpreeuw]

Ah, sorry, that must be import sourcextractor as sx.

[marcschefer]

Hi @HannoSpreeuw ,

Just so there is no confusion, be aware that the Python wrapper is currently only available in the #522 development branch and is not currently fully tested or ready for production use.

We plan to develop it further but it may still be some time until we have a fully production ready version.

[HannoSpreeuw]

Thanks.
But what is currently the recommended way to launch SourceExtractor++ modules from a Python script?

[marcschefer]

Currently running the executable, for example with os.system() is the only solution with the released version. We know that Python integration is a very requested feature but unfortunately it's not yet fully ready.

[mkuemmel]

The last exercise of the test data set mentioned in the README and described here shows what is implemented in the branch mentioned by @marcschefer . This is the directions the development will likely go.

Until then there is no recommended way executing SE++ from python, and as a matter of fact in the Euclid satellite project we use a wrapper with the module "subprocess" at its core.