Best performance when doing QueryATNF

[noREAVER]

Hi
I noticed a big difference in search speed, 20x slower when doing from local db than from default (web?)
The context is that i have a list of pulsars i will search (60-70sh), for which I do individual queries.
I assume this is from 2 reasons:

• local db is opened>parsed>queried for each time i launch a query.
• the default, without specifying a local db, assuming the search is done through the web? search engine is much more optimized, maybe cached?
  I know I might be wrong by doing it this way, I have an idea how to do it the right way, like passing a full list for a single query and then
  process the table separately.

I thought the object returned after a first query with params, then reuse it for searches later, but I think is not supporting this.

Any recommandations @mattpitkin ?

[mattpitkin]

Hi @noREAVER. Thanks for your question - it makes me realise that I really should properly update the documentation at some point!

The QueryATNF class will only access the web the very first time you call it unless you specifically set check_update=True, or cache=False, or you specify requiring a version that hasn't previously been cached. It only accesses the web to download the full database tarball, so it's never trying to use the catalogue's web interface (an early version of psrqpy did do this). All subsequent instantiations of QueryATNF will load a cached pickle file containing the full catalogue.

So, if I newly installed psrqpy and started a Python session and did:

from psrqpy import QueryATNF
q = QueryATNF()

it would download the latest ATNF catalogue tarball and generate a cache of that. In any subsequent use of

q = QueryATNF(...)

even in new Python sessions, it would use the cached database.

Any instantiation of the QueryATNF object will contain the full database, but if you specify particular pulsars or conditions then it just masks the unwanted results from the output. So, in general, I'd recommend specifying required pulsars not through the QueryATNF call, but instead through the attributes/methods of the object. E.g., generally just do:

q = QueryATNF()

without any keyword arguments. Then if you want to get a specific pulsar you can do multiple things:

# get table of parameters for the Crab pulsar
q["J0534+2200"]

# or...
q.get_pulsar("J0534+2200")

# or...
q.psrs = ["J0534+2200"]
# then look at q which is now masked to only show the Crab pulsar
q
<Table length=1>
   PSRJ    PSRJ_REF     RAJ      RAJ_ERR ...       TYPE_REF      BINCOMP_ORIG BINCOMP_REF
                                         ...                                             
  str10     object     str12     float64 ...        str19           object        str1   
---------- -------- ------------ ------- ... ------------------- ------------ -----------
J0534+2200       -- 05:34:31.973   0.005 ... cdt69,fhm+69,hjm+70           --   

In this latter case, as q still contains the whole catalogue, you could switch to another pulsars, e.g., the Vela pulsar, with:

q.psrs = ["J0835-4510"]
q
<Table length=1>
   PSRJ    PSRJ_REF      RAJ       RAJ_ERR RAJ_REF ...    TYPE_ORIG      TYPE_REF  BINCOMP_ORIG BINCOMP_REF
                                                   ...                                                     
  str10     object      str14      float64   str6  ...      str15         str11       object        str1   
---------- -------- -------------- ------- ------- ... --------------- ----------- ------------ -----------
J0835-4510       -- 08:35:20.61149   2e-05  dlrm03 ... HE[hg73,tfko75] hg73,tfko75           --            

You could reset to show all pulsars again with:

q.psrs = None

Similarly, if you wanted to just show pulsars based on a certain condition, e.g., "F0 > 100", you could do:

q.condition = "F0 > 100"
len(q)
529

and reset to have no condition with:

q.condition = None