docs(library/ra3u5GxySF6SkjCCKo7E-w) passes some extra stream which trips (?) solr

[romanchyla]

firstly, the deployment must have proper config:

          value: https://dev.adsabs.harvard.edu/v1/biblib/libraries
        - name: RATELIMIT_STORAGE_URL
--
        - name: VAULT_ENDPOINT
          value: https://dev.adsabs.harvard.edu/v1/vault/query

then, only POST requests allow contacting bigquery - which is what we do - but actually, i think we want to allow also normal query endpoints (GET) to do it

nevermind, so this query will fail:

curl 'https://dev.adsabs.harvard.edu/v1/search/bigquery?foo=bar&fl=identifier%2C%5Bcitations%5D%2Cabstract%2Caff%2Cauthor%2Cbibcode%2Ccitation_count%2Ccomment%2Cdoi%2Cid%2Ckeyword%2Cpage%2Cproperty%2Cpub%2Cpub_raw%2Cpubdate%2Cpubnote%2Cread_count%2Ctitle%2Cvolume%2Clinks_data%2Cesources%2Cdata%2Cemail%2Cdoctype&q=docs(library%2Fra3u5GxySF6SkjCCKo7E-w)&rows=25&sort=date%20desc%2C%20bibcode%20desc&start=0' -H 'Authorization: Bearer:......' -X POST

with

 curl 'https://dev.adsabs.harvard.edu/v1/search/bigquery?foo=bar&fl=identifier%2C%5Bcitations%5D%2Cabstract%2Caff%2Cauthor%2Cbibcode%2Ccitation_count%2Ccomment%2Cdoi%2Cid%2Ckeyword%2Cpage%2Cproperty%2Cpub%2Cpub_raw%2Cpubdate%2Cpubnote%2Cread_count%2Ctitle%2Cvolume%2Clinks_data%2Cesources%2Cdata%2Cemail%2Cdoctype&q=docs(library%2Fra3u5GxySF6SkjCCKo7E-w)&rows=25&sort=date%20desc%2C%20bibcode%20desc&start=0' -H 'Authorization: Bearer:znycM3rMKREttSiVXvAriJrmu074t6e7yIVxjchX' -X POST
{"responseHeader":{"status":500,"QTime":3,"params":{"q":"docs(library/ra3u5GxySF6SkjCCKo7E-w)","x-amzn-trace-id":"Root=1-5cabcac3-c075d900528d3d98c65dd590","fl":"identifier,[citations],abstract,aff,author,bibcode,citation_count,comment,doi,id,keyword,page,property,pub,pub_raw,pubdate,pubnote,read_count,title,volume,links_data,esources,data,doctype","foo":"bar","start":"0","sort":"date desc, bibcode desc","fq":"{!bitset}","rows":"25","wt":"json"}},"error":{"metadata":["error-class","org.apache.solr.common.SolrException","root-error-class","org.apache.solr.common.SolrException"],"msg":"org.apache.solr.common.SolrException: Uff, uff, I have no idea how to map this field (--f7ab6787d6218b8760061cbd39b7556d) values into docids! Call 911","trace":"org.apache.solr.common.SolrException: org.apache.solr.common.SolrException: Uff, uff, I have no idea how to map this field (--f7ab6787d6218b8760061cbd39b7556d) values into docids! Call 911\n\tat org.apache.solr.search.BitSetQParserPlugin$1.parse(BitSetQParserPlugin.java:208)\n\tat org.apache.solr.search.QParser.getQuery(QParser.java:140)\n\tat org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:207)\n\tat org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:269)\n\tat org.apache.solr.handler.component.BigQuerySearchHandler.handleRequest(BigQuerySearchHandler.java:82)\n\tat org.apache.solr.core.SolrCore.execute(SolrCore.java:2213)\n\tat org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)\n\tat org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:460)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:303)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:254)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)\n\tat org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)\n\tat org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:518)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)\n\tat org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: org.apache.solr.common.SolrException: Uff, uff, I have no idea how to map this field (--f7ab6787d6218b8760061cbd39b7556d) values into docids! Call 911\n\tat org.apache.solr.search.BitSetQParserPlugin$DataProcessor.getBits(BitSetQParserPlugin.java:385)\n\tat org.apache.solr.search.BitSetQParserPlugin$1.parse(BitSetQParserPlugin.java:158)\n\t... 33 more\n","code":500}}```

so, the values passed inside the stream:

- are they encoded/compressed?
- is there some extra thing/stream?
- it should start with 'bibcode\n' if we do it, why doesn't it?

[romanchyla]

ok, so this is what gets passed into solr (from python side):

--8daf4a4ecb8841a288baf6645d7544e1=Content-Disposition: form-data; name="library/ra3u5GxySF6SkjCCKo7E-w"; filename="library/ra3u5GxySF6SkjCCKo7E-w"

[romanchyla]

ok, i think i got it

In [25]: r = requests.post('http://localhost:9983/solr/collection1/debug/dump?wt=json', data=dict(foo='bar'), files=dict(file=('foo', "bibcode\nfoo\nbar", "big-query/csv")), headers={'Content-Type': 'big-query/cs
...: v'})

In [26]: print r.text
{"responseHeader":{"status":0,"QTime":0,"handler":"org.apache.solr.handler.DumpRequestHandler","params":{"wt":"json"}},"params":{"wt":"json","echoHandler":"true","echoParams":"explicit"},"streams":[{"name":null,"sourceInfo":null,"size":270,"contentType":"big-query/csv","stream":"--12d235fec771658854ade10c2d723163\r\nContent-Disposition: form-data; name="foo"\r\n\r\nbar\r\n--12d235fec771658854ade10c2d723163\r\nContent-Disposition: form-data; name="file"; filename="foo"\r\nContent-Type: big-query/csv\r\n\r\nbibcode\nfoo\nbar\r\n--12d235fec771658854ade10c2d723163--\r\n"}],"context":{"webapp":"/solr","path":"/debug/dump","httpMethod":"POST"}}

In [27]: r = requests.post('http://localhost:9983/solr/collection1/debug/dump?wt=json', data=dict(foo='bar'), files=dict(file=('foo', "bibcode\nfoo\nbar", "big-query/csv")))

In [28]: print r.text
{"responseHeader":{"status":0,"QTime":0,"handler":"org.apache.solr.handler.DumpRequestHandler","params":{"foo":"bar","wt":"json"}},"params":{"echoHandler":"true","wt":"json","echoParams":"explicit","foo":"bar"},"streams":[{"name":"foo","sourceInfo":"file","size":15,"contentType":"big-query/csv","stream":"bibcode\nfoo\nbar"}],"context":{"webapp":"/solr","path":"/debug/dump","httpMethod":"POST"}}