Can I use the feedback endpoint without the UI?

[asharm0662]

Hi,

I am wanting to train an underlying QA model. I found out about the Annotation and feedback REST API here

How can I use these end point as a stand alone endpoints and submit the feedback directly to this endpoint without the UI?

Where can I find what format the labels need to be created under and use a template to manually create annotations? Can I get a csv example file for the labels that can be exported in SQuAD form?

Thanks in advance.

[brandenchan]

Hi @asharm0662 , cool that you've found these! So it is definitely possible to interact with the REST API without a UI. Here you will find the definition of the REST API endpoints.

Can you clarify for me what you need in terms of the label creation? The Annotation tool allows you to upload files, perform Question Answering annotation and then export this in either CSV or SQuAD format.

[asharm0662]

Hey @brandenchan ,

Thanks for the quick response. I think my question should be re-framed.

If am able to submit a question and get an answer back with below code snippet:

pipe = ExtractiveQAPipeline(reader, retriever)



prediction = pipe.run(
    query="How much did large caps in the S&P 500 post gains for?", params={"Retriever": {"top_k": 15},"Reader": {"top_k": 3}}
    )
   
#print(prediction)
print_answers(prediction, details="all")

1. What all do I submit from to the POST /feedback endpoint in the body that is relevant to the parameters in the BODY of the API call?
2. How do I submit a particular query to POST /feedback endpoint with the fields below from the prediction variable?
   a. Am i thinking incorrectly of the POST /feedback endpoint with the above questions? If yes, how should I use the POST /feedback endpoint then?
3. Do you have a small code snippet of working with the POST /feedback API endpoint I can model?
4. With your link to the API REST Endpoints ,do I not need the swagger UI turned on? Ideally how would create a dataset for training without any need for the UI being turned ?
5. Is there a certain format for the dataset for training I can add data too? For example question, correct answer, and results returned, does a dataset like this exist that I can fill out for training?

The prediction variable printed out has these fields available:

[   <Answer {'answer': '+10.4%', 'type': 'extractive', 'score': 0.928941935300827, 'context': ' performance table. Large caps in the S&P 500 have now posted a gain of +10.4% over the last twelve months compared to a loss of -3.3% for the small c', 'offsets_in_document': [{'start': 188, 'end': 194}], 'offsets_in_context': [{'start': 72, 'end': 78}], 'document_id': 'eac9cf8d01365823019c37235ce15b48', 'meta': {'name': 'Choice Equities Capital Management Q2 2019.pdf', '_split_id': 0}}>,
    <Answer {'answer': 'no one knows for sure', 'type': 'extractive', 'score': 0.04849616065621376, 'context': 'Of course no one knows for sure, but a growing economy, improving liquidity conditions and a moderate discount in relative valuations support their ca', 'offsets_in_document': [{'start': 10, 'end': 31}], 'offsets_in_context': [{'start': 10, 'end': 31}], 'document_id': 'f4dc06d105103867a7040d4832491008', 'meta': {'name': 'Choice Equities Capital Management Q2 2019.pdf', '_split_id': 36}}>,
    <Answer {'answer': '-.5%', 'type': 'extractive', 'score': 0.01743756467476487, 'context': 'our portfolio had both winners and losers in the quarter, netting out to -.5% net performance for the quarter. This latest update now means $1 investe', 'offsets_in_document': [{'start': 88, 'end': 92}], 'offsets_in_context': [{'start': 73, 'end': 77}], 'document_id': '2a2dc6e5c83cfc062978b2826d6ce87e', 'meta': {'name': 'Choice Equities Capital Management Q2 2019.pdf', '_split_id': 1}}>]

Thank you in advance.

[brandenchan]

Ok I will try my best to address these points. I believe the workflow will be something like this. Your user submits a POST request to the /query endpoint. They will receive the model's prediction that will list answers. You will then submit a POST request to the /feedback endpoint where you copy over the predicted answer and also provide the user feedback. The same can also be done for systems that return documents instead of answers I believe.

1. and 2) So this is the API structure expected when you submit a POST request to the /feedback

{
  "id": "string",
  "query": "string",
  "document": {
    "content": "string",
    "content_type": "text",
    "id": "string",
    "meta": {},
    "score": 0,
    "embedding": "string"
  },
  "is_correct_answer": true,
  "is_correct_document": true,
  "origin": "user-feedback",
  "answer": {
    "answer": "string",
    "type": "extractive",
    "score": 0,
    "context": "string",
    "offsets_in_document": [
      {
        "start": 0,
        "end": 0
      }
    ],
    "offsets_in_context": [
      {
        "start": 0,
        "end": 0
      }
    ],
    "document_id": "string",
    "meta": {}
  },
  "no_answer": true,
  "pipeline_id": "string",
  "created_at": "string",
  "updated_at": "string",
  "meta": {},
  "filters": {}
}

Fields like query, document and answer you should be able to copy from the result returned by your query. The most important fields for you to fill will be

"is_correct_answer": true,
"is_correct_document": true,

3. Unfortunately we do not have an example code snippet yet.

4. The Haystack REST API is a wrapper around a running Haystack pipeline. This can run independently of a UI. When the UI runs, it is making calls to the REST API. What I can suggest is that you have a look at our Demo which makes use of the feedback endpoint. If you tick the box for "Evaluation Mode" you will see the thumbs up and thumbs down which will provide feedback.

5. The data format we use for training Question Answering systems is essentially the SQuAD format. If you do submit the feedback to the REST API, there is an /export-feedback endpoint. Another way you can see the format we use is to make use of our annotation tool which also has a SQuAD export.

[asharm0662]

thank you ! exactly what I was looking for.

[vibha0411]

Hi,
I am trying to run feedback.py as uvicorn feedback:app
also I have installed haystack in my conda environment.
However, I still get the error ModuleNotFoundError: No module named 'haystack'
Not sure how else should I run it

[vibha0411]

In short I want to know how can I run feedback.py independently..any thoughts on this @brandenchan