Assistants API Integration

[randomwangran]

It looks like nobody has proposed any new issues here. Maybe it's the
challenge to bring all those new things into this great emacs ai
package. I'll just bring some basic the things into users attention. I
hope as we further learn the basics of how the Assistant API works, We
can extend our package with even more power.

The documentaion is Here OpenAI Platform

• Step 1: Create an Assistant

  I don't know if we can set up the assistent setting from the
  web interface instead of the Python script. Because what I
  found is that on the web interface, we have more easily
  control over the things. Ideally, I'm thinking about once we
  create a new assistant, we have a unique ID for that system,
  and then we can use that ID to communicate the things that we
  are interested in.

• Step 2: Create a Thread

• Step 3: Add a Message to a Thread

  Step two and three looks to me is the same thing, which we
  kind of start a new section to talk to this assistant. But,
  yeah, different from our previous function in this package,
  now we have ability to upload files, and in the near future we
  can upload images in the coming months.

• Step 4: Run the Assistant

• Step 5: Display the Assistant's Response

  Steps four and five is essentially we are sending our input to
  API and we expect data or output from the API and then we
  decode the content and properly process within Emacs, either
  we do region replacement or we do some magic Org-mode
  manipulations.

For me, I need to figure out a minimum elisp function to properly set
up the request information and then gather feedback from the current
API and from there I can probably tweak about the new API.

[karthink]

Support for the assistants API is not currently planned, as this is OpenAI only and I prefer to work on features that work across the board.

Moving this to a discussion in case people have suggestions or thoughts.

[ahmed-shariff]

I was playing around with setting this up. Here's my current minimal implementation of using the assistant API. My understanding is that the gptel-request API is built with single-step calls in mind. As pointed out above, the assistant API involves multiple steps. I am working around that by wrapping multiple calls to gptel-requests with different backends for each endpoint.

Note that, I have created an assistant on the openai dashboard and using the id here.

(defvar-local gptel-openai-assistant-thread-id nil)

(cl-defstruct (gptel-openai-assistant
               (:constructor gptel-openai--make-assistant-step)
               (:copier nil)
               (:include gptel-openai))
  step)

(cl-defmethod gptel--parse-response ((backend gptel-openai-assistant) response info)
  (pcase (gptel-openai-assistant-step backend)
    (:threads
     (with-current-buffer (plist-get info :buffer)
       (setq-local gptel-openai-assistant-thread-id (plist-get response :id))))
    (:messages
     "")
    (:runs
     ;; TODO: peridically poll the run and check if it's completed. If so, get last message
     ;; https://platform.openai.com/docs/assistants/deep-dive#polling-for-updates
     (error "not implemented yet!"))
    (_ (error "Unknown step"))))

(cl-defmethod gptel--request-data ((backend gptel-openai-assistant) prompts)
  (pcase (gptel-openai-assistant-step backend)
    (:threads nil)
    (:messages
     (list :role "user"
           :content `[(:type "text" :text ,(plist-get (car (last prompts)) :content))]))
    (:runs
     (let ((prompts-plist
            ;; Using cached assistant-id
            `(:assistant_id ,(gethash 'openai-assistant-id configurations)
              :stream ,(or (and gptel-stream gptel-use-curl
                                (gptel-backend-stream gptel-backend))
                           :json-false))))
       (when gptel-temperature
         (plist-put prompts-plist :temperature gptel-temperature))
       (when gptel-max-tokens
         (plist-put prompts-plist (if (memq gptel-model '(o1-preview o1-mini))
                                      :max_completion_tokens :max_tokens)
                    gptel-max-tokens))
       ;; Merge request params with model and backend params.
       (gptel--merge-plists
        prompts-plist
        (gptel-backend-request-params gptel-backend)
        (gptel--model-request-params  gptel-model))))
    (_ (error "Unknown step"))))

(cl-defmethod gptel-curl--parse-stream ((backend gptel-openai-assistant) info)
  (pcase (gptel-openai-assistant-step backend)
    (:threads
     (user-error "no streaming happening here!"))
    (:messages 
     (user-error "no streaming happening here!"))
    (:runs
     (let* ((content-strs))
       (condition-case nil
           (while (re-search-forward "^data:" nil t)
             (save-match-data
               (unless (looking-at " *\\[DONE\\]")
                 ;; TODO: Handle annotations
                 (when-let* ((response (gptel--json-read))
                             (content (map-nested-elt
                                       response '(:delta :content 0 :text :value))))
                   (push content content-strs)))))
         (error
          (goto-char (match-beginning 0))))
       (apply #'concat (nreverse content-strs))))))

(setq gptel-openai-assistant--threads-start-backend
      (gptel-openai--make-assistant-step
       :name "gptel-openai-assistant--threads-start"
       :step :threads
       :host "api.openai.com"
       :key 'gptel-api-key
       :stream nil
       :header (lambda () (when-let (key (gptel--get-api-key))
                            `(("Authorization" . ,(concat "Bearer " key))
                              ("OpenAI-Beta" . "assistants=v2"))))
       :protocol "https"
       :endpoint "/v1/threads"
       :models (gptel--process-models gptel--openai-models)
       :url (concat "https://api.openai.com/v1/threads")))

(setq gptel-openai-assistant--add-message-backend
      (gptel-openai--make-assistant-step
       :name "gptel-openai-assistant--add-message"
       :step :messages
       :host "api.openai.com"
       :key 'gptel-api-key
       :stream nil
       :header (lambda () (when-let (key (gptel--get-api-key))
                            `(("Authorization" . ,(concat "Bearer " key))
                              ("OpenAI-Beta" . "assistants=v2"))))
       :protocol "https"
       :endpoint "/v1/threads/thread_id/messages"
       :models (gptel--process-models gptel--openai-models)
       :url (lambda ()
              (if gptel-openai-assistant-thread-id
                  (format "https://api.openai.com/v1/threads/%s/messages" gptel-openai-assistant-thread-id)
                (user-error "No thread in current buffer to add messages!")))))

(setq gptel-openai-assistant--runs-backend
      (gptel-openai--make-assistant-step
       :name "gptel-openai-assistant--runs"
       :step :runs
       :host "api.openai.com"
       :key 'gptel-api-key
       :header (lambda () (when-let (key (gptel--get-api-key))
                            `(("Authorization" . ,(concat "Bearer " key))
                              ("OpenAI-Beta" . "assistants=v2"))))
       :protocol "https"
       :endpoint "/v1/threads/thread_id/runs"
       :models (gptel--process-models gptel--openai-models)
       :stream t
       :url (lambda ()
              (if gptel-openai-assistant-thread-id
                  (format "https://api.openai.com/v1/threads/%s/runs" gptel-openai-assistant-thread-id)
                (user-error "No thread in current buffer to start run!")))))

(defun gptel-openai-assistant-start-thread (&optional callback)
  "Start a assistant thread in the current buffer."
  (let ((gptel-backend gptel-openai-assistant--threads-start-backend))
    (funcall (if gptel-use-curl
                 #'gptel-curl-get-response #'gptel--url-get-response)
             `(:data ,(gptel--request-data gptel-openai-assistant--threads-start-backend nil)
                     :buffer ,(current-buffer))
             callback)))

(defun gptel-openai-assistant-add-messages (&optional callback)
  "Add a message to the current thread in the buffer."
  (let ((gptel-backend gptel-openai-assistant--add-message-backend))
    (gptel-request nil :stream nil :callback callback)))

(defun gptel-openai-assistant-run ()
  "Start a run on the thread in the current buffer."
  ;; TODO: should messages already in the thread be filtered out?
  (let ((gptel-backend gptel-openai-assistant--runs-backend))
    (gptel-request nil :stream t)))

(defun gptel-openai-assistant-send ()
  "Add message to the current thread and run it. Create thread if one is not initialized."
  (interactive)
  (let (prompt position)
    (cl-labels ((send-message-and-run (&optional _ _)
                (gptel-openai-assistant-add-messages (lambda (_ _)
                                                       (gptel-openai-assistant-run)))))
    (if gptel-openai-assistant-thread-id
        (send-message-and-run)
      (gptel-openai-assistant-start-thread #'send-message-and-run)))))

[karthink]

Wow that's clever. Nice!

My understanding is that the gptel-request API is built with single-step calls in mind.

Since I'm working on adding tool-use to gptel (see the feature-tool-use branch), this is no longer the case. Tool use requires multiple back-and-forth exchanges where the state needs to be maintained. So each request is driven by and drives a finite state machine. The state machine has

• A state, which is a label like INIT, WAIT, TOOL, DONE etc. (You can define an arbitrary set of labels and transitions)
• A list of state handlers for each state, that are called when the machine transitions to that state.
• A context stored in a plist that is updated by various state handlers as required. The context is used to determine which state to transition to next. You can add arbitrary keys to the context to store whatever info you need.
• The context stores all the state data pertaining to the request, like the backend, model, buffer, insert markers, the latest http status, tool calls, various handlers like the parser and response transformer, and more. You can pass the state machine around or store it in a variable to cleanly pause, resume or duplicate requests from any intermediate state (even across Emacs sessions), or dynamically change request parameters.

Screenshot of the request context inspector now built-into gptel:

While none of the tool-use code changes the request backend dynamically, all properties of the context are mutable.

There are no backward-incompatible changes so your above solution should continue to work. In fact I found and fixed several subtle state-management bugs in the main branch. But if you're interested in working on this feature further, I suggest using gptel-fsm and basing it off the feature-tool-use branch instead, which will be merged sometime this month. The state machine should be able to encode the state of the assistant interaction much more cleanly.

[karthink]

The individual components are documented but the package might be missing some "big picture" explanation of how the state machine driven control flow works -- the kind of thing that's better served by a quick pencil sketch. If you need help feel free to ask here.

[karthink]

Oh, a couple more things, for when you dive into the code:

• The abbreviation fsm in gptel's code stands for finite state machine.
• The above inspector panel can be accessed via (gptel--inspect-fsm) to check the latest state machine used by gptel-request in the current buffer, or (gptel--inspect-fsm some-fsm) if you have a handle to some-fsm.
• gptel--inspect-fsm omits a few verbose parts of the fsm context, including the messages vector in the data, which can be very large. However the fsm contains everything. You can access the full structure using elisp, but I recommend using the inspector package (from MELPA) as it makes it easy to navigate structs and other elisp objects interactively.

This might help "load" the shape of the fsm into your head, hopefully making it easier to write your own control flow for the assistants API.

[ahmed-shariff]

Appreciate the clarifications. I just saw this, and figured this part out the hard way 😅

Since #514 is focused on tool-use, and this thread is more focused on fsm, I'll share some initial thoughts on how I am going to go about this here:

1. For the state transitions I'll add a TYPE -> WAIT. I'm thinking another state parameter in info would let me track that.
2. The backend would have an internal state, i.e., THREAD-STARTED -> MESSAGE-SENT -> RUN, which would get processed when requests are made (in WAIT). So the fsm would be INIT -> WAIT (THREAD-STARTED) -> TYPE -> WAIT (MESSAGE-SENT) -> TYPE -> WAIT (RUN) -> TYPE -> DONE.
3. The part that I am not sure how to best approach is the url. At different states, the end-point used is different. Which means, either I do what I had done above - have different backends and swap them out based on the state - or make the gptel-backend-url aware of the states. I think the latter would be a better solution. Probably have a buffer local var for that? or is there plans to make the url function aware of the fsm?

[karthink]

1. Yes, you can set up any transition table you need, with whatever state labels are appropriate for managing the assistant state. The only thing to keep in mind is that some state transitions are hard-coded in state handlers. For example, gptel-request runs (gptel--fsm-transition fsm) at the end, as does gptel--handle-wait. The state it transitions to is determined by the transition table and not by the handler, but if you reuse these handlers you'll have to account for this in your handlers/transition-table setup.
2. I don't fully follow but that looks plausible to me 👍. Just make sure that you set up the right handlers for each state. I also suggest using as few states as you absolutely need, otherwise it adds to the cognitive load of debugging issues, even with the fsm inspection facilities now available.
3. Some notes:

As you suggest, it would be cleaner to have a single assistant gptel-backend and change the url depending on the state. You can do this via your state handler as follows:

1. Create a new backend for each assistant session: (gptel--make-openai-assistant ..). Note the double dash, we want to avoid registering this backend in gptel--known-backends as this is meant for this session only.
2. In your state handler, change the backend url to whatever it needs to be next:

(defun gptel-openai-assistant--handle-FOO (fsm) ;handler for state FOO
  (let* ((info (gptel-fsm-info fsm))
         (backend (plist-get info :backend)))
    ;; Change the backend url to whatever it next needs to be
    (setf (gptel-backend-url backend) "api.openai.com/...")
    ;; Do your other updates here, possibly modifying INFO

    ;; Only if you need to move to the next state right away:
    (gptel--fsm-transition fsm)
    ;; Otherwise the next call to `gptel-openai-assistant-send' etc should
    ;; transition the fsm forward.
    ))

Feel free to make liberal use of gptel--inspect-fsm to make sure the transitions are what you expect. For introspection, you can also add gptel--inspect-fsm to the list of state handlers to automatically pop up the inspector whenever a state transition occurs.

[ahmed-shariff]

Ah, nice! Thanx for the suggestions man. I'll try this and see where it goes.

[ahmed-shariff]

So my above solution only works if I had the fix I had discussed in #545 😅

Anyhow I ended up reimplementing everything with fsm in mind:

Here's the current implementation I have

;; This gets created when a new session/thread is started
(cl-defstruct (gptel-openai-assistant-session
               (:constructor gptel-openai--make-assistant-session)
               (:copier nil)
               (:include gptel-openai))
  (steps
   '(:threads :messages :runs)
   :documentation "The sequence of the steps.")
  (step-idx -1 :documentation "Current step in steps sequence.")
  cached-prompts
  (step
   :threads
   :documentation "The current step of the session."))

(defvar gptel-openai-assistant--known-steps nil)

(cl-defstruct (gptel-openai-assistant-step
               (:constructor gptel-openai-assistant--make-step))
  (url
   :documentation "String of function that returns the url as string")
  (request-data-fn
   :documentation "Takes backend and prompts as arg. See `gptel--request-data'")
  (parse-response-fn
   :documentation "Takes response and info as args. See `gptel--parse-response'")
  (parse-stream-fn
   :documentation "Takes info as args. See `gptel-curl--parse-stream'"))

(cl-defun gptel-openai-assistant--add-known-step (step &key url request-data-fn parse-response-fn parse-stream-fn)
  (setf (alist-get step gptel-openai-assistant--known-steps)
        (gptel-openai-assistant--make-step
         :url url :request-data-fn request-data-fn :parse-response-fn parse-response-fn :parse-stream-fn parse-stream-fn)))

(gptel-openai-assistant--add-known-step
 :threads
 :url "https://api.openai.com/v1/threads"
 :parse-response-fn (lambda (response info)
                      (with-current-buffer (plist-get info :buffer)
                        (setq-local gptel-openai-assistant-thread-id (plist-get response :id))
                        nil)))

(gptel-openai-assistant--add-known-step
 :messages
 :url (lambda ()
        (if gptel-openai-assistant-thread-id
            (format "https://api.openai.com/v1/threads/%s/messages" gptel-openai-assistant-thread-id)
          (user-error "No thread in current buffer to add messages!")))
 :request-data-fn (lambda (backend prompts)
                    (list :role "user"
                          :content
                          `[(:type "text"
                             :text ,(plist-get (car (last prompts)) :content))])))

(gptel-openai-assistant--add-known-step
 :runs
 :url (lambda ()
        (if gptel-openai-assistant-thread-id
            (format "https://api.openai.com/v1/threads/%s/runs" gptel-openai-assistant-thread-id)
          (user-error "No thread in current buffer to add messages!")))
 :request-data-fn (lambda (backend prompts)
                    (let ((_ (em gptel-stream gptel-use-curl (gptel-backend-stream backend) 333333333333))
                          (prompts-plist
                           `(:assistant_id ,(gethash 'openai-assistant-id configurations)
                                           :stream ,(or (and gptel-stream gptel-use-curl
                                                             (gptel-backend-stream backend))
                                                        :json-false))))
                      (when gptel-temperature
                        (plist-put prompts-plist :temperature gptel-temperature))
                      (when gptel-max-tokens
                        (plist-put prompts-plist (if (memq gptel-model '(o1-preview o1-mini))
                                                     :max_completion_tokens :max_tokens)
                                   gptel-max-tokens))
                      ;; Merge request params with model and backend params.
                      (gptel--merge-plists
                       prompts-plist
                       (gptel-backend-request-params backend)
                       (gptel--model-request-params  gptel-model))))
 :parse-response-fn (lambda (response info)
                      ;; TODO: peridically poll the run and check if it's completed. If so, get last message
                      ;; https://platform.openai.com/docs/assistants/deep-dive#polling-for-updates
                      (error "not implemented yet!"))
 :parse-stream-fn (lambda (info)
                    (let* ((content-strs))
                      (condition-case nil
                          (while (re-search-forward "^data:" nil t)
                            (save-match-data
                              (if (looking-at " *\\[DONE\\]")
                                  ;; Going to next step.....
                                  (gptel-openai-assistant--update-session-step info)
                                (when-let* ((response (gptel--json-read))
                                            (content (map-nested-elt
                                                      response '(:delta :content 0 :text :value)))
                                            )
                                  (push content content-strs)
                                  (when-let (annotations (map-nested-elt
                                                          response '(:delta :content 0 :text :annotations)))
                                    (cl-loop for annotation in annotations
                                             for file-id = (map-nested-elt
                                                            annotation '(:file_citation :file_id))
                                             if file-id do (push (format "(%s)" file-id) content-strs)))))))
                        (error
                         (goto-char (match-beginning 0))))
                      (apply #'concat (nreverse content-strs)))))

(defun gptel-openai-assistant--update-session-step (info &optional backend)
  (cl-assert (or info backend))
  (let* ((backend (or backend
                          (plist-get info :backend)))
         (new-step
          (when-let* ((step-idx (cl-incf (gptel-openai-assistant-session-step-idx backend)))
                      (_ (length> (gptel-openai-assistant-session-steps backend) step-idx)))
            (nth step-idx (gptel-openai-assistant-session-steps backend))))
         (target-buffer (or (and info
                                 (plist-get info :buffer))
                            ;; When being called first time, assuming it is
                            ;; done from the current-buffer
                            (current-buffer)))
         stream)
    (setf 
     (gptel-openai-assistant-session-step backend) new-step
     (gptel-backend-url backend) (with-current-buffer target-buffer
                                   (when new-step
                                     (if-let (step (alist-get new-step gptel-openai-assistant--known-steps))
                                         (let ((url (em (gptel-openai-assistant-step-url step) new-step)))
                                           (cl-typecase url
                                             (string url)
                                             (function (funcall url))
                                             (t (error "Unknown value for url (%s) in step (%s)" url new-step))))
                                       (error "Unknown step %s" new-step))))

     ;; This doesn't currently work unless I have set the process sentinel/fileter functions
     ;; stream (pcase new-step
     ;;          (:threads nil)
     ;;          (:messages nil)
     ;;          (:runs t)))
     )

    (when new-step
      (plist-put info :data (gptel--request-data backend (gptel-openai-assistant-session-cached-prompts backend))))
    (when info
      (plist-put info :wait new-step)
      ;; ;; KLUDGE: copied from `gptel-request'. Can this be managed upstream?
      ;; (plist-put info :stream (and stream
      ;;                              ;; TODO: model parameters?
      ;;                              gptel-stream gptel-use-curl
      ;;                              (gptel-backend-stream backend)))
      ;; KLUDGE: copied from `gptel-curl-get-response'. Can this be managed upstream?
      ;; (plist-put info :parser (cl--generic-method-function
      ;;                          (if stream
      ;;                              (cl-find-method
      ;;                               'gptel-curl--parse-stream nil
      ;;                               (list (aref backend 0) t))
      ;;                            (cl-find-method
      ;;                             'gptel--parse-response nil
      ;;                             (list (aref backend 0) t t)))))
      )))

(cl-defmethod gptel--request-data ((backend gptel-openai-assistant-session) prompts)
  (when (eq 0 (gptel-openai-assistant-session-step-idx backend))
    (setf (gptel-openai-assistant-session-cached-prompts backend) prompts))
  (let* ((step (gptel-openai-assistant-session-step backend))
         (step-info (alist-get step gptel-openai-assistant--known-steps)))
    (if step-info
        (when-let (fn (gptel-openai-assistant-step-request-data-fn step-info))
          (funcall fn backend prompts))
      (error "Unknown step %s" step-info))))

(cl-defmethod gptel--parse-response ((_ gptel-openai-assistant-session) response info)
  (prog1
      (let* ((step (gptel-openai-assistant-session-step (plist-get info :backend)))
             (step-info (alist-get step gptel-openai-assistant--known-steps)))
        (if step-info
            (when-let (fn (gptel-openai-assistant-step-parse-response-fn step-info))
              (funcall fn response info))
          (error "Unknown step %s" step)))
    (gptel-openai-assistant--update-session-step info)))

(cl-defmethod gptel-curl--parse-stream ((_ gptel-openai-assistant-session) info)
  (let* ((step (gptel-openai-assistant-session-step (plist-get info :backend)))
         (step-info (alist-get step gptel-openai-assistant--known-steps)))
    (if step-info
        (when-let (fn (gptel-openai-assistant-step-parse-stream-fn step-info))
          (funcall fn info))
      (error "Unknown step %s" new-step))))

(defun gptel-openai-assistant-make-session (steps &optional stream)
  (gptel-openai--make-assistant-session
   :name "gptel-openai-assistant-session"
   :step (car steps)
   :steps steps
   :host "api.openai.com"
   :key 'gptel-api-key
   :header (lambda () (when-let (key (gptel--get-api-key))
                        `(("Authorization" . ,(concat "Bearer " key))
                          ("OpenAI-Beta" . "assistants=v2"))))
   :protocol "https"
   :endpoint "/v1/threads/thread_id/runs"
   :stream stream))

(defun gptel-openai-assistant-start-thread ()
  "Start a assistant thread in the current buffer."
  (interactive)
  ;; The info hasn't yet been created either.
  ;; FIXME: is this needed?
  (setq-local gptel-backend (gptel-openai-assistant-make-session '(:threads)))
  (gptel-openai-assistant--update-session-step nil gptel-backend)
  (gptel-request nil :stream nil))

(defun gptel-openai-assistant-add-message ()
  "Start a assistant thread in the current buffer."
  (interactive)
  ;; The info hasn't yet been created either.
  ;; FIXME: is this needed?
  (setq-local gptel-backend (gptel-openai-assistant-make-session '(:messages)))
  (gptel-openai-assistant--update-session-step nil gptel-backend)
  (gptel-request nil :stream nil))

(defun gptel-openai-assistant-start-run ()
  "Start a assistant thread in the current buffer."
  (interactive)
  ;; The info hasn't yet been created either.
  ;; FIXME: is this needed?
  (setq-local gptel-backend (gptel-openai-assistant-make-session '(:runs) t))
  (gptel-openai-assistant--update-session-step nil gptel-backend)
  (gptel-request nil :stream t))

(defun gptel-openai-assistant-send ()
  "Add message to the current thread and run it. Create thread if one is not initialized."
  (interactive)
  (setq-local gptel-backend (gptel-openai-assistant-make-session '(:threads :messages)))
  (gptel-openai-assistant--update-session-step nil gptel-backend)
  (gptel-request nil :stream t))

What have I done:

• Adding transition from TYPE to WAIT
• Also added a DELAY state. I'll explain in a bit why I added this. As it is now, the TYPE state can transition to DELAY. And after some specified time (in :delay in info), the fsm would move to whichever state it would have moved to if the :delay wasn't specified.
• What I had previously would do the following
  1. Create a thread using non-streaming (:threads)
  2. Send the message (i.e., the actual prompt) using non-streaming (:messages)
  3. Request to run the thread. (:runs) I had used the streaming option. I had done this mostly because it was the method of least resistance.

Earlier, I had used the callbacks to chain the method, which works fine as the fsm is started anew for each step (i.e., a new fsm for each of :threads, :messages and :runs).
But when I try to chain them using a single fsm, I run into issues as the fsm as it is now doesn't allow mixing streaming and non-streaming. I've left in my attempts to reconcile this by modifying the data in backend and info. But since the subprocess's sentinal and filter methods are set during fsm's INIT, those don't really help.

So either I should figure out how to mix streaming and non-streaming, or implement all steps for streaming/non-streaming. I am not entirely sure how to do the former.

When doing the latter:

• To implement streaming for the :threads and :messages steps, I need to figure out how to get that data from the streaming setup. I would appreciate any pointers on this.
• To implement the non-streaming stack, I need GET methods. That is, after starting a run, we'd need to poll the endpoints with GET until we get the complete message. The DELAY state was added to allow for this to happen. Only after adding that did I realize POST is hard coded in gptel. I could advice the gptel-curl--get-args function and work around that. Wanted to get your opinion on that before I do that.

[ahmed-shariff]

Well, implementing the streaming based option turned out much simpler. Here's the full working code with some bug fixes on the above:

EDIT: missed pre-amble

(defun gptel--wait-again-p (info)
  "Check if the fsm should WAIT."
  (plist-get info :wait))

(defun gptel--delay-p (info)
  "Test fsm transition to DELAY."
  (plist-get info :delay))

(defun gptel-handle-on-wait-callback (fsm)
  "Handle on-wait-callback."
  (when-let ((callback (plist-get (gptel-fsm-info fsm) :on-wait-callback)))
    (funcall callback)))

(defun gptel--handle-delay (fsm)
  "Handle the delay state."
  (let ((delay (plist-get (gptel-fsm-info fsm) :delay)))
    (cl-assert (numberp delay))
    (plist-put (gptel-fsm-info fsm) :delay nil)
    (run-at-time delay nil (lambda () (gptel--fsm-transition fsm)))))

;; Adding TYPE -> WAIT & TYPE -> DELAY transitions
(setf (alist-get 'TYPE gptel-request--transitions) '((gptel--delay-p . DELAY)
                                                     (gptel--error-p . ERRS)
                                                     (gptel--tool-use-p . TOOL)
                                                     (gptel--wait-again-p . WAIT)
                                                     (t . DONE)))

;; MAYBE: this is useful to be connected to other states as well?
;; From DELAY go to what was expected
(setf (alist-get 'DELAY gptel-request--transitions) '((gptel--error-p . ERRS)
                                                     (gptel--tool-use-p . TOOL)
                                                     (gptel--wait-again-p . WAIT)
                                                     (t . DONE)))

(setf (alist-get 'DELAY gptel-request--handlers) '(gptel--handle-delay))

(setf (alist-get 'WAIT gptel-request--handlers) '(gptel-handle-on-wait-callback gptel--handle-wait))

(defvar-local gptel-openai-assistant-thread-id nil)

(cl-defstruct (gptel-openai-assistant-session
               (:constructor gptel-openai--make-assistant-session)
               (:copier nil)
               (:include gptel-openai))
  (steps
   '(:threads :messages :runs)
   :documentation "The sequence of the steps.")
  (step-idx -1 :documentation "Current step in steps sequence.")
  cached-prompts
  (step
   :threads
   :documentation "The current step of the session."))

(defvar gptel-openai-assistant--known-steps nil)

(cl-defstruct (gptel-openai-assistant-step
               (:constructor gptel-openai-assistant--make-step))
  (url
   :documentation "String of function that returns the url as string")
  (request-data-fn
   :documentation "Takes backend and prompts as arg. See `gptel--request-data'")
  (parse-response-fn
   :documentation "Takes response and info as args. See `gptel--parse-response'")
  (parse-stream-fn
   :documentation "Takes info as args. See `gptel-curl--parse-stream'"))

(cl-defun gptel-openai-assistant--add-known-step (step &key url request-data-fn parse-response-fn parse-stream-fn)
  (setf (alist-get step gptel-openai-assistant--known-steps)
        (gptel-openai-assistant--make-step
         :url url :request-data-fn request-data-fn :parse-response-fn parse-response-fn :parse-stream-fn parse-stream-fn)))

(gptel-openai-assistant--add-known-step
 :threads
 :url "https://api.openai.com/v1/threads"
 :parse-response-fn (lambda (response info)
                      (with-current-buffer (plist-get info :buffer)
                        (setq-local gptel-openai-assistant-thread-id (plist-get response :id))
                        nil))
 :parse-stream-fn (lambda (info)
                    (goto-char (point-min))
                    (re-search-forward "
?\n
?\n" nil t)
                    (condition-case err
                        (when-let* ((response (gptel--json-read))
                                    (_ (equal "thread" (plist-get response :object))))
                          (gptel-openai-assistant--wait-or-update-step info)
                          (with-current-buffer (plist-get info :buffer)
                            (setq-local gptel-openai-assistant-thread-id (plist-get response :id)))
                          nil)
                      (json-parse-error
                       (goto-char (match-beginning 0)))
                      (error
                       (signal (car err) (cdr err))))))

(gptel-openai-assistant--add-known-step
 :messages
 :url (lambda ()
        (if gptel-openai-assistant-thread-id
            (format "https://api.openai.com/v1/threads/%s/messages" gptel-openai-assistant-thread-id)
          (user-error "No thread in current buffer to add messages!")))
 :request-data-fn (lambda (backend prompts)
                    (list :role "user"
                          :content
                          `[(:type "text"
                             :text ,(plist-get (car (last prompts)) :content))]))
 :parse-stream-fn (lambda (info)
                    (goto-char (point-min))
                    (re-search-forward "
?\n
?\n" nil t)
                    (condition-case nil
                        (when-let* ((response (gptel--json-read))
                                    (_ (equal "thread.message" (plist-get response :object))))
                          (gptel-openai-assistant--wait-or-update-step info)
                          nil)
                      (json-parse-error
                       (goto-char (match-beginning 0)))
                      (error
                       (signal (car err) (cdr err))))))

(gptel-openai-assistant--add-known-step
 :runs
 :url (lambda ()
        (if gptel-openai-assistant-thread-id
            (format "https://api.openai.com/v1/threads/%s/runs" gptel-openai-assistant-thread-id)
          (user-error "No thread in current buffer to add messages!")))
 :request-data-fn (lambda (backend prompts)
                    (let ((prompts-plist
                           `(:assistant_id ,(gethash 'openai-assistant-id configurations)
                                           :stream ,(or (and gptel-stream gptel-use-curl
                                                             (gptel-backend-stream backend))
                                                        :json-false))))
                      (when gptel-temperature
                        (plist-put prompts-plist :temperature gptel-temperature))
                      (when gptel-max-tokens
                        (plist-put prompts-plist (if (memq gptel-model '(o1-preview o1-mini))
                                                     :max_completion_tokens :max_tokens)
                                   gptel-max-tokens))
                      ;; Merge request params with model and backend params.
                      (gptel--merge-plists
                       prompts-plist
                       (gptel-backend-request-params backend)
                       (gptel--model-request-params  gptel-model))))
 :parse-response-fn (lambda (response info)
                      ;; TODO: peridically poll the run and check if it's completed. If so, get last message
                      ;; https://platform.openai.com/docs/assistants/deep-dive#polling-for-updates
                      (error "not implemented yet!"))
 :parse-stream-fn (lambda (info)
                    (let* ((content-strs))
                      (condition-case err
                          (while (re-search-forward "^data:" nil t)
                            (save-match-data
                              (if (looking-at " *\\[DONE\\]")
                                  (prog1 nil
                                    (gptel-openai-assistant--wait-or-update-step info))
                                (when-let* ((response (gptel--json-read))
                                            (content (map-nested-elt
                                                      response '(:delta :content 0 :text :value)))
                                            )
                                  (push content content-strs)
                                  (when-let (annotations (map-nested-elt
                                                          response '(:delta :content 0 :text :annotations)))
                                    (cl-loop for annotation in annotations
                                             for file-id = (map-nested-elt
                                                            annotation '(:file_citation :file_id))
                                             if file-id do (push (format "(%s)" file-id) content-strs)))))))
                        ((json-parse-error json-end-of-file)
                         (goto-char (match-beginning 0)))
                        (error
                         (signal (car err) (cdr err))))
                      (apply #'concat (nreverse content-strs)))))

(defun gptel-openai-assistant--wait-or-update-step (info)
  (if (length>
       (gptel-openai-assistant-session-steps (plist-get info :backend))
       (1+ (gptel-openai-assistant-session-step-idx (plist-get info :backend))))
      (progn
        (plist-put info :wait t)
        (plist-put info :on-wait-callback
                   (lambda ()
                     (gptel-openai-assistant--update-session-step info))))
    (plist-put info :wait nil)))

(defun gptel-openai-assistant--update-session-step (info &optional backend)
  (cl-assert (or info backend))
  (let* ((backend (or backend
                          (plist-get info :backend)))
         (new-step
          (when-let* ((step-idx (cl-incf (gptel-openai-assistant-session-step-idx backend)))
                      (_ (length> (gptel-openai-assistant-session-steps backend) step-idx)))
            (nth step-idx (gptel-openai-assistant-session-steps backend))))
         (target-buffer (or (and info
                                 (plist-get info :buffer))
                            ;; When being called first time, assuming it is
                            ;; done from the current-buffer
                            (current-buffer)))
         stream)
    (setf 
     (gptel-openai-assistant-session-step backend) new-step
     (gptel-backend-url backend) (with-current-buffer target-buffer
                                   (when new-step
                                     (if-let (step (alist-get new-step gptel-openai-assistant--known-steps))
                                         (let ((url (gptel-openai-assistant-step-url step)))
                                           (cl-typecase url
                                             (string url)
                                             (function (funcall url))
                                             (t (error "Unknown value for url (%s) in step (%s)" url new-step))))
                                       (error "Unknown step %s" new-step))))

     ;; This doesn't currently work unless I have set the process sentinel/fileter functions
     ;; stream (pcase new-step
     ;;          (:threads nil)
     ;;          (:messages nil)
     ;;          (:runs t)))
     )

    (when new-step
      (plist-put info :data (gptel--request-data backend (gptel-openai-assistant-session-cached-prompts backend))))
    (when info
      (plist-put info :wait new-step)
      (plist-put info :on-wait-callback nil)
      ;; ;; KLUDGE: copied from `gptel-request'. Can this be managed upstream?
      ;; (plist-put info :stream (and stream
      ;;                              ;; TODO: model parameters?
      ;;                              gptel-stream gptel-use-curl
      ;;                              (gptel-backend-stream backend)))
      ;; KLUDGE: copied from `gptel-curl-get-response'. Can this be managed upstream?
      ;; (plist-put info :parser (cl--generic-method-function
      ;;                          (if stream
      ;;                              (cl-find-method
      ;;                               'gptel-curl--parse-stream nil
      ;;                               (list (aref backend 0) t))
      ;;                            (cl-find-method
      ;;                             'gptel--parse-response nil
      ;;                             (list (aref backend 0) t t)))))
      )))

(cl-defmethod gptel--request-data ((backend gptel-openai-assistant-session) prompts)
  (when (eq 0 (gptel-openai-assistant-session-step-idx backend))
    (setf (gptel-openai-assistant-session-cached-prompts backend) prompts))
  (let* ((step (gptel-openai-assistant-session-step backend))
         (step-info (alist-get step gptel-openai-assistant--known-steps)))
    (if step-info
        (when-let (fn (gptel-openai-assistant-step-request-data-fn step-info))
          (funcall fn backend prompts))
      (error "Unknown step %s" step-info))))

(cl-defmethod gptel--parse-response ((_ gptel-openai-assistant-session) response info)
  (prog1
      (let* ((step (gptel-openai-assistant-session-step (plist-get info :backend)))
             (step-info (alist-get step gptel-openai-assistant--known-steps)))
        (if step-info
            (when-let (fn (gptel-openai-assistant-step-parse-response-fn step-info))
              (funcall fn response info))
          (error "Unknown step %s" step)))
    (gptel-openai-assistant--update-session-step info)))

(cl-defmethod gptel-curl--parse-stream ((_ gptel-openai-assistant-session) info)
  (let* ((step (gptel-openai-assistant-session-step (plist-get info :backend)))
         (step-info (alist-get step gptel-openai-assistant--known-steps)))
    (if step-info
        (when-let (fn (gptel-openai-assistant-step-parse-stream-fn step-info))
          (funcall fn info))
      (error "Unknown step %s" new-step))))

(defun gptel-openai-assistant-make-session (steps &optional stream)
  (gptel-openai--make-assistant-session
   :name "gptel-openai-assistant-session"
   :step (car steps)
   :steps steps
   :host "api.openai.com"
   :key 'gptel-api-key
   :header (lambda () (when-let (key (gptel--get-api-key))
                        `(("Authorization" . ,(concat "Bearer " key))
                          ("OpenAI-Beta" . "assistants=v2"))))
   :protocol "https"
   :endpoint "/v1/threads/thread_id/runs"
   :stream stream))

(defun gptel-openai-assistant-start-thread ()
  "Start a assistant thread in the current buffer."
  (interactive)
  ;; The info hasn't yet been created either.
  ;; FIXME: is this needed?
  (when (or (not gptel-openai-assistant-thread-id)
            (y-or-n-p "There is already a thread in the buffer. Create a new thread? "))
    (setq-local gptel-backend (gptel-openai-assistant-make-session '(:threads)))
    (gptel-openai-assistant--update-session-step nil gptel-backend)
    (gptel-request nil :stream nil)))

(defun gptel-openai-assistant-add-message ()
  "Start a assistant thread in the current buffer."
  (interactive)
  ;; The info hasn't yet been created either.
  ;; FIXME: is this needed?
  (setq-local gptel-backend (gptel-openai-assistant-make-session '(:messages)))
  (gptel-openai-assistant--update-session-step nil gptel-backend)
  (gptel-request nil :stream nil))

(defun gptel-openai-assistant-start-run ()
  "Start a assistant thread in the current buffer."
  (interactive)
  ;; The info hasn't yet been created either.
  ;; FIXME: is this needed?
  (setq-local gptel-backend (gptel-openai-assistant-make-session '(:runs) t))
  (gptel-openai-assistant--update-session-step nil gptel-backend)
  (gptel-request nil :stream t))

(defun gptel-openai-assistant-send ()
  "Add message to the current thread and run it. Create thread if one is not initialized."
  (interactive)
  (setq-local gptel-backend (gptel-openai-assistant-make-session
                             (if gptel-openai-assistant-thread-id
                                 '(:messages :runs)
                               '(:threads :messages :runs))
                             t))
  (gptel-openai-assistant--update-session-step nil gptel-backend)
  (gptel-request nil :stream t))

[ahmed-shariff]

Thanx for taking a look at this. The easiest way I found to understand what is going on is to use the openai's playground. It shows the calls being made and the responses in the logs.
The loop I am trying to get to work is the following:

1. create a thread
2. add a message to the thread
3. run the thread and get the output (using streaming)

When there are new messages/prompts, steps (2) and (3) are repeated.

You are right, this went a bit out of hand 😅
Even if I do, I still have to handle how data is being processed, both, to make requests and in the responses. Each endpoint requires different urls, data, headers, and response structures. Which means custom gptel--reqest-data and gptel-curl--parse-stream/gptel--parse-response methods for each one of them. As far as I understand, state handlers alone would not be enough for this. The reason I have this gptel-openai-assistant-step struct is to avoid having to extend the backends for each endpoint. That struct is just a container to manage functionality for different endpoints/steps (see the gptel--reqest-data, gptel-curl--parse-stream and gptel--parse-response methods above). This also would make it easier to add more steps/endpoints down the line. The above 3 steps are the simplest loop I could find. I also would like to start to experiment with adding more functionality, like managing threads, vector stores, files, and whatnot.

In terms of the state management, what I have here can be moved up the fsm. What I have here is to define a sequence of steps/states as opposed to defining transitions through a table. For example in gptel-openai-assistant-send, I am specifying the steps to go through as either threads -> messages -> runs or messages -> runs depending on the state of the buffer. The gptel-openai-assistant--wait-or-update-step and gptel-openai-assistant--update-session-step (naming things is hard man!) do the state management, stepping through the sequence.

[ahmed-shariff]

EDIT: I missed some of the pre-able where I am modifying the table and handlers a bit, added them in the above post.

[karthink]

I think I understand the basics of the assistants API now. I'm trying to understand your code.

What is configurations in (gethash 'openai-assistant-id configurations)?

[karthink]

One thing I noticed is that you're creating a new backend for each message or run, like in

(setq-local gptel-backend (gptel-openai-assistant-make-session '(:messages)))

Is there some reason you can't (re)use a single backend for this?

[ahmed-shariff]

What is configurations in (gethash 'openai-assistant-id configurations)?

That's my way of storing and a bunch of kv pairs. I should have scrubbed that out when I posted it here 😅

Is there some reason you can't (re)use a single backend for this?

With the code I have here, I can't. I need to reset the internal state of the backend before I start the next request. I am working on trying to get that part to work.

[ahmed-shariff]

so, yet another implementation.

What changed:

• I was trying to avoid writing network code and use what was available in gptel. The whole "steps" thing was a result of that. Here I am copying parts of the network code from gptel and modifying it to fit the needs here better and completely ditchingthe "steps" part.
• Adding an AWAIT state to the fsm. The above means the WAIT has to be postponed until the thread is created and the messages are added to the thread. The fsm would go into this AWAIT state and wait for the transition to be called.
• Adding two methods gptel-backend--on-start-of-state and gptel-backend--on-end-of-state to the gptel-backend. Right now only the former is used. This allows the backend to have be aware of state transition, and potentially change the flow when needed. Here, when in INIT the backend will force a transition to AWAIT and allow the setup to happen. The wait-again code I had previously is used here to allow the fsm to go back to WAIT after the setup is done.
• Also, now the backend can be (re)used with the gptel-menu
• I still have not implemented the non-streaming version. But it's far more straightforward to do now.

Some setup for by adding states to the fsm:

(require 'gptel)

(defun gptel--wait-again-p (info)
  "Check if the fsm should WAIT."
  (plist-get info :wait))

(defun gptel--delay-p (info)
  "Test fsm transition to DELAY."
  (plist-get info :delay))

(defun gptel--await-p (info)
  "Test fsm transition to AWAIT."
  (plist-get info :await))

(defun gptel-handle-on-wait-again (fsm)
  "Handle on-wait-callback."
  (plist-put (gptel-fsm-info fsm) :wait nil))

;; Not used right now
(defun gptel-handle-on-wait-callback (fsm)
  "Handle on-wait-callback."
  (when-let ((callback (plist-get (gptel-fsm-info fsm) :on-wait-callback)))
    (funcall callback)
    (plist-put (gptel-fsm-info fsm) :on-wait-callback nil)))

(defun gptel--handle-delay (fsm)
  "Handle the delay state."
  (let ((delay (plist-get (gptel-fsm-info fsm) :delay)))
    (cl-assert (numberp delay))
    (plist-put (gptel-fsm-info fsm) :delay nil)
    (run-at-time delay nil (lambda () (gptel--fsm-transition fsm)))))

;; Adding TYPE -> WAIT & TYPE -> DELAY transitions
(setf (alist-get 'TYPE gptel-request--transitions) '((gptel--await-p . AWAIT)
                                                     (gptel--delay-p . DELAY)
                                                     (gptel--error-p . ERRS)
                                                     (gptel--tool-use-p . TOOL)
                                                     (gptel--wait-again-p . WAIT)
                                                     (t . DONE)))

;; Not used right now - but will be useful when trying to poll using non streaming option
;; MAYBE: this is useful to be connected to other states as well?
;; From DELAY go to what was expected
(setf (alist-get 'DELAY gptel-request--transitions) '((gptel--await-p . AWAIT)
                                                      (gptel--error-p . ERRS)
                                                      (gptel--tool-use-p . TOOL)
                                                      (gptel--wait-again-p . WAIT)
                                                      (t . DONE)))


;; From AWAIT go to what was expected
(setf (alist-get 'AWAIT gptel-request--transitions)
      '((gptel--delay-p . DELAY)
        (gptel--error-p . ERRS)
        (gptel--tool-use-p . TOOL)
        (gptel--wait-again-p . WAIT)
        (t . DONE)))

(push '(gptel--await-p . AWAIT) (alist-get 'INIT gptel-request--transitions))
(push '(gptel--await-p . AWAIT) (alist-get 'WAIT gptel-request--transitions))
(push '(gptel--await-p . AWAIT) (alist-get 'TOOL gptel-request--transitions))

(setf (alist-get 'DELAY gptel-request--handlers) '(gptel--handle-delay))

(setf (alist-get 'WAIT gptel-request--handlers) '(gptel-handle-on-wait-callback gptel-handle-on-wait-again gptel--handle-wait))

(cl-defgeneric gptel-backend--on-start-of-state (backend state info)
  "When a the gptel fsm starts a new STATE, this will be called.
This method is called before the handlers of the new state are invoked.")

(cl-defmethod gptel-backend--on-start-of-state ((backend gptel-backend) state info))

(cl-defgeneric gptel-backend--on-end-of-state (backend state info)
  "When a the gptel fsm transitions to a new state, this will be called with the old STATE.")

(cl-defmethod gptel-backend--on-end-of-state ((backend gptel-backend) state info))

(defun gptel--fsm-transition-handle-backend-on-start-on-end (oldfn fsm &optional new-state)
  "Advice for `gptel--fsm-transition'.
Invokes the `gptel-backend--on-start-of-state' & `gptel-backend--on-end-of-state'."
  (let* ((info (gptel-fsm-info fsm))
         (backend (plist-get info :backend)))
    (gptel-backend--on-end-of-state backend (gptel-fsm-state fsm) info)
    ;; the start is being called before the handlers of the next state..
    (gptel-backend--on-start-of-state backend (gptel--fsm-next fsm) info)
    (funcall oldfn fsm new-state)))

(advice-add #'gptel--fsm-transition :around #'gptel--fsm-transition-handle-backend-on-start-on-end)

Implementation of openai-assistant:

(defvar-local gptel-openai-assistant-thread-id nil)

;; TODO: Get this from api?
(defvar-local gptel-openai-assistant-assistant-id <assistant id copied from openai dashboard>)

(cl-defstruct (gptel-openai-assistant
               (:constructor gptel-openai--make-assistant)
               (:copier nil)
               (:include gptel-openai))
  messages-data)

(defvar url-http-end-of-headers)
(defun gptel--openai-assistant-url-retrive (method data url info callback)
  "Get data from URL with DATA using METHOD (POST/GET).
INFO is info from gptel
CALLBACK is called with the response from calling url-retrive."
  (let ((url-request-method "POST")
        (url-request-extra-headers
         (append `(("Content-Type" . "application/json")
                   ("Authorization" . ,(concat "Bearer " (gptel--get-api-key)))
                   ("OpenAI-Beta" . "assistants=v2"))))
        (url-request-data data))
    (url-retrieve (cl-typecase url
                    (string url)
                    (function (funcall url))
                    (t (error "Unknown value for url (%s)" url)))
                  (lambda (_)
                    (pcase-let ((`(,response ,http-status ,http-msg ,error)
                                 (custom--url-parse-response))
                                (buf (current-buffer)))
                      (plist-put info :http-status http-status)
                      (plist-put info :status http-msg)
                      (when error
                        (plist-put info :error error))
                      (with-current-buffer (plist-get info :buffer)
                        (funcall callback response))
                      (kill-buffer buf)
                      ))
                  nil t t)))

;; copied from `gptel--url-parse-response' beacuse we don't want the following:
;; (gptel--parse-response backend response proc-info)
(defun custom--url-parse-response ()
  "Parse response from url-retrive."
  (when gptel-log-level             ;logging
    (save-excursion
      (goto-char url-http-end-of-headers)
      (when (eq gptel-log-level 'debug)
        (gptel--log (gptel--json-encode (buffer-substring-no-properties (point-min) (point)))
                    "response headers"))
      (gptel--log (buffer-substring-no-properties (point) (point-max))
                  "response body")))
  (if-let* ((http-msg (string-trim (buffer-substring (line-beginning-position)
                                                     (line-end-position))))
            (http-status
             (save-match-data
               (and (string-match "HTTP/[.0-9]+ +\\([0-9]+\\)" http-msg)
                    (match-string 1 http-msg))))
            (response (progn (goto-char url-http-end-of-headers)
                             (condition-case nil
                                 (gptel--json-read)
                               (error 'json-read-error)))))
      (cond
       ;; FIXME Handle the case where HTTP 100 is followed by HTTP (not 200) BUG #194
       ((or (memq url-http-response-status '(200 100))
            (string-match-p "\\(?:1\\|2\\)00 OK" http-msg))
        (list response
              http-status http-msg))
       ((plist-get response :error)
        (list nil http-status http-msg (plist-get response :error)))
       ((eq response 'json-read-error)
        (list nil http-status (concat "(" http-msg ") Malformed JSON in response.") "json-read-error"))
       (t (list nil http-status (concat "(" http-msg ") Could not parse HTTP response.")
                "Could not parse HTTP response.")))
    (list nil (concat "(" http-msg ") Could not parse HTTP response.")
          "Could not parse HTTP response.")))

(defun gptel-openai-assistant-start-thread (info &optional callback)
  "Use the threads endpoint to start new thread.
Set the `gptel-openai-assistant-thread-id' of the buffer.
INFO is the info plist from gptel.
CALLBACK is invoked without any args after successfully creating a thread."
  (gptel--openai-assistant-url-retrive
   "POST" nil "https://api.openai.com/v1/threads"
   info
   (lambda (response)
     (with-current-buffer
         (plist-get info :buffer)
       (setq-local gptel-openai-assistant-thread-id (plist-get response :id))
       (when callback (funcall callback))))))

(defun gptel-openai-assistant-add-message (info callback)
  "Use the messages endpoint to start new thread.
Needs the `gptel-openai-assistant-thread-id' of the buffer to be set.
INFO is the info plist from gptel.
CALLBACK is invoked without any args after successfully creating a thread."
  (gptel--openai-assistant-url-retrive
   "POST"
   (encode-coding-string
    (gptel--json-encode (gptel-openai-assistant-messages-data (plist-get info :backend)))
    'utf-8)
   (with-current-buffer
       (plist-get info :buffer)
     (if gptel-openai-assistant-thread-id
         (format "https://api.openai.com/v1/threads/%s/messages"
                 gptel-openai-assistant-thread-id)
       (user-error "No thread in current buffer to add messages!")))
   info
   (lambda (response)
     (when callback (funcall callback)))))

(cl-defmethod gptel--request-data ((backend gptel-openai-assistant) prompts)
  (setf (gptel-openai-assistant-messages-data backend)
        (list :role "user"
              :content
              `[(:type "text"
                       :text ,(plist-get (car (last prompts)) :content))]))
  (let ((prompts-plist
         `(:assistant_id ,gptel-openai-assistant-assistant-id
                         :stream ,(or (and gptel-stream gptel-use-curl
                                           (gptel-backend-stream backend))
                                      :json-false))))
    (when gptel-temperature
      (plist-put prompts-plist :temperature gptel-temperature))
    (when gptel-max-tokens
      (plist-put prompts-plist (if (memq gptel-model '(o1-preview o1-mini))
                                   :max_completion_tokens :max_tokens)
                 gptel-max-tokens))
    ;; Merge request params with model and backend params.
    (gptel--merge-plists
     prompts-plist
     (gptel-backend-request-params backend)
     (gptel--model-request-params  gptel-model))))

(cl-defmethod gptel--parse-response ((_ gptel-openai-assistant) response info)
  ;; TODO: peridically poll the run and check if it's completed. If so, get last message
  ;; https://platform.openai.com/docs/assistants/deep-dive#polling-for-updates
  (error "not implemented yet!"))

(cl-defmethod gptel-curl--parse-stream ((_ gptel-openai-assistant) info)
  (let* ((content-strs))
    (condition-case err
        (while (re-search-forward "^data:" nil t)
          (save-match-data
            (unless (looking-at " *\\[DONE\\]")
              (when-let* ((response (gptel--json-read))
                          (content (map-nested-elt
                                    response '(:delta :content 0 :text :value)))
                          )
                (if-let* ((annotations (map-nested-elt
                                        response '(:delta :content 0 :text :annotations)))
                          (_ (length> annotations 0)))
                    (cl-loop for annotation across annotations
                             for file-id = (map-nested-elt
                                            annotation '(:file_citation :file_id))
                             if file-id
                             do (push (format "[[file_citation:%s][%s]]" file-id content) content-strs))
                  (push content content-strs))))))
      ((json-parse-error json-end-of-file search-failed)
       (goto-char (match-beginning 0)))
      (error
       (signal (car err) (cdr err))))
    (apply #'concat (nreverse content-strs))))

(cl-defmethod gptel--handle-openai-assistant-await (fsm)
  (let* ((info (gptel-fsm-info fsm))
         (await-manual-state (plist-get info :await)))
    (when (gptel-openai-assistant-p (plist-get info :backend))
     ;; This tells us await manual is happening from after the init
      (pcase await-manual-state
        (:send-message
         (plist-put info :await nil)
         (cl-labels ((send-message ()
                       (gptel-openai-assistant-add-message
                        info
                        (lambda ()
                          (plist-put info :wait t)
                          (gptel--fsm-transition fsm)))))
           (if gptel-openai-assistant-thread-id
               (send-message)
             (gptel-openai-assistant-start-thread
              info
              (lambda ()
                (if (plist-get info :error)
                    (gptel--fsm-transition fsm)
                  (send-message)))))))
        (:await-runs-complete
         (plist-put info :await nil)
         ;; TODO: The polling can happen here
         (error "Not implemented yet"))))))
     
(cl-defmethod gptel-backend--on-end-of-state ((backend gptel-openai-assistant) state info)
  (when (eq state 'INIT)
    (plist-put info :await :send-message)))

;;;###autoload
(defun gptel-openai-make-assistant ()
  "Create a openai-assistant backend."
  (gptel-openai--make-assistant
   :name "gptel-openai-assistant"
   :host "api.openai.com"
   :key 'gptel-api-key
   :models (gptel--process-models gptel--openai-models)
   :header (lambda () (when-let (key (gptel--get-api-key))
                        `(("Authorization" . ,(concat "Bearer " key))
                          ("OpenAI-Beta" . "assistants=v2"))))
   :protocol "https"
   :endpoint "/v1/threads/thread_id/runs"
   :url (lambda ()
          (if gptel-openai-assistant-thread-id
              (format "https://api.openai.com/v1/threads/%s/runs" gptel-openai-assistant-thread-id)
            (user-error "No thread in current buffer to add messages!")))
   :stream t))

;;;###autoload
(defun gptel-openai-assistant-create-new-thread ()
  "Create a new thread in the current buffer."
  (interactive)
  (gptel-openai-assistant-start-thread `(:buffer ,(buffer-name))))

(push 'gptel--handle-openai-assistant-await (alist-get 'AWAIT gptel-request--handlers))

(setf (alist-get "openai-assistant" gptel--known-backends
                 nil nil #'equal)
      (gptel-openai-make-assistant))

[karthink]

You're partially correct, but perhaps not for the reason you think?

I sometimes switch between backends and also modify the other parameters (i.e.., context, instructions, tool-use, etc.) in the same session using gptel-menu. To allow this to happen, my understanding is the backend needs to be registered as one of the known backends

The backend needs to be registered in gptel--known-backends for you to change the backend/model using the menu option (-m). Otherwise it won't show up in the list of backends. This is the only menu item for which registering the backend matters.

Everything else just sets independent elisp variables or transient infixes, so it doesn't matter how the request is sent afterwards.

so that I can do gptel-menu > update parameters/arguments > gptel-send. Having a custom send would not allow this, right? or am I misunderstanding the flow?

The answer depends on what exactly you're doing.

If you use a custom send command (i.e. one that uses custom transitions/handlers, OR does extra processing as in Method 1)

It's not clear what you mean by gptel-send here, because there are actually two versions:

• M-x gptel-send.
• M-x gptel--suffix-send, the transient suffix that is run when you press RET in the transient menu.

Case 1:

1. Set a bunch of options from gptel-menu, quit the menu.
2. Run M-x gptel-assistant-send, your custom send function.

All the variables you set in the menu will be respected by gptel-assistant-send.

Case 2:

1. Set a bunch of options from gptel-menu.
2. Press RET in the menu, which calls gptel--suffix-send.

All the options you set in the menu are respected, but not the extra processing you do in gptel-assistant-send using Method 1.

The solution: find a way to replace gptel--suffix-send with gptel--suffix-assistant-send in gptel-menu, another custom command that does the processing you need. Or find a way to pass gptel--suffix-send a custom fsm, or fsm transitions/handlers.

If you modify gptel-request--transitions and gptel-request--handlers (i.e. no custom send command, reuse gptel-send)

In this case gptel-send and gptel--suffix-send will automatically do the right thing -- i.e. work as assistant commands in an assistant buffer, but the the transitions/handlers have to be modified carefully to not clobber the use of gptel-send in non-assistant contexts. I cover this below.

Running the INIT handlers

By having the caveat that the handler of a state would not get executed the first time if it is the initial state would mean during the initial step of the fsm, the work of the particular handler would need to be delegated to another state. At which point, it may not make sense to have any handlers for a state that is the initial state - might as well have the initial state transition to followup/prep state which would trigger all handlers every time the fsm arrives at that state. Not sure if that makes sense

It makes sense. I agree that the INIT state is useless when the request is initiated. It works as expected from then on, if there's ever a SOME-STATE -> INIT transition.

However this is not easy to fix in the way that you need it. I can run the INIT handlers when the fsm is initialized, but that won't help since you need the payload to be ready first, i.e. for gptel-request to have run already.

Essentially, you need a state in between INIT and WAIT to prepare the assistant thread. Let's call it ASST. You would modify the table and handlers so it looks like:

(defvar gptel-request--transitions
  `((INIT . ((,#'gptel-openai-assistant-p   . ASST) ;<= THIS IS NEW
             (t                       . WAIT)))
    (ASST . ((t                       . WAIT))) ;<= THIS IS NEW
    (WAIT . ((t                       . TYPE)))
    (TYPE . ((,#'gptel--error-p       . ERRS)
             (,#'gptel--tool-use-p    . TOOL)
             (t                       . DONE)))
    (TOOL . ((,#'gptel--error-p       . ERRS)
             (,#'gptel--tool-result-p . WAIT)
             (t                       . DONE)))))

(defvar gptel-request--handlers
  `((ASST ,#'gptel-openai-assistant--handle-prep) ;<= THIS IS NEW
    (WAIT ,#'gptel--handle-wait)
    (TYPE ,#'gptel--handle-pre-insert)
    (ERRS ,#'gptel--handle-error ,#'gptel--fsm-last)
    (TOOL ,#'gptel--handle-tool-use)
    (DONE ,#'gptel--handle-post-insert ,#'gptel--fsm-last)))

Now you have a state (ASST), controlled by you, where you can do whatever prep you need in between creating the payload (in gptel-request) and sending it (in gptel--handle-wait). gptel-openai-assistant--handle-prep can set up the assistant thread asynchronously, and run (gptel--fsm-transition) in its callback to move to WAIT.

There are two ways to use this solution.

1. You can give the transitions/handlers custom names and define a custom send command, but this has the Case 2 problem from above.
2. If you overload the defaults, assuming gptel-openai-assistant-p does its job, you don't need a custom send command, and non-assistant gptel-send invocations should not be affected at all.

General thoughts on the FSM

The idea behind creating a generic state machine interface was that folks could write packages with custom workflows and custom UIs reusing gptel-request (which does a bunch of plumbing that would be tedious to replicate from scratch). I didn't foresee that they might want to reuse gptel's UI, which is pretty basic, but with custom workflows.

Ultimately, there is a little bit of coupling between gptel's default transitions/handlers and its UI, but only because of gptel--suffix-send. We can find a way to feed this transient suffix a custom fsm/fsm handlers if required.

[ahmed-shariff]

Thanks for taking the time to provide a detailed response, it is much appreciated.

The backend needs to be registered in gptel--known-backends for you to change the backend/model using the menu option (-m). Otherwise it won't show up in the list of backends. This is the only menu item for which registering the backend matters.

I hadn't realized that. I'll keep that in mind.

It's not clear what you mean by gptel-send here

I was referring to gptel--suffix-send.

I didn't foresee that they might want to reuse gptel's UI, which is pretty basic, but with custom workflows.

I think this is something the fsm implementation affords. Otherwise, I am not sure this would have been possible without some extensive advising around some of the core functions. In this regard, I think this would be a much preferable way when building new workflows:

If you modify gptel-request--transitions and gptel-request--handlers (i.e. no custom send command, reuse gptel-send)

In a way, it would provide a standard way to implement a lot of new workflows while using the same UI. But you are right, if I am not careful, I can easily mess up the base flow of gptel-send/gptel--suffix-send. In fact, there were a few times I found myself stuck in an infinite loop while experimenting with that. I've learned the hard way to have customization to the global transition table to always limit itself to the backend I am writing (e.g., with gptel-openai-assistant-p).

We can find a way to feed this transient suffix a custom fsm/fsm handlers if required.

That would be a cool feature to have with gptel. This would allow implementing new workflows without stepping on the global or even other workflows. But off the top of my head, I can see it getting a bit wild. For instance, the fsm and backend may be tied like in my case - where the transition-table and handlers are specific to the backend I am using. This would require some way to assess compatibility between backend and fsm. One naive solution might be to have a method that specializes on the backend to initialize and return an fsm which is used by gptel-request when not passed in as a parameter. But I am not sure what the implications of that are. Either way, if you need contributions to try and get this working, I'd be happy to help.

Essentially, you need a state in between INIT and WAIT to prepare the assistant thread. Let's call it ASST.

Yes, this would be the solution to go with for now.

However this is not easy to fix in the way that you need it. I can run the INIT handlers when the fsm is initialized, but that won't help since you need the payload to be ready first, i.e. for gptel-request to have run already.

This was why the patch I had shown was at the end of gptel-request, after the payload has been prepped and right before control is passed to the fsm.

[karthink]

We can find a way to feed this transient suffix a custom fsm/fsm handlers if required.

That would be a cool feature to have with gptel. This would allow implementing new workflows without stepping on the global or even other workflows.

Assuming the openai assistant session is tied to a buffer, an easy way to achieve this for the assistant is to just set gptel-request--transitions and gptel-request--handlers buffer-locally. Then both gptel-send and gptel--suffix-send should work seamlessly with your custom handlers without having to worry about breaking gptel-send in non-assistant buffers.

gptel already sets the backend, model and request parameters buffer-locally anyway.

[ahmed-shariff]

True. But that also means, that if I switch to a different backend in the same session, those variables would also need to change?

[karthink]

Oh yes, unfortunately. A more bespoke solution is probably required to feed gptel--suffix-send the right transition table/handlers. My current assessment is that

• If you carefully modify the current transition table/handlers, all of gptel's UI works without any changes. You can move ahead with this solution for now.
• If you define a new transition table/handlers, everything except gptel--suffix-send works.

[ahmed-shariff]

So, here's a full working version for both streaming and non-streaming versions. One issue that I ran into with implementing the non-streaming option was with gptel always sending POST requests. However, ideally, the GET method would be used to get the messages after the run has completed. As a workaround I am using the modify messages endpoint where I request an empty modification, which also returns the messages I am looking for.

;; Openai assistant api ******************************************************************
(defvar-local gptel-openai-assistant-thread-id nil)

;; TODO: Get this from api?
(defvar-local gptel-openai-assistant-assistant-id "<assistant id from dashboard>")

;; Helper functions **********************************************************************
(defvar url-http-end-of-headers)
(defun gptel-openai-assistant--url-retrive (method data url info callback)
  "Get data from URL with DATA using METHOD (POST/GET).
INFO is info from gptel
CALLBACK is called with the response from calling url-retrive."
  (let ((url-request-method method)
        (url-request-extra-headers
         (append `(("Content-Type" . "application/json")
                   ("Authorization" . ,(concat "Bearer " (gptel--get-api-key)))
                   ("OpenAI-Beta" . "assistants=v2"))))
        (url-request-data data))
    (when gptel-log-level               ;logging
      (when (eq gptel-log-level 'debug)
        (gptel--log (gptel--json-encode
                     (mapcar (lambda (pair) (cons (intern (car pair)) (cdr pair)))
                             url-request-extra-headers))
                    "request headers"))
      (when url-request-data
        (gptel--log url-request-data "request body")))
    (url-retrieve (cl-typecase url
                    (string url)
                    (function (funcall url))
                    (t (error "Unknown value for url (%s) in step" url)))
                  (lambda (_)
                    (pcase-let ((`(,response ,http-status ,http-msg ,error)
                                 (gptel-openai-assistant--url-parse-response))
                                (buf (current-buffer)))
                      (plist-put info :http-status http-status)
                      (plist-put info :status http-msg)
                      (when error
                        (plist-put info :error error))
                      (with-current-buffer (plist-get info :buffer)
                        (funcall callback response))
                      (kill-buffer buf)
                      ))
                  nil t t)))

;; copied from `gptel--url-parse-response' beacuse we don't want the following:
;; (gptel--parse-response backend response proc-info)
(defun gptel-openai-assistant--url-parse-response ()
  "Parse response from url-retrive."
  (when gptel-log-level             ;logging
    (save-excursion
      (goto-char url-http-end-of-headers)
      (when (eq gptel-log-level 'debug)
        (gptel--log (gptel--json-encode (buffer-substring-no-properties (point-min) (point)))
                    "response headers"))
      (gptel--log (buffer-substring-no-properties (point) (point-max))
                  "response body")))
  (if-let* ((http-msg (string-trim (buffer-substring (line-beginning-position)
                                                     (line-end-position))))
            (http-status
             (save-match-data
               (and (string-match "HTTP/[.0-9]+ +\\([0-9]+\\)" http-msg)
                    (match-string 1 http-msg))))
            (response (progn (goto-char url-http-end-of-headers)
                             (condition-case nil
                                 (gptel--json-read)
                               (error 'json-read-error)))))
      (cond
       ;; FIXME Handle the case where HTTP 100 is followed by HTTP (not 200) BUG #194
       ((or (memq url-http-response-status '(200 100))
            (string-match-p "\\(?:1\\|2\\)00 OK" http-msg))
        (list response
              http-status http-msg))
       ((plist-get response :error)
        (list nil http-status http-msg (plist-get response :error)))
       ((eq response 'json-read-error)
        (list nil http-status (concat "(" http-msg ") Malformed JSON in response.") "json-read-error"))
       (t (list nil http-status (concat "(" http-msg ") Could not parse HTTP response.")
                "Could not parse HTTP response.")))
    (list nil (concat "(" http-msg ") Could not parse HTTP response.")
          "Could not parse HTTP response.")))


;; Core function to work with gptel ******************************************************
(cl-defstruct (gptel-openai-assistant
               (:constructor gptel-openai--make-assistant)
               (:copier nil)
               (:include gptel-openai))
  messages-data)

(defun gptel-openai-assistant-start-thread (info &optional callback)
  "Use the threads endpoint to start new thread.
Set the `gptel-openai-assistant-thread-id' of the buffer.
INFO is the info plist from gptel.
CALLBACK is invoked without any args after successfully creating a thread."
  (gptel-openai-assistant--url-retrive
   "POST" nil "https://api.openai.com/v1/threads"
   info
   (lambda (response)
     (with-current-buffer
         (plist-get info :buffer)
       (setq-local gptel-openai-assistant-thread-id (plist-get response :id))
       (when callback (funcall callback))))))

(defun gptel-openai-assistant-add-message (info callback)
  "Use the messages endpoint to start new thread.
Needs the `gptel-openai-assistant-thread-id' of the buffer to be set.
INFO is the info plist from gptel.
CALLBACK is invoked without any args after successfully creating a thread."
  (gptel-openai-assistant--url-retrive
   "POST"
   (encode-coding-string
    (gptel--json-encode (gptel-openai-assistant-messages-data (plist-get info :backend)))
    'utf-8)
   (with-current-buffer
       (plist-get info :buffer)
     (if gptel-openai-assistant-thread-id
         (format "https://api.openai.com/v1/threads/%s/messages"
                 gptel-openai-assistant-thread-id)
       (user-error "No thread in current buffer to add messages!")))
   info
   (lambda (response)
     (when callback (funcall callback)))))

(cl-defmethod gptel--request-data ((backend gptel-openai-assistant) prompts)
  (setf (gptel-openai-assistant-messages-data backend)
        (list :role "user"
              :content
              `[(:type "text"
                       :text ,(plist-get (car (last prompts)) :content))]))
  (if (and gptel-stream gptel-use-curl
           (gptel-backend-stream backend))
      (let ((prompts-plist
             `(:assistant_id ,gptel-openai-assistant-assistant-id
                             :stream ,(or (and gptel-stream gptel-use-curl
                                               (gptel-backend-stream backend))
                                          :json-false))))
        (when gptel-temperature
          (plist-put prompts-plist :temperature gptel-temperature))
        (when gptel-max-tokens
          (plist-put prompts-plist (if (memq gptel-model '(o1-preview o1-mini))
                                       :max_completion_tokens :max_tokens)
                     gptel-max-tokens))
        ;; Merge request params with model and backend params.
        (gptel--merge-plists
         prompts-plist
         (gptel-backend-request-params backend)
         (gptel--model-request-params  gptel-model)))
    `(:metadata nil)))

(cl-defmethod gptel--parse-response ((_ gptel-openai-assistant) response info)
  (string-join
   (mapcar
    (lambda (content)
      (let ((str (map-nested-elt content '(:text :value))))
        (mapcar
         (lambda (annotation)
           (setf str
                 (string-replace
                  (plist-get annotation :text)
                  (format "[file_citation:%s]" (map-nested-elt annotation '(:file_citation :file_id)))
                  str)))
         (map-nested-elt content '(:text :annotations)))
        str))
      (plist-get response :content))
     " "))

(cl-defmethod gptel-curl--parse-stream ((_ gptel-openai-assistant) info)
  (let* ((content-strs))
    (condition-case err
        (while (re-search-forward "^data:" nil t)
          (save-match-data
            (unless (looking-at " *\\[DONE\\]")
              (when-let* ((response (gptel--json-read))
                          (content (map-nested-elt
                                    response '(:delta :content 0 :text :value)))
                          )
                (if-let* ((annotations (map-nested-elt
                                        response '(:delta :content 0 :text :annotations)))
                          (_ (length> annotations 0)))
                    (cl-loop for annotation across annotations
                             for file-id = (map-nested-elt
                                            annotation '(:file_citation :file_id))
                             if file-id
                             do (push (format "[file_citation:%s]" file-id content) content-strs))
                  (push content content-strs))))))
      ((json-parse-error json-end-of-file search-failed)
       (goto-char (match-beginning 0)))
      (error
       (signal (car err) (cdr err))))
    (apply #'concat (nreverse content-strs))))

(defun gptel-openai-assistant--handle-await (fsm)
  (let* ((info (gptel-fsm-info fsm))
         (await-manual-state
          (or (plist-get info :openai-assistant-await)
              (let ((history (plist-get info :history)))
                (when (and
                       (eq (car history) 'INIT)
                       (null (cdr history)))
                  :send-message)))))
    (plist-put info :openai-assistant-await nil)
    (when (gptel-openai-assistant-p (plist-get info :backend))
      (with-current-buffer (plist-get (gptel-fsm-info fsm) :buffer)
        (gptel--update-status " Waiting..." 'warning))
      ;; This tells us await manual is happening from after the init
      (pcase await-manual-state
        (:send-message
         (cl-labels ((send-message ()
                       (gptel-openai-assistant-add-message
                        info
                        (lambda ()
                          (if (plist-get info :stream)
                              (plist-put info :openai-assistant-wait t)
                            (plist-put info :openai-assistant-await :openai-assistant-runs))
                          (gptel--fsm-transition fsm)))))
           (if gptel-openai-assistant-thread-id
               (send-message)
             (gptel-openai-assistant-start-thread
              info
              (lambda ()
                (if (plist-get info :error)
                    (gptel--fsm-transition fsm)
                  (send-message)))))))
        (:openai-assistant-runs
         (gptel-openai-assistant--url-retrive
          "POST"
          (encode-coding-string
           (gptel--json-encode `(:assistant_id ,gptel-openai-assistant-assistant-id))
           'utf-8)
          (with-current-buffer
              (plist-get info :buffer)
            (if gptel-openai-assistant-thread-id
                (format "https://api.openai.com/v1/threads/%s/runs"
                        gptel-openai-assistant-thread-id)
              (user-error "No thread in current buffer to add messages!")))
          info
          (lambda (response)
            (unless (plist-get info :error)
              (plist-put info :openai-assistant-run-id (plist-get response :id))
              (plist-put info :openai-assistant-await :openai-assistant-runs-completed-p)
              (plist-put info :openai-assistant-delay 0.5))
            (gptel--fsm-transition fsm))))
        (:openai-assistant-runs-completed-p
         (gptel-openai-assistant--url-retrive
          "GET"
          nil
          (with-current-buffer
              (plist-get info :buffer)
            (if gptel-openai-assistant-thread-id
                (format "https://api.openai.com/v1/threads/%s/runs/%s"
                        gptel-openai-assistant-thread-id
                        (plist-get info :openai-assistant-run-id))
              (user-error "No thread in current buffer to add messages!")))
          info
          (lambda (response)
            (unless (plist-get info :error)
              (pcase (plist-get response :status)
                ((or "in_progress" "queued")
                 (plist-put info :openai-assistant-await :openai-assistant-runs-completed-p)
                 (plist-put info :openai-assistant-delay 0.5))
                ("completed"
                 (plist-put info :openai-assistant-await :openai-assistant-list-messages))
                (status (error "Unhandle state for run %s" status))))
            (gptel--fsm-transition fsm))))
        (:openai-assistant-list-messages
         (gptel-openai-assistant--url-retrive
          "GET"
          nil
          (with-current-buffer
              (plist-get info :buffer)
            (if gptel-openai-assistant-thread-id
                (format "https://api.openai.com/v1/threads/%s/messages?run_id=%s"
                        gptel-openai-assistant-thread-id
                        (plist-get info :openai-assistant-run-id))
              (user-error "No thread in current buffer to add messages!")))
          info
          (lambda (response)
            (unless (plist-get info :error)
              (plist-put info :openai-assistant-message-id (map-nested-elt response '(:data 0 :id)))
              (plist-put info :openai-assistant-wait t))
            (gptel--fsm-transition fsm))))
        (_ (error "Unknown await state %s" await-manual-state))))))

;;;###autoload
(defun gptel-make-openai-assistant ()
  "Create a openai-assistant backend."
  (gptel-openai--make-assistant
   :name "gptel-openai-assistant"
   :host "api.openai.com"
   :key 'gptel-api-key
   :models (gptel--process-models gptel--openai-models)
   :header (lambda () (when-let (key (gptel--get-api-key))
                        `(("Authorization" . ,(concat "Bearer " key))
                          ("OpenAI-Beta" . "assistants=v2"))))
   :protocol "https"
   :endpoint "/v1/threads/thread_id/runs"
   :url (lambda ()
          (if gptel-openai-assistant-thread-id
              (format "https://api.openai.com/v1/threads/%s/runs" gptel-openai-assistant-thread-id)
            (user-error "No thread in current buffer to add messages!")))
   :stream t))

;;;###autoload
(defun gptel-openai-assistant-create-new-thread ()
  "Create a new thread in the current buffer."
  (interactive)
  (gptel-openai-assistant-start-thread `(:buffer ,(buffer-name))))

;; Modify gptel vars *********************************************************************

(setf (alist-get "openai-assistant" gptel--known-backends
                 nil nil #'equal)
      (gptel-make-openai-assistant))

(defun gptel-openai-assistant--backend-is-oaia-p (info)
  "Check if backend is openai-assistant."
  (gptel-openai-assistant-p (plist-get info :backend)))

(defun gptel-openai-assistant--init-to-await (info)
  "If in first INIT, move to AWAIT"
  (and (gptel-openai-assistant--backend-is-oaia-p info)
       ;; If :history is not set or empty, means state is at INIT
       (null (plist-get info :history))))

(defun gptel-openai-assistant--wait-again-p (info)
  "Check if the fsm should WAIT."
  (and (gptel-openai-assistant--backend-is-oaia-p info)
       (plist-get info :openai-assistant-wait)))

(defun gptel-openai-assistant--delay-p (info)
  "Test fsm transition to DELAY."
  (and (gptel-openai-assistant--backend-is-oaia-p info)
       (plist-get info :openai-assistant-delay)))

(defun gptel-openai-assistant--await-p (info)
  "Test fsm transition to AWAIT."
  (and (gptel-openai-assistant--backend-is-oaia-p info)
       (plist-get info :openai-assistant-await)))

(defun gptel-openai-assistant--handle-wait (fsm)
  "Handle wait-again."
  (let ((info (gptel-fsm-info fsm)))
    (when (gptel-openai-assistant--backend-is-oaia-p info)
      (unless gptel-openai-assistant-thread-id
        (user-error "No thread in current buffer to add messages!"))
      (plist-put info :openai-assistant-wait nil)
      (setf (gptel-backend-url (plist-get info :backend))
            (if (plist-get info :stream)
                (format "https://api.openai.com/v1/threads/%s/runs" gptel-openai-assistant-thread-id)
              ;; FIXME: gptel-always sends POST. Ideally, this should be the GET endpoint
              ;; So using the endpoint for updating messages to get messages related to a run.
              (format "https://api.openai.com/v1/threads/%s/messages/%s"
                      gptel-openai-assistant-thread-id
                      (plist-get info :openai-assistant-message-id)))))))

(defun gptel-openai-assistant--handle-delay (fsm)
  "Handle the delay state."
  (let ((info (gptel-fsm-info fsm)))
    (and (gptel-openai-assistant--backend-is-oaia-p info)
         (let ((delay (plist-get info :openai-assistant-delay)))
           (cl-assert (numberp delay))
           (plist-put info :openai-assistant-delay nil)
           (run-at-time delay nil (lambda () (gptel--fsm-transition fsm)))))))

;; INIT should transition to AWAIT
(push '(gptel-openai-assistant--init-to-await . AWAIT) (alist-get 'INIT gptel-request--transitions))

;; Adding TYPE -> WAIT/DELAY/AWAIT
;; DELAY & AWAIT has to happen at the very beginning
;; WAIT should be tested right before DONE
(setf (alist-get 'TYPE gptel-request--transitions)
      (append 
       '((gptel-openai-assistant--delay-p . DELAY)
         (gptel-openai-assistant--await-p . AWAIT))
       ;; *sigh*
       (cl-loop for transition in (alist-get 'TYPE gptel-request--transitions)
                ;; Insert the wait again, right before DONE
                if (eq (cdr transition) 'DONE)
                collect '(gptel-openai-assistant--wait-again-p . WAIT) into retval
                collect transition into retval
                finally return retval)))

;; From DELAY go to what was expected
;; Same as TYPE but without the DELAY itself!
(setf (alist-get 'DELAY gptel-request--transitions) (cl-loop for transition in (alist-get 'TYPE gptel-request--transitions)
                                                             unless (eq (cdr transition) 'DELAY)
                                                             collect transition))

;; From AWAIT go to what was expected. Same as TYPE
(setf (alist-get 'AWAIT gptel-request--transitions) (alist-get 'TYPE gptel-request--transitions))

;; AWAIT should be tested for first in WAIT and TOOL
(push '(gptel-openai-assistant--await-p . AWAIT) (alist-get 'WAIT gptel-request--transitions))
(push '(gptel-openai-assistant--await-p . AWAIT) (alist-get 'TOOL gptel-request--transitions))

;; Adding DELAY handler
(push '(DELAY gptel-openai-assistant--handle-delay) gptel-request--handlers)

;; Handle the wait-again in WAIT
(push 'gptel-openai-assistant--handle-wait (alist-get 'WAIT gptel-request--handlers))

;; Handle AWAIT
(push '(AWAIT gptel-openai-assistant--handle-await) gptel-request--handlers)

[karthink]

In the above code, gptel-request--handlers needs to be changed to gptel-send--handlers, which see.

I took a longer look at the code today. Still not sure how many of these transitions can occur, like WAIT -> AWAIT or TYPE -> AWAIT, but it appears to work fine so far. It looks like you've combined what could be a few different state machine states into the AWAITstate -- is this assessment accurate?

For things like tool-use and file-uploads, which I imagine is most of the reason to use the assistants API, you'll have to re-use gptel-openai's methods. I don't know if the difference in JSON output between the chat and assistants API extends also to how tool-use is reported. If it does, we're in trouble and you'll have to write a fairly annoying function, as you can see in gptel-openai's current implementation of gptel-curl--parse-stream.

[ahmed-shariff]

Still not sure how many of these transitions can occur, like WAIT -> AWAIT or TYPE -> AWAIT, but it appears to work fine so far.

Right now, these don't happen. I left them in there so that I can handle tool-use/function-calls and code-interpreter as well.

It looks like you've combined what could be a few different state machine states into the AWAIT state -- is this assessment accurate?

Yes, that is correct. It might be better to rename that openai-assistant-await or something similar to avoid confusion. I am also doing this to avoid adding transitions/handlers for each sub-state - that would be too many states to juggle, especially to make sure they are setup correctly. But the downside of what I have done is I have slightly less visibility when I am using fsm introspection.

I don't know if the difference in JSON output between the chat and assistants API extends also to how tool-use is reported

As far as I understand the tool-use structure itself seems to be the same. Where and in which endpoint it is triggered seems to be different. For now, I am exclusively using the assistant API for file-search, I'll play around with the rest when I get some time.

[ahmed-shariff]

But the downside of what I have done is I have slightly less visibility when I am using fsm introspection.

Now that I think about it, tampering with the :history in info shouldn't break anything in gptel, correct? i.e., I can push the substrates to history, right?

[karthink]

Right now, these don't happen. I left them in there so that I can handle tool-use/function-calls and code-interpreter as well.

Okay.

...that would be too many states to juggle, especially to make sure they are setup correctly.

Perhaps. I'll leave it to you.

Now that I think about it, tampering with the :history in info shouldn't break anything in gptel, correct? i.e., I can push the substrates to history, right?

I forgot to mention -- it is OK to push things onto :history, but it's meant for introspection only. I would try to avoid using the value of :history as a test when determining transitions, as you do in one place right now. While nothing breaks today, I don't know about the future. it's also no longer a state transition machine if you use the history instead of the state to make decisions!

[ahmed-shariff]

Yeah, I was using the :history to see if its the first transition - incase some other fsm flow results in coming back to INIT. Not sure if it is an actual concern or I am overthinking it 😅 . A better option would be to set a different flag once I come to the AWAIT state which I can check in the INIT handler.

[karthink]

As far as I understand the tool-use structure itself seems to be the same. Where and in which endpoint it is triggered seems to be different. For now, I am exclusively using the assistant API for file-search, I'll play around with the rest when I get some time.

You can use cl-call-next-method in your implementation of gptel-curl--parse-stream and gptel--parse-response to run the openai methods and capture the tool calls. (See the newly added gptel-perplexity backend in gptel-openai-extras.el for an example)

[ahmed-shariff]

That's neat, I'll take a closer look at the newer implementations and see how I use that.

[karthink]

@ahmed-shariff Where are we on the Assistants API integration? Is there anything I need to do?

[karthink]

regarding the TODO's itself, if you have a specific design in mind

I don't yet have an idea of how to supply a transition table to gptel--suffix-send (in the transient menu). Designating a column for adding custom entries to gptel-menu is simple, I just have to pick a location in the menu. The menu gets redesigned now and then, so the "custom column/group" needs to be able to work anywhere in the larger menu.

as in, gptel-send is a specific implementation of or uses gptel-menu?

Yeah, that's one way to solve it. But the way it's currently engineered both gptel-send and gptel--suffix-send are just calls to gptel-request, so they are more siblings derived from gptel-request than from each other. gptel-request can take a custom :fsm, so I just have to find a way to get this fsm through gptel--suffix-send to it.

[karthink]

Oh, I think I found a way. Are you okay with using a buffer-local variable to store the fsm/transition table? I think I can pass it through to gptel--suffix-send.

[ahmed-shariff]

That would be similar to the variables that get set based on the scope? If so shouldn't make a difference, right? Since it is all being passed to gptel-request?

[karthink]

Right, this problem again: #539 (reply in thread)

I might have confused myself here, I need to think about it more.

[ahmed-shariff]

To address that would mean either the transition table would need to be aware of the backend or vice versa. i.e., if the table is changed if the backend is not compatible with the table it should error and vice versa?

[ahmed-shariff]

I've been working on adding function call feature openai assistant. The implementation is currently tucked away in my (very messy) config. I'll putt this out to a seeprate repo when I get some time.

So, the good news is open ai decided to stick to the same structure of defining tools. But where and how these calls are seen are different. I had to modify the gptel-curl--parse-stream, gptel--inject-prompt and gptel--parse-tool-results to get tool-use working. I am still working on getting this to work with non-stream option. For now, I'm posting here these changes.

Another piece to be conscious about is the tool configured for the assistant need to be set in gptel-tools, otherwise gptel--handle-tool-use will not work. For now I make sure I am making sure this is set with the gptel-menu. Ideally I'd need to make sure the tools on the assistant is registered locally with gptel as well. There's also the option to override the functions setup on assistant.

1. gptel--parse-tool-results:

(cl-defmethod gptel--parse-tool-results ((_ gptel-openai-assistant) tool-use)
  "Return a prompt containing tool call results in TOOL-USE."
  ;; (declare (side-effect-free t))
  (mapcar
   (lambda (tool-call)
     (list
      :output (plist-get tool-call :result)
      :tool_call_id (plist-get tool-call :id)))
   tool-use))

2. gptel--inject-prompt:

(cl-defmethod gptel--inject-prompt ((_ gptel-openai-assistant) data new-prompt &optional _position)
  ;; The tool calls are handled differently in oai
  ;; FIXME: What are the other use cases for inject?
  (let ((prompts (plist-get data :tool_outputs)))
    (plist-put data :tool_outputs (vconcat prompts new-prompt))))

3. The modifications I needed on the openai's gptel-curl--parse-stream. I am posting the diff here so it would be easier for you to know what changes. Based on this I am not sure it would be trivial to implement this in a way where cl-call-next-method would be used.

@@ -195,9 +195,7 @@ information if the stream contains it."
                             (args (apply #'concat (nreverse (plist-get info :partial_json))))
                             (func (plist-get (car tool-use) :function)))
                   (plist-put func :arguments args) ;Update arguments for last recorded tool
-                  (gptel--inject-prompt
-                   (plist-get info :backend) (plist-get info :data)
-                   `(:role "assistant" :content :null :tool_calls ,(vconcat tool-use))) ; :refusal :null
+                  (plist-put info :data (list :stream (plist-get info :stream)))
                   (cl-loop
                    for tool-call in tool-use ; Construct the call specs for running the function calls
                    for spec = (plist-get tool-call :function)
@@ -207,13 +205,23 @@ information if the stream contains it."
                                                        (plist-get spec :arguments))))
                    into call-specs
                    finally (plist-put info :tool-use call-specs)))
-              (when-let* ((response (gptel--json-read))
-                          (delta (map-nested-elt response '(:choices 0 :delta))))
-                (if-let* ((content (plist-get delta :content))
+              (when-let* ((response (gptel--json-read)))
+                (when (and (null (plist-get info :openai-assistant-run-id))
+                           (string-equal (plist-get response :object) "thread.run"))
+                  (plist-put info :openai-assistant-run-id (plist-get response :id)))
+                (if-let* ((content (map-nested-elt response '(:delta :content 0 :text :value)))
                           ((not (eq content :null))))
-                    (push content content-strs)
+                    (if-let* ((annotations (map-nested-elt
+                                            response '(:delta :content 0 :text :annotations)))
+                              (_ (length> annotations 0)))
+                        (cl-loop for annotation across annotations
+                                 for file-id = (map-nested-elt
+                                                annotation '(:file_citation :file_id))
+                                 if file-id
+                                 do (push (format "[file_citation:%s]" file-id content) content-strs))
+                      (push content content-strs))
                   ;; No text content, so look for tool calls
-                  (when-let* ((tool-call (map-nested-elt delta '(:tool_calls 0)))
+                  (when-let* ((tool-call (map-nested-elt response '(:delta :step_details :tool_calls 0)))

The complete gptel-curl--parse-stream

(cl-defmethod gptel-curl--parse-stream ((_ gptel-openai-assistant) info)
  (let* ((content-strs))
    (condition-case err
        (while (re-search-forward "^data:" nil t)
          (save-match-data
            (if (looking-at " *\\[DONE\\]")
                ;; The stream has ended, so we do the following thing (if we found tool calls)
                ;; - pack tool calls into the messages prompts list to send (INFO -> :data -> :messages)
                ;; - collect tool calls (formatted differently) into (INFO -> :tool-use)
                (when-let* ((tool-use (plist-get info :tool-use))
                            (args (apply #'concat (nreverse (plist-get info :partial_json))))
                            (func (plist-get (car tool-use) :function)))
                  (plist-put func :arguments args) ;Update arguments for last recorded tool
                  (plist-put info :data (list :stream (plist-get info :stream)))
                  (cl-loop
                   for tool-call in tool-use ; Construct the call specs for running the function calls
                   for spec = (plist-get tool-call :function)
                   collect (list :id (plist-get tool-call :id)
                                 :name (plist-get spec :name)
                                 :args (ignore-errors (gptel--json-read-string
                                                       (plist-get spec :arguments))))
                   into call-specs
                   finally (plist-put info :tool-use call-specs)))
              (when-let* ((response (gptel--json-read)))
                (when (and (null (plist-get info :openai-assistant-run-id))
                           (string-equal (plist-get response :object) "thread.run"))
                  (plist-put info :openai-assistant-run-id (plist-get response :id)))
                (if-let* ((content (map-nested-elt response '(:delta :content 0 :text :value)))
                          ((not (eq content :null))))
                    (if-let* ((annotations (map-nested-elt
                                            response '(:delta :content 0 :text :annotations)))
                              (_ (length> annotations 0)))
                        (cl-loop for annotation across annotations
                                 for file-id = (map-nested-elt
                                                annotation '(:file_citation :file_id))
                                 if file-id
                                 do (push (format "[file_citation:%s]" file-id content) content-strs))
                      (push content content-strs))
                  ;; No text content, so look for tool calls
                  (when-let* ((tool-call (map-nested-elt response '(:delta :step_details :tool_calls 0)))
                              (func (plist-get tool-call :function)))
                    (if (plist-get func :name) ;new tool block begins
                        (progn
                          (when-let* ((partial (plist-get info :partial_json)))
                            (let* ((prev-tool-call (car (plist-get info :tool-use)))
                                   (prev-func (plist-get prev-tool-call :function)))
                              (plist-put prev-func :arguments ;update args for old tool block
                                         (apply #'concat (nreverse (plist-get info :partial_json)))))
                            (plist-put info :partial_json nil)) ;clear out finished chain of partial args
                          ;; Start new chain of partial argument strings
                          (plist-put info :partial_json (list (plist-get func :arguments)))
                          ;; NOTE: Do NOT use `push' for this, it prepends and we lose the reference
                          (plist-put info :tool-use (cons tool-call (plist-get info :tool-use))))
                      ;; old tool block continues, so continue collecting arguments in :partial_json 
                      (push (plist-get func :arguments) (plist-get info :partial_json)))))))))
      ((json-parse-error json-end-of-file search-failed)
       (goto-char (match-beginning 0)))
      (error
       (signal (car err) (cdr err))))
    (apply #'concat (nreverse content-strs))))