Save and restore state, incl dataframes

[vnijs]

Hi @schloerke: @cpsievert suggested connecting with you about the new state save and restore features. Hopefully this path is appropriate.

I cobbled together a working solution to save and restore data, incl dataframes, but it seems a bit clunky (i.e., state is being updated many times to avoid losing information on browser refresh). See example below. Would love to hear your suggestions.

A few additional questions:

• The shiny bookmarks folder can fill up quickly with this approach. My ideal would be to store state for a specific user in one file (or folder with a data and a json file) only on (1) browser refresh and (2) a user button click. I see a setting for ".dir". Is there a setting to control the file name for state?
• If you "Add data" in the app and then refresh, the "Select data" dropdown visibly flips from C to A to C. Not an issue in this app but if an analysis would run based on the value of "Select data" ...
• Any suggestions on how to best approach save state to a specific file/folder and then restoring state from a specific file / folder in shiny-for-python would be very interesting.

import os

from chatlas import ChatOpenAI
from shiny import App, ui, reactive, render
from shiny.bookmark import BookmarkState, RestoreState
from starlette.requests import Request
from dotenv import load_dotenv
import pandas as pd
import pickle
from pathlib import Path

load_dotenv()

chat_client = ChatOpenAI(
    api_key=os.getenv("OPENAI_API_KEY"),
    model="gpt-4o",
    system_prompt="You are a helpful assistant.",
)


DATASETS_PATH = Path("datasets.pkl")


def app_ui(request: Request):
    return ui.page_sidebar(
        ui.sidebar(
            ui.h4("Test"),
            ui.input_select("select_letter", "Select letter", choices=["X", "Y", "Z"]),
            ui.input_select("select_data", "Select data", choices=["A", "B"]),
            # ui.output_ui("ui_select_data"),  # Use different ID for output
            ui.input_action_button("add_data", "Add Data"),
            ui.input_action_button("remove_data", "Remove Data"),
            ui.output_ui("data"),
            width=400,
        ),
        ui.chat_ui(
            id="chat",
            messages=["Hello! How can I help you today?"],
        ),
        width=400,
    )


def server(input, output, session):
    if DATASETS_PATH.exists() and DATASETS_PATH.stat().st_size > 0:
        with open(DATASETS_PATH, "rb") as f:
            _datasets = pickle.load(f)
    else:
        _datasets = {
            "A": pd.DataFrame({"a": [1, 2], "b": [3, 4], "c": [5, 6]}),
            "B": pd.DataFrame({"b": [3, 4], "c": [7, 8], "d": [9, 10]}),
        }

    datasets = reactive.value(_datasets)
    available_datasets = reactive.value(list(_datasets.keys()))

    chat = ui.Chat(id="chat")

    @reactive.Effect
    @reactive.event(input.add_data)
    def _():
        tmp = datasets.get()
        tmp["C"] = pd.DataFrame({"c": [5, 6], "d": [7, 8], "e": [9, 10]})
        datasets.set(tmp)
        tmp = list(tmp.keys())
        available_datasets.set(tmp)
        ui.update_select("select_data", choices=tmp, selected="C")

    @reactive.Effect
    @reactive.event(input.remove_data)
    def _():
        tmp = datasets.get()
        del tmp[input.select_data()]
        datasets.set(tmp)

        tmp = list(tmp.keys())
        available_datasets.set(tmp)
        ui.update_select("select_data", choices=tmp, selected=tmp[0])

    @render.ui
    def ui_select_data():  # Match the output ID
        choices = available_datasets.get()
        return ui.input_select("select_data", "Select data", choices=choices)

    @render.ui
    @reactive.event(input.select_data)
    def data():
        selected = input.select_data()
        data_dict = datasets.get()
        if selected in data_dict:
            return ui.output_data_frame("selected_data")
        return ui.p("No data selected")

    @render.data_frame
    def selected_data():
        selected = input.select_data()
        data_dict = datasets.get()
        return data_dict.get(selected)

    chat.enable_bookmarking(chat_client)

    @chat.on_user_submit
    async def handle_user_input(user_input: str):
        response = await chat_client.stream_async(user_input)
        await chat.append_message_stream(response)

    @session.bookmark.on_restore
    def _(state: RestoreState) -> None:
        if "available_datasets" in state.values and "select_data" in state.values:
            tmp = available_datasets.get()
            ui.update_select(
                "select_data", choices=tmp, selected=state.values["select_data"]
            )

    @session.bookmark.on_bookmark
    def _(state: BookmarkState) -> None:
        state.values["available_datasets"] = available_datasets.get()
        state.values["select_data"] = input.select_data()
        state.values["select_letter"] = input.select_letter()

    @reactive.Effect
    @reactive.event(
        input.select_data, input.select_letter, input.add_data, input.remove_data
    )
    async def _():
        await session.bookmark()

    @reactive.Effect
    @reactive.event(input.add_data, input.remove_data)
    def _():
        with open(DATASETS_PATH, "wb") as f:
            pickle.dump(datasets.get(), f)


app = App(app_ui, server, bookmark_store="server")

[schloerke]

Related: #1958 (comment)

[schloerke]

If you "Add data" in the app and then refresh, the "Select data" dropdown visibly flips from C to A to C. Not an issue in this app but if an analysis would run based on the value of "Select data" ...

There are two approaches on how to fix this:

1. set the UI during render

2. use dynamic UI where there is no select drop down and it is only added after the data has been restored.

[schloerke]

The shiny bookmarks folder can fill up quickly with this approach. My ideal would be to store state for a specific user in one file (or folder with a data and a json file) only on (1) browser refresh and (2) a user button click. I see a setting for ".dir". Is there a setting to control the file name for state?

"store state for a specific user in one file". Shiny does not know of the user and the demo app does not expose it either.

For example purposes, let propose you have access to inputs$user_name that has their info. When restoring the data, this becomes really tricky. You'll have no information on what the username is unless you read every saved file. (Or at least match against every possible state file name if some special file path was used.)

Let imagine then we're deployed on connect and session.user is populated. We're going to leverage App's set_bookmark_save_dir(fn) and set_bookmark_restore_dir(fn).

(UNTESTED. Please let me know your final solution!)

# (UNTESTED. Please let me know your final solution!)

app = App(app_ui, server, bookmark_store="server")

bookmark_dir = Path("bookmarks")

# Docs on the headers:
# https://docs.posit.co/connect/admin/appendix/advanced-user-group/#content-credentials
# Ex: session.http_conn.headers["shiny-server-credentials"]

async def user_restore_dir(_id_ignored: str) -> Path:
    from shiny.session import require_active_session
    from shiny import reactive

    user = require_active_session(None).user

	# Return your own bookmark dir location!
    user_bookmark_dir = bookmark_dir / user
    return user_bookmark_dir


async def user_save_dir(_id_ignored: str) -> Path:
    user_bookmark_dir = await user_restore_dir(_id_ignored)
    user_bookmark_dir.mkdir(parents=True, exist_ok=True)
    return user_bookmark_dir


app.set_bookmark_save_dir_fn(user_save_dir)
app.set_bookmark_restore_dir_fn(user_restore_dir)

---

If this approach is used, the ID being displayed is meaningless. Therefore, we can ignore the query string update within the app.

	# (UNTESTED)

    # Do not auto-update the query string
    chat.enable_bookmarking(chat_client, bookmark_on=None)
	
	# Also update the bookmark on message change
    @reactive.Effect
    @reactive.event(
        input.select_data,
        input.select_letter,
        input.add_data,
        input.remove_data,

        # Reactively react to chat messages
        # This is a workaround to ensure the bookmark is updated when the chat messages change
        # This approach may change in the future
        lambda: chat.messages(format=MISSING),
    )
    async def _():
        await session.bookmark()

	# Add placeholder method (to do nothing) which hides the modal that shows up by default
    @session.bookmark.on_bookmarked
    async def _(_url_ignored: str):
        pass

[schloerke]

If you "Add data" in the app and then refresh, the "Select data" dropdown visibly flips from C to A to C. Not an issue in this app but if an analysis would run based on the value of "Select data" ...

There are two approaches on how to fix this:

1. set the UI during render

UNTESTED.

def app_ui(request: Request):

    data_choices = ["A", "B"]

    from shiny.bookmark._restore_state import get_current_restore_context

    restore_ctx = get_current_restore_context()
    if restore_ctx:
        # Only need to restore the choices as the selected value is restored by the
        # bookmark
        data_choices = state.values["available_datasets"]

    return ui.page_sidebar(
        ui.sidebar(
            ui.h4("Test"),
            ui.input_select("select_letter", "Select letter", choices=["X", "Y", "Z"]),
            ui.input_select("select_data", "Select data", choices=data_choices),
            # ui.output_ui("ui_select_data"),  # Use different ID for output
            ui.input_action_button("add_data", "Add Data"),
            ui.input_action_button("remove_data", "Remove Data"),
            ui.output_ui("data"),
            width=400,
        ),
        ui.chat_ui(
            id="chat",
            messages=["Hello! How can I help you today?"],
        ),
        width=400,
    )

I believe a restore_value(id) helper function should be exported. Related: #1904

[schloerke]

Any suggestions on how to best approach save state to a specific file/folder and then restoring state from a specific file / folder in shiny-for-python would be very interesting.

Note... during on_bookmark callbacks, you can save to the state's .dir (which is also module scoped when necessary). The files will be available to you during the on_restore/on_restored callbacks.

[schloerke]

@vnijs I hope this helps you get moving!

Please let me know of any rough edges you come across. I'd be happy to fix them / update docs.

[vnijs]

Thank for the detailed replies @schloerke! Much appreciated and I will try each of your suggestions.

We would have a user_name if using shiny-server pro and/or we could use a session identifier in the url. Joe Cheng suggested the below for me years ago and that has worked really well in R-Shiny

https://github.com/radiant-rstats/radiant.data/blob/master/inst/app/www/js/session.js

Below the sequence of call I use in radiant.data and several other apps.

https://github.com/radiant-rstats/radiant.data/blob/b42fc2a29c1ddbf1e98584d0164d8c22b174bb8b/inst/app/server.R#L27
https://github.com/radiant-rstats/radiant.data/blob/b42fc2a29c1ddbf1e98584d0164d8c22b174bb8b/inst/app/radiant.R#L87-L107
https://github.com/radiant-rstats/radiant.data/blob/b42fc2a29c1ddbf1e98584d0164d8c22b174bb8b/inst/app/radiant.R#L43-L73
https://github.com/radiant-rstats/radiant.data/blob/b42fc2a29c1ddbf1e98584d0164d8c22b174bb8b/inst/app/tools/data/manage_ui.R#L835-L846

I will try to implement something similar for shiny for python very soon.

On a related note, I'd love to hear your thoughts on the feature request linked below. Feasible? Of interest? I wouldn't know where to start in the shiny code base, tbh, but with some pointers I'd be happy to give this a shot as a PR if you would consider it.

#1981

[schloerke]

On a related note, I'd love to hear your thoughts on the feature request linked below. Feasible? Of interest? I wouldn't know where to start in the shiny code base, tbh, but with some pointers I'd be happy to give this a shot as a PR if you would consider it.

github.com/issues/created?issue=posit-dev%7Cpy-shiny%7C1981

Commenting in the Issue.