Retrieving and invoking user-selectable menu items or shortcut keys in applications

[cheuk-cheng]

After a user launches an app and while the focus is on the app's main window, the user can invoke Arvis and then Arvis can dynamically retrieve all available menu item names/descriptions (and perhaps also equivalent keyboard shortcuts) of the app. For example, in Visual Studio Code, there is a "New File" menu item and its keyboard shortcut is -N. Then the user can create a sequence of steps relatively easy inside Arvis by visually selecting a series of actions (including invoking a menu item like "New File" which may be displayed as [Visual Studio Code] -> New File (-N) inside Arvis.

For example, a sequence to comment out the current clipboard's content can be like invoking [Visual Studio Code] File -> New File (-N) followed by [Visual Studio Code] Edit -> Paste (-V) followed by [Visual Studio Code] Edit -> Toggle Block Comment (/) followed by [Visual Studio Code] Edit -> Copy (-C)

The above block commenting the clipboard's content can be achieved with scripting but not all users know how to write scripts. Letting the user visually pick and choose available menu items inside an application and chain the steps and save them for later use will be quite powerful.

There is a macOS app called Paletro that somehow can obtain the menu item names and keyboard shortcuts of multiple applications and show the commands to the user in a command palette.

[jopemachine]

Could you tell me in detail what items you want to add to the menu?

Below list is just my guess.
Is this kind of what you meant? If not, I'd appreciate writing some items you want to add.

• Show installed workflow preference
• Show store preference
• Install workflow, plugin

[cheuk-cheng]

HI Gyubong. This was just an idea as it might not be easy or even impossible to implement. What I was thinking was to provide a way in Arvis for users to visually pick what menu items to run in a launched app because not all apps could be controlled on the command line and not all users knew how to write scripts. However, users would be able to achieve what they wanted by launching their installed apps and clicking menu bar items. For example, most users would know that "File | New File" would create a new file in any app and "Edit | Paste" would paste the clipboard content to an app.

So my imaginative example of the idea would be:

1. User launched an app like Visual Studio Code (or any installed app like an image editing app).
2. Press Arvis shortcut key to launch Arvis
3. Arvis would read all menu items (with the names and shortcuts like "File -> New File" and "command-key+N") from the launched app (which had the focus before switching to Arvis). The menu items + shortcut keys from the app would be shown by Arvis to the user in a list or checkboxes, etc.
4. User would select a sequence of menu items by clicking list items or checkboxes created in step 3. For example, "File menu | New File" then "Edit menu | Paste" then "Edit menu | Toggle Comment Block" then "Edit menu | Copy"
5. User could save the app name (Visual Studio Code) in step 1 and the chosen sequence of menu items in step 4. She could name the sequence "Convert clipboard to comment block"

User could run the saved sequence in step 5 later to convert textual clipboard content to a comment block in Arvis. I was not sure if it was possible to do the above in general for all apps that had menu items (with a new Arvis plugin or workflow). Also, even if it was possible how Arvis could avoid causing screen flashes (to switch to another app in Arvis and to invoke a sequence of menu items by clicking the menu bar or by pressing some shortcut keys). The whole idea was to provide a way in Arvis for users to take advantage of apps they already had and functions that were already provided by an app instead of asking users to write a plugin or workflow for everything. With Visual Studio Code, being able to use just its built-in functions through menu items or their shortcut keys could do a lot to change text in the clipboard. Another example would be using an installed image editing app's menu items in Arvis to resize an image, rotate an image, enlarge an image, etc.

Thank you for reading the above.

[jopemachine]

Thanks for your suggestion.

I like your approach,

I think it will be very helpful to automate generating kinds of workflow if it is possible.

But unfortunately, I think reading other app's menu is generally not feasible.

If there's a technically possible way, I'll reopen the issue and consider it later :)

[cheuk-cheng]

Thanks Gyubong. As I mentioned before there was an app for macOS called "Paletro" which could read menu items (and their textual descriptions) from different apps and then presented those items in a "Visual Studio Code" type of command palette when the user presses command-shift-P. I have tried the app and it is able to retrieve menu items for many apps including those written in Swift and Electron. So I think it is technically possible to retrieve the menu items of an app. Can you download a trial version of the app at https://appmakes.io/paletro and test it out?

[jopemachine]

I tried paletro, this feature is quite impressive.

I'm not sure but, I think maybe there are some kind of APIs allowing accessing other App's menus in macOS with swift,

Which means this feature might be only feasible in macOS.

So I'm not planning to dedicate to implement this feature at this time.

But I'll leave the issue to reopen for the sake of future possibilities, and add Help wanted tag.

[cheuk-cheng]

Fair enough since Arvis is cross-platform and not just for macOS. By the way, the latest raycast app update will support (iOS like) shortcuts that are coming in the next macOS version. Perhaps you can consider supporting them in future.

Thanks.