Switch from Bash to Python #873
Replies: 32 comments 123 replies
-
I've been definitely starting to think that working in Bash has started to become cumbersome. Working out SSO and API in the past week had me literally yelling 'why did I take on maintaining a bash-only program' 😅
Distribution would definitely be better served as some kind of AppImage or via package managers. Plus it would've been easier to find dependencies like
I'm most familiar with Python. Nowhere near an exit, but then again, I never was with Bash either. It seems like the best option. Aside from the redirector already being written in C, I don't see any reason not to use Python. I had actually started working on something of a UI concept in Python. Regardless, I'm down for it. Thanks for everything you've managed to do here. |
Beta Was this translation helpful? Give feedback.
-
|
I was also considering a TUI. Click seems pretty good, I think we can go that route. Less need for design work. |
Beta Was this translation helpful? Give feedback.
-
|
So, I just whipped up a single example for a GUI and TUI. For testing purposes, it's all in one file, but they can pretty easily be split into multiple, that way we can better manage individual steps. GUI with QT framework, the same as MO2 uses. Separate window. codeimport sys
from PySide6 import QtCore, QtGui, QtWidgets
class MO2LintApp(QtWidgets.QWidget):
def __init__(self):
super().__init__()
self.title = "Choose Game"
self.setWindowTitle("MO2-LINT - " + self.title)
self.header = QtWidgets.QLabel(self.title,
alignment=QtCore.Qt.AlignCenter)
self.header.setFont(QtGui.QFont("Arial", 12, QtGui.QFont.Weight.Bold))
self.header.setAlignment(QtCore.Qt.AlignTop | QtCore.Qt.AlignHCenter)
self.body = QtWidgets.QVBoxLayout()
self.instruct = QtWidgets.QLabel("Please select the game you.",
alignment=QtCore.Qt.AlignCenter)
self.instruct.setFont(QtGui.QFont("Arial", 12, QtGui.QFont.Weight.Normal))
self.gameList = QtWidgets.QListWidget()
self.gameList.addItems(["The Elder Scrolls V: Skyrim Special Edition", "The Witcher 3: Wild Hunt", "Fallout 4", "Custom Game"])
self.body.addWidget(self.instruct)
self.body.addWidget(self.gameList)
self.backButton = QtWidgets.QPushButton("Back")
#self.backButton.clicked.connect(self.close)
self.nextButton = QtWidgets.QPushButton("Next")
self.nextButton.clicked.connect(self.close)
self.navButtonLayout = QtWidgets.QHBoxLayout()
self.navButtonLayout.addWidget(self.backButton)
self.navButtonLayout.addWidget(self.nextButton)
self.layout = QtWidgets.QVBoxLayout(self)
self.layout.addWidget(self.header)
self.layout.addLayout(self.body)
self.layout.addLayout(self.navButtonLayout)
if __name__ == "__main__":
app = QtWidgets.QApplication([])
widget = MO2LintApp()
widget.resize(400, 200)
widget.show()
app.setApplicationName("Mod Organizer 2 Linux Installer (MO2-LINT)")
app.setApplicationVersion("7.0.0-indev")
sys.exit(app.exec())
TUI with Textural framework. This is all displayed in the terminal. code#!/usr/bin/env python3
from rich.markdown import Markdown
from textual.app import App, ComposeResult
from textual.screen import Screen
from textual.widgets import Static, Button, ListView, ListItem, Label
from textual.containers import Horizontal
INSTRUCTIONS = """\
# Choose Game
Please select the game you wish to manage with Mod Organizer 2 from the options below:
"""
class StepSelectGame(Screen):
DEFAULT_CSS = """
StepSelectGame {
width: 100%;
height: 100%;
background: $surface;
}
StepSelectGame Container {
padding: 1;
color: $foreground;
}
StepSelectGame Static {
padding: 1;
margin: 0 1;
}
StepSelectGame Horizontal {
align: center bottom;
height: auto;
layout: horizontal;
overflow: hidden hidden;
}
StepSelectGame Horizontal Button {
width: 50%;
}
StepSelectGame ListView {
padding: 1;
margin: 0 1;
}
StepSelectGame ListView Static {
padding: 0;
margin: 1 1;
}
"""
def compose(self) -> ComposeResult:
yield Static(Markdown(INSTRUCTIONS))
yield ListView(
ListItem(Label("The Elder Scrolls V: Skyrim Special Edition"), id="skyrimse"),
ListItem(Label("The Witcher 3: Wild Hunt"), id="witcher3"),
ListItem(Label("Fallout 4"), id="fallout4"),
ListItem(Label("Custom Game"), id="custom"),
id="game-list"
)
with Horizontal():
yield Button("Back", id="back")
yield Button("Next", id="next")
def on_mount(self) -> None:
pass
def on_button_pressed(self, event) -> None:
if getattr(event.button, "id", None) == "back":
#from .StepWelcome import StepWelcome
#self.app.push_screen(StepWelcome())
pass
if getattr(event.button, "id", None) == "next":
#from .StepSelectExtra import StepSelectExtra
#self.app.push_screen(StepSelectExtra())
self.close()
class MO2LintApp(App):
def on_mount(self) -> None:
self.push_screen(StepSelectGame())
self.title = "Mod Organizer 2 Linux Installer"
if __name__ == "__main__":
app = MO2LintApp()
app.run()
Of course, we can develop CLI right alongside that. Keep it simple as mo2-lint <game> <dir> --script-extender --plugin <plugin1> --plugin <plugin2> --no-uior something like that. Or prioritize CLI, with a UI as a secondary objective. |
Beta Was this translation helpful? Give feedback.
-
|
If I were doing this myself I'd lean towards getting a minimum viable product CLI working just to facilitate development, but if the UI is more interesting to you I'd say go for it. Would also serve the purpose of giving something to hook "step" functions to. Much as I love Textual, I think I'd personally prefer QT just to keep inline with MO2. That said, I also don't think it's that important. |
Beta Was this translation helpful? Give feedback.
-
100%, function first over form. I'm just still thinking over the plan 😅 having some kind of pseudo-"end product" helps me think over the different aspects and make sure all the boxes have been checked off. but I'll probably start on actual (CLI) development later tonight or tomorrow |
Beta Was this translation helpful? Give feedback.
-
|
It's 1AM so it took a moment, but I figured out Click and managed to get parameters set up, as well as proof-of-concept parsing for the game info, so the values you see there are read from the json file. 
|
Beta Was this translation helpful? Give feedback.
-
|
Ooh, that looks nice. I might start trying to puzzle out how such a rewrite might fit together over the holiday weekend if I'm not to swamped by requests from family. |
Beta Was this translation helpful? Give feedback.
-
|
A bit of brainstorming, these are the functions we'd need to hit, general goals, a few ideas and considerations Primary Functions(Not necessarily in this order)
Secondary Functions
Goals
Ideas:
Unknowns
|
Beta Was this translation helpful? Give feedback.
-
|
Drew up an example filetree, just to help kinda visualize and figure out where the gaps are. modorganizer2-linux-installer/
├─ .github/
│ ├─ ISSUE_TEMPLATE/
│ │ ├─ config.yml # Forces issue templates
│ │ ├─ report-bug.yml # Issue Template - Bug Report
│ │ ├─ request-feature.yml # Issue Template - Feature Request
│ │ └─ request-game.yml # Issue Template - Game Request
│ │
│ ├─ README/
│ │ ├─ modorganizer2-linux-installer.svg # Logo
│ │ ├─ V5-README.md # "RockerBacon Era" README (for archival purposes)
│ │ └─ V6-README.md # Pre-Python (current) README (for archival purposes)
│ │
│ ├─ workflows/
│ │ ├─ build.yml # Action for compiling release [workflow_dispatch]
│ │ └─ formatting.yml # Action for ensuring formatting (linter, black style) [pull_request]
│ ├─ CONTRIBUTING.md # Contribution guidelines and environment setup
│ └─ README.md # Current README
│
├─ configs/
│ ├─ gamesinfo.json # Game Info
│ └─ pluginsinfo.json # Plugin Info
│
├─ src/
│ ├─ ui/ # User interface (replaces 'step'?)
│ │ ├─ precheck.py # Pre-run check (update check, root check, dependencies (if any))
│ │ ├─ select-game.py # Choose game and load related info
│ │ └─ etc...
│ ├─ util/ # Utilities/Helpers (downloader, extracter, etc)
│ │ ├─ nxm/ # Nexus related functions (nxm handlers, api/sso)
│ │ ├─ steam/ # Steam related functions (library scan, prefix handling(?), redirector install, redirector source(?))
│ │ ├─ proton/ # Proton related functions (prefix cleanup, configuration)
│ │ ├─ download.py # Download manager
│ │ ├─ helper.py # File extraction helper
│ │ └─ logger.py # Logging helper
│ ├─ __init__.py # Initializing script
│ └─ uninstall.py # Uninstall process
│
├─ steam-redirector/ # Redirector source(?)
├─ .gitignore
├─ LICENSE
├─ Makefile # Compiling
├─ requirements.txt # Python dependencies
└─ shell.nix # Nix packaging |
Beta Was this translation helpful? Give feedback.
-
|
Damn, you've been busy! 😆 Some thoughts:
|
Beta Was this translation helpful? Give feedback.
-
At least, interpret whatever we can... In practice, their logging doesn't seem to contain much useful information -.-
True. As much as I'd prefer to prioritize Steam, since I have no clue how Heroic or Lutris handles games, it should definitely be up to the user.
The majority of the info is fairly must-be-this. The only thing I can think of being dynamic is the script extenders, which we can keep some kind of db for user's choice.
Definitely looking like a 'Settings' menu is called for. List of present installations with 'Modify' and 'Uninstall' buttons
I may be slow-braining it, but it doesn't seem to be a detriment to them being a CLI wrapper.
No arguments to state. SQL or JSON would work. JSON may be more user friendly, but that could serve more detrimental than anything.
No arguments here, either. Though I'm unsure what you mean by the 'configuration', aside from the compilation files.
Existence should be sufficient. Maybe checksums for the redirector replacement, just to avoid launching a badly compiled exe, but that's all I can think of.
I'm in agreement on the 'multi-distro' maintenance. If it's really needed/requested, we can cross that bridge then.
Full agreement there. I never agreed with the full purge the script did after failure, but never took the time to replace it. |
Beta Was this translation helpful? Give feedback.
-
Good point, was being overly broad there. Mostly what I'm concerned with allowing the user to change is the target installation, which I think I better explained with the Steam vs. Heroic vs. Lutris bit.
It's not, but I was hoping for something more python-native to hook into. A silly hope perhaps, given the nature of winetricks. The point is: it'll work.
Users breaking stuff is a valid concern, but its also my belief that anyone using MO2 should understand what they're doing well enough to not grenade their system, and/or be able to fix it if they do. Maybe an invalid assumption? Either way, I also should've mentioned that eschewing SQLite or similar keeps our dependencies down. I wouldn't be against using it if there were a good argument for, but I don't think the amount/complexity of data we need to store warrants it.
Specifically the file that the redirector looks at in the game directory to determine which MO2 instance it's supposed to use. On further consideration, it's somewhat challenging to store that information elsewhere, the first thing that comes to mind is looking up the redirector's own path in a global configuration. The downside of this being that it breaks if the game moves, but conversely other stuff breaks currently if the MO2 instance moves. I think I still prefer having all the values in a central location so they're easier to find and update if the user does want to move something. |
Beta Was this translation helpful? Give feedback.
-
I don't have too much C knowledge tbh 😅 But I think Python is the best bet language-wise |
Beta Was this translation helpful? Give feedback.
-
|
For now, let's put the redirector at the bottom of the priority list. It's functional as is, and if we need to rewrite it for easier integration with Python, we can cross that bridge then. |
Beta Was this translation helpful? Give feedback.
-
|
@ashtonqlb brought up SulfurNitride/NaK, another MO2 installer. Thought we might be able to pull some inspiration from it. Thing is, I can't make heads or tails of the project. It's about as AI slop as you can get. (That's not an allegation, their README states it's gen'd with Claude) LINT won't be a 'vibe coding' project. With the amount of hallucinations it gets, and it's tendencies to make scripts 10x more complex than needed.. any code that's clearly been gen'd will be declined. Use AI for the occasional debug, syntax check, or whatever? That's your deal, not mine. Just don't poison this codebase. This isn't specifically directed at anyone. Just a general statement. |
Beta Was this translation helpful? Give feedback.
-
|
rewrite branch created and prepped |
Beta Was this translation helpful? Give feedback.
-
|
Random question @Furglitch; is there a reason you know of that we're using the JDK and not the JRE? |
Beta Was this translation helpful? Give feedback.
-
Not a clue, came like that when I adopted the script. Frankly I'm not even sure why we're grabbing Java, I don't know what depends on it. I'll be honest, whenever I needed JRE on my desktop (back before I swapped to Linux) I just downloaded the JDK because it was easier to find a chocolatey package for that. I wouldn't be surprised if there was similar reasoning here. |
Beta Was this translation helpful? Give feedback.
-
|
I completely missed this line in the README
So it's purely for Skyrim LE and Enderal LE. That being said, I still don't know why JDK is chosen over JRE. The earliest mention I can find is issue #42 and the earliest commit is 1104ff0, but neither provide any insight... |
Beta Was this translation helpful? Give feedback.
-
|
That makes sense. Shouldn't matter, let's try the JRE for a lighter download and install. |
Beta Was this translation helpful? Give feedback.
-
|
I just noticed, it's actually the JRE. Their naming is just a bit confusing. |
Beta Was this translation helpful? Give feedback.
-
Steam RedirectorAdditional thought on the redirector; I'm not sure how feasible this is - having never played with python EXE compilation - but it would be neat if we could have the utility dynamically "compile" a python script into the redirector on the fly. It adds complexity, but has the benefits of removing the less-understood C dependency, removing the required extra compilation before packaging, and reduces the number of files we rely on being in the correct place at install time. Maybe something to look into later. |
Beta Was this translation helpful? Give feedback.
-
|
I just want to make sure I'm understanding this correctly.. you mean to hardcode the MO2.exe path into the redirector? Or is there something else I'm missing here? |
Beta Was this translation helpful? Give feedback.
-
|
Here's what I'm currently thinking: a rewritten redirector is included in Python form, and packed in an EXE to the appropriate location when an instance of MO2 is installed. The redirector can then look up and launch the target MO2 instance from the global config based on its parent directory. |
Beta Was this translation helpful? Give feedback.
-
|
That's pretty much on par with what I thought you meant lmao |
Beta Was this translation helpful? Give feedback.
-
|
Maybe I'm the one misunderstanding then. Either way, the desire is to keep the MO2 path out of the game's directory so the user (or we) doesn't need to go hunting for it. But we can talk about that down the road. |
Beta Was this translation helpful? Give feedback.
-
|
Just discovered another fun little problem, for BG3 at least. Norbyte's BG3SE hooks the executable by name, meaning that if we rename the file to After a bit of fiddling, I was able to get around this by jamming the original Given that other injectors (SKSE, Reshade, etc.) seem to work fine, this may be a BGSE-specific problem. Should we try to come up with a workaround, or reach out to Norbyte to see if we can get them to change how they hook the process? |
Beta Was this translation helpful? Give feedback.
-
Data Structure DiscussionAs aforementioned, I'm looking at how we might transition game info, resource info, and plugin info (or more uniformly structure in this case) into JSON; and how to best hold the "state" data. WIP in my fork of the rewrite tree. Some thoughts so far:
|
Beta Was this translation helpful? Give feedback.
-
My own convenience moreso. The URL contained all the needed information for the API download, and could just be grabbed by [right click > copy link]ing the manual download button of the mod lol
I was also thinking this could allow us to provide downloads for multiple versions of script extenders. Basically give the user a drop down with available versions, so that they're not restricted to the latest. Y'know, for those who want to mod Skyrim SE over AE. No objections, overall |
Beta Was this translation helpful? Give feedback.
-
|
Ah, this'd also facilitate storing/installing multiple versions then. For some reason I wasn't seeing the download link when I was looking earlier, probably a me problem 🤷🏻. Since that's the case, I have mixed feelings about using the URL. It's easier for contributors and makes it simpler to figure out what you're looking at from the JSON file. On the downside, we have to parse it anyways as aforementioned, and it'll break if the nexus ever changes the link scheme (whereas the API should be more static). |
Beta Was this translation helpful? Give feedback.
-
|
I kinda misspoke. The Nexus URL was chosen because the game_scriptextender_urls array was already in place, so I figured it would be more straightforward to implement that over creating new variables. So... laziness, on my part lmao. But yeah, I'm definitely down for switching off the URLs. |
Beta Was this translation helpful? Give feedback.
-
|
Linting is implemented as a pre-commit hook and a pull request action. |
Beta Was this translation helpful? Give feedback.
-
Development EnvironmentI'll try and get started on this over the holiday weekend. To reiterate, I'm thinking a script that builds a...
Goals:
|
Beta Was this translation helpful? Give feedback.
-
|
Made some more progress, but hit a hurdle; PT expects appcache.vdf to be present and populated. This isn't a traditional VDF, but rather a binary VDF containing nested VDFs. I'm fairly confident I can get it working, but it'll take some reverse-engineering, so don't hold your breath. |
Beta Was this translation helpful? Give feedback.
-
|
More progress made (see notes). I think I've mostly figured out the structure of the file, but still working on determining what the minimum components necessary are. Currently crashing when PT tries to parse it, but crashing further in now. 🙃 |
Beta Was this translation helpful? Give feedback.
-
|
It's also just occurred to me that the VDF module might be capable of exporting a binary VDF on its own. If so, I'd prefer that over my DIY implementation. Will check when my brain is less fried. |
Beta Was this translation helpful? Give feedback.
-
|
Status update; the fake appinfo.vdf (wrong name earlier) no longer causes Protontricks to implode, now it's just mad that it can't find a Proton version. I have a direction to start chasing that, just need to do some more digging. On a side note, my reverse-engineering of the file format is becoming fairly robust IMHO, and I'll probably include that in the documentation for the dev environment. |
Beta Was this translation helpful? Give feedback.
-
|
From library_root = Path('/scratch/tmp/steamdev')
protontricks_main(('--no-runtime', '-c', 'echo $WINEPREFIX', '489830'), library_root, library_root)Output: Got some cleanup to do, but I think I'm on the home stretch for Steam, which should be the hardest one. The generated library is pretty big (1.5 GiB) for just one game, since almost all of that is the prefix. We could maybe get away with just making one prefix and symlinking it into every compatdata folder, but that isn't how the actual library works, and would result in us sharing registries for each MO2 instance (not good), but I suppose I could link everything but the regestries... Maybe something to revisit once I get it in a working state. |
Beta Was this translation helpful? Give feedback.
-
|
step/load_gameinfo.sh has been converted. ...I think. I don't own any supported games on Epic to test all the cases with. I had to juryrig the game_info.json with info for Assassin's Creed 1 (and a bunch of placeholders) in order to see if it would even attempt, and those tests were successful. Also added log output with a |
Beta Was this translation helpful? Give feedback.
-
|
steps clean_game_prefix and configure_steam_wineprefix have been added as configure_prefix |
Beta Was this translation helpful? Give feedback.
-
|
Need opinions:
|
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
That's a good point, but if they need to be in the repository anyways, we might as well pack them. |
Beta Was this translation helpful? Give feedback.
-
|
Implemented |
Beta Was this translation helpful? Give feedback.
-
|
Implemented script extender download and installation, with the checksum validation for existing downloads. Took a moment to sort out, I'll manage the rest of the external resources tomorrow. |
Beta Was this translation helpful? Give feedback.
-
|
Finished the external resources download and installation. The file is a bit big and unwieldy at nearly 500 lines.. gonna have to work on consolidating functions. Aside from that, only thing now is the NXM:// handler, Steam redirector, and game specific workarounds. |
Beta Was this translation helpful? Give feedback.
-
|
External resources handling is refactored. Halved the linecount/filesize :) |
Beta Was this translation helpful? Give feedback.
-
|
Sorry for the longer than expected absence. My homelab corrupted thanks to a faulty breaker so I spent most of last week getting it back online. Updates to the above list:
Up next, in order:
|
Beta Was this translation helpful? Give feedback.
-
|
Been busy myself, no worries. I was going to say we can probably leave the redirector at the bottom of the list, but there's also the question of control flow. I highly doubt that my somewhat misguided change has affected any games we currently support (and haven't seen any issues as such), but rewriting that while we remember what it's doing probably isn't a bad idea. Still looking for thoughts on whether it makes sense to expose arguments in the game info (see aforementioned discussion for explanation of added complexity). Along with the uninstall functionality, it'd also be nice to allow the user to list known instances. Should just be a matter of formatting and printing instances out of the state. |
Beta Was this translation helpful? Give feedback.
-
|
Additional thought on that, an automagic way to re-install the redirector after a Steam update would be nice (thanks for breaking my BG3 install again guys 😑). Yes, it's trivial to put the exe's back where they're supposed to be, but maybe not for the casual user. |
Beta Was this translation helpful? Give feedback.
-
|
Sorry for another disappearance. Wife wanted a Minecraft modpack, and wife requests take priority xD I've already started again, working on implementing the existing redirector into the script as is. As recommended, I'll rewrite the redirector as a final step.
I wonder what the best way to do that would be... My initial thought is a systemd service, but what would we be checking for? A hash comparison would probably be part of it, but when would we check? How would we know that the update happened? |
Beta Was this translation helpful? Give feedback.
-
|
That's a neat idea, but I was thinking more user-driven; like an option they can specify to reinstall the redirector for a known instance. I suppose we could also expand the install process to short-circuit on existing installs, but a dedicated verb makes more sense to me, both for control flow and for user understanding. Something like mo2lint.py fix bg3 |
Beta Was this translation helpful? Give feedback.
-
|
That seems more reasonable, and much easier to implement xD Added it to the list. |
Beta Was this translation helpful? Give feedback.
-
|
The --list and --uninstall command have been added. --listSimply lists all the instances found in the state file --uninstallLists all the instances and lets the user choose to remove one or all of them from the system. It restores the game .exe file from the backup, deletes the mo2-lint instance folder, and removes the instance entry from the state file. Additionally, let's the user choose between a permanent deletion or sending to trash Additionally, using |
Beta Was this translation helpful? Give feedback.
-
|
It's nearly complete! Three steps remain:
I'm planning on handling these in this order as it doesn't make much sense to rewrite the redirector twice (once for the Python translation, once to adapt to the refactor. |
Beta Was this translation helpful? Give feedback.
-
|
Needing an opinion on the translations for protontricks... Where
So the result with log level info is: (terminal output of .log snippet at the bottom) It's fairly minimal with that, and I don't believe much more is needed, but wanted to run it past and see if anyone had other ideas. .log snippet while installing tricks |
Beta Was this translation helpful? Give feedback.
-
|
That looks good to me. Appreciate you stripping out the x64 nonsense. |
Beta Was this translation helpful? Give feedback.
-
Yeah, that'll still be included in trace logs, but for end-user stuff it's just clutter. The rewrite is 95% finished, Most of the refactor is done, now I'm just doing bug testing and then I'll get to the redirector |
Beta Was this translation helpful? Give feedback.
-
|
On refactoring, I'm wondering if it might be tidier to use command groups instead of options for other functions (like list and uninstall). So the main install function would be a non-command group invocation IMO this is cleaner from a user perspective; as each task is a distinct command, rather than an option for which the acceptable combinations has to be parsed from the help text. It also makes it easier for us to keep track of what task we're performing, without having to do stuff like modorganizer2-linux-installer/src/mo2-lint/__init__.py Lines 236 to 245 in b8ac860 IIRC you can lift an option up to the module-level so you don't have to re-define it for each command function. |
Beta Was this translation helpful? Give feedback.
-
Huh. I didn't see that. Definitely the way to go, I'll work on implementing that once I have the refactor stable. |
Beta Was this translation helpful? Give feedback.
-
|
Up to this point I've had the game/plugin/resource files in a JSON structure, but I'm starting to lean towards switching to YAML. It's a bit more user-readable and as far as I can tell, takes up less file space. (that being said, I know my original JSON was a bit messy, what with redeclaring script extender versions for each launcher instead of declaring launchers that each SE is compatible with) Examples:JSON[
"oblivion": {
"display_name": "Oblivion",
"nexus_slug": "oblivion",
"launcher_ids": {
"steam": 22330,
"gog": 1458058109
},
"subdirectory": "Oblivion",
"executable": "OblivionLauncher.exe",
"tricks": [
"d3dcompiler_43",
"d3dx9"
],
"script_extenders": {
"epic": null,
"gog": [
{
"version": "22.13",
"runtime": "1.10.163",
"download_url": "https://github.com/llde/xOBSE/releases/download/22.13/xOBSE-22.13.zip",
"checksum": "92d18411a9da803ffaac8c0b005cb6b37c4f9ce2f1571215f39363760d832bf2",
"file_whitelist": [
"Data",
"obse_1_2_416.dll",
"obse_editor_1_2.dll",
"obse_loader.exe"
]
}
],
"steam": [
{
"version": "22.13",
"runtime": "1.10.163",
"download_url": "https://github.com/llde/xOBSE/releases/download/22.13/xOBSE-22.13.zip",
"checksum": "92d18411a9da803ffaac8c0b005cb6b37c4f9ce2f1571215f39363760d832bf2",
"file_whitelist": [
"Data",
"obse_1_2_416.dll",
"obse_editor_1_2.dll",
"obse_steam_loader.dll",
"obse_loader.exe"
]
}
]
}
}
]YAMLoblivion:
name: 'The Elder Scrolls IV: Oblivion'
nexus_slug: oblivion
launcher_ids:
steam: 22330
gog: 1458058109
subdirectory: Oblivion
executable: OblivionLauncher.exe
tricks:
- "d3dcompiler_43"
- "d3dx9"
script_extender:
- version: "22.13"
runtime:
steam:
- "1.2.0416"
gog:
- "1.2.0416"
download:
checksum: 92d18411a9da803ffaac8c0b005cb6b37c4f9ce2f1571215f39363760d832bf2
direct:
url: "http://www.uesp.net/downloads/files/obse/obse_22_13.zip"
file_whitelist:
- "Data"
- "obse_1_2_416.dll"
- "obse_editor_1_2.dll"
- "obse_loader.exe"
- "obse_steam_loader.dll"I think my main gripe is just the {} structuring |
Beta Was this translation helpful? Give feedback.
-
|
We could keep the gameinfo split into files per-game to cut down on table nesting, not sure how much that helps... Ultimately doesn't matter to me. I usually avoid TOML for anything non-trivial in structure, but this isn't that complicated. I do kinda like the idea of implementing it in Python though. Regardless of how we package the distributable, we could load user modules at runtime like we'd need to do with a data file anyways. It'd also open the door to users doing silly/interesting stuff inside custom infos, not sure if that's a pro or a con. |
Beta Was this translation helpful? Give feedback.
-
|
Just throwing this into the discussion: https://github.com/ModOrganizer2/modorganizer-basic_games/blob/fda5d6ab1e1d70669693e4fa791f607dae79bc9a/games/game_cyberpunk2077.py#L200 MO2's game plugins are already written in python and contain most of the metadata the installer would need apart from the protontricks and the scriptextender links. Might needs some stubs to replace MO classes but they could be imported pretty easily since the rewrite is done in Python anyway. |
Beta Was this translation helpful? Give feedback.
-
I'm not a big fan of this approach. Yes, it would make things easier, but I would prefer to concretely ensure the games are properly tested and compatible. This kinda feels like a 'add it all and see what works, deal with problems as they come' approach
Either way, thanks for linking that. I was able to look it over and get all the missing Epic IDs that we haven't had so far - da29015 |
Beta Was this translation helpful? Give feedback.
-
Yeah I get that. But it would definitely be nice if I as a user could extend the list easily. Like edit a yaml file to add unsupported games. And instead of adding all games MO2 supports, you could define a list of supported titles and have all unsupported titles be in a separate "advanced" section in the UI to select with a big disclaimer that these titles are unsupported. So you don't have to deal with people reporting issues for unsupported titles but give users the ability to test these games on their own. In the end I would definitely hope that with the Python rewrite I can still add games on my own, like I did with Stalker SoC which works perfectly fine with MO2 on Windows and Linux but doesn't have an official MO2 plugin and support. |
Beta Was this translation helpful? Give feedback.
-
That's the intent, one way or another. Stealing the information from MO2 is an interesting idea, even if we filter it to games that we know work (at least for the basic flow). |
Beta Was this translation helpful? Give feedback.
-
|
Hi, I'm wondering whether the redirector is actually still needed? The main argument in #275 (comment) against using a Steam shortcut was that it seems to be unstable. I don't know if Steam has made changes to their VDF structure in the past 4 years. On the other hand, I was playing the Tuxborn Wabbajack list for several weeks now, which is installed using a Steam shortcut and doesn't use a redirector or similar mechanism, but simply calls the SKSE launcher. I'm also working on a setup for linux, which will allow for easy developing of mods and/or modlists using MO2. This setup also runs with a steam shortcut, currently calling the native skyrim launcher, but eventually should end up calling the SKSE launcher. No redirector is needed for this. In both occasions I didn't come upon any issues which were related to using a Steam shortcut. So, have you checked about the state of actually using native steam mechanics, and additionally applying tricks in the games compatdata folder would be a working alternative? I'm aware you are currently working on placing the redirector outside the game directory, which would be, for me, a must-have for the installer to be an alternative to a manual setup, but still having no actual need anymore for this would, in my opinion, even better. |
Beta Was this translation helpful? Give feedback.
-
|
DLL replacement is clever, but I think it's a much better fit for game-specific stuff, or something like ReShade where you're dependent on a specific library anyways. The more that we can follow the MO2 pattern of not messing with the actual game install the better IMHO. We could bake which DLL to steal into the game info and redirect it like the executable now. Would be beneficial to solve the launcher or multi-executable issue as mentioned if shortcuts don't work out. Also, still a little fuzzy on shortcuts. @Furglitch , you're making it sound like the shortcut gets tied to an existing game, not sure I've seen that behavior before (I've only ever used them to launch stuff like Lutris on my deck). You said it works with Wabajack? Does that run the game in its prefix afterwards, or just output to the data folder? |
Beta Was this translation helpful? Give feedback.
-
|
Don't worry @Furglitch, I don't find you to be confrontational at all! On the contrary I appreciate you giving my ideas serious consideration. I do think replacing a DLL is feasible as far as the game folder is concerned: to use OptiScaler as an example again, it is designed so that it can replace any of dxgi.dll, winmm.dll, version.dll, dbghelp.dll, d3d12.dll, wininet.dll, or winhttp.dll. There is typically no conflict with any file already in the game folder because most of these DLLs typically come from System32. Further, the OptiScaler DLL is designed to forward all function calls to the real DLL, so it doesn't break any program behavior. This would be even easier for us are there are no functions we're interested in intercepting -- we only want a constructor that launches MO2. We can simply pick one DLL name for each game that we're reasonably sure doesn't conflict with any popular modding tools for it and keep it in gameinfo; or, like OptiScaler, we can ask the user which DLL to replace. The hard part, in my opinion, is that we would have to maintain the extra C/C++ code to declare the exported functions from the original DLL and pass calls on to the same. We could borrow OptiScaler's code or use a tool like Admittedly, I do think the other solutions are better on the grounds alone that they require far less work from us. I just wanted to point out that it is a plausible avenue if we decide the extra work is worth it. |
Beta Was this translation helpful? Give feedback.
-
As I said, it's been a while since I tried it out, but if memory serves... I'd make a shortcut to an MO2 instance for, say, Fallout 4. When I'd launch the game from MO2, both the MO2 shortcut and the Fallout 4 install would be marked as 'Running' in Steam. I might be remembering behavior from Windows or maybe an unintended benefit of how I set something up, so take what I say with a grain of salt. I'll have to test it tomorrow when I'm off work. @daemon-mouse |
Beta Was this translation helpful? Give feedback.
-
|
Hi! I've switched my gaming rig to CachyOS and been tinkering around with your MO2 installer for a few days now. The problem with this game is that it doesn't like it's main executable to be renamed, it MUST be named XR_3DA.exe so the steam-redirector is definitely a hindrance and I didn't like my game folder being modified even if it's just the instance_path.txt and the redirector. Validing my game files would also mean that Steam will override the steam-redirector. I'm not sure if many games have this behavior but I think it would be best if the MO2 Linux Installer wouldn't touch the game files at all. My solution was to simply write a script that is added to the Launch Options of the game, see: https://github.com/shawly/modorganizer2-linux-installer/blob/master/handlers/modorganizer2-steam-launch-wrapper.sh I think it would be perfectly fine for the setup wizard to tell the user to modify their game's launch options and add I think adding a launch wrapper script is much more flexible and allows for more customization than using a redirector executable or hook. One could also make the
Personally I think using a DLL to inject some redirection logic would be rather frail, much more complex and it would require setting WINEDLLOVERRIDES anyway in the Steam launch options so that "redirection" |
Beta Was this translation helpful? Give feedback.
-
I meant Steam shortcuts @tim-elmer. The entries in the Steam UI where you start a game, being it a steam or a non-steam game.
AFAICS the game runs in its prefix. It shows up in
@Furglitch for a more recent implementation you might take a look at the Jackify project, which features a (mostly, it uses part of the wabbajack code in its engine) native Wabbajack layer for linux boxes. I used this tool to install the Tuxborn wabbajack modlist. In this file is the code for the handling of shortcuts. For my own setup, I was using the native steam "install non-steam game" menu and pointed it to the location of the ModOrganizer.exe file unpacked from the MO2 zip file and configured MO2 as a portable instance. |
Beta Was this translation helpful? Give feedback.
-
|
Alright! As of 82c4597, the only things left to do are refactor the NXM handler and the Redirector rewrite. Then maybe one more general refactor, to make sure things are clean. At least, I believe that's all that needs to be done. Some testing would be great. @tim-elmer @waebbl @shawly @daemon-mouse @AceOfKestrels If you do indulge me, I'd recommend using a directory separate from your current MO2 installs, just in case something goes wrong. A build of the most recent commit is available in the Action artifacts. https://github.com/Furglitch/modorganizer2-linux-installer/actions/runs/21856057709 As for using it, I've been writing documentation alongside the code. Improvements since my last update:
Things I have tested:
Then I did it all again until no weird issues were left over Known issues:
If you do test, please let me know of any issues you run into, or any edge cases I may have missed! Thanks a ton. Ideas I need feedback on:
Again, thanks for all your help thus far. I might take a small 1-2 day break as my server's busted and I gotta get that back up since I have a few people who use the services on it. |
Beta Was this translation helpful? Give feedback.
-
That should be resolved by #913. I've posted the full artifact list at the bottom of the discussion.
So, after some testing (definitely reproducible)... I'll add some logic to detect whether or not the run is in-dev or a binary, alter the path resulting accordingly.
Essentially, yes. We'd have one copy of the redirector (say in
I could see that being an issue, definitely.
Currently the state file already keeps track of the plugins installed in an instance, so it'd just be a matter of loading that list instead of the list from the plugin_info file.
I'm the same way, Most of my instances live on the same drive as the game, i.e.
On scenario A, yeah it's definitely niche... It's already implemented to a point. Let's the user choose if it has multiple installs. For Scenario B... My reasoning is partially selfish lol. Sometimes I just have multiple modlists, like for FO4, I most often play Fallout Anomaly + a personally made list, and up to this point have just been swapping the directories.
Clarified in 98189a2
Added in d20bcb0 |
Beta Was this translation helpful? Give feedback.
-
If we want one shortcut per game, I think the obvious answer would be to pass the game ID as an argument. Not sure how to handle that otherwise. If we just have one shortcut, we just have one shortcut and no contextual knowledge (unless someone can find a USB mind-reader) as to what game the user wants to launch, so we'd need to fall back on a selector menu. I think I'd prefer having an additional shortcut per instance, rather than a menu, but that's just my opinion.
What I meant was that I can see an argument for us automatically installing mods - like a script extender, but I personally think we should try to stay out of the game's install directory if at all possible. I like the idea of automatically adding root builder and any script extenders, I'm just nervous about spawning a new genre of support issues regarding people not understanding how root builder works.
The point I was trying to make is that we should only allow removal of plugins that either we installed, or that we understand the structure of (e.g. user manually installed root builder). |
Beta Was this translation helpful? Give feedback.
-
Explained in more detail in #917 |
Beta Was this translation helpful? Give feedback.
-
|
Kinda rolling back to the whole script-extender thing.. Maybe we make it so that script extender installs for root builder, but only if the user has |
Beta Was this translation helpful? Give feedback.
-
|
That seems reasonable. If a user doesn't want to deal with root builder, they can just install the script extender to the game directly (and in the case of SKSE, at least, remove it fairly easily later if so desired). If they do plan on using root builder, they probably appreciate a clean game root anyways, so getting the extender automatically would be a bonus. |
Beta Was this translation helpful? Give feedback.
-
|
Is there a build that currently runs? I wanna see if will work on nixos or if a flake will be needed. |
Beta Was this translation helpful? Give feedback.
-
|
There's currently no dedicated Nix support as I'm more focused on getting the core of the script working. That being said, you're welcome to try the test build here. Please look over my last post, since I list out the known issues there. And know that any instances you make with that build will not be compatible with the full release, as we'll be using a completely different method of directing the game to MO2, and the general 'code is subject to change'. Please let me know your findings so I can tackle it when it comes time. |
Beta Was this translation helpful? Give feedback.
-
|
Hey @ShadowReaper420 I got an email that you left a comment about an error, but I can't seem to load it (Deleted?). Sorry for the delayed response. That error you got should be fixed by #913. Try this build: https://github.com/Furglitch/modorganizer2-linux-installer/actions/runs/21931386763 |
Beta Was this translation helpful? Give feedback.
-
|
I deleted it because I thought I misread something in your above message about a known issue and forgot to repost it lol. I'll see if I can get it working in a flake tonight if I am successful would you like me to make a PR for the flake? |
Beta Was this translation helpful? Give feedback.
-
|
I've started reviewing the new Python code now that I have a bit of time. Would you be interested in making this a uv project? It would make it much easier for others to contribute by locking in dependencies, rather than using a requirements.txt file. This prevents situations where two developers see different behavior because they crafted their virtual environments at different times — using uv, every developer's environment would be identical and easy to set up. To boot, it would make the Makefile much smaller. The only downside is that developers would need to install uv, but this is becoming increasingly common for Python programmers. If you're interested, let me know and I can submit a PR converting the project over to uv so you can see what I mean and try it for yourself. On another note, I absolutely love seeing the use of ruff and pydantic here! |
Beta Was this translation helpful? Give feedback.
-
|
Looking it over, uv sounds great! Haven't heard of it before myself (this is my first major python project) so honestly the suggestion is more than welcome. And yeah, ruff was somewhat of a must-have for me. And pydantic was tim's suggestion, so he gets that award lmao |
Beta Was this translation helpful? Give feedback.
-
|
+1 UV is very nice. I've actually been using it for my fork and just throwing away its config files before committing. |
Beta Was this translation helpful? Give feedback.
-
|
I'll just post the artifact list here so that people can test with the most recent version lol https://github.com/Furglitch/modorganizer2-linux-installer/actions/workflows/build-artifact.yml |
Beta Was this translation helpful? Give feedback.
-
|
So, I'm getting a certification error keeping MO2 from getting downloaded using the artifact from 14th of Feb. |
Beta Was this translation helpful? Give feedback.
-
Missed this, but my fiance was experiencing the same thing. Resolved with 6fdd1b0 |
Beta Was this translation helpful? Give feedback.
-
|
I'm going to be out of state this week for work training (that I've already received.. yay), but when I come back, I'll start work on the handler and redirector. |
Beta Was this translation helpful? Give feedback.
-
|
Just curious about how the current instances of MO2 with the switch to the Python version will be handled. Will this do an in-place upgrade of the existing instances, or will all prior version instances be incompatible and need to be fully replaced? If they need to be fully replaced, will there be any sort of an option to import mods from prior version instances? |
Beta Was this translation helpful? Give feedback.
-
|
All instances using v6 will be incompatible with v7. While not automated, instructions will be provided on moving modlists to v7. It should be as simple as dragging over the |
Beta Was this translation helpful? Give feedback.
-
|
Good to know what to prep for ... drag & drop is easy enough. Thanks. |
Beta Was this translation helpful? Give feedback.
-
|
Also, there's nothing saying you must update. The bash version will continue working until you're ready to do so, I just wouldn't try using both at once since the nexus handler won't play nicely with that. |
Beta Was this translation helpful? Give feedback.
-
|
I want to apologize once again for another extended absence. Things have been... hectic. My home server went down (not fixed), my computer decided to die on me (fixed), i've been out of town for work training quite a bit, and we're all of a sudden moving apartments at the end of the month. Thanks for the patience. 5524a96 fixes the broken-until-now NXM:// handler, at least as far as I could think to test it. I'll be actively working on the new redirector system over, and hope to have it done in the next week. |
Beta Was this translation helpful? Give feedback.
-
|
I already have a working concept of the launch options that doesn't require any executables to be overwritten, which will allow for single executable games to work without issue (which would also resolve #877) 
I still need to refine it. Right now it's separate script, not at all integrated into mo2-lint. but it works. |
Beta Was this translation helpful? Give feedback.
-
|
The redirector is finished!* At first I thought we wouldn't need it at all, but it turns out proton/wine doesn't play well with symlinks as a launch option So we still have the redirector, but it's fully in python and with the launch option implementation, we no longer need to replace the game's executable to run! Which brings the added bonus of bringing LOOT functionality back to single-executable games like Starfield (tested) :) I also managed to implement the same functionality in Heroic! So all three launchers (Steam, Heroic, and Lutris) are fully supported with the new redirector and launch option system. If anyone wants to give it a try, the lastest artifact is here. * I haven't tested as thoroughly as possible since it's 0400, but initial tests look good! Tests were run with Oblivion on all three launchers, and Starfield on Steam, with basic MO2 launching, game launching, LOOT tests, and game launching. I'll run another batch of testing (this run) tomorrow, and I would appreciate if anyone else could also give it a try and report any issues they encounter. @tim-elmer | @daemon-mouse |
Beta Was this translation helpful? Give feedback.
-
|
Not 100 percent sure what you're looking for with the filetree, game root looks like: That's irritating with the launch options. Looked at the metadata editor, it seems like I missed that when I was looking for an implementation of the appinfo.vdf 😑. Ah well, I learned a fair bit working on my implementation. It's odd that the options disappear when the computer restarts rather than Steam. Makes me wonder if something is getting placed in a temp file somewhere? |
Beta Was this translation helpful? Give feedback.
-
|
((dafuq somehow I edited your reply-))
On Epic, Oblivion installs different copies of the game with subfolders for different localizations, so for example, inside I was trying to see if DA Origins did the same thing, even with you being on GOG. Could you give me the full path to DAOriginsLauncher.exe? I might just have the wrong game install directory for GOG. I've been mostly testing with Oblivion since that's the only game I have copies of for all three launchers.
The only thing (to my memory) that should be getting put in temp is the internal config files like gameinfo.. just because that's how Python handles copying them over.. All I can come up that might work with is setting appinfo.vdf to read-only, but that could come at the cost of breaking Steam's game install/update/validate processes, if my thinking is correct... |
Beta Was this translation helpful? Give feedback.
-
|
Weird. Maybe GH lets you do that as the maintainer? IDK I see; it's at
Yeah, I wouldn't do that. The reason I was wondering about a temp file is that if Steam refreshes the file on launch, I'd expect it to remove the option after I restart it, but it doesn't. It also doesn't seem to modify the file on launch. Interesting. |
Beta Was this translation helpful? Give feedback.
-
|
@tim-elmer I think we're seeing the same thing: mo2-lint finds the game install but then "protontricks exited with code 1". It's a SystemExit of 1 at util.wine.protontricks.run line 38, trying to do a simple '-l'. So it's not just that protontricks can't find the game, it's maybe the first time pt() gets called. I'd assume it needs to be configured on SteamOS to function? In my terminal I can run I'll install the protontricks python package directly and poke around. |
Beta Was this translation helpful? Give feedback.
-
|
Python protontricks is working fine on my Deck in a normal python venv, but when I try to run it through uv, protontricks can't find winetricks. That's causing the exit code of 1, and the protontricks stderr message seems to get captured and only show in the SystemError object, not reaching the log. When I use normal python to load the venv created by uv it also can't find winetricks, so the issue shouldn't be uv itself, just how it configures the venv. If I can't put together a PR easily I'll update with my findings in a new comment. |
Beta Was this translation helpful? Give feedback.
-
|
Another week out of town for work training starting tomorrow afternoon. Hopefully the morning is slow so I can churn out fixes for some of the major issues. |
Beta Was this translation helpful? Give feedback.
-
|
I got MO2 launching for morrowind and oblivion on steam deck but not for fallout3 and falloutnv. I still don't know why uv-installed protontricks needs to be pointed at winetricks when the standard python/pip-installed package finds something (not sure where). My manual workaround is to download winetricks earlier than the rest of the external resources and set $WINETRICKS; I submitted a PR that implements this. I also found a shortfall in the launch option selector solution. Games launched via the command Btw @Furglitch nxm links have never worked for me on steam deck even with the bash version of the code. I was able to configure the link handler but I think it took editing the .desktop file to show up in the system link handler GUI. MO2 still doesn't react tho |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
I love that this project exists, I appreciate the work that's gone into it, and I love that I've been able to make some small contributions to it. That said, every time I try to work on this codebase I get a headache.
As rockerbacon mentioned back in 2020, the main reason for using bash was ease of integration and rapid iteration. Valid as that may have been, I'm beginning to wonder if the project wouldn't be better served by an easier to understand language.
The project has grown significantly in features, and I'm concerned that maintainability may begin to suffer as the logic becomes harder to understand. We're also increasing our dependency load as such, and it'd be very nice to be able to provide the user a complete or automatic package rather than "install x, y, and z". I've also run into the hurdle of debugging bash scripts multiple times, and can't help but think that hunting down issues would be easier with an actual debugger.
As for what to move to, I'd lean towards Python due to its ubiquity and friendliness, but I'm open to other suggestions - ideally minimizing dependencies (Python, for example, is already on most systems, but something like .NET is much less common) - and maximizing ease of adoption (we don't really need C's performance, and neither Furglitch nor myself are terribly familiar with it).
Thoughts? (apologies for the issue, we don't have discussions enabled)
Beta Was this translation helpful? Give feedback.
All reactions