The Outer Worlds 2

[item520]

20150913 GBE

STEAM_LOG_2131539534.log

[otavepto]

Some notes

Early shutdown then reinit

[8 ms, 8302 us] [tid 30284] SteamAPI_Shutdown()

Then later after 4 complete seconds!

[4092 ms, 4092435 us] [tid 28792] Mine_Connect() 

No parser can get achievements list/json, since the game isn't out yet at this time, which will be a problem on the long run. Many games expect a callback/callresult when they trigger a known achievement.

No calls to Steam_Client::RegisterCallback()

gbe_fork/dll/steam_client.cpp

Lines 487 to 492 in 507ac2f

	void Steam_Client::RegisterCallback( class CCallbackBase *pCallback, int iCallback)
	{
	int base_callback = (iCallback / 100) * 100;
	int callback_id = iCallback % 100;
	bool isGameServer = CCallbackMgr::isServer(pCallback);
	PRINT_DEBUG("isGameServer %u %i %i", isGameServer, iCallback, base_callback);

Last line
[6675 ms, 6675536 us] [tid 26348] SteamCallResults::addCallResult() 1281
Is coming from this call because the background thread started running
[6674 ms, 6674411 us] [tid 26348] Steam_Client::background_thread_proc() run @@@@@@@@@@@@@@@@@@@@@@@@@@@

gbe_fork/dll/steam_networking_utils.cpp

Lines 706 to 712 in 507ac2f

	void Steam_Networking_Utils::RunCallbacks()
	{
	if (init_relay && !relay_initialized) {
	relay_initialized = true;
	SteamRelayNetworkStatus_t data = get_network_status();
	callbacks->addCBResult(data.k_iCallback, &data, sizeof(data));
	}

Init relay access, as mentioned in this comment, was a previous fix

gbe_fork/dll/dll/steam_networking_utils.h

Lines 23 to 38 in 507ac2f

	class Steam_Networking_Utils :
	public ISteamNetworkingUtils001,
	public ISteamNetworkingUtils002,
	public ISteamNetworkingUtils003,
	public ISteamNetworkingUtils
	{
	class Settings *settings{};
	class Networking *network{};
	class SteamCallResults *callback_results{};
	class SteamCallBacks *callbacks{};
	class RunEveryRunCB *run_every_runcb{};
	std::chrono::time_point<std::chrono::steady_clock> initialized_time = std::chrono::steady_clock::now();
	FSteamNetworkingSocketsDebugOutput debug_function{};
	bool relay_initialized = false;
	bool init_relay = true; // Initializing relay immediately when we ask for it. Fixing Elden Ring SeamlessCoop

But since the game (seemingly) didn't register any callback handler, this shouldn't be a problem I guess?

The only interesting call from the logs is this one Steam_Apps::BIsSubscribed() but nothing else is happening.

---

There's barely anything happening in the steam ecosystem!, the long time between events (in the scale of seconds) is very unusual.

• Either this log is coming from a secondary exe other than the main one, which is unlikely since this is a direct experimental dll replacement
• The game is using the steam wrapper DRM (very likely)
• The game does some integrity checks on the dll

If this game is using steam stub you have to remove it first.

Also according to this the cloud saves directories seem to be an issue in this game as well, these are the folders you need to create manually from SteamDB.
PR #358 partially addresses that scenario, it seems to be recurring lately.

[gameclamp]

steamclient_experimental work

[Bleibeidl]

Someone fixed the issue in regards to the regular version of GBE-fork.
I asked for the reason and he allowed me to hand it over here:
"The issue is that the emulator (for some reason, I haven't fully traced why that happens) frees the allocated heap with SteamUtils10 interface, after it is created.
So when the game request the interface, it points to nothing. I just disabled the heapfree code."

[otavepto]

@Bleibeidl Thanks for the input, this actually helped.

This is the exception (at RVA 0x1D7653F)

FF15 B3AE4008   | call qword ptr ds:[<&SteamInternal_ContextInit>] // XXX no print_debug("initialize")
48:8B08         | mov rcx,qword ptr ds:[rax]                       // RCX = &SteamClient
48:8B01         | mov rax,qword ptr ds:[rcx]                       // RAX = &SteamClient::vftable
FF50 48         | call qword ptr ds:[rax+48]                       // SteamClient::vftable->GetISteamUtils() <<< XXX nullptr!

[otavepto]

Ping @Detanup01 and @universal963 here's a decompilation of SteamAPI_Shutdown() from the original .so lib

SteamAPI_Shutdown

void SteamAPI_Shutdown(void)
{
  // ...

  if (DAT_steamclient_handle != (void *)0x0) {
    dlclose(DAT_steamclient_handle);
    DAT_0014c648 = 0;
    DAT_0014c640 = 0;
    if (DAT_0014c660 == '\0') {
      if (DAT_0014c608 != (void *)0x0) {
        dlclose(DAT_0014c608);
      }
      DAT_0014c608 = (void *)0x0;
    }
    else {
      if (DAT_0014c5e8 != (void *)0x0) {
        dlclose(DAT_0014c5e8);
      }
      DAT_0014c5e8 = (void *)0x0;
    }

    // <<<<<<<<<<<<<<<<<<<<<<<<<<
    // never goes back to 0 !
    DAT_global_ctr = DAT_global_ctr + 1;
  }

  DAT_steamclient_handle = (void *)0x0;
  return;
}

And here the decomp for SteamInternal_ContextInit() from the same .so

SteamInternal_ContextInit

void * SteamInternal_ContextInit(SteamInitCtx_t *param_1)
{
  void **ppvVar1;
  
  ppvVar1 = &param_1->ctx;
  if (DAT_global_ctr == param_1->counter) {
    return ppvVar1;
  }
  pthread_mutex_lock(&UNION_pthread_mutex_t_0014c6e0);
  if (param_1->counter != DAT_global_ctr) {
    (*(code *)param_1->pfn)(ppvVar1);
    param_1->counter = DAT_global_ctr;
  }
  pthread_mutex_unlock(&UNION_pthread_mutex_t_0014c6e0);
  return ppvVar1;
}

So it turns out, this global counter in the original lib

gbe_fork/dll/dll.cpp

Line 280 in 507ac2f

static uintp global_counter{};

Is never decreased! and this condition

gbe_fork/dll/dll.cpp

Lines 290 to 291 in 507ac2f

	struct ContextInitData *contextInitData = (struct ContextInitData *)pContextInitData;
	if (contextInitData->counter != global_counter) {

Is evaluated to true (in the original lib) when the game calls this API, and this required line

gbe_fork/dll/dll.cpp

Line 293 in 507ac2f

contextInitData->pFn(&contextInitData->ctx);

Is called, which properly initializes the pointers.

But, in our case, the global counter is decreased to 0 on the call to SteamAPI_Shutdown(), then when the game initializes the dll again, this variable in incremented to 1.

Here's the lifetime of this stupid variable:

Global counter lifetime

1. Game calls steam init
2. Global counter = 1
3. Game calls SteamInternal_ContextInit()
   • Context counter = 0, and global counter = 1
   • SteamInternal_ContextInit() then decides to call context->pfn(&context->ctx)
   • SteamInternal_ContextInit() sets context counter = 1 to prevent subsequent calls
4. Game calls Steam_Utils::IsSteamRunningOnSteamDeck()
5. Game calls SteamAPI_Shutdown()
6. Global counter = 0
7. After many seconds, game calls steam init again
8. Global counter = 1
9. Game calls SteamInternal_ContextInit()

• Context counter = 1, and global counter = 1
• SteamInternal_ContextInit() then ignores the call to context->pfn(&context->ctx)

The obvious solution is to mimic the original behavior (and also add the std::lock_guard), but as you can see, this global counter is used in other places to take some decisions when it is 0.
I'm scared to change this variable, so I'll leave to you. You now have the full details of the issue, I'm running from this one 😆

EDIT: scrap all of that, after debugging the original lib with x64dbg, indeed the global counter = 3, but also the local context counter = 3 !
The difference is that the memory is never deallocated, this needs more investigation.

You have to place a breakpoint on the first call to SteamInternal_ContextInit() after the first shutdown.

Here the counters are different (context = 1, global = 3) and the first call to SteamInternal_ContextInit() after the shutdown will indeed call context->pfn(&context-ctx).

So, the context object still holds the old counter value after shutdown (= 1), but since in the original lib the global counter is always incrementing, it reached 3 (init - shutdown - init) and the context is initialized again.

[universal963]

Thanks for the detailed info! So it looks like another game made by devs can't code😅
Currently I have not came up with a better idea. I think we can create a new section or even new ini specially used for compatibility for all these stupid games and then create an option inside of it to specifically tweak this behavior for this game only and/or other games needed.

[otavepto]

I've had this idea for some time, but given how difficult these options would be for the user I end up with workarounds.
In this issue I noticed that the global counter has 2 meanings:

• From Valve's perspective, it's more of a flag that conveys the message that the last steamclient (or SDK in general) state was invalidated, more accurately it would be named global_context_state.
  Value=1 means the first ever init, any subsequent number means a destructive action was taken (shutdown or another call to init) which explains why they compare context->ctr != global_ctr

  They could've made it boolean is_context_init_required each time the SDK is initialized or shutdown, but there might be a more useful meaning for it to be a number.

  • Any non-successful action is ignored (failure to init for example) and the counter stays the same
  • Any redundant action is also ignored (2 or more consecutive calls to init for example) and the counter stays the same

• From emu's perspective, it's a counter to track how many times init or shutdown was called, and only destroy the steamclient instance once it reaches 0, so more accurately speaking it would be named init_shutdown_requests.

  • The emu also ignores increasing or decreasing the counter on non-successful calls
    Here in init
    

    gbe_fork/dll/dll.cpp

    Lines 342 to 345 in 507ac2f

    	STEAMAPI_API steam_bool S_CALLTYPE SteamAPI_Init()
    	{
    	PRINT_DEBUG_ENTRY();
    	if (user_steam_pipe) return true;

    
    And here in shutdown
    

    gbe_fork/dll/dll.cpp

    Lines 368 to 379 in 507ac2f

    	STEAMAPI_API void S_CALLTYPE SteamAPI_Shutdown()
    	{
    	PRINT_DEBUG_ENTRY();
    	// if nothing is initialized just return, see the note in SteamGameServer_Shutdown()
    	// guard against sloppy games programming
    	// don't shutdown if this isn't a client app (maybe gameserver?)
    	if (!user_steam_pipe &&
    	!(steamclient_instance && steamclient_instance->IsUserLogIn())) {
    	PRINT_DEBUG("[WARNING] app is trying to shutdown as client, but it isn't initialzed (is this a gameserver?)");
    	return;
    	}

    
    

    gbe_fork/dll/dll.cpp

    Lines 983 to 998 in 507ac2f

    	STEAMAPI_API void SteamGameServer_Shutdown()
    	{
    	PRINT_DEBUG_ENTRY();
    	// appid 35140 despite being a regular game (not a server) will still call this function
    	// its steam_api.dll (old api) will check first if "g_pSteamClientGameServer" is null, and if so it just returns
    	// hence avoiding sloppy programming
    	// if nothing is initialized just return
    	// don't shutdown if this isn't a gameserver app (maybe client?)
    	if (!server_steam_pipe &&
    	!g_pSteamClientGameServer && // old steam_api.dll checks for this
    	!(steamclient_instance && steamclient_instance->IsServerInit())) {
    	PRINT_DEBUG("[WARNING] app is trying to shutdown as gameserver, but it isn't initialzed (is this a client?)");
    	return;
    	}

  • These same conditions also act as guard for it against redundant calls

The emu and the original lib both have the necessary logic guards, but the meaning may be a little off.
I was thinking we can mimic the original lib by simply adding another counter for the context-init struct thing and leave the scary global_counter alone.
This new counter will be incremented or decremented in the exact same places as the current global_counter, but it will be read/used only by SteamInternal_ContextInit().
This might solve the problem without user intervention, and probably many similar problems that went unnoticed for long time.

[Detanup01]

Aint Steam docs say only initialize once, then destroy/uninit/shutdown when the game shut downs too??
Idk who to blame at this point. gameDevs for doing this, epic for maybe did this, valve to making other devs to do this?
(All of them!)

[universal963]

@otavepto Looks like no other better ideas, at least I don't have one😄

[otavepto]

@universal963 and @Detanup01
One workaround coming up.
Btw, here's one of the biggest and most blatant lies in the SDK.

From the docs about callresults

NOTE: Callbacks and call results are not interchangeable. An event will only come through one or the other, not both. You must make sure that you are registering for the right type of event!

Fast forward to 2024-2025, as proven by appid 756800, this is actually not true, otherwise the game gets stuck in the initial loading screen

gbe_fork/dll/steam_user_stats_stats.cpp

Lines 291 to 294 in 507ac2f

	// appid 756800 expects both: a callback (global event occurring in the Steam environment),
	// and a callresult (the specific result of this function call)
	// otherwise it will lock-up and hang at startup
	SteamAPICall_t ret{};

When you check the docs for ISteamUserStats::RequestUserStats and scroll down to the callbacks section
You can see that UserStatsReceived_t is triggered as a callback!.

So on one hand

Callbacks and call results are not interchangeable

But, on the other hand, the API function ISteamUserStats::RequestUserStats

Returns: SteamAPICall_t to be used with a UserStatsReceived_t call result
And that same call-result is defined in the callbacks section of the interface.

Because of this, I had to scatter a bunch of callbacks after each call-result, like this

gbe_fork/dll/steam_user_stats_stats.cpp

Lines 676 to 678 in 507ac2f

	auto ret = callback_results->addCallResult(data.k_iCallback, &data, sizeof(data));
	callbacks->addCBResult(data.k_iCallback, &data, sizeof(data));
	return ret;

[Stonga2]

@Stonga2 I already unlocked 2 achievements without a problem, the reason it doesn't work is simply because the game is crashing 😄 (assuming you're using this fork) You still need to generate proper achievements.json, I assume that would be possible now. And don't forget to create the folders I mentioned above to solve the issue with saving.

Not sure why you have to do all of that but it seems this isn't the build from this repo. Anyway glad you got it working

I'm using this repo and you're right, the game did crash on every startup with the regular and experimental Goldberg steam_api64.dll. Using Steamless on the .exe made no difference. I had to either use another modified game .exe combined with the Cold Loader or the StubDRM64.dll inside "extra_dlls" folder with the original game .exe to make the game launch.

The other thing I had wrong wad the location of the "steam_settings" folder. I rarely use the Cold Loader so I didn't know you had to have it located in the same folder as the Cold Loader.

If your pull request prevents the game from crashing, I can ignore all these convoluted solutions and go back to a basic Goldberg setup with only the .dll and Steamless on the game .exe.

@otavepto The new repo works like a charm. Thank you for fixing this in such a short amount of time. Amazing.

[universal963]

@otavepto Indeed, during my own tests many API behaviors are different from or even contradictory to how the doc states, rendering the doc some kind of joke. For the callback and callresult stuff, even my incomplete tests showed many APIs returned both, and for some rare APIs (only several, like ISteamRemoteStorage::PublishWorkshopFile()) they could even return different callresult with the same SteamAPICall_t based on how you registered it.