Frustrations and Complexities of Updating appsettings.json During Installation

[nickhoeferpickpro]

Hello WiX Community,

I’ve come across a significant challenge while trying to perform what seems like it should be a simple task: updating an appsettings.json file during the installation of a .NET application. The goal is to merge settings from an old configuration file with a new one during the upgrade process, retaining only the settings from the old existing config while adding any potentially new configurations in the new version.

The issue arises when attempting to execute this code within a deferred custom action:

public ActionResult UpdateAppSettings(ISession session, IFileService fileService)
{
    try
    {
        // Goal: Update the new configuration file with settings from the old configuration file.
        // However, we only want to keep the settings that are present in BOTH the old and the new configuration files.
        // In other words, we are doing the following:
        // 1. Identify common settings that exist in both the old and the new configuration files.
        // 2. For these common settings, use the values from the old configuration file.
        // 3. Any new settings that are only in the new configuration file should remain unchanged.
        // 4. Any old settings that are only in the old configuration file should be discarded.

        // Example:
        // Old Config: { "setting1": "oldValue1", "setting2": "oldValue2" }
        // New Config: { "setting2": "newValue2", "setting3": "newValue3" }
        // Merged Config: { "setting2": "oldValue2", "setting3": "newValue3" }

        // we can get the INSTALLFOLDER and TempFolder property in this deferred custom action because we already set it in the immediate custom action
        var installFolder = PropertyUtil.GetProperty(session, fileService, "INSTALLFOLDER");
        var tempFolder = PropertyUtil.GetProperty(session, fileService, "TempFolder");
		
        string newConfigFilePath = Path.Combine(installFolder, "appsettings.json");
        string newConfigContent = fileService.ReadAllText(newConfigFilePath);

        string backupFilePath = Path.Combine(tempFolder, "appsettings.json.bak");
        string oldConfigContent = fileService.ReadAllText(backupFilePath);

        JObject newConfig = JObject.Parse(newConfigContent);
        JObject oldConfig = JObject.Parse(oldConfigContent);

        // now rename the bak file with the current timestamp
        string timestamp = DateTime.Now.ToString("MM-dd-yyyy-HH-mm-ss");
        string version = PropertyUtil.GetProperty(session, fileService, "ProductVersion");

        var configBackupFolder = Path.Combine(installFolder, "configBackup");
        if (!fileService.Exists(configBackupFolder))
        {
            fileService.CreateDirectory(configBackupFolder);
        }
        string backupFilePathWithTimestamp = Path.Combine(configBackupFolder, $"appsettings.v{version}-{timestamp}.json.bak");
        fileService.Move(backupFilePath, backupFilePathWithTimestamp);

        // now merge the old and new settings
        var mergedConfig = MergeSettings(newConfig, oldConfig);

        // Write the updated JSON back to the file.
        fileService.WriteAllText(newConfigFilePath, mergedConfig.ToString());

        // now move the other backup files back to the install folder
        // but put it in a configBackup subfolder
        // create the folder if it doesn't exist

        var backupFiles = fileService.GetFiles(tempFolder, "appsettings*.json.bak");
        foreach (var backupFile in backupFiles)
        {
            var fileName = Path.GetFileName(backupFile);
            var destFile = Path.Combine(configBackupFolder, fileName);
            session.Log($"MyLog Moving backup file {backupFile} to {destFile}");
            fileService.Move(backupFile, destFile);
        }


        return ActionResult.Success;
    }
    catch (Exception ex)
    {
        session.Log($"MyLog Error updating settings: {ex}");
        return ActionResult.Failure;
    }
}

private static JObject MergeSettings(JObject newConfig, JObject oldConfig)
{
    foreach (var property in newConfig.Properties())
    {
        if (oldConfig.TryGetValue(property.Name, out JToken? oldToken))
        {
            if (property.Value.Type == JTokenType.Object && oldToken.Type == JTokenType.Object)
            {
                MergeSettings((JObject)property.Value, (JObject)oldToken);
            }
            else
            {
                newConfig[property.Name] = oldToken;
            }
        }
    }
    return newConfig;
}

Despite configuring my deferred custom action to run after InstallFiles—which should ensure that the appsettings.json is in place—the custom action is unable to locate the file in the INSTALLFOLDER. The file's absence is unexpected since the installer should have already copied it to the destination directory. This behavior occurs during upgrade sequences, where the action's purpose is to merge the existing settings with the new ones, crucial for the application's functionality.

Here's the relevant section from my WiX setup:

<SetProperty 
	After="FindRelatedProducts" 
	Id="FirstInstall" 
	Value="true" Condition="NOT Installed AND NOT WIX_UPGRADE_DETECTED AND NOT WIX_DOWNGRADE_DETECTED" />
<SetProperty 
	After="SetFirstInstall" 
	Id="Upgrading" 
	Value="true" Condition="WIX_UPGRADE_DETECTED AND NOT (REMOVE=&quot;ALL&quot;)" />
<SetProperty 
	After="RemoveExistingProducts" 
	Id="RemovingForUpgrade" 
	Sequence="execute" Value="true" Condition="(REMOVE=&quot;ALL&quot;) AND UPGRADINGPRODUCTCODE" />
<SetProperty 
	After="SetUpgrading" 
	Id="Uninstalling" 
	Value="true" Condition="Installed AND (REMOVE=&quot;ALL&quot;) AND NOT (WIX_UPGRADE_DETECTED OR UPGRADINGPRODUCTCODE)" />
<SetProperty 
	After="SetUninstalling" 
	Id="Maintenance" 
	Value="true" Condition="Installed AND NOT Upgrading AND NOT Uninstalling AND NOT UPGRADINGPRODUCTCODE" />


<Binary Id="MyCustomActionDll" SourceFile="$(var.CustomActionDLL)" />

<CustomAction Id="InstallServiceCertificate" BinaryRef ="MyCustomActionDll" DllEntry="InstallServiceCertificate" Execute="deferred" Impersonate="no" Return="check"/>
<CustomAction Id="UninstallServiceCert" BinaryRef ="MyCustomActionDll" DllEntry="UninstallServiceCert" Execute="deferred" Impersonate="no" />
<CustomAction Id="BackupExisting" BinaryRef="MyCustomActionDll" DllEntry="BackupExisting" Execute="deferred" Return="check" />
<CustomAction Id="UpdateAppSettings" BinaryRef="MyCustomActionDll" DllEntry="UpdateAppSettings" Execute="deferred" Return="check" />
<CustomAction Id="PersistProperties" BinaryRef="MyCustomActionDll" DllEntry="PersistProperties" Execute="immediate" Return="check" />

<InstallExecuteSequence>
	<Custom Action="PersistProperties" Before="InstallFinalize" Condition="FirstInstall OR Upgrading OR Maintenance OR Uninstalling"/>
	<Custom Action="InstallServiceCertificate" Before="InstallFinalize" Condition="FirstInstall"/>
	<Custom Action="UninstallServiceCert" Before="InstallFinalize" Condition="Uninstalling"/>
	<Custom Action="BackupExisting" Before="InstallFinalize" Condition="Upgrading"/>
	<Custom Action="UpdateAppSettings" After="InstallFiles" Condition="Upgrading"/>
	
</InstallExecuteSequence>

The core issue is that even though the UpdateAppSettings custom action is scheduled correctly and the INSTALLFOLDER property is set, the file does not appear to be accessible or present at the time the custom action executes. This is contrary to what one would expect given the sequence specified in the WiX script.

I’ve been through numerous posts and documentation, tried various combinations of deferred and immediate custom actions, and encountered every roadblock you can imagine, not to mention the countless hours I've spent watching all of the youtube videos from the deployment dojo – from permission issues to the notorious "file not found" errors because the files are still in their temporary state.

I’m aware of the philosophical stance in the WiX community that custom actions are generally a sign of installation failure, or at best, a necessary evil. While I understand the sentiment behind minimizing the use of custom actions, as highlighted in Rob Mensching's blog post "Zataoca: Custom actions are (generally) an admission of failure," I find myself at an impasse.

Adding to the confusion is the state of the official WiX documentation. The current resources seem to offer a fragmented view—focusing on isolated details without much-needed context or comprehensive coverage. This absence of structured and thorough documentation leaves developers piecing together information like a jigsaw puzzle, often having to resort to trial and error to understand the full picture of how WiX operates, particularly with complex custom actions.

The disorganized state of the documentation has only amplified the difficulties I've encountered with this installation procedure. A process that should be straightforward is now mired in opacity, leading to an excessive amount of time spent on what should be a routine task.

Here are the options as I see them:

Wait for a solution to be implemented in some library, delaying the project indefinitely.
Attempt a workaround that might involve less-than-ideal practices and potential future headaches.
Continue to pour hours into finding a solution that allows the custom action to execute as intended.
The frustration stems from the necessity of the task at hand. We're not discussing a frivolous or optional feature – this is a core functionality that needs to be addressed for the application to work correctly post-upgrade. I urge the WiX team to consider enhancing the documentation to provide a clearer, more sequential guide that walks through the process, especially for deferred custom actions.

To the experts and veterans here: what is the recommended course of action when faced with such a scenario? Is there a middle ground where practicality meets best practice? And how do we reconcile the need to deliver on time with the desire to adhere to the principles of good setup design?

I’m reaching out to the community in hopes of finding guidance, or perhaps to spark a discussion on how we can better handle these types of common tasks within the WiX framework.

Thank you for your time and assistance.

[nickhoeferpickpro]

I've come up with a workaround and .... lets just say I don't think this is Rob approved.

/// <summary>
/// This method is called from the installer to invoke a method in a class in a dll with elevated privileges
/// This will let us get around the issue of not being able to run an deferred custom outside of InstallInitialize and InstallFinalize
/// </summary>
/// <param name="session">Interface wrapper for the current session of the custom action.</param>
/// <param name="assembly">The assembly containing the class with the method to invoke</param>
/// <param name="type">The type containing the method to invoke</param>
/// <param name="methodInfo">The method to invoke</param>
/// <returns></returns>
public ActionResult InvokeElevatedImmediate(ISession session, Assembly assembly, Type type, MethodInfo methodInfo)
{
    var assemblyName = assembly.Location;
    var namespaceName = type.Namespace!;
    var typeName = type.Name;
    var methodName = methodInfo.Name;
    session.Log($"MyLog Invoking {assemblyName} {namespaceName}.{typeName}.{methodName} in powershell with elevated privileges");
    var powerShellScriptString = @"
    add-type -path '" + assemblyName + @"'
    [" + namespaceName + "." + typeName + @"]::" + methodName + @"()
";
    RunCommand("powershell.exe", $"-NoProfile -ExecutionPolicy unrestricted -Command \"{powerShellScriptString}\"");
    return ActionResult.Success;
}

private static void RunCommand(string command, string arguments)
{
    using var process = new Process();
    process.StartInfo.FileName = command;
    process.StartInfo.Verb = "runas"; // <-- this is what allows us to run with elevated privileges
    process.StartInfo.Arguments =  arguments;
    process.StartInfo.UseShellExecute = true; // <-- this is required to run with elevated privileges
    process.StartInfo.RedirectStandardOutput = false; // <-- this is required when UseShellExecute is true
    process.StartInfo.RedirectStandardError = false;
    process.StartInfo.CreateNoWindow = true;
    process.Start();
    process.WaitForExit();
}

and the wxs:

<SetProperty 
	After="FindRelatedProducts" 
	Id="FirstInstall" 
	Value="true" Condition="NOT Installed AND NOT WIX_UPGRADE_DETECTED AND NOT WIX_DOWNGRADE_DETECTED" />
<SetProperty 
	After="SetFirstInstall" 
	Id="Upgrading" 
	Value="true" Condition="WIX_UPGRADE_DETECTED AND NOT (REMOVE=&quot;ALL&quot;)" />
<SetProperty 
	After="RemoveExistingProducts" 
	Id="RemovingForUpgrade" 
	Sequence="execute" Value="true" Condition="(REMOVE=&quot;ALL&quot;) AND UPGRADINGPRODUCTCODE" />
<SetProperty 
	After="SetUpgrading" 
	Id="Uninstalling" 
	Value="true" Condition="Installed AND (REMOVE=&quot;ALL&quot;) AND NOT (WIX_UPGRADE_DETECTED OR UPGRADINGPRODUCTCODE)" />
<SetProperty 
	After="SetUninstalling" 
	Id="Maintenance" 
	Value="true" Condition="Installed AND NOT Upgrading AND NOT Uninstalling AND NOT UPGRADINGPRODUCTCODE" />



<Binary Id="MyCustomActionDll" SourceFile="$(var.CustomActionDLL)" />

<CustomAction Id="InstallServiceCertificate" BinaryRef ="MyCustomActionDll" DllEntry="InstallServiceCertificate" Execute="deferred" Impersonate="no" Return="check"/>
<CustomAction Id="UninstallServiceCert" BinaryRef ="MyCustomActionDll" DllEntry="UninstallServiceCert" Execute="deferred" Impersonate="no" />
<CustomAction Id="SetComputerName" BinaryRef="MyCustomActionDll" DllEntry="SetComputerName" Execute="immediate" Return="check" />
<CustomAction Id="InvokeBackupExistingSettings" BinaryRef="MyCustomActionDll" DllEntry="InvokeBackupExistingSettings" Execute="immediate" Return="check" />
<CustomAction Id="InvokeUpdateAppSettings" BinaryRef="MyCustomActionDll" DllEntry="InvokeUpdateAppSettings" Execute="immediate" Return="check" />

<InstallExecuteSequence>
	<Custom Action="InstallServiceCertificate" Before="InstallFinalize" Condition="FirstInstall"/>
	<Custom Action="UninstallServiceCert" Before="InstallFinalize" Condition="Uninstalling"/>
	<Custom Action="InvokeBackupExistingSettings" Before="RemoveExistingProducts" Condition="Upgrading"/>
	<Custom Action="InvokeUpdateAppSettings" After="InstallFinalize" Condition="Upgrading"/>
	<Custom Action="SetComputerName" After="InstallInitialize" Condition="FirstInstall"/>
</InstallExecuteSequence>

I'd love to know if anyone else has a better solution.

[nickhoeferpickpro]

I should point out that this has it's own problems because Windows will flag it as a threat.

[chrpai]

I didn't read everything you wrote but I know how I dealt with this in .NET framework. It's much simpler and I'm sure if you reframe the problem you can think of a way to do it in .NET. I haven't really bothered to do it yet because most of my .NET apps are in house server apps where I can just bake the settings into a file for each life cycle.

So in App.Config there was a file attribute on the AppSettings element that specified a relative path file that contained override files. You could have:

foo.exe.config -> foo.exe.config.override

This solved the merge dilema by keeping built/static installer owned data in one file (foo.exe.config) and configuration data or user provided data in a seperate file (foo.exe.config.override)

My doing this MSI could always safely overwrite the first file and would never touch the second file because it didn't create it or know about it unless done by custom action in which case you owned the data following the remember property pattern.

I've used this countless times and it works perfectly.

[chrpai]

Update:

Consider this snippet:

        IConfiguration builder = new ConfigurationBuilder().AddJsonFile("appsettings.json", false).AddJsonFile("override.json", true).Build();
        string port = builder.GetSection("ServerSettings")["Port"];

AppSettings.json:

{
"ServerSettings": {
"Port": "80"
}
}

Override.json ( can exist or can not exist )
{
"ServerSettings": {
"Port": "81"
}
}

If override.json doesn't exist, port will equal 80. If Override.json does exist it'll equal 81. This greatly simplifies the installer.

[nickhoeferpickpro]

Hey, thanks for your suggestion about using appsettings.json and override.json. I've been pondering over it and here's where I stand: The key for me is automating the configuration update process. While having two files separates default and customizable settings, I realized I could still automate handling both. However, this brings me back to wondering about the advantage of splitting into two files. In my project, nearly all settings in the config file are subject to change or customization. So, if the goal is full automation, the two-file approach might not offer significant benefits over a single, well-managed file. I'm curious to know if there's an aspect I might be overlooking about the two-file method, especially in the context of automated updates. Your thoughts?

[chrpai]

Trust me, it works great! :) This was a problem I first encountered with XML files. In scenario where build 1 has XML1 and user then modifies XML1 to be XML1+U (either via installer or post install / first run doesn't matter ) and build 2 has XML2, the installer is wrong if it overwrite the file because it looses the +U and it's wrong if it doesn't overwrite the file because it loses the 1->2 changes. This leaves you to write custom actions to migrate the schema or migrate the user data and thats a crazy complex road to go down as you found out.

By splitting them into two files the first file can always 100% owned by the installer and always overwritten as it should never contain user data. The second file can be managed by the installer (custom actions) and/or application.

I've done this countless times. Consulting services are available if you want help. The first hour is always free.

[nickhoeferpickpro]

Here, let me give a more realistic hypothetical example to provide more context

Old Config:

{
  "API": {
    "Endpoint": "http://oldapi.example.com", // <- this will stay the same
    "Timeout": 30                            // <- this will be removed 
  },
  "Theme": "Dark"                            // <- this will stay the same
}

New Config:

{
  "API": {
    "Endpoint": "http://default.example.com", // <- this will be discarded because we already have this 
    "Retries": 3                              // <- this will be added
  },
  "Theme": "Light",                           // <- this will be discarded because we already have this 
  "Analytics": {                              // <- this will be added
    "Enabled": true                           // <- this will be added
  }
}

Resulting Config after update:

{
  "API": {
    "Endpoint": "http://oldapi.example.com",  // <- same
    "Retries": 3                              // <- new
  },
  "Theme": "Dark",                            // <- same
  "Analytics": {                              // <- new
    "Enabled": true                           // <- new
  }
}

[wmanning]

If I'm understanding correctly you'll have to rearchitect your app a bit to use @chrpai 's method. The "user" file will be consulted first by the app and if it doesn't contain the needed "config item" then the "installer owned" file would be consulted by the app for the "default".

If the user changes config/settings in the app then the app should only update the "user" file. The "installer" file will already remain unchanged and used as a backup.

The advantage is you got a backup say if the user wanted to set the config back to defaults. You can store it in "readonly" area say in Program Files (so no one can touch it). Second advantage is the app knows exactly what the user changed as it's all in the "user" file; although maybe you are storing default config in code so not an advantage in that case perhaps.

[nickhoeferpickpro]

Even if I do use a second file, that still doesn't solve the problem of not having elevated privileges to modify the file in a way that is completely automated during the upgrade process.

Perhaps is misphrased the problem. The Installer does not have the permissions necessary at the point where it is needed. I need access to the INSTALLFOLDER just before InstallInitialize and just after InstallFinalize.

I really do appreciate your feedback though.

[wmanning]

I don't think the INSTALLFOLDER property is available to the deferred custom action. Typically you have to specify which properties you want available to the deferred custom action.
References:

• https://stackoverflow.com/questions/11233267/how-to-pass-customactiondata-to-a-customaction-using-wix
• https://learn.microsoft.com/en-ca/windows/win32/msi/obtaining-context-information-for-deferred-execution-custom-actions?redirectedfrom=MSDN Specifically under where it says: "Only the following limited set of properties is always accessible to custom actions during script execution."

---

I see it's appsettings.json file which is in the installer directory (aka Program Files or Programs Files (x86)). So no point in splitting it in two if both will remain there.

IMHO best practice is usually you have appsettings.json be the "installer owned" file that's fixed and then the "user" file would be in "Program Data" folder or in a user data folder which do not require elevated privileges. Caveat on Program Data though: https://stackoverflow.com/questions/22107812/privileges-owner-issue-when-writing-in-c-programdata.

• Then have the app manage the "user" file and not the installer.

[chrpai]

Yes, a slight rewrite is needed. Not a problem if you've encapsulated your settings. There used to be a blog article saying car manufacturers have to design their vehicles to fit on car trailers and developers should design their vehicles to fit inside the MSI box. This example also works for MSIX. When I meet with customers I make this reccomendation and it works every time.

[nickhoeferpickpro]

sigh
It's unfortunate that windows installer is so convoluted. It certainly has it's share of technical debt. I haven't looked into MSIX yet but the windows installer certainly needs an overhaul.
Perhaps I was naively hoping that WIX would abstract the ugly parts of it but it seems it still bleeds through.

Thanks for the insight anyway.

[nickhoeferpickpro]

I don't think the INSTALLFOLDER property is available to the deferred custom action. Typically you have to specify which properties you want available to the deferred custom action.

I've already handled that in code not listed here:

// we can get the INSTALLFOLDER and TempFolder property in this deferred custom action because we already set it in the immediate custom action
var installFolder = PropertyUtil.GetProperty(session, fileService, "INSTALLFOLDER");

I see it's appsettings.json file which is in the installer directory (aka Program Files or Programs Files (x86)). So no point in splitting it in two if both will remain there.

I agree and I'm still looking into this.

[chrpai]

Since your JSON has "theme" I'm going to assume it's got a UI which means you can have code on first run.

Installler installs INSTALLLOCATION\default.json with contents:

{
"API": {
"Endpoint": "http://oldapi.example.com", // <- this will stay the same
"Timeout": 30 // <- this will be removed
},
"Theme": "Dark" // <- this will stay the same
}

Your application code has first run that decides which of these should be sticky and which will be allowed to float. It creates C:\ProgramData...\machine-wide-override.json and/or populates the values that are sticky and don't already exist.

You upgrade your application and you simply replace INSTALLLOCATION\default.json with the newest version in the build.

Your application code has first run that once again decides which should be sticky and which will be allowed to float. It updates machine-wide-override.json with any additional settings that are sticky that don't yet exist in machine-wide-override.json.

Now you have no custom actions in the installer and your business logic is written in .net where you are more easily able to manage it.

[nickhoeferpickpro]

Since your JSON has "theme" I'm going to assume it's got a UI which means you can have code on first run.

Well no, that was just some made up config file because I didn't want to share the exact config publicly.

But, when you say "first run", do you mean on app startup?
If so, that presents an interesting idea I had not thought of. I'm going to explore that in combination with using the ProgramData folder.

Thanks for the inspiration.

[chrpai]

Yes. Glad to help. Let me know if your interested in taking a look at IsWiX. https://github.com/iswix-llc/iswix-tutorials I think you'll find that if you simplify your requirements to directories, files, shortcuts, services and possibly IIS registration that creating a WiX MSI for this scenario can be done practically without any typing at all.

[nickhoeferpickpro]

Thanks for the suggestion, it looks interesting. Is that like a wrapper around WIX?
However, I want to try and rely libraries/frameworks with a larger community with active development.
Thanks anyway.

[chrpai]

It's not a wrapper. It's a project template (scaffolding) with graphical designers that author WiX code. It's not a framework or a library. It's not used on the build server and doesn't prevent you from still authoring wix by hand. It's been in active development for 12+ years. I wouldn't dismiss it so quickly.