Support for serializing / writing packages

[tuokri]

I know it would be a massive effort, but is serializing / writing packages "back" to files after manipulating them something that is planned to be supported?

I am currently doing small asset patching / manipulation using the library, but I am planning on attempting on building something that requires completely removing / changing entire assets within packages. I will probably end up doing the stream manipulation manually in the end, but it would be helpful if the library offered utilities to ensure package binary format remains valid after changing a lot of things.

[EliotVU]

A few years back I had started work on re-vamping the library from the ground up, with the intention to bring the capability to re-serialize entire packages, even moving objects to other packages.

So in a way sure, but I have suspended the work on that for now, as the effort required to get there kept increasing as more problems became apparent.

Adding more support for in-place serializing would definitely be handy, maybe I can cherry pick some of the previous work into the current develop branch.

[EliotVU]

I have been re-visiting the UDefaultProperty class to make it possible to re-serialize it into any stream.

So far it is possible to directly set the value of any tagged property and invoke Serialize:

    private byte _ScriptPropertyValueHost;

    [TestMethod]
    public unsafe void TestScriptPropertySerialization()
    {
        var scriptPackage = UE2PackageContentTests.GetScriptPackageLinker();
        Assert.IsNotNull(scriptPackage);
        scriptPackage.InitializePackage();

        var defaultPropertiesClass = scriptPackage.FindObject<UClass>("DefaultProperties");

        using var stream = UnrealPackageUtilities.CreateTempPackageStream();
        using var linker = new UnrealPackage(stream);
        linker.Build = new UnrealPackage.GameBuild(linker);
        linker.Summary = new UnrealPackage.PackageFileSummary
        {
            Version = 128
        };

        fixed (byte* p = &_ScriptPropertyValueHost)
        {
            var scriptProperty = new UDefaultProperty(defaultPropertiesClass, null, (IntPtr)p)
            {
                Type = PropertyType.ByteProperty,
                Size = 1,
            };

            long backup = stream.Position;

            _ScriptPropertyValueHost = 33;
            scriptProperty.Serialize(stream);
            Assert.AreEqual(33, scriptProperty.AsByte);

            _ScriptPropertyValueHost = 28;
            scriptProperty.Serialize(stream);
            Assert.AreEqual(28, scriptProperty.AsByte);

            scriptProperty.AsByte = 14;
            scriptProperty.Serialize(stream);
            Assert.AreEqual(14, scriptProperty.AsByte);
            Assert.AreEqual(14, _ScriptPropertyValueHost);

            stream.Position = backup;

            scriptProperty.Deserialize(stream);
            Assert.AreEqual(33, scriptProperty.AsByte);
        }
    }

[EliotVU]

Well, got carried away with this! I've implemented serialization support for the entire UnrealPackage header (up to the first UObject in file)

See e9d8296

[tuokri]

This looks awesome, I'll have a look at some point soon for sure!

[EliotVU]

I've made some progress towards this: 46f9cf7 (Allows re-serialization of all byte-code tokens back to the struct)

Also, all UObject classes can now be re-serialized back to the package as well, but that is still undergoing polishment.

[EliotVU]

I've implemented 3a2d281 Serialize(IUnrealStream) for all classes that UELib is capable of deserializing; additionally also re-factored all UObject derivative classes to encapsulate serialized fields, and default any fields where it makes sense to.

[EliotVU]

Putting the new serialize implementations to the test:

🤩