serialize/unserialize of records with 'array' columns fails

[tlt-miamed]

This bug happens only on edge cases. Let me describe the scenario first:

schema.yml:

Model:
  columns:
    details: { type: array, notnull: true }
  options:
    symfony: { form: false, filter: false }
    type: InnoDB


RelatedModel:
  columns:
    model_id: { type: integer(8), unsigned: true, notnull: true }
  relations:
    Model:    { class: Model, foreign: id, local: model_id, foreignAlias: RelatedModels, type: one, foreignType: many }
  options:
    symfony:  { form: false, filter: false }
    type: InnoDB

Important here is that 'Model' contains a column of type 'array' and 'Model' has a 'RelatedModel'.

Now the database content:
The database should contain at least one 'Model' (id: 1) connected with one 'RelatedModel' (id: 1).
The 'Model'.'details' should contain an array with at least 20 entries.

Now lets provoke the error. I found these two methods:

Method 1: Load form database with cache
If this query hits the cache the unserialize will fail.

$foo = ModelTable::getInstance()
    ->createQuery('m')
    ->leftJoin('m.RelatedModels rm')
    ->select('m.id, m.details, rm.id')
    ->where('m.id = ?', 1)
    ->useResultCache(true)
    ->execute();

Method 2: serialize/unserialize with references

$foo = ModelTable::getInstance()
    ->createQuery('m')
    ->leftJoin('m.RelatedModels rm')
    ->select('m.id, m.details, rm.id')
    ->where('m.id = ?', 1)
    ->execute();

$foo->serializeReferences(true);
unserialize(serialize($foo));

both of these examples will create an error similar to this:

>> sfWebDebugLogger  Notice at /workdir/vendor/lexpress/doctrine1/lib/Doctrine/Collection.php on line 176 (unserialize(): Error at offset 596 of 657 bytes)
NOTICE |13:06:01: {sfWebDebugLogger}  Notice at /workdir/vendor/lexpress/doctrine1/lib/Doctrine/Collection.php on line 176 (unserialize(): Error at offset 596 of 657 bytes)

Notice: unserialize(): Error at offset 596 of 657 bytes in /workdir/vendor/lexpress/doctrine1/lib/Doctrine/Collection.php on line 176

Call Stack:
    0.0005     241824   1. {main}() /workdir/symfony:0
    0.3185   25599144   2. sfSymfonyCommandApplication->run() /workdir/symfony:19
    0.3207   25603696   3. sfTask->runFromCLI() /workdir/vendor/lexpress/symfony1/lib/command/sfSymfonyCommandApplication.class.php:76
    0.3207   25605088   4. sfBaseTask->doRun() /workdir/vendor/lexpress/symfony1/lib/task/sfTask.class.php:98
    0.3664   37410728   5. testTask->execute() /workdir/vendor/lexpress/symfony1/lib/task/sfBaseTask.class.php:70
    0.3757   39949400   6. Doctrine_Query_Abstract->execute() /workdir/lib/task/testTask.class.php:45
    0.6709   42200896   7. Doctrine_Query_Abstract->_constructQueryFromCache() /workdir/vendor/lexpress/doctrine1/lib/Doctrine/Query/Abstract.php:1085
    0.6709   42200944   8. unserialize() /workdir/vendor/lexpress/doctrine1/lib/Doctrine/Query/Abstract.php:1235
    0.6717   42437752   9. Doctrine_Collection->unserialize() /workdir/vendor/lexpress/doctrine1/lib/Doctrine/Query/Abstract.php:1235
    0.6717   42437800  10. unserialize() /workdir/vendor/lexpress/doctrine1/lib/Doctrine/Collection.php:176
    0.6718   42441352  11. Doctrine_Record->unserialize() /workdir/vendor/lexpress/doctrine1/lib/Doctrine/Collection.php:176
    0.6718   42441968  12. unserialize() /workdir/vendor/lexpress/doctrine1/lib/Doctrine/Record.php:869
    0.6718   42455632  13. Doctrine_Collection->unserialize() /workdir/vendor/lexpress/doctrine1/lib/Doctrine/Record.php:869
    0.6718   42455680  14. unserialize() /workdir/vendor/lexpress/doctrine1/lib/Doctrine/Collection.php:176

>> sfWebDebugLogger  Warning at /workdir/vendor/lexpress/doctrine1/lib/Doctrine/Collection.php on line 178 (Invalid argument supplied for foreach())
WARNING|13:06:01: {sfWebDebugLogger}  Warning at /workdir/vendor/lexpress/doctrine1/lib/Doctrine/Collection.php on line 178 (Invalid argument supplied for foreach())

Warning: Invalid argument supplied for foreach() in /workdir/vendor/lexpress/doctrine1/lib/Doctrine/Collection.php on line 178

Call Stack:
    0.0005     241824   1. {main}() /workdir/symfony:0
    0.3185   25599144   2. sfSymfonyCommandApplication->run() /workdir/symfony:19
    0.3207   25603696   3. sfTask->runFromCLI() /workdir/vendor/lexpress/symfony1/lib/command/sfSymfonyCommandApplication.class.php:76
    0.3207   25605088   4. sfBaseTask->doRun() /workdir/vendor/lexpress/symfony1/lib/task/sfTask.class.php:98
    0.3664   37410728   5. testTask->execute() /workdir/vendor/lexpress/symfony1/lib/task/sfBaseTask.class.php:70
    0.3757   39949400   6. Doctrine_Query_Abstract->execute() /workdir/lib/task/testTask.class.php:45
    0.6709   42200896   7. Doctrine_Query_Abstract->_constructQueryFromCache() /workdir/vendor/lexpress/doctrine1/lib/Doctrine/Query/Abstract.php:1085
    0.6709   42200944   8. unserialize() /workdir/vendor/lexpress/doctrine1/lib/Doctrine/Query/Abstract.php:1235
    0.6717   42437752   9. Doctrine_Collection->unserialize() /workdir/vendor/lexpress/doctrine1/lib/Doctrine/Query/Abstract.php:1235
    0.6717   42437800  10. unserialize() /workdir/vendor/lexpress/doctrine1/lib/Doctrine/Collection.php:176
    0.6718   42441352  11. Doctrine_Record->unserialize() /workdir/vendor/lexpress/doctrine1/lib/Doctrine/Collection.php:176
    0.6718   42441968  12. unserialize() /workdir/vendor/lexpress/doctrine1/lib/Doctrine/Record.php:869
    0.6718   42455632  13. Doctrine_Collection->unserialize() /workdir/vendor/lexpress/doctrine1/lib/Doctrine/Record.php:869


06.09.2018 01:06:01 - Task ./symfony, t:t caught exception of class Doctrine_Exception with message Couldn't find class /workdir/vendor/lexpress/doctrine1/lib/Doctrine/Table.php 310
#0 /workdir/vendor/lexpress/doctrine1/lib/Doctrine/Table.php(261): Doctrine_Table->initDefinition()
#1 /workdir/vendor/lexpress/doctrine1/lib/Doctrine/Connection.php(1148): Doctrine_Table->__construct(NULL, Object(Doctrine_Connection_Mysql), true)
#2 /workdir/vendor/lexpress/doctrine1/lib/Doctrine/Collection.php(182): Doctrine_Connection->getTable(NULL)
#3 [internal function]: Doctrine_Collection->unserialize('a:6:{s:4:"data"...')
#4 /workdir/vendor/lexpress/doctrine1/lib/Doctrine/Record.php(869): unserialize('a:15:{s:3:"_id"...')
#5 [internal function]: Doctrine_Record->unserialize('a:15:{s:3:"_id"...')
#6 /workdir/vendor/lexpress/doctrine1/lib/Doctrine/Collection.php(176): unserialize('a:6:{s:4:"data"...')
#7 [internal function]: Doctrine_Collection->unserialize('a:6:{s:4:"data"...')
#8 /workdir/vendor/lexpress/doctrine1/lib/Doctrine/Query/Abstract.php(1235): unserialize('a:3:{i:0;C:31:"...')
#9 /workdir/vendor/lexpress/doctrine1/lib/Doctrine/Query/Abstract.php(1085): Doctrine_Query_Abstract->_constructQueryFromCache('a:3:{i:0;C:31:"...')
#10 /workdir/lib/task/testTask.class.php(45): Doctrine_Query_Abstract->execute()
#11 /workdir/vendor/lexpress/symfony1/lib/task/sfBaseTask.class.php(70): testTask->execute(Array, Array)
#12 /workdir/vendor/lexpress/symfony1/lib/task/sfTask.class.php(98): sfBaseTask->doRun(Object(sfCommandManager), NULL)
#13 /workdir/vendor/lexpress/symfony1/lib/command/sfSymfonyCommandApplication.class.php(76): sfTask->runFromCLI(Object(sfCommandManager), NULL)
#14 /workdir/symfony(19): sfSymfonyCommandApplication->run()
#15 {main}

                        
  Couldn't find class   
                        


Fatal error: Call to a member function evictAll() on null in /workdir/vendor/lexpress/doctrine1/lib/Doctrine/Connection.php on line 1267

Call Stack:
    0.0005     241824   1. {main}() /workdir/symfony:0
    1.0430   42803488   2. sfDatabaseManager->shutdown() /workdir/vendor/lexpress/symfony1/lib/database/sfDatabaseManager.class.php:0
    1.0430   42803616   3. sfDoctrineDatabase->shutdown() /workdir/vendor/lexpress/symfony1/lib/database/sfDatabaseManager.class.php:137
    1.0430   42803792   4. Doctrine_Manager->closeConnection() /workdir/vendor/lexpress/symfony1/lib/plugins/sfDoctrinePlugin/lib/database/sfDoctrineDatabase.class.php:152
    1.0430   42803976   5. Doctrine_Connection->close() /workdir/vendor/lexpress/doctrine1/lib/Doctrine/Manager.php:583
    1.0430   42804560   6. Doctrine_Connection->clear() /workdir/vendor/lexpress/doctrine1/lib/Doctrine/Connection.php:1296

[tlt-miamed]

I could trace down the problem to the custom serialization of Doctrine_Record and Doctrine_Collection.

see #54

[tlt-miamed]

To make it easier to understand the problem. Here is an abstract example of what happens in Doctrine:

$bar = new Model();
$str = serialize($bar);
$res = unserialize($str);

class RelatedModel
{
}

class Collection implements Serializable
{
    private $data = null;

    private $_snapshot = null;

    public function __construct()
    {
        $this->data = $this->_snapshot = [new RelatedModel()];
    }

    public function serialize()
    {
        return serialize(get_object_vars($this));
    }

    public function unserialize($serialized)
    {
        $vars = unserialize($serialized);

        foreach ($vars as $k => $v) {
            $this->$k = $v;
        }
    }
}

class Model implements Serializable
{
    private $_data = ['arrayField' => ['one', 'one', 'one', 'one', 'one', 'one', 'one', 'one', 'one',]];

    private $relation = null;

    public function __construct()
    {
        $this->relation = ['RelatedModels' => new Collection()];
    }

    public function serialize()
    {
        $vars = get_object_vars($this);

        // fields of type array are serialized before the rest
        $vars['_data']['arrayField'] = serialize($vars['_data']['arrayField']);

        return serialize($vars);
    }

    public function unserialize($serialized)
    {
        $vars = unserialize($serialized);

        foreach ($vars as $k => $v) {
            $this->$k = $v;
        }

        $this->_data['arrayField'] = unserialize($this->_data['arrayField']);
    }
}

This will result in this serialized string:

C:5:"Model":330:{a:2:{s:5:"_data";a:1:{s:10:"arrayField";s:132:"a:9:{i:0;s:3:"one";i:1;s:3:"one";i:2;s:3:"one";i:3;s:3:"one";i:4;s:3:"one";i:5;s:3:"one";i:6;s:3:"one";i:7;s:3:"one";i:8;s:3:"one";}";}s:8:"relation";a:1:{s:13:"RelatedModels";C:10:"Collection":82:{a:2:{s:4:"data";a:1:{i:0;O:12:"RelatedModel":0:{}}s:9:"_snapshot";a:1:{i:0;r:19;}}}}}}

The problem lies in r:19;. This is a reference which should point to O:12:"RelatedModel":0:{} because RelatedModel is reference twice in Collection but the number is wrong.

As far as I know serialize gives every object in the serialized sting a number to reference it later but the calculated number is wrong. I think the problem lies in Model::serialize().

On serialize PHP serializes $bar in this order

Model {
  arrayField
  Collection {
    RelatedModel in data
    RelatedModel in _snapshot (as Reference)
  }
}

and every node in the result will get a number to reference it later

On unserialize we change the order (due to the custom serialization)

Model {
  Collection {
    RelatedModel in data
    RelatedModel in _snapshot (fails to find the reference because arrayField was not handled yet)
  }
  arrayField
}

and fail because arrayField is out of order.

[tlt-miamed]

Attention: This bug can also lead to corrupt data. If the arrayField contains only a small array the reference will point to a node in the serialized string which exists but is wrong.

[alquerci]

PHP does not support the double serialization, very weird.

<?php

$value = [['foo'], 'bar'];
$serialized = $value;
$serialized[0] = serialize($serialized[0]);
$serialized = serialize($serialized);
$unserialized = unserialize($serialized);
$unserialized[0] = unserialize($unserialized[0]);

$value == $unserialized // true

[tlt-miamed]

@alquerci your example should work and works.

The problem only exists if serialize uses references in the serialized string.

As in my abstract example we use in Collection::data and Collection::_snapshot the same object. This will create a reference in the serialized string (r:19;). But because we serialize/unserialize the array out of order the reference counter gets out of sync.

[tlt-miamed]

Here a shorter example to illustrate the problem

<?php

class RelatedModel 
{ }

class Model implements Serializable
{
    private $doubleSerialized = ['one', 'one', ];

    private $obj1 = null;
    private $obj2 = null;

    public function __construct()
    {
        $this->obj1 = $this->obj2 = new RelatedModel();
    }

    public function serialize()
    {
        $vars                     = get_object_vars($this);
        $vars['doubleSerialized'] = serialize($vars['doubleSerialized']);

        return serialize($vars);
    }

    public function unserialize($serialized)
    {
        $vars = unserialize($serialized);

        foreach ($vars as $k => $v) {
            $this->$k = $v;
        }

        $this->doubleSerialized = unserialize($this->doubleSerialized);
    }
}

$bar = new Model();
$str = serialize($bar); // 'C:5:"Model":122:{a:3:{s:16:"doubleSerialized";s:34:"a:2:{i:0;s:3:"one";i:1;s:3:"one";}";s:4:"obj1";O:12:"RelatedModel":0:{}s:4:"obj2";r:7;}}'
$res = unserialize($str); // throws error

[tlt-miamed]

Another prerequisite is that you use double serialization in a custom serialize function. Outside of Serializable::serialize() the reference counter gets reset. That's why this example works:

<?php

class RelatedModel
{ }

$object = new RelatedModel();
$value  = [
'foo'  => ['one', 'one'],
'obj1' => $object,
'obj2' => $object,
];

$value['foo']        = serialize($value['foo']);
$serialized          = serialize($value); // 'a:3:{s:3:"foo";s:34:"a:2:{i:0;s:3:"one";i:1;s:3:"one";}";s:4:"obj1";O:12:"RelatedModel":0:{}s:4:"obj2";r:3;}'
$unserialized        = unserialize($serialized);
$unserialized['foo'] = unserialize($unserialized['foo']);

$value === $unserialized; // true