Sync Issues

[maforget]

I've been having issues with Syncing for a while and just recently did some debugging, so I will use this to collect my findings and things to do.

My main problem was that I had the same series syncing every time even if there was no change or when the books weren't opened or accessed in anyway. I've encountered 3 issues once of which might be a bug or some behavior from the Android version which might not be fixable.

Syncing behavior

When syncing it iterates through all the books needed to be synced and add them to the provider.

ComicRackCE/ComicRack.Engine/Sync/StorageSync.cs

Lines 199 to 208 in 7d30682

	provider.Add(value, hashSet.Contains(value.Id), pagePool, delegate
	{
	UpdateProgress(progress, percentStart + i * 65 / count);
	}, delegate
	{
	UpdateProgress(progress, percentStart + i * 65 / count, msg);
	}, delegate
	{
	UpdateProgress(progress, percentStart + ++i * 65 / count);
	});

Most of the issues are found in that Add method

ComicRackCE/ComicRack.Engine/Sync/SyncProviderBase.cs

Lines 183 to 306 in 7d30682

	public void Add(ComicBook book, bool optimize, IPagePool pagePool, Action workingCallback, Action sendCallback, Action completedCallback)
	{
	if (pendingException != null)
	{
	Exception ex = pendingException;
	pendingException = null;
	throw ex;
	}
	book = (ComicBook)book.Clone();
	book.Series = book.ShadowSeries;
	book.Title = book.ShadowTitle;
	book.Volume = book.ShadowVolume;
	book.Number = book.ShadowNumber;
	book.Count = book.ShadowCount;
	book.Format = book.ShadowFormat;
	ComicBook existing = BooksOnDevice.FindItemById(book.Id);
	if (existing != null && PagesAreSame(existing, book))
	{
	if (ContentIsSame(existing, book) && !existing.ComicInfoIsDirty)
	{
	if (completedCallback != null)
	{
	writeQueue.AddItem(book, delegate
	{
	completedCallback();
	});
	}
	return;
	}
	for (int i = 0; i < book.Pages.Count; i++)
	{
	ComicPageInfo page = book.GetPage(i);
	existing.UpdatePageType(i, page.PageType);
	existing.UpdateBookmark(i, page.Bookmark);
	}
	book.Pages.Clear();
	book.Pages.AddRange(existing.Pages);
	writeQueue.AddItem(book, delegate
	{
	using (ItemMonitor.Lock(deviceAccessLock))
	{
	WriteBookInfo(book, Path.GetFileName(existing.FilePath));
	}
	if (completedCallback != null)
	{
	completedCallback();
	}
	});
	return;
	}
	ExportSetting portableFormat = GetPortableFormat(Device, optimize);
	string temp = EngineConfiguration.Default.GetTempFileName();
	string fileBaseName = portableFormat.GetTargetFileName(book, 0);
	portableFormat.ForcedName = Path.GetFileName(temp);
	portableFormat.TargetFolder = Path.GetDirectoryName(temp);
	while (writeQueue.Count > Math.Max(EngineConfiguration.Default.SyncQueueLength, 5))
	{
	Thread.Sleep(1000);
	}
	ComicExporter export = new ComicExporter(ListExtensions.AsEnumerable<ComicBook>(book), portableFormat, 0);
	export.Progress += delegate
	{
	if (writeQueue.Count == 0 && workingCallback != null)
	{
	workingCallback();
	}
	};
	try
	{
	export.Export(pagePool);
	}
	catch (Exception ex2)
	{
	FileUtility.SafeDelete(temp);
	throw new InvalidOperationException(book.Caption + ": " + ex2.Message, ex2);
	}
	writeQueue.AddItem(book, delegate
	{
	using (ItemMonitor.Lock(deviceAccessLock))
	{
	try
	{
	if (pendingException == null)
	{
	if (completedCallback != null)
	{
	sendCallback();
	}
	if (existing != null)
	{
	Remove(existing);
	}
	long num = GetFreeSpace() - (long)EngineConfiguration.Default.FreeDeviceMemoryMB * 1024L * 1024;
	if (FileUtility.GetSize(temp) > num)
	{
	throw new StorageSync.DeviceOutOfSpaceException(StringUtility.Format(TR.Messages["DeviceOutOfSpace", "Device '{0}' does not have enough free space"], Device.Name));
	}
	string uniqueFileName = GetUniqueFileName(fileBaseName);
	using (FileStream data = File.OpenRead(temp))
	{
	WriteFile(uniqueFileName, data);
	}
	book.Pages.Clear();
	book.Pages.AddRange(export.ComicInfo.Pages);
	book.FilePath = uniqueFileName;
	WriteBookInfo(book, uniqueFileName);
	BooksOnDevice.Add(book);
	if (completedCallback != null)
	{
	completedCallback();
	}
	}
	}
	catch (Exception ex3)
	{
	pendingException = ex3;
	}
	finally
	{
	FileUtility.SafeDelete(temp);
	}
	}
	});
	}

The provider find if the books exists on the device using the Guid. If it does it it uses the PagesAreSame method to determine if the pages are the same, and if not forces it to resend the books, that was the source of my 1st issue.

ComicRackCE/ComicRack.Engine/Sync/SyncProviderBase.cs

Lines 446 to 464 in 7d30682

	public static bool PagesAreSame(ComicBook device, ComicBook library, bool withBookmarks = false)
	{
	ComicPageInfo[] array = device.Pages.ToArray();
	ComicPageInfo[] array2 = library.Pages.ToArray();
	if (array.Length < array2.Length)
	{
	return false;
	}
	for (int i = 0; i < Math.Min(array.Length, array2.Length); i++)
	{
	ComicPageInfo comicPageInfo = array[i];
	ComicPageInfo comicPageInfo2 = array2[i];
	if (comicPageInfo.PageType != comicPageInfo2.PageType || (comicPageInfo2.ImageHeight != 0 && comicPageInfo2.ImageWidth != 0 && comicPageInfo.IsDoublePage != comicPageInfo2.IsDoublePage) || comicPageInfo.PagePosition != comicPageInfo2.PagePosition || (withBookmarks && comicPageInfo.Bookmark != comicPageInfo2.Bookmark))
	{
	return false;
	}
	}
	return true;
	}

Then if the pages are the same it uses the ContentIsSame method to determine if the metadata is the same and if not resend the metadata. That was the source of the 2nd issue, particularly the ComicInfo.IsSameContent used in it.

ComicRackCE/ComicRack.Engine/Sync/SyncProviderBase.cs

Lines 466 to 473 in 7d30682

	public static bool ContentIsSame(ComicBook a, ComicBook b)
	{
	if (a.IsSameContent(b, withPages: false) && a.Rating == b.Rating && a.OpenedCount == b.OpenedCount && a.LastPageRead == b.LastPageRead && a.OpenedTime == b.OpenedTime && a.AddedTime == b.AddedTime && a.ReleasedTime == b.ReleasedTime)
	{
	return PagesAreSame(a, b, withBookmarks: true);
	}
	return false;
	}

ComicRackCE/ComicRack.Engine/ComicInfo.cs

Lines 1474 to 1485 in 7d30682

	public bool IsSameContent(ComicInfo ci, bool withPages = true)
	{
	if (ci != null && ci.Writer == Writer && ci.Publisher == Publisher && ci.Imprint == Imprint && ci.Inker == Inker && ci.Penciller == Penciller && ci.Title == Title && ci.Number == Number && ci.Count == Count && ci.Summary == Summary && ci.Series == Series && ci.Volume == Volume && ci.AlternateSeries == AlternateSeries && ci.AlternateNumber == AlternateNumber && ci.AlternateCount == AlternateCount && ci.StoryArc == StoryArc && ci.SeriesGroup == SeriesGroup && ci.Year == Year && ci.Month == Month && ci.Day == Day && ci.Notes == Notes && ci.Review == Review && ci.Genre == Genre && ci.Colorist == Colorist && ci.Editor == Editor && ci.Translator == Translator && ci.Letterer == Letterer && ci.CoverArtist == CoverArtist && ci.Web == Web && ci.LanguageISO == LanguageISO && ci.PageCount == PageCount && ci.Format == Format && ci.AgeRating == AgeRating && ci.BlackAndWhite == BlackAndWhite && ci.Manga == Manga && ci.Characters == Characters && ci.Teams == Teams && ci.MainCharacterOrTeam == MainCharacterOrTeam && ci.Locations == Locations && ci.ScanInformation == ScanInformation && ci.Tags == Tags)
	{
	if (withPages)
	{
	return ci.Pages.PagesAreEqual(Pages);
	}
	return true;
	}
	return false;
	}

When checking the content it also checks the LastPageRead and updates it in the XML on the device. This is where the 3rd issue arises and might be (partly) responsible for the 2nd issue also, more tests are required.

ComicRackCE/ComicRack.Engine/Sync/SyncProviderBase.cs

Lines 220 to 231 in 7d30682

	writeQueue.AddItem(book, delegate
	{
	using (ItemMonitor.Lock(deviceAccessLock))
	{
	WriteBookInfo(book, Path.GetFileName(existing.FilePath));
	}
	if (completedCallback != null)
	{
	completedCallback();
	}
	});
	return;

Why it Happened

1. I had the same series re-syncing every time, nothing worked like removing the list, deleting the books would fix it. The series in question was an anniversary release, but they had released a year later another version but with new covers, so instead of keeping both I just added new covers to the old release. What I did was just use 7-Zip to insert the new cover and just named the new cover P0001a.jpg. The lists of pages wasn't correctly updated and that is where the issue came from.
   • The before last pages are usually double inner covers, adding the new pages made it so the DB had the double pages width set to say the ImageIndex 52, when it was now the 53rd.
   • That meant that when calling PagesAreSame is would check IsDoublePage comparing the devices pages from the DB.
   • On the device the list of pages index and it's size are determined when the book is optimized so it uses the actual size of pages from the conversion, while the DB only uses the DB size.
   • So it would check an index that was double sized on the device while it wasn't in the DB (because it was now another index).

The solution was simple to open the books info and let the size update correctly so the correct Index is the double page and let the DB update correctly. It also might have required to Refresh the pages, because I believe the size are determined when the books are cached and uses the cache to update them. So when I initially added the new cover it probably took the size of the old index. So when doing things like that selecting all pages and refreshing them to clear the cache might be required.

---

2. The other issue came when ComicInfo.IsSameContent was checked. Since the addition of Tags & Translator to the ComicInfo, I've added them to that function.
   • Just a note that the XML saved on the devices isn't the regular ComicInfo.xml but a full xml that is pretty much a ComicBook.xml, so it includes all the metadata like LastPageRead, even custom values.
   • When the device updates the XML it seems to strip these values, so things like setting a Tags or a Translator that wasn't supported initially doesn't seem to be kept.
   • So when you compare the content then these new field don't match so the new XML file is updated. It is faster than the previous issue since only the metadata is updated and not the whole book.

So what I did internally was add a check to the IsSameContent so that when used for syncing it ignores these new tags. Would need testing to determine if the issue isn't related to the 3rd issue or is because of new fields.

public bool IsSameContent(ComicInfo ci, bool withPages = true, bool isSync = false)
		{
			if (ci != null && ci.Writer == Writer && ci.Publisher == Publisher && ci.Imprint == Imprint && ci.Inker == Inker && ci.Penciller == Penciller && ci.Title == Title && ci.Number == Number && ci.Count == Count && ci.Summary == Summary && ci.Series == Series && ci.Volume == Volume && ci.AlternateSeries == AlternateSeries && ci.AlternateNumber == AlternateNumber && ci.AlternateCount == AlternateCount && ci.StoryArc == StoryArc && ci.SeriesGroup == SeriesGroup && ci.Year == Year && ci.Month == Month && ci.Day == Day && ci.Notes == Notes && ci.Review == Review && ci.Genre == Genre && ci.Colorist == Colorist && ci.Editor == Editor && ci.Letterer == Letterer && ci.CoverArtist == CoverArtist && ci.Web == Web && ci.LanguageISO == LanguageISO && ci.PageCount == PageCount && ci.Format == Format && ci.AgeRating == AgeRating && ci.BlackAndWhite == BlackAndWhite && ci.Manga == Manga && ci.Characters == Characters && ci.Teams == Teams && ci.MainCharacterOrTeam == MainCharacterOrTeam && ci.Locations == Locations && ci.ScanInformation == ScanInformation && (isSync ? true : ci.Tags == Tags && ci.Translator == Translator))
			{
				if (withPages)
				{
					return ci.Pages.PagesAreEqual(Pages);
				}
				return true;
			}
			return false;
		}

---

3. This seems to be an issue with the Android version. If say you want to update the metadata like LastPageRead from the computer to the devices, the XML is updated. I can see that the XML is correctly updated with all the correct info, but upon the device refreshing it resets the value to it's own known value.
   • So if you read a book on the computer and sync the new location, the XML is updated to the new page, but the device will just reset the value to (I believe) is it's own internal DB value.
   • It also sets the ExtraSyncInformation stating that the content was changed.
   • The only way to sync progress was to delete the book from the device. So using options like sync only unread would work correctly, because it would delete the old read books, it is just when updating current books that it wasn't working.

So if the behavior is from the android program it might not be possible to fix. Some additional research is required to know if there is anything that can be done from our side. There must be an internal database used when the folder isn't writable it can't update the book XML like when reading files stored in the secondary location or just manually copying files. There is clearly something something else keeping track. Would need to find an old rooted phone and read what is stored in the android program database.

Maybe changing values in ExtraSyncInformation while writing the info on the device would modify the behavior.

So to reproduce it is easy.

• Sync a new unread/unopened book.
• Read the book on the computer.
• Sync with the device.
• New XML file is created with the new value, but the new progress is discarded.