Building better integration tests

[plauche]

No description provided.

[John-LittleBearLabs]

@plauche

This pull request is still a work in progress
This branch has conflicts that must be resolved

[John-LittleBearLabs]

I feel like there should be a TODO to have the get_all* functions return an iterator rather than allocate. This is a good example of why.

[plauche]

I think this could easily have been done inside of sqlite instead of returning all of the CIDs...but this function doesn't end up really getting used, so it can probably get removed.

But yes I agree that a good refactor would be returning iterator from those get_all* functions.

[John-LittleBearLabs]

I mean, you don't provide the last received CID.

Since UDP can go out-of-order and drop middle parts, I wonder if there might be another model that could be followed. Like if you got block 1,2,3,5,6,7 & 8 but missed 4, do you really want to ask for everything starting with 4 to be retransmitted? Couldn't you just as easily start a request for a new DAG rooted at CID of block # 4 ?

[plauche]

Hmm I'm not sure we could start a transmission for a DAG rooted at the CID of block 4, because it wouldn't have the links to resolve the rest of the DAG like the root CID would. But I do think we could use some windowing information to make the first resume request a bit better.

Here is the current behavior for the scenario you laid out:
Let's assume we have a window size of 8 blocks on both sides, so we missed block 4 from that window of 1-8.

1. The receiver would calculate that it has received 7 CIDs for this dag, so it would send that number in a ResumePriorDagTransmit message to the transmitter.
2. The transmitter would receive the Resume... message and determine that the window containing blocks 1...8 needs to be resent
3. The whole window would be transmitted again and then the iterative windowing process would resume

Here is a slightly better way that doesn't involve reworking how we specify missing blocks:
Again assuming a window size of 8 blocks on both sides.

1. The receiver would call a new function, get_missing_blocks_and_window, which takes a root cid and window size, and returns the last window num it was receiving blocks for, and any missing CIDs from that window (based on how large it should be), which is placed in a ResumePriorDagTransmit message and sent to the transmitter
2. The transmitter would send the blocks associated with the CIDs in the received Resume... message and set the window num in the session accordingly
3. After the last partly-received window has been correctly transmitted, then the iterative windowing processing resumes

[John-LittleBearLabs]

Just to clarify: I'm not suggesting a change here and now. Just thinking about these things. As I wrap my head around the project I can't help but noodle a bit.

I'm not sure we could start a transmission for a DAG rooted at the CID of block 4, because it wouldn't have the links to resolve the rest of the DAG like the root CID would.

It would only have its own block and children (grand children of the original root, perhaps) if it has any. Every node in a DAG is a root of a smaller DAG, because 🌴 😄

What I was really getting at is one could have a request that said, "Here's some CIDs I want" and pass in the missing CIDs, and that's logically the same as calling TransmitDag (with your own address as the target) repeatedly - once for every link in the DAG you don't have a block for.

But yeah, if you have a window as a parameter in the request message, then yes you can do the exact equivalent thing. If you can detect a missing block that implies you have a block with a link to it, so you could just as easily 'resume' that block with a window that only covers links you know you don't have.

I don't know if you need the transmitter to initiate the resume stuff, though? myceli could see what blocks it's missing at startup (I imagine when the sat gets powered on it runs main(), yes?) and just start requesting them. Or if that's too much (might accidentally un-GC you), one could have a database record of roots of DAGs that were originally sent, and at startup start requesting links missing only from them (or if the DAG is complete remove it from the table).