Defragment framework interfaces into context based calls

[hongchaodeng]

Problem Description

The latest build failure was caused in the race:

1. Everything worked correctly at epoch 6
2. After finished sending the data to parent and before new epoch set, a slave died. It restarted in a new node and repeated the same computation.
3. In regression framework,

func (t *dummySlave) ParentDataReady(parentID uint64, req string, resp []byte) {
        ...
    if len(children) != 0 {
        t.framework.FlagMetaToChild("ParamReady")
    } else {
        // On leaf node, we can immediately return by and flag parent
        // that this node is ready.
        t.framework.FlagMetaToParent("GradientReady")
    }
}

In FlagMetaToX, it prepended epoch to the string and set it in etcd. However, the current implementation is wrong about synchronizing epoch here:

func (f *framework) FlagMetaToChild(meta string) {
    value := fmt.Sprintf("%d-%s", f.epoch, meta)
    f.etcdClient.Set (ChildMetaPath, value, 0 )
}

The problem is, a new epoch 7 was set, and it flagged meta with the epoch 7. But it's actually dealing with data in epoch 6!

Analysis

Apparently there are many ways to resolve it. We can stop all work on hand at the end of each epoch. But this is not code scalable -- if we write more remote requests, we also need to consistently close them, and heavy burden on debugging. Or we can just let user pass the epoch to FlagMetaToChild. This violates our design, and I am aware that keeping track of epoch, etc. info should be done in framework, not user level.

Proposed Solution

After a second thought to framework interface, actually there are two types of work in it. One type is epoch specific: FlagMetaToX, DataRequest, (maybe) IncEpoch. The rest of APIs are the other type, epoch agnostic.

I propose to defragment FlagMetatoX, DataRequest into a different interface:

type Context:
  GetEpoch, GetFromID
  FlagMetatoChild, FlagMetatoParent, DataRequest

And change Task interface:

func (t *dummySlave) ParentDataReady(ctx Context, req string, resp []byte) {
    if len(children) != 0 {
        ctx.FlagMetaToChild("ParamReady")
    } else {
        // On leaf node, we can immediately return by and flag parent
        // that this node is ready.
        ctx.FlagMetaToParent("GradientReady")
    }
}

In this way, the context will help us track epoch and it's easy for use to synchronized them under the framework hood.

[hongchaodeng]

@xiang90 @xiaoyunwu
I am aware of the latest build failure and I have some thoughts on it. Help read and discuss it here. Let's define the context interface quickly as first step and then I can start implementing it.

[xiaoyunwu]

"In FlagMetaToX, it prepended epoch to the string and set it in etcd."
Why do we need to set that in etcd?

[hongchaodeng]

@xiaoyunwu

meritop/framework/bootstrap.go

Line 231 in 7e62eb3

// epoch is prepended to meta. When a new one starts and replaces

[hongchaodeng]

More about the "epoch-meta":

1. At start-up, a new node will read meta from neighbors.
2. But it didn't know whether the meta is flagged at current or previous epoch.
3. it parses epoch from "epoch-meta" and only passes the meta to user if epoch is current.

[xiaoyunwu]

ok, I got it, you mean the epoch is in meta, which is in etcd.

But I still do not understand why we have issues in
func (f *framework) FlagMetaToChild(meta string) {
value := fmt.Sprintf("%d-%s", f.epoch, meta)
f.etcdClient.Set (ChildMetaPath, value, 0 )
}

Can you show a list of event that problem will arise? What if we prepend the epoch info to req in the framework, so that we know which epoch is this request about? Basically doing the same thing like we did to meta.

[hongchaodeng]

Request has epoch. In spirit of #101 , we add epoch to all communication between frameworks.

What if we prepend the epoch info to req in the framework, so that we know which epoch is this request about?

This is user API. User didn't add epoch to the meta. We need to separate framework and task layers. See my Analysis section at the top.

[xiaoyunwu]

I know, user call our framework with req, in our implementation we prepend epcoh, before we return result, we take out epcoh from req, that way, user do not have to know anything, but we have info we need to do the right thing.

[hongchaodeng]

That's right. That's why I am proposing to give context to user and that context carries epoch, etc. info.

[xiaoyunwu]

but can we encode context into req then take it out?

On Thu, Jan 15, 2015 at 5:16 PM, Hongchao Deng notifications@github.com
wrote:

That's right. That's why I am proposing to give context to user and that
context carries epoch, etc. info.

—
Reply to this email directly or view it on GitHub
#107 (comment)
.

[hongchaodeng]

There are many choices to give context to user:

1. We can give context at SetEpoch which is the only thing we need to solve this problem.
2. We can give context at all callbacks (DataReady, MetaReady, SetEpoch...) which provides more flexibility to user.
3. We can also give context in other ways like encoding it into req.

It's decision making. I only care letting user call APIs in this way.

There is one more important thing to discuss: what should be called via Context not Framework? My list:

• FlagMetaToX
• DataRequest
• IncEpoch

[xiaoyunwu]

So u do not think pretend epoch would work?

[hongchaodeng]

I didn't catch your question.. Could you use example?

[xiaoyunwu]

I think we should stay in simple if possible, what do u think?

[hongchaodeng]

Do you have simpler solution for this?

[xiaoyunwu]

My preference is that we encoding the context into req and also response,
this way, user has one less thing to worry about. What do you think?

Otherwise, we have to again explain what is context, and how to use it.
Unless there are absolute reason for that, we should stay away.

On the separate note, I am wondering whether we can use the following
interface
ParentDataReady(parentID uint64, req string, resp ByteSink)
ChildDataReady(childID uint64, req string, resp ByteSink)

where ByteSink

type ByteSink interface{
ReadAll(r io.Reader)
}

this way the application can get access to the raw data stream directly,
that can be a big saving when we copy gigabytes data around.

We can also make

// These are payload for application purpose.
ServeAsParent(fromID uint64, req string) []byte
ServeAsChild(fromID uint64, req string) []byte
use writer interface
ServeAsParent(fromId uint64, req string, io.Writer)
for the same reason.

Xiaoyun

On Thu, Jan 15, 2015 at 5:23 PM, Hongchao Deng notifications@github.com
wrote:

There are many choices to give context to user:

1. We can give context at SetEpoch if we assume this is only epoch
   specific context
2. We can give context at all callbacks?
3. We can also give context in other ways like encoding it into req.

It's decision making. I only care letting user call APIs in this way.

There is one more important thing to discuss: what should be called via
Context not Framework? My list:

• FlagMetaToX
• DataRequest
• IncEpoch

—
Reply to this email directly or view it on GitHub
#107 (comment)
.