RFC: command hooks in cooker

[pboettch]

Before starting to hack around in the code individually, we should specify what we expect and how hooks should be configured and used in cooker.

I start with some questions:

Shall a hook be called pre-<cmd> and post-<cmd> where cmd is one of the low-level-commands of cooker (i.e.: init, update, generate, build)? Is this finely grained enough? Should we have local-conf-generate-pre/post-hook and bblayers.conf-generate-hook instead of a single generate-hook?

For commands which have items (layers/sources, builds) is the pre and post-hook called for each of them?

In what way do we configure a hook in the menu? Is there a special-section hooks or does each item has a pre-hook/post-hook? generate and init will be at global scope.

IMO, cooker would need a refactoring to include this kind of feature: splitting the file into classes, trying to integrate coverage-tests by using unit-tests (+ the existing functional tests).

[cpb-]

First of all I totally agree with the need for refactoring and splitting the cooker.py file, it starts to be a bit long to read and understand.

Concerning the hooks, I think that items-oriented sub-commands (fetch and build) will need to be handled at two-levels: some high-level hooks that are called before doing anything and after all the sub-command is done. And low-level hooks (pre/post-fetch and pre/post-build) that have to be specified for each item that need them.
For instance in sources section:

{ "url": "git://git.yoctoproject.org/poky", "branch": "kirkstone", "rev": "yocto-4.0", "pre-fetch": "remove-my-patch", "post-fetch": "apply-my-patch" },

Now, where will the hooks lie? Do we had a default hooks/ directory beside layers, builds and downloads and a hooks_dir command line parameter? Or do we accept relative/absolute paths to find the scripts?

How to specify the high-level hooks that are global for a sub-command.
Do we need a "hooks" section in the menu for high-level hooks ?

"hooks" : {
  "post-fetch" : "backup-downloads-dir",
  "post-build" : "upload-images"
},

"builds" : {
  "pi4" : {
    "target": "core-image-base",
    "local.conf": [
        "..."
   ], 
   "post-build": "upload-image-to-tftp-server"
  }
}

[pboettch]

Concerning the hooks, I think that items-oriented sub-commands (fetch and build) will need to be handled at two-levels: some high-level hooks that are called before doing anything and after all the sub-command is done.

Fetch? The cooker-command for that is update.

And low-level hooks (pre/post-fetch and pre/post-build) that have to be specified for each item that need them. For instance in sources section:

{ "url": "git://git.yoctoproject.org/poky", "branch": "kirkstone", "rev": "yocto-4.0", "pre-fetch": "remove-my-patch", "post-fetch": "apply-my-patch" },

In this example the very first time pre-fetch (pre-update) will fail, how is this information transferred to the post-update-hook?

Now, where will the hooks lie? Do we had a default hooks/ directory beside layers, builds and downloads and a hooks_dir command line parameter? Or do we accept relative/absolute paths to find the scripts?

Hooks cannot be fetched, they have to live alongside the menu-file, reachable without any other command issued by cooker. But! If we add an init-hook, only the init-hook-script has to be next to the menu, this script could get the other hook from somewhere.

Relative paths are OK for me.

How to specify the high-level hooks that are global for a sub-command. Do we need a "hooks" section in the menu for high-level hooks ?

"hooks" : {
  "post-fetch" : "backup-downloads-dir",
  "post-build" : "upload-images"
},

"builds" : {
  "pi4" : {
    "target": "core-image-base",
    "local.conf": [
        "..."
   ], 
   "post-build": "upload-image-to-tftp-server"
  }
}

This would make post-build a build in the cooker-sens impossible, or at least I don't like it. For global hooks a dedicated section is the way.

We also need to specify which arguments are passed to each hook.

[pboettch]

If we inspire us by git, its hooks are just files with a filename. If we chose this, then no changes are needed for the menu at all, but during cooker init we would need to specify a hook-dir. Inside we would search for special filenames and issue the hook if the name matches.

The hook pre-update-source would be called for each source and the user would need to check whether the source is the one he wants to patch based on the arguments given by cooker.

Here's the complete list:

pre-init
post-init
pre-update
foreach source
  pre-update-source
  post-update-source
post-updatee
foreach build
  pre-generate build
  post-generate build
foreach build
  pre-build build
  post-build build

Do we need a global pre/post-build and pre/post/generate?

[cpb-]

Fetch? The cooker-command for that is update.

Yes, sorry, it was late...

In this example the very first time pre-fetch (pre-update) will fail, how is this information transferred to the post-update-hook?

The hook scripts have to be robust, and fail only in critical situations. If a hook fails, Cooker stops its action, that's all.

Hooks cannot be fetched, they have to live alongside the menu-file, reachable without any other command issued by cooker. But! If we add an init-hook, only the init-hook-script has to be next to the menu, this script could get the other hook from somewhere.

Seems a bit complicated (but hooks are not easy beasts).

"builds" : {
"pi4" : {
"target": "core-image-base",
"local.conf": [
"..."
],
"post-build": "upload-image-to-tftp-server"
}
}

This would make post-build a build in the cooker-sens impossible, or at least I don't like it. For global hooks a dedicated section is the way.

The post-build field above is only for pi4 build, not global.

[cpb-]

If we inspire us by git, its hooks are just files with a filename. If we chose this, then no changes are needed for the menu at all, but during cooker init we would need to specify a hook-dir. Inside we would search for special filenames and issue the hook if the name matches.

Using pre-defined names for the hooks sounds really interesting, it would be very light for the menu.

[cpb-]

I like the idea that depending on local configuration (the hook-dir option during cooker init) the behaviour of a menu could be different.

This could be a way to handle an old issue (site specific setting) without adding any configuration file (cf. #95)