Unable to install a custom pack in docker image for K8 setup !

[NitinNahal]

In one of my projects (moving the stackstorm setup from standalone to K8), I am trying to follow the documentation to create a docker image for the custom app notifier as mentioned in - https://github.com/StackStorm/st2packs-dockerfiles#building-the-st2packs-image

Here is my directory structure first -

nnahal$ ls -lrth
-rw-r--r-- 1 nnahal staff 324B Oct 30 06:24 Makefile
-rw-r--r-- 1 nnahal staff 2.0K Oct 30 06:24 README.md
drwxr-xr-x 4 nnahal staff 128B Oct 30 06:24 st2packs-builder
drwxr-xr-x 4 nnahal staff 128B Oct 30 06:24 st2packs-runtime
drwxr-xr-x 6 nnahal staff 192B Nov 1 12:13 st2packs-image
nnahal$ cd st2packs-image/
nnahal$ ls
Dockerfile README.md hooks packs
nnahal$ cd packs/
nnahal$ ls
notifier
nnahal$ ls -lrth notifier/
total 24
-rw-r--r-- 1 nnahal staff 10B Nov 1 14:18 README.md
drwxr-xr-x 4 nnahal staff 128B Nov 1 14:19 actions
drwxr-xr-x 3 nnahal staff 96B Nov 1 14:19 config
-rw-r--r-- 1 nnahal staff 150B Nov 1 14:19 config.schema.yaml
-rw-r--r-- 1 nnahal staff 212B Nov 1 14:19 pack.yaml
drwxr-xr-x 3 nnahal staff 96B Nov 1 14:19 rules

My pwd and Dockerfile are as follows -

nnahal$ pwd
/Users/nnahal/st2packs-dockerfiles

nnahal$ ls -lrth
total 16
-rw-r--r-- 1 nnahal staff 324B Oct 30 06:24 Makefile
-rw-r--r-- 1 nnahal staff 2.0K Oct 30 06:24 README.md
drwxr-xr-x 4 nnahal staff 128B Oct 30 06:24 st2packs-builder
drwxr-xr-x 4 nnahal staff 128B Oct 30 06:24 st2packs-runtime
drwxr-xr-x 6 nnahal staff 192B Nov 1 12:13 st2packs-image

nnahal$ cat st2packs-image/Dockerfile
ARG PACKS=“file:///tmp/stackstorm-st2”
FROM stackstorm/st2packs:builder AS builder
COPY packs/notifier /tmp/stackstorm-st2/
# Install custom packs
RUN /opt/stackstorm/st2/bin/st2-pack-install ${PACKS}
###########################
# Minimize the image size. Start with alpine:3.8,
# and add only packs and virtualenvs from builder.
FROM stackstorm/st2packs:runtime

Now when I try to build this image , I am able to see some steps but do not see my pack being taken into consideration nor do I see the other steps mentioned in Docker file (ls etc).

Below is the output of my build step -

st2packs-dockerfiles nnahal$ docker build -t nnahal/notifier:1.0 st2packs-image
[+] Building 2.0s (9/9) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 37B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/stackstorm/st2packs:runtime 1.7s
=> [auth] stackstorm/st2packs:pull token for registry-1.docker.io 0.0s
=> [stage-1 1/1] FROM docker.io/stackstorm/st2packs:runtime@sha256:33f4e77bdd2e942750d8631668ccc9973f12719dac96dc4081e99d30934da69c 0.0s
=> [builder 1/3] FROM docker.io/stackstorm/st2packs:builder 0.2s
=> => resolve docker.io/stackstorm/st2packs:builder 0.2s
=> CACHED [stage-1 2/1] COPY --from=builder /opt/stackstorm/packs /opt/stackstorm/packs 0.0s
=> CACHED [stage-1 3/1] COPY --from=builder /opt/stackstorm/virtualenvs /opt/stackstorm/virtualenvs 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:a73aab89e80875efa15debca79b573268d94c8249f814e373e5a6b6c86661b05 0.0s
=> => naming to docker.io/nnahal/notifier:1.0

Can someone please help me to understand if I am doing something wrong here ? When I see this image and check its contents, it does not have my pack installed, which is visible from above build logs.

[amanda11]

Could you try with the names matching e.g. in the Dockerfile use:

ARG PACKS=“file:///tmp/notifier”
FROM stackstorm/st2packs:builder AS builder
COPY packs/notifier /tmp/notifier/

[NitinNahal]

Thank you for replying.

I tried with this file as you suggested -

ARG PACKS=“file:///tmp/notifier”
FROM stackstorm/st2packs:builder AS builder
COPY packs/notifier /tmp/notifier/

# Install custom packs
RUN /opt/stackstorm/st2/bin/st2-pack-install ${PACKS}

###########################
# Minimize the image size. Start with alpine:3.8,
# and add only packs and virtualenvs from builder.
FROM stackstorm/st2packs:runtime

But when I build -

nnahal$ docker build -t nnahal/notifier:1.0 st2packs-image/
[+] Building 2.0s (9/9) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 401B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/stackstorm/st2packs:runtime 1.7s
=> [auth] stackstorm/st2packs:pull token for registry-1.docker.io 0.0s
=> [builder 1/3] FROM docker.io/stackstorm/st2packs:builder 0.2s
=> => resolve docker.io/stackstorm/st2packs:builder 0.2s
=> [stage-1 1/1] FROM docker.io/stackstorm/st2packs:runtime@sha256:33f4e77bdd2e942750d8631668ccc9973f12719dac96dc4081e99d3 0.0s
=> CACHED [stage-1 2/1] COPY --from=builder /opt/stackstorm/packs /opt/stackstorm/packs 0.0s
=> CACHED [stage-1 3/1] COPY --from=builder /opt/stackstorm/virtualenvs /opt/stackstorm/virtualenvs 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:a73aab89e80875efa15debca79b573268d94c8249f814e373e5a6b6c86661b05 0.0s
=> => naming to docker.io/nnahal/notifier:1.0

But still notifier pack is not installed in the image !

[NitinNahal]

Another strange thing is I added one additional line RUN ls -lrt packs , just to see if even the build process is able to reach this part and prints the content. But when I build the image this ls output is not there so I feel strange that is it even taking these changes into consideration.

Kindly help/suggest.

[NitinNahal]

Hello,

So I have an update on this,

With some help of colleagues we noticed that image build up somehow was not working on mac os and also there is a missing path to certificates which we follow internally. So the update is that with ubuntu os we are now able to build the image and I have pushed the image in docker.hub:

https://hub.docker.com/repository/docker/nnahal/st2packs

But still trying to install the image with changes in values.yaml as follows -

    # Custom packs images settings.
    #
    # For each given st2packs container you can define repository, name, tag and pullPolicy for this image below.
    # Multiple pack images can help when dealing with frequent updates by only rebuilding smaller images for desired packs
    # E.g. having all desired StackStorm-Exchange packs in one image and several custom packs in additional images
    images:
      - repository: nnahal
        name: st2packs
        tag: "1"
        pullPolicy: Always

I run the command -

nnahal$ helm install st2 stackstorm/stackstorm-ha -f stackstorm-k8s/values.yaml
Error: INSTALLATION FAILED: failed post-install: timed out waiting for the condition

I re tried the command with increasing timeout and debug mode -

nnahal$ sudo helm install st2 stackstorm/stackstorm-ha -f stackstorm-k8s/values.yaml --debug --timeout 900s
install.go:178: [debug] Original chart version: ""
install.go:195: [debug] CHART PATH: /Users/nnahal/Library/Caches/helm/repository/stackstorm-ha-0.100.0.tgz

client.go:128: [debug] creating 49 resource(s)
client.go:310: [debug] Starting delete for "st2-job-st2-apikey-load" Job
client.go:128: [debug] creating 1 resource(s)
client.go:540: [debug] Watching for changes to Job st2-job-st2-apikey-load with timeout of 15m0s
client.go:568: [debug] Add/Modify event for st2-job-st2-apikey-load: ADDED
client.go:607: [debug] st2-job-st2-apikey-load: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:568: [debug] Add/Modify event for st2-job-st2-apikey-load: MODIFIED
client.go:310: [debug] Starting delete for "st2-job-st2-key-load" Job
client.go:128: [debug] creating 1 resource(s)
client.go:540: [debug] Watching for changes to Job st2-job-st2-key-load with timeout of 15m0s
client.go:568: [debug] Add/Modify event for st2-job-st2-key-load: ADDED
client.go:607: [debug] st2-job-st2-key-load: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:568: [debug] Add/Modify event for st2-job-st2-key-load: MODIFIED
client.go:607: [debug] st2-job-st2-key-load: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:568: [debug] Add/Modify event for st2-job-st2-key-load: MODIFIED
client.go:310: [debug] Starting delete for "st2-job-st2-register-content" Job
client.go:128: [debug] creating 1 resource(s)
client.go:540: [debug] Watching for changes to Job st2-job-st2-register-content with timeout of 15m0s
client.go:568: [debug] Add/Modify event for st2-job-st2-register-content: ADDED
client.go:607: [debug] st2-job-st2-register-content: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:568: [debug] Add/Modify event for st2-job-st2-register-content: MODIFIED
client.go:607: [debug] st2-job-st2-register-content: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:568: [debug] Add/Modify event for st2-job-st2-register-content: MODIFIED
Error: INSTALLATION FAILED: failed post-install: job failed: BackoffLimitExceeded
helm.go:84: [debug] failed post-install: job failed: BackoffLimitExceeded
INSTALLATION FAILED
main.newInstallCmd.func2
	helm.sh/helm/v3/cmd/helm/install.go:127
github.com/spf13/cobra.(*Command).execute
	github.com/spf13/[email protected]/command.go:856
github.com/spf13/cobra.(*Command).ExecuteC
	github.com/spf13/[email protected]/command.go:974
github.com/spf13/cobra.(*Command).Execute
	github.com/spf13/[email protected]/command.go:902
main.main
	helm.sh/helm/v3/cmd/helm/helm.go:83
runtime.main
	runtime/proc.go:255
runtime.goexit
	runtime/asm_amd64.s:1581

helm version -

nnahal$ helm version
version.BuildInfo{Version:"v3.9.4", GitCommit:"dbc6d8e20fe1d58d50e6ed30f09a04a77e4c68db", GitTreeState:"clean", GoVersion:"go1.17.13"}

Any suggestions ? @amanda11 , Thank you in advance.

[NitinNahal]

Thanks @armab for your support,

So I finally got it working today !

Problem : I forgot that in my config.schema.yaml I had a parameter to be required which was not being passed into the config file in the values.yaml.
Solution : Please remember to put required params in the configs which is part of the values.yaml.

i.e.

  packs:
    # Custom StackStorm pack configs. Each record creates a file in '/opt/stackstorm/configs/'
    # https://docs.stackstorm.com/reference/pack_configs.html#configuration-file
    # NOTE: This takes precedence over the contents of st2.packs.volumes.configs (if defined) on helm upgrade.
    configs:
      core.yaml: |
        ---
        # example core pack config yaml
      notifier.yaml: |
        ---
        # example notifier pack config yaml
        certificate_path: "/usr/local/share/ca-certificates/"          # <----- was missing this line
    # Custom packs images settings.
    #
    # For each given st2packs container you can define repository, name, tag and pullPolicy for this image below.
    # Multiple pack images can help when dealing with frequent updates by only rebuilding smaller images for desired packs
    # E.g. having all desired StackStorm-Exchange packs in one image and several custom packs in additional images
    images:
      - repository: docker.io/nnahal
        name: st2packs
        tag: 1
        pullPolicy: Always

This was a small problem but took a lot of time.

Summarising our discussion : It is clear that the only way to proceed in such case is to observe which job is failing. If everything works well, pack installation last job ideally is to do st2-job-st2-register-content. In case you do not see it, chances are something is wrong internally.

• As soon as you start install of helm chart with -f values.yaml, open another terminal,
• run : kubectl get pods
• Observe which pod results in crash consistently and without coming to status RUNNING or COMPLETED
• Once identified it will be good to run kubectl logs -f -l job-name=<job-name> and see the last message.
• To get this job name, run : kubectl describe pods and look for failed job, under it you will find the Labels which is basically the container identification forking these jobs. In labels there is job-name, use it in last command to get errors.

In my case error was this variable not being found in the values.yaml which the pack expects.

[arm4b]

Nice, @NitinNahal! Glad things eventually worked out.

Also thanks for wrapping up example steps to debug the issue 👍

[NitinNahal]

Thanks @armab , I just want to mark this discussion answered for now but I do not see any option where I can do so. Is it possible for you ?

[arm4b]

@NitinNahal Looks like it's not possible to mark the reply in a sub-thread as an answer.

Could you perhaps copy-paste the same details you wrote here (#5794 (reply in thread)) and post it outside of the thread?

We could mark it as an answer then. That sounds like a perfect detailed answer that will help others in the future.

[NitinNahal]

Thanks @armab , I added it below 👍

[NitinNahal]

Thanks @armab for your support,

So I finally got it working today !

Problem : I forgot that in my config.schema.yaml I had a parameter to be required which was not being passed into the config file in the values.yaml.
Solution : Please remember to put required params in the configs which is part of the values.yaml.

i.e.

  packs:
    # Custom StackStorm pack configs. Each record creates a file in '/opt/stackstorm/configs/'
    # https://docs.stackstorm.com/reference/pack_configs.html#configuration-file
    # NOTE: This takes precedence over the contents of st2.packs.volumes.configs (if defined) on helm upgrade.
    configs:
      core.yaml: |
        ---
        # example core pack config yaml
      notifier.yaml: |
        ---
        # example notifier pack config yaml
        certificate_path: "/usr/local/share/ca-certificates/"          # <----- was missing this line
    # Custom packs images settings.
    #
    # For each given st2packs container you can define repository, name, tag and pullPolicy for this image below.
    # Multiple pack images can help when dealing with frequent updates by only rebuilding smaller images for desired packs
    # E.g. having all desired StackStorm-Exchange packs in one image and several custom packs in additional images
    images:
      - repository: docker.io/nnahal
        name: st2packs
        tag: 1
        pullPolicy: Always

This was a small problem but took a lot of time.

Summarising our discussion : It is clear that the only way to proceed in such case is to observe which job is failing. If everything works well, pack installation last job ideally is to do st2-job-st2-register-content. In case you do not see it, chances are something is wrong internally.

• As soon as you start install of helm chart with -f values.yaml, open another terminal,
• run : kubectl get pods
• Observe which pod results in crash consistently and without coming to status RUNNING or COMPLETED
• Once identified it will be good to run kubectl logs -f -l job-name=<job-name> and see the last message.
• To get this job name, run : kubectl describe pods and look for failed job, under it you will find the Labels which is basically the container identification forking these jobs. In labels there is job-name, use it in last command to get errors.

In my case error was this variable not being found in the values.yaml which the pack expects.