Is there any way to execute concurrent step or action ?

[BragdonD]

Hey, I have an usecase where I need to perform an HTTP request to a REST API, which will perform another request to another REST API. To check that this request is performed I use tcpdump in my terminal. It works well but I would like to make fully autonomous using runn.

I tried to use the concurrency field on include notebook in step. However, it did not work.

main runbook:

steps:
  init:
    include: ./init_log.yaml
  list_logs:
    http:
      /mypath
        get:
          headers:
            Authorization: "Bearer xxxx"
          body:
            application/json: null
    test: |
      current.res.status == 200
  export_log:
    include: 
      path: ./export_log_http.yaml
  network_test:
    include: ./export_log_network.yaml
  servers:
    include: ./export_log_server.yaml
  cleanup:
    include: ./cleanup.yaml
    force: true

network runbook:

debug: true
if: included
vars:
  netInt: ${NETWORK_INTERFACE:-ens33}
  host: ${PACKAGE_HOSTNAME:-localhost}
  port: ${PACKAGE_PORT:-9000}
concurrency: export-log-request
steps:
  export_log:
    exec:
      command: |
        tcpdump -i {{ vars.netInt }} -c 1 'host {{ vars.host }} and port {{ vars.port }}'
    test:
      current.stdout contains "1 packet captured"

http runbook:

debug: true
runners:
  http:
    endpoint: ${TEST_HTTP_ENDPOINT:-http://localhost:9004}
    trace: true 
if: included
concurrency: export-log-request
steps:
  export_log:
    http:
      /mypath
        post:
          headers:
            Authorization: "Bearer xxxx"
          body: null

netcat server to simulate the other service:

debug: true
if: included
vars:
  host: ${PACKAGE_HOSTNAME:-localhost} # There is probably a way to use this with netcat
  port: ${PACKAGE_PORT:-9000}
concurrency: export-log-request
steps:
  export_log:
    exec:
      command: |
        nc -l -p {{ vars.port }} -w 1

Is there any way to perform this operation like I want in the main ?

[BragdonD]

OK, after reading the doc again, I understand that the concurrency keyword means that only one run book should be executed at a time.

I still can't run different step at the same time. I guess there is just no way to do it ?

However, even running the different run books separately with the following command failed.

runn run ./export_log*.yaml --scopes run:exec --verbose --concurrent "on"

I'm stuck at the end of the run and the launch order is not deterministic. Sometimes the netcat server is launched before the HTTP run, so it succeeds, and sometimes it's the last run book to run, so it fails.

To solve this problem I use the background keyword set to true to launch the different shell command. But now I have 2 other problems:

• the test on the tcpdump capture always fail because stdout and stderr are nil.
• I need to launch the shell command before the HTTP step, and I can't use the answer from the previous HTTP step anymore. I tried to use steps[my_step_name].res.body[0].id and it is telling me that res is always nil.

[BragdonD]

the test on the tcpdump capture always fail because stdout and stderr are nil.

I found an alternative approach to check the command output. Instead of using the standard test with current.stderr and current.stdout, I redirect the output to a temporary file using Bash. This can sometimes lead to race conditions due to the multi-threaded nature of the process, but I added a sleep command in the exec step to give the capture process enough time to finish writing.

After that, I run a cat command on the temporary file, which is where I perform the actual test.

Side Note: It is not working with background: true on the command

[k1LoW]

Hi @BragdonD

I still can't run different step at the same time

The steps in the runbook are always executed sequentially.

ref: https://github.com/k1LoW/runn/blob/main/docs/designs/runbook.md

Maybe you can find a way to run them in parallel, but at least I haven't come up with one.

[BragdonD]

Hi @k1LoW,

I did see that, but thank you for the confirmation.

I would like to know if there is a way to use the exec command with background set to true and still use the output ?

When I am using the output it is null, which I suppose is normal in the context of the step, but even when redirecting the output of the command to a file using &> path/to/myfile, I get an empty file. I don't know if this is a normal behavior.

Maybe you can find a way to run them in parallel, but at least I haven't come up with one.

Are you open to suggestion/PR or not ?

[k1LoW]

When I am using the output it is null, which I suppose is normal in the context of the step, but even when redirecting the output of the command to a file using &> path/to/myfile, I get an empty file. I don't know if this is a normal behavior.

The redirect should be handled successfully.

$ cat bk.yml
steps:
  -
    exec:
      command: |
        sleep 1 && echo hoge > log.txt
      background: true
  -
    exec:
      command: |
        sleep 3
  -
    exec:
      command: |
        cat log.txt
    test: |
      current.stdout contains 'hoge'

$ runn run bk.yml --scopes run:exec
.

1 scenario, 0 skipped, 0 failures

[k1LoW]

@BragdonD

If possible, could I get a complete sample?

I would be even happier if it worked on macOS.

[BragdonD]

@k1LoW, do you have tcpdump on macOs ?

At the moment I don't have access to my work computer so I can't check the exact runbook, but it was something similar to the following:

steps:
  -
    exec: # tcpdump output only on stderr
      command: |
        tcpdump -c 1 -i lo 'host 127.0.0.1 and port 9000' 2> ./network.log
      background: true
  -
    exec: # I did have a real API on my work computer but this should be enough to make an HTTP request
      command: |
        nc -l 127.0.0.1 9000
      background: true
  -
    exec:
      command: |
        curl http://127.0.0.1:9000
  -
    exec: # Sleep to let the tcpdump command writes the data
      command: |
        sleep 3 & cat ./network.log
    test: |
      current.stdout contains '1 packet captured'

The stdout stream was always empty for the cat command.

[k1LoW]

The test has failed, but it appears to be working as intended.

~/src/github.com/k1LoW/runn (main)> runn -v
runn version 0.129.3
~/src/github.com/k1LoW/runn (main)> cat 1212.yml
steps:
  -
    exec: # tcpdump output only on stderr
      command: |
        tcpdump -c 1 -i any 'port 9000' 2> ./network.log
      background: true
  -
    exec: # I did have a real API on my work computer but this should be enough to make an HTTP request
      command: |
        nc -l 127.0.0.1 9000
      background: true
  -
    exec:
      command: |
        curl http://127.0.0.1:9000
  -
    exec: # Sleep to let the tcpdump command writes the data
      command: |
        sleep 3 & cat ./network.log
    test: |
      current.stdout contains '1 packet captured'
~/src/github.com/k1LoW/runn (main)> sudo runn run 1212.yml --scopes run:exec --debug
Run "exec" on "Generated by `runn new`".steps[0]
-----START COMMAND-----
tcpdump -c 1 -i any 'port 9000' 2> ./network.log
-----END COMMAND-----

Run "exec" on "Generated by `runn new`".steps[1]
-----START COMMAND-----
nc -l 127.0.0.1 9000
-----END COMMAND-----

Run "exec" on "Generated by `runn new`".steps[2]
-----START COMMAND-----
curl http://127.0.0.1:9000
-----END COMMAND-----
-----START STDOUT-----

-----END STDOUT-----
-----START STDERR-----
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (52) Empty reply from server

-----END STDERR-----

Run "exec" on "Generated by `runn new`".steps[3]
-----START COMMAND-----
sleep 3 & cat ./network.log
-----END COMMAND-----
-----START STDOUT-----
tcpdump: data link type PKTAP
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type PKTAP (Apple DLT_PKTAP), snapshot length 524288 bytes

-----END STDOUT-----
-----START STDERR-----

-----END STDERR-----
Run "test" on "Generated by `runn new`".steps[3]
F

1) 1212.yml 2e78bd5ab156060f7a3fa88350271cbdc5a5bbb7
  Failure/Error: test failed on "Generated by `runn new`".steps[3]: condition is not true

  Condition:
    current.stdout contains '1 packet captured'

    │
    ├── current.stdout => "tcpdump: data link type PKTAP
    │   tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
    │   listening on any, link-type PKTAP (Apple DLT_PKTAP), snapshot length 524288 bytes
    │   "
    └── "1 packet captured"

  Failure step (1212.yml):
  16   -
  17     exec: # Sleep to let the tcpdump command writes the data
  18       command: |
  19         sleep 3 & cat ./network.log
  20     test: |
  21       current.stdout contains '1 packet captured'


1 scenario, 0 skipped, 1 failure
~/src/github.com/k1LoW/runn (main)> cat network.log
tcpdump: data link type PKTAP
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type PKTAP (Apple DLT_PKTAP), snapshot length 524288 bytes
1 packet captured
28 packets received by filter
0 packets dropped by kernel
~/src/github.com/k1LoW/runn (main)>

The output of 1 packet captured appears to be output when tcpdump ends.

[BragdonD]

@k1LoW, yes, I can get the beginning of the command, but I can't get what should be displayed later when the packet is received. I need the end of the output for the test to make sens

[k1LoW]

@BragdonD
I fixed the command chaining ( & -> && ) and it worked.

~/src/github.com/k1LoW/runn (main)> cat 1212.yml
steps:
  -
    exec: # tcpdump output only on stderr
      command: |
        tcpdump -c 1 -i any 'port 9000' 2> ./network.log
      background: true
  -
    exec: # I did have a real API on my work computer but this should be enough to make an HTTP request
      command: |
        nc -l 127.0.0.1 9000
      background: true
  -
    exec:
      command: |
        curl http://127.0.0.1:9000
  -
    exec: # Sleep
      command: |
        sleep 3 && cat network.log
    test: |
      current.stdout contains '1 packet captured'


~/src/github.com/k1LoW/runn (main)> sudo runn run 1212.yml --scopes run:exec --debug
Password:
Run "exec" on "Generated by `runn new`".steps[0]
-----START COMMAND-----
tcpdump -c 1 -i any 'port 9000' 2> ./network.log
-----END COMMAND-----

Run "exec" on "Generated by `runn new`".steps[1]
-----START COMMAND-----
nc -l 127.0.0.1 9000
-----END COMMAND-----

Run "exec" on "Generated by `runn new`".steps[2]
-----START COMMAND-----
curl http://127.0.0.1:9000
-----END COMMAND-----
-----START STDOUT-----

-----END STDOUT-----
-----START STDERR-----
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (52) Empty reply from server

-----END STDERR-----

Run "exec" on "Generated by `runn new`".steps[3]
-----START COMMAND-----
sleep 3 && cat network.log
-----END COMMAND-----
-----START STDOUT-----
tcpdump: data link type PKTAP
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type PKTAP (Apple DLT_PKTAP), snapshot length 524288 bytes
1 packet captured
43 packets received by filter
0 packets dropped by kernel

-----END STDOUT-----
-----START STDERR-----

-----END STDERR-----
Run "test" on "Generated by `runn new`".steps[3]
.

1 scenario, 0 skipped, 0 failures
~/src/github.com/k1LoW/runn (main)>

[BragdonD]

@k1LoW, ok so I made a small mistake in my example and it differs from what I have at work. Indeed it work well when the tcpdump command is launch from the same notebook as the cat command.

However at work, the tcpdump command was launch from an included notebook. And it does not create the file when redirecting the output:

# Main notebook
debug: true
runners:
  http:
    endpoint: ${TEST_HTTP_ENDPOINT:-http://localhost:9004}
steps:
  init:
    include: ../utils/init_log.yaml
  nc_server:
    include: ./export_log_server.yaml
  network:
    include: ./export_log_network.yaml
  list_logs: # Get the log id
    http:
      /logs:
        get:
          headers:
            Authorization: "Bearer xxxx"
          body:
            application/json: null
  export_log: # Export the log
    include: 
      path: ./export_log_http.yaml
      vars:
        logId: '{{ previous.res.body[0].id }}'  
  network_test:
    exec:
      command: |
        sleep 3 && cat ./network.log
    test:
      current.stdout contains '1 packet captured'
  cleanup:
    include: ../utils/cleanup.yaml
    force: true

# netcat server
desc: |
  Launch a netcat server to make the http connection succeed.
debug: true
if: included
vars:
  host: ${PACKAGE_HOSTNAME:-localhost} # There is probably a way to use this with netcat
  port: ${PACKAGE_PORT:-9000}
steps:
  export_log:
    exec:
      command: |
        nc -l -p {{ vars.port }} -w 1
      background: true

desc: |
  Check if a network package is sent to a given application on a
  a specified network interface.
debug: true
if: included
steps:
  network_test:
    exec:
      command:
        tcpdump -c 1 -i any 'port 9000' 2> network.log
      background: true

Test output:

sudo runn run ./export_log_main.yaml --scopes run:exec,read:parent --verbose
=== Test the export of log files.  
 (./export_log_main.yaml)
Run "include" on "Test the export of log files".steps.servers
Run "exec" on "Launch a netcat server to make the http connection succeed.\n".steps.export_log
-----START COMMAND-----
nc -l -p 9000 -w 1
-----END COMMAND-----
    --- (servers) ... ok
        === Launch a netcat server to make the http connection succeed.
 (export_log_server.yaml)
            --- (export_log) ... ok

Run "include" on "Test the export of log files".steps.network
Run "exec" on "Check if a network package is sent to a given application on a\na specified network interface.\n".steps.network_test
-----START COMMAND-----
tcpdump -c 1 -i any 'port 9000' 2> network.log
-----END COMMAND-----
    --- (network) ... ok
        === Check if a network package is sent to a given application on a
a specified network interface.
 (export_log_network.yaml)
            --- (network_test) ... ok

Run "http" on "Test the export of log files".steps.list_logs
-----START HTTP REQUEST-----
GET /logs/op HTTP/1.1
Host: localhost:9004
Authorization: Bearer xxxx
Content-Type: application/json
X-Runn-Trace: {"id":"d821a1a91836aefc046dec380927d48fc503e07e?step=3"}


-----END HTTP REQUEST-----
-----START HTTP RESPONSE-----
HTTP/1.1 200 OK
Content-Length: 276
Content-Type: application/json
Date: Mon, 14 Apr 2025 06:52:16 GMT

[{"id":"4f86dd9683b5b12ada64aaaacccadeb2d797e006"}]
-----END HTTP RESPONSE-----
Run "test" on "Test the export of log files".steps.list_logs
    --- (list_logs) ... ok

Run "include" on "Test the export of log files".steps.export_log
Run "http" on "Send the http request for the export logs request.\n".steps.export_log
-----START HTTP REQUEST-----
POST /export/logs/op HTTP/1.1
Host: localhost:9004
Authorization: Bearer xxxx
Content-Type: application/json
X-Runn-Trace: {"id":"d821a1a91836aefc046dec380927d48fc503e07e?step=4\u0026step=0"}

{"logIds":["4f86dd9683b5b12ada64aaaacccadeb2d797e006"]}
-----END HTTP REQUEST-----
-----START HTTP RESPONSE-----
HTTP/1.1 500 Internal Server Error
Content-Length: 196
Content-Type: text/plain; charset=utf-8
Date: Mon, 14 Apr 2025 06:52:16 GMT
X-Content-Type-Options: nosniff

unable to export log files of type op: unable to perform http request: Head "http://localhost:9000/": dial tcp 127.0.0.1:9000: connect: connection refused

-----END HTTP RESPONSE-----
    --- (export_log) ... ok
        === Send the http request for the export logs request.
 (export_log_http.yaml)
            --- (export_log) ... ok

Run "exec" on "Test the export of log files".steps.network_test
-----START COMMAND-----
sleep 3 && cat ./network.log
-----END COMMAND-----
-----START STDOUT-----

-----END STDOUT-----
-----START STDERR-----
cat: ./network.log: No such file or directory

-----END STDERR-----
Run "test" on "Test the export of log files".steps.network_test
    --- (network_test) ... fail
        Failure/Error: test failed on "Test the export of log".steps.network_test: condition is not true
        
        Condition:
          current.stdout contains '1 packet captured'
          │
          ├── current.stdout => ""
          └── "1 packet captured"
          
        Failure step (./export_log_main.yaml):
        46   network_test:
        47     exec:
        48       command: |
        49         sleep 3 && cat ./network.log
        50     test:
        51       current.stdout contains '1 packet captured'


1) ./export_log_main.yaml d821a1a91836aefc046dec380927d48fc503e07e
  Failure/Error: test failed on "Test the export of log files"
  
  Condition:
    current.stdout contains '1 packet captured'
    │
    ├── current.stdout => ""
    └── "1 packet captured"
    
  Failure step (./export_log_main.yaml):
  46   network_test:
  47     exec:
  48       command: |
  49         sleep 3 && cat ./network.log
  50     test:
  51       current.stdout contains '1 packet captured'


1 scenario, 0 skipped, 1 failure

[k1LoW]

@BragdonD

After all, it is difficult to debug without actual failing samples.
No network.log means tcpdump's working directory may be off.

[BragdonD]

@k1LoW, both runbooks are in the same directory thought.

ll
total 28
drwxrwxr-x 2 user user 4096 avril 14 08:55 ./
drwxrwxr-x 4 user user 4096 avril 14 10:03 ../
-rw-rw-r-- 1 user user 1106 avril 11 15:23 export_log_http.yaml
-rw-rw-r-- 1 user user 532 avril 14 08:52 export_log_network.yaml
-rw-rw-r-- 1 user user 355 avril 11 15:24 export_log_server.yaml
-rw-rw-r-- 1 user user 1827 avril 14 08:55 export_log.yaml