SSM Port Forwarding fails on multiple simultaneous connections #1189
Description
Confirm by changing [ ] to [x] below to ensure that it's a bug:
- I've gone though the API reference
- I've checked AWS Forums and StackOverflow for answers
- I've searched for previous similar issues and didn't find any solution
Describe the bug
I am trying to establish tunnel from my laptop into EC2 instance that is running mongodb. It works fine, if I do it on command line with
aws ssm start-session --target i-... --document-name AWS-StartPortForwardingSession --parameters '{"portNumber":["27017"],"localPortNumber":["27017"]}'
on terminal 1 and
mongodump --port 27017 --db yletunnus --out ./dump --username root --password ... --authenticationDatabase admin --verbose --host localhost
on terminal 2.
If I try to handlee the first command with
// startSession creates tunnel via AWS Systems Manager (SSM)
func (client *SSM) startSession(instanceID, documentName string, params map[string][]string) (*ssm.StartSessionOutput, error) {
return client.sdk.StartSession(context.TODO(), &ssm.StartSessionInput{
Target: &instanceID,
DocumentName: &documentName,
Parameters: params,
})
}
func (client *SSM) StartPortForwardingSession(instanceID string, localPort, remotePort int) (*ssm.StartSessionOutput, error) {
params := map[string][]string{
"portNumber": {strconv.Itoa(remotePort)},
"localPortNumber": {strconv.Itoa(localPort)},
}
return client.startSession(instanceID, "AWS-StartPortForwardingSession", params)
}
...
portSession, err := ssm.StartPortForwardingSession(secondaryID, 27017, 27017)
It will create session that appears in SessionManager, but then it disappears and mongodump fails with
2021-03-19T12:17:34.886+0000 will listen for SIGTERM, SIGINT, and SIGKILL
2021-03-19T12:18:04.847+0000 Failed: can't create session: could not connect to server: server selection error: server selection timeout, current topology: { Type: Single, Servers: [{ Addr: host.docker.internal:27017, Type: Unknown, State: Connected, Average RTT: 0, Last error: connection() : dial tcp 192.168.65.2:27017: connect: connection refused }, ] }
and it does exit 1
Version of AWS SDK for Go?
github.com/aws/aws-sdk-go-v2 v1.3.0
github.com/aws/aws-sdk-go-v2/service/ssm v1.2.0
Version of Go (go version)?
1.15.7
To Reproduce (observed behavior)
Assuming you have EC2 running mongodb somewhere, use example above to open port forwarding and issue time.Sleep(120*time.Second) while issueing mongodump command (example also above) on another terminal.
Expected behavior
mongodump should be able to create multiple tcp connection via port forwarding and print out something like:
2021-03-19T14:20:25.103+0200 will listen for SIGTERM, SIGINT, and SIGKILL
2021-03-19T14:20:25.841+0200 enqueued collection 'foo.tempTokens'
2021-03-19T14:20:25.908+0200 enqueued collection 'foo.currentTermsOfService'
...
instead of
Failed: can't create session: could not connect to server
Additional context
This requires amazon-ssm-agent 3.0.222.0 or later on EC2 instance. (https://aws.amazon.com/about-aws/whats-new/2020/10/port-forwarding-sessions-created-sessions-manager-support-multiple-simultaneous-connections/). You also need sessionmanagerplugin (installs under /usr/local in MacOS) on your local machine.
Current work-around is to execute awscli command from golang code.