iptables blocking fly port results in no error message and app crash #5570

hejkerooo · 2020-05-08T12:37:34Z

hejkerooo
May 8, 2020

Summary

iptables blocking fly port results in no error message and app crash

After redirecting to this URL

Steps to reproduce

Starting from docker-compose up, what steps can be followed to reproduce
the problem?

Please use https://gist.github.com for large snippets!

Run docker-compose up -d on remote machine (i.e. some VPS)
Block all ports via IPTables on the remote machine except 8080 (default concourse port)

iptables -P INPUT DROP

# Accept incomming TCP connections from eth0 on port 80 and 443
iptables -A INPUT -i eth0 -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -i eth0 -p tcp --dport 443 -j ACCEPT

run fly --target example login --concourse-url=<remote_url> on local machine (i.e. your computer)
Go to the URL which CLI suggests to go (i.e. https://example.com/login?fly_port=34685)
Observe Fly CLI

Expected results

Any error message would be displayed, the app should not crash

Actual results

The web page is showing login successful!
FLY CLI hangs out, kills concourse process

Docker logs: https://pastebin.com/nSfRr6Kt

navigate to the following URL in your browser:

  https://example.com/login?fly_port=34685
or enter token manually: 

// 5-10 seconds of lag
error: Unexpected Response
Status: 504 Gateway Time-out
Body:
<html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.16.1 (Ubuntu)</center>
</body>
</html>

Process restarts after few seconds

Additional context

The most unexpected thing there is a process being killed but this kind of request looks like an unhandled exception.

~/.flyrc is not being created

Triaging info

Concourse version: 6.0
Browser (if applicable): Chrome
Did this used to work? I don't know

jamieklassen · 2020-05-08T12:51:06Z

jamieklassen
May 8, 2020
Collaborator

What does 'remote machine' mean here? the docker container running the concourse web process? or the machine running the fly commands?
What does 'kills concourse process' mean? the concourse web process is killed? If that's the case, I'm curious as to what you see in the logs - does the concourse web process output a stack trace?
Can you be more specific about 'Block all ports via IPTables'? Can you give specific iptables CLI invocations that could bring the 'remote machine' to this state?

0 replies

hejkerooo · 2020-05-08T13:03:58Z

hejkerooo
May 8, 2020
Author

@pivotal-jamie-klassen I've updated issue description with that information.
I couldn't find in logs any stack trace.

0 replies

jamieklassen · 2020-05-08T13:21:17Z

jamieklassen
May 8, 2020
Collaborator

there are only 6 lines of output in the linked logs - is that the whole log?

0 replies

hejkerooo · 2020-05-08T13:23:23Z

hejkerooo
May 8, 2020
Author

It's repeated few times (as I was trying few times), but there is no stack trace or anything

0 replies

jamieklassen · 2020-05-08T13:26:58Z

jamieklassen
May 8, 2020
Collaborator

The only thing that seems to be happening in those logs is the worker is attempting to get some data from the atc - for example, the container sweeper is making a 'sweep-containers' request to the TSA, and the TSA is attempting to query the ListDestroyingContainers endpoint on the ATC, but the ATC is returning a 500 status. What I'm surprised by here is that there do not seem to be any logs from the ATC itself. Why is it giving a 500 on that request?

Before getting much more into it, what does your docker-compose file look like? Based on the concourse_1 prefix on the logs and intermingled 'tsa' and 'worker' logs on the same container it sounds like you are running everything in a single container with the quickstart command, similar to https://github.com/concourse/docs/blob/master/docker-compose.yml?

More specifically, where is your database?

0 replies

hejkerooo · 2020-05-08T13:35:50Z

hejkerooo
May 8, 2020
Author

I'm running exactly this same docker-compose.yml with one modification - I've changed my CONCOURSE_EXTERNAL_URL to my server domain (like https://example.com)

more logs: https://pastebin.com/NfrDqk00
https://pastebin.com/f0Q1E8X2

Sorry for not enough details

My database is located on this same machine (remote machine) along with concourse

0 replies

vito · 2020-05-08T13:45:44Z

vito
May 8, 2020
Maintainer

I'm not seeing a crash in the logs. 🤔 A crash to means concourse exits and dumps a stack trace to stderr. These logs seem to show that it just can't connect to the database.

Just a guess, but if it seems like it's crashing because of the exit status 1 logs, that's actually just a protocol level thing between the workers and web nodes. Kind of confusing that it's worded that way though.

(Gonna convert this to a discussion.)

1 reply

hejkerooo May 8, 2020
Author

I couldn't access concourse for a few seconds so I assumed it has crashed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Concourse

iptables blocking fly port results in no error message and app crash #5570

Uh oh!

{{title}}

Uh oh!

Replies: 7 comments 1 reply

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Concourse

iptables blocking fly port results in no error message and app crash #5570

Uh oh!

hejkerooo May 8, 2020

Summary

Steps to reproduce

Expected results

Actual results

Additional context

Triaging info

Replies: 7 comments · 1 reply

Uh oh!

jamieklassen May 8, 2020 Collaborator

Uh oh!

hejkerooo May 8, 2020 Author

Uh oh!

jamieklassen May 8, 2020 Collaborator

Uh oh!

hejkerooo May 8, 2020 Author

Uh oh!

jamieklassen May 8, 2020 Collaborator

Uh oh!

hejkerooo May 8, 2020 Author

Uh oh!

vito May 8, 2020 Maintainer

Uh oh!

hejkerooo May 8, 2020 Author

hejkerooo
May 8, 2020

Replies: 7 comments 1 reply

jamieklassen
May 8, 2020
Collaborator

hejkerooo
May 8, 2020
Author

jamieklassen
May 8, 2020
Collaborator

hejkerooo
May 8, 2020
Author

jamieklassen
May 8, 2020
Collaborator

hejkerooo
May 8, 2020
Author

vito
May 8, 2020
Maintainer

hejkerooo May 8, 2020
Author