Problems With Sniffing

[blachniet]

I'm running Elasticsearch v1.4.4 in a Docker container. I kept having trouble getting the client to work properly. I was trying to run the sample in the README (obviously pointing to my Docker container instead of localhost). It was taking ~30 seconds to create the client, and then would fail to create the index with the error: no Elasticsearch node available.

As soon as I set turned off sniffing when creating the client (elastic.SetSniff(false)), everything worked perfectly. It doesn't really bother me that I have to turn sniffing off, but I wanted to put this issue out to see if anyone else had seen an issue like this.

P.S. @olivere - The documentation is awesome! 👍

[olivere]

@blachniet Thank's for using Elastic.

Yes, Docker seems to be a problem because (I assume) running ES inside a container is returning IP addresses that are not available from outside. There was a discussion about this in #42 IIRC.

Can you post the results of curl -XGET <esserver>/_nodes/http?pretty=1? The http_address is what Elastic (and the other official clients) use to find a new host. If that address is not available from outside Docker, we have a problem...

[blachniet]

Ah, that makes sense. It makes me feel better knowing I wasn't the only one seeing this. I'm perfectly fine leaving the sniffing disabled.

curl -XGET http://192.168.59.103:9200/_nodes/http?pretty
{
  "cluster_name" : "elasticsearch",
  "nodes" : {
    "PGNiMR5YSPaxhOzDDRn_DQ" : {
      "name" : "Aunt May Parker",
      "transport_address" : "inet[/172.17.0.2:9300]",
      "host" : "b9d7c0daafe2",
      "ip" : "172.17.0.2",
      "version" : "1.4.4",
      "build" : "c88f77f",
      "http_address" : "inet[/172.17.0.2:9200]",
      "http" : {
        "bound_address" : "inet[/0:0:0:0:0:0:0:0:9200]",
        "publish_address" : "inet[/172.17.0.2:9200]",
        "max_content_length_in_bytes" : 104857600
      }
    }
  }
}

[olivere]

Unfortunately, this is what happens here: Elastic will query 192.168.59.103 and registers the IP address returned there (inet[/172.17.0.2:9200]). Then it tries to query 172.17.0.2, which probably won't work.

I haven't found a solution to this other than (somehow) getting the ES in Docker to return the external IP. I guess other clients have the same problem.

If you have an idea how to solve this (or a document describing the way to configure Docker), let me know :-)

[olivere]

Okay. Let me know if I can do anything here. Closing for now.

[emilgpa]

Hi,
I have the same problem here:

panic: main: conn db: no Elasticsearch node available

goroutine 1 [running]:
log.Panicf(0x84de50, 0x11, 0xc2080c7e90, 0x1, 0x1)
    /usr/local/go/src/log/log.go:314 +0xd0
main.init·1()
    /Users/emilio/go/src/monoculum/init.go:40 +0x348
main.init()
    /Users/emilio/go/src/monoculum/main.go:334 +0xa4

goroutine 526 [select]:
net/http.(*persistConn).roundTrip(0xc2088ad1e0, 0xc2086a9d50, 0x0, 0x0, 0x0)
20:30:13 app         |  /usr/local/go/src/net/http/transport.go:1082 +0x7ad
net/http.(*Transport).RoundTrip(0xc20806c000, 0xc2086f6000, 0xc20873ff50, 0x0, 0x0)
20:30:13 app         |  /usr/local/go/src/net/http/transport.go:235 +0x558
20:30:13 app         | net/http.send(0xc2086f6000, 0xed4f18, 0xc20806c000, 0x21, 0x0, 
20:30:13 app         | 0x0)
    /usr/local/go/src/net/http/client.go:219
20:30:13 app         |  +0x4fc
net/http.(*Client).send(0xc08b00, 0xc2086f6000, 0x21
20:30:13 app         | , 0x0, 0x0)
    /usr/local/go/src/net/http/client.go:142 +0x15b
20:30:13 app         | net/http.(*Client).doFollowingRedirects(0xc08b00, 0xc2086f6000, 0x97cd00, 0x0, 0x0, 0x0)
20:30:13 app         |  /usr/local/go/src/net/http/client.go:367 +0xb25
net/http.(*Client).Do(0xc08b00, 0xc2086f6000, 0xc20873fce0, 0x0, 
20:30:13 app         | 0x0)
    /usr/local/go/src/net/http/client.go
20:30:13 app         | :174 +0xa4
github.com/olivere/elastic.(*Client).sniffNode(0xc208659d10, 0xc208569920, 0x15
20:30:13 app         | , 0x0, 0x0, 0x0)
20:30:13 app         |  /Users/emilio/go/src/github.com/olivere/elastic/client.go:543
20:30:13 app         |  +0x16a
20:30:13 app         | 
github.com/olivere/elastic.func·014(0xc208569920, 0x15
20:30:13 app         | )
    /Users/emilio/go/src/github.com/olivere/elastic/client.go:508 +0x47
20:30:13 app         | created by github.com/olivere/elastic.(*Client).sniff
    /Users/emilio/go/src/github.com/olivere/elastic/client.go:508 +0x744

goroutine 525 [chan receive]:
20:30:13 app         | database/sql.(*DB).connectionOpener(0xc2086de960)
    /usr/local/go/src/database/sql/sql.go:589 +0x4c
created by database/sql.Open
    /usr/local/go/src/database/sql/sql.go:452 +0x31c

goroutine 529 [IO wait]:
20:30:13 app         | net.(*pollDesc).Wait(0xc2084fe370, 0x72, 0x0
20:30:13 app         | , 
20:30:13 app         | 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc2084fe370, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc2084fe310, 0xc208709000, 0x1000, 0x1000, 0x0, 0xed4d48, 0xc2086a9ec8)
    /usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc20896a800, 0xc208709000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:121 +0xdc
net/http.noteEOFReader.Read(0xef0410, 0xc20896a800, 0xc2088ad238, 0xc208709000, 0x1000, 0x1000, 0xeb7010, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1270 +0x6e
net/http.(*noteEOFReader).Read(0xc208569b40, 0xc208709000, 0x1000, 0x1000, 0xc207f6957f, 0x0, 0x0)
    <autogenerated>:125 +0xd4
bufio.(*Reader).fill(0xc2088f3c80)
    /usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).Peek(0xc2088f3c80, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:132 +0xf0
net/http.(*persistConn).readLoop(0xc2088ad1e0)
    /usr/local/go/src/net/http/transport.go:842 +0xa4
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:660 +0xc9f

goroutine 530 [select]:
net/http.(*persistConn).writeLoop(0xc2088ad1e0)
    /usr/local/go/src/net/http/transport.go:945 +0x41d
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:661 +0xcbc

This occurs sometimes... not always...

curl -XGET 127.0.0.1:9200/_nodes/http?pretty=1
{
  "cluster_name" : "elasticsearch",
  "nodes" : {
    "3l_Ing0oSfWu5U63US5kxg" : {
      "name" : "Rattler",
      "transport_address" : "inet[192.168.1.91/192.168.1.91:9300]",
      "host" : "Mac-Emilio",
      "ip" : "192.168.1.91",
      "version" : "1.3.4",
      "build" : "a70f3cc",
      "http_address" : "inet[/192.168.1.91:9200]",
      "http" : {
        "bound_address" : "inet[/0:0:0:0:0:0:0:0:9200]",
        "publish_address" : "inet[/192.168.1.91:9200]",
        "max_content_length_in_bytes" : 104857600
      }
    }
  }
}

[olivere]

@DasHaus and @blachniet, can you guys please share some details about your configuration? I will try to reproduce the issue. You are both using Docker, are you? How many ES instances do you use? What's your typical load? When and how often do you get errors?

@DasHaus The stack trace seems odd. According to the log the error occurs in line 543 of client.go, which just does a HTTP request:

res, err := c.c.Do((*http.Request)(req))

I cannot see what could go wrong here. It doesn't look like you're using a custom transport, do you?

When I know your configuration, I can at least try to set up a similar environment and run the cluster test. Maybe I will get similar errors.

Thanks in advance.

[emilgpa]

I don't use Docker, I have installed it manually. I use only one instance. The load is very low (just me with the application that i'am creating). The error occurs when the server restarts by fresh (sometimes the error is raised, other not). I create the connection like this (I don't use a custom transport):

u := elastic.SetURL("127.0.0.1:9200")
m, err := elastic.NewClient(u)

Thanks!

[olivere]

Is it just a typo, or are you using the URL without the scheme?

When I use it like that, I'll get the same error as you do. When I change it to ...

u := elastic.SetURL("http://127.0.0.1:9200")

... everything is okay.

[emilgpa]

Yes, is a typo, really I have http://127.0.0.1:9200.

It is strange because not always occurs the error...

[blachniet]

I've tried 2 different Docker configurations. Below is the common details between the 2.

• Elasitcsearch 1.5.0 Container
• 1 Elasticsearch node
• Very light load: 1 document (less than 300 bytes) pushed every 10 seconds
• Rare, light querying

I ran the Elasticsearch container on Boot2Docker, on the same device as the application interacting with it. When that failed due to the problem described in the original post, I tried running the Elasticsearch container on a separate Ubuntu server with Docker installed. I ran into the same issue with this approach.

I'm fairly new to Docker, so I'm not sure if there is some way I can configure my Docker containers or daemon to address this problem. In the meantime, I have disabled sniffing.

[blachniet]

According this post from 2012, it sounds like there should be a network.host option that can be set in the config that may solve this problem. I'll try it out and let you know how it goes.

[olivere]

Thank you, guys. I think we have two separate issues here: 1) Docker and 2) Problems while connecting. I don't think they are related. So I started #58 for the connection issue, and keep the Docker stuff here.