fix #2209 Several improvements in some edgecase querydsl scenarios (#2235)

* This commit brings about several improvements in some edgecase querydsl
scenarios

We always flattened situations like:

query1 && query2 && query3 into a single bool query with 3 must clauses.

But if the queries are booleans themselves that can't be merged we would
generate deeply nested pairs of two.

For instance we have 3 bool queries with only should clauses

bq1 && bq2 && bq3

ideally we'd generate:

bool:
  must:
     bool: should:,
     bool: should:,
     bool: should:

However due to the bitwise binary nature the way they were combined was:

Step1:

bq1 && b2 - both sides only have should clauses merging them would be
wrong so we wrap to `result1`

bool:
  must:
     bool: should:,
     bool: should:

now we do `result1 && bq2` since result1 is a must we can not merge the
should from `bq2` on it and thus we need to wrap to `result2`

bool:
  should:
    bool:
      must:
        bool: should:,
	bool: should:
    bool: should: //bq2

This is still sementically correct but now immagine you are doing this
in a loop combining thousands of these. Admittingly an edgecase it would
cause a nasty stackoverflow and query graphs that are way too deep
unnecessary as is showcased by #2209

This is now fixed in a way that does not need constant traversal of to
see if a bool must or should clause consists of only bools.

Similary the same happened if you'd || or a bunch of booleanqueries only
containing musts.

A third fix is that we no longer modify the original query when stealing
clauses. This could cause nasty errors when reusing a static bool query
e.g

```csharp
var q = Query<Project>.Bool(b => b.Should(ATermQuery).Must(ATermQuery));
var c = new QueryContainer();
for(int i = 0;i<1000;i++) c &= q;
```

The & or | bitwise operators were not pure and could potentially steal clauses from
`q` and adding them to the new result. Cuasing problems when we want to
apply q again and again and again.

* attempt to minimize allocations in the bool dsl by reusing lists and already instantiated bool queries

* documentation commit, hacked around asciidotnet not supporting tables just yet :) russcam/asciidotnet#2

* updated bool dsl performance section in the documentation

* inverted if statement for readabillity and removed some of the tests in the documentation output

Author: Mpdreamz

docs/client-concepts/low-level/connecting.asciidoc

@@ -91,9 +91,7 @@ var client = new ElasticLowLevelClient(config);
 var result = client.Search<SearchResponse<object>>(new { size = 12 });
 ----
 <1> Disable automatic proxy detection. When called, defaults to `true`.
-
 <2> Enable compressed request and responses from Elasticsearch (Note that nodes need to be configured to allow this. See the {ref_current}/modules-http.html[http module settings] for more info).
-
 <3> By default responses are deserialized directly from the response stream to the object you tell it to. For debugging purposes, it can be very useful to keep a copy of the raw response on the result object, which is what calling this method will do.
 
 `.ResponseBodyInBytes` will only have a value if the client configuration has `DisableDirectStreaming` set 
@@ -131,13 +129,9 @@ config = config
     .BasicAuthentication("username", "password");
 ----
 <1> Allows you to set querystring parameters that have to be added to every request. For instance, if you use a hosted elasticserch provider, and you need need to pass an `apiKey` parameter onto every request.
-
 <2> Sets proxy information on the connection.
-
 <3> [[request-timeout]] Sets the global maximum time a connection may take. Please note that this is the request timeout, the builtin .NET `WebRequest` has no way to set connection timeouts (see http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.timeout(v=vs.110).aspx[the MSDN documentation on `HttpWebRequest.Timeout` Property]).
-
 <4> As an alternative to the C/go like error checking on `response.IsValid`, you can instead tell the client to <<thrown-exceptions, throw exceptions>>.
-
 <5> forces all serialization to be indented and appends `pretty=true` to all the requests so that the responses are indented as well
 
 NOTE: Basic authentication credentials can alternatively be specified on the node URI directly:
@@ -365,9 +359,7 @@ public class MyJsonNetSerializer : JsonNetSerializer
 }
 ----
 <1> Call this constructor if you only need access to `JsonSerializerSettings` without local state
-
 <2> Call OverwriteDefaultSerializers if you need access to `JsonSerializerSettings` with local state
-
 <3> You can inject contract resolved converters by implementing the ContractConverters property. This can be much faster then registering them on `JsonSerializerSettings.Converters`
 
 You can then register a factory on `ConnectionSettings` to create an instance of your subclass instead.

docs/common-options/date-math/date-math-expressions.asciidoc

@@ -16,35 +16,35 @@ please modify the original csharp file found at the link and submit the PR with
 == Date Math Expressions
 
 The date type supports using date math expression when using it in a query/filter
-Whenever durations need to be specified, eg for a timeout parameter, the duration can be specified 
+Whenever durations need to be specified, eg for a timeout parameter, the duration can be specified
 
-The expression starts with an "anchor" date, which can be either now or a date string (in the applicable format) ending with `||`. 
-It can then follow by a math expression, supporting `+`, `-` and `/` (rounding). 
-The units supported are 
+The expression starts with an "anchor" date, which can be either now or a date string (in the applicable format) ending with `||`.
+It can then follow by a math expression, supporting `+`, `-` and `/` (rounding).
+The units supported are
 
 * `y` (year)
 
 * `M` (month)
 
-* `w` (week) 
+* `w` (week)
 
-* `d` (day) 
+* `d` (day)
 
-* `h` (hour) 
+* `h` (hour)
 
 * `m` (minute)
 
 * `s` (second)
 
-as a whole number representing time in milliseconds, or as a time value like `2d` for 2 days. 
+as a whole number representing time in milliseconds, or as a time value like `2d` for 2 days.
 
 :datemath: {ref_current}/common-options.html#date-math
 
 Be sure to read the Elasticsearch documentation on {datemath}[Date Math].
 
 === Simple Expressions
 
-You can create simple expressions using any of the static methods on `DateMath` 
+You can create simple expressions using any of the static methods on `DateMath`
 
 [source,csharp]
 ----
@@ -74,7 +74,7 @@ the resulting date math will assume the whole string is the anchor
 Expect(nonsense).WhenSerializing<Nest.DateMath>(nonsense)
 .Result(dateMath => ((IDateMath)dateMath)
         .Anchor.Match(
-            d => d.Should().NotBe(default(DateTime)), 
+            d => d.Should().NotBe(default(DateTime)),
             s => s.Should().Be(nonsense)
         )
     );
@@ -94,15 +94,15 @@ the anchor will be an actual `DateTime`, even after a serialization/deserializat
 Expect("2015-05-05T00:00:00").WhenSerializing<Nest.DateMath>(date)
 .Result(dateMath => ((IDateMath)dateMath)
     .    Anchor.Match(
-            d => d.Should().Be(date), 
+            d => d.Should().Be(date),
             s => s.Should().BeNull()
         )
     );
 ----
 
 === Complex Expressions
 
-Ranges can be chained on to simple expressions 
+Ranges can be chained on to simple expressions
 
 [source,csharp]
 ----
@@ -129,7 +129,7 @@ Expect("now+1d-1m/d").WhenSerializing(
 ----
 
 When anchoring dates, a `||` needs to be appended as clear separator between the anchor and ranges.
-Again, multiple ranges can be chained 
+Again, multiple ranges can be chained
 
 [source,csharp]
 ----
@@ -139,3 +139,33 @@ Expect("2015-05-05T00:00:00||+1d-1m").WhenSerializing(
         .Subtract(TimeSpan.FromMinutes(1)));
 ----
 
+=== Fractional times
+
+DateMath expressions do not support fractional numbers so unlike `Time` DateMath will
+pick the biggest integer unit it can represent
+
+[source,csharp]
+----
+Expect("now+25h").WhenSerializing(
+    Nest.DateMath.Now.Add(TimeSpan.FromHours(25)));
+----
+
+where as `Time` on its own serializes like this 
+
+[source,csharp]
+----
+Expect("1.04d").WhenSerializing(new Time(TimeSpan.FromHours(25)));
+
+Expect("now+90001s").WhenSerializing(
+    Nest.DateMath.Now.Add(TimeSpan.FromHours(25).Add(TimeSpan.FromSeconds(1))));
+
+Expect("now+90000001ms").WhenSerializing(
+    Nest.DateMath.Now.Add(TimeSpan.FromHours(25).Add(TimeSpan.FromMilliseconds(1))));
+
+Expect("now+1y").WhenSerializing(
+    Nest.DateMath.Now.Add("1y"));
+
+Expect("now+52w").WhenSerializing(
+    Nest.DateMath.Now.Add(TimeSpan.FromDays(7 * 52)));
+----
+

docs/query-dsl/bool-dsl/bool-dsl.asciidoc

@@ -364,3 +364,60 @@ Assert(
     c => AssertDoesNotJoinOntoLockedBool(c, "leftBool"));
 ----
 
+=== Perfomance considerations
+
+If you have a requirement of combining many many queries using the bool dsl please take the following into account.
+
+You *can* use bitwise assignments in a loop to combine many queries into a bigger bool.
+
+In this example we are creating a single bool query with a 1000 must clauses using the `&=` assign operator.
+
+[source,csharp]
+----
+var c = new QueryContainer();
+
+var q = new TermQuery { Field = "x", Value = "x" };
+
+c &= q;
+----
+
+....
+|===
+|     Median|     StdDev|       Gen 0|  Gen 1|  Gen 2|  Bytes Allocated/Op
+|  1.8507 ms|  0.1878 ms|    1,793.00|  21.00|      -|        1.872.672,28
+|===
+....
+
+As you can see while still fast its causes a lot of allocations to happen because with each iteration
+we need to re evaluate the mergability of our bool query.
+
+Since we already know the shape of our bool query in advance its much much faster to do this instead:
+
+[source,csharp]
+----
+QueryContainer q = new TermQuery { Field = "x", Value = "x" };
+
+var x = Enumerable.Range(0, 1000).Select(f => q).ToArray();
+
+var boolQuery = new BoolQuery
+{
+    Must = x
+};
+----
+
+....
+|===
+|      Median|     StdDev|   Gen 0|  Gen 1|  Gen 2|  Bytes Allocated/Op
+|  31.4610 us|  0.9495 us|  439.00|      -|      -|            7.912,95
+|===
+....
+
+ The drop both in performance and allocations is tremendous!
+
+NOTE: If you assigning many bool queries prior to NEST 2.4.6 into a bigger bool using an assignment loop
+the client did not do a good job flattening the result in the most optimal way and could
+cause a stackoverflow when doing ~2000 iterations. This only applied to bitwise assigning many `boolean` queries.
+Other queries behave fine in earlier versions. Since NEST 2.4.6 you can combine as many bool queries
+as you'd like this way too.
+See https://github.com/elastic/elasticsearch-net/pull/2235[PR #2335 on github for more information]
+

docs/query-dsl/bool-dsl/operators/and-assign-many-manual-bools-usage.asciidoc

@@ -0,0 +1,160 @@
+:ref_current: https://www.elastic.co/guide/en/elasticsearch/reference/2.3
+
+:github: https://github.com/elastic/elasticsearch-net
+
+:nuget: https://www.nuget.org/packages
+
+////
+IMPORTANT NOTE
+==============
+This file has been generated from https://github.com/elastic/elasticsearch-net/tree/2.x/src/Tests/QueryDsl/BoolDsl/Operators/AndAssignManyManualBoolsUsageTests.cs. 
+If you wish to submit a PR for any spelling mistakes, typos or grammatical errors for this file,
+please modify the original csharp file found at the link and submit the PR with that change. Thanks!
+////
+
+[[and-assign-many-manual-bools-usage]]
+== And Assign Many Manual Bools Usage
+
+[source,csharp]
+----
+var q = Query<Project>.Bool(b => b.MustNot(ATermQuery));
+
+var container = AndAssignManyBoolQueries(q);
+
+container.Bool.MustNot.Should().NotBeEmpty().And.HaveCount(Iterations);
+
+container.Bool.MustNot.Cast<IQueryContainer>().Should().OnlyContain(s => s.Term != null);
+
+AssertBoolQuery(q, b => b.MustNot.Should().NotBeNullOrEmpty());
+----
+
+[source,csharp]
+----
+var q = Query<Project>.Bool(b => b.Must(ATermQuery));
+
+var container = AndAssignManyBoolQueries(q);
+
+DefaultMustAssert(container);
+
+container.Bool.Must.Cast<IQueryContainer>().Should().OnlyContain(s => s.Term != null);
+
+AssertBoolQuery(q, b => b.Must.Should().NotBeNullOrEmpty());
+----
+
+[source,csharp]
+----
+var q = Query<Project>.Bool(b => b.Should(ATermQuery).Must(ATermQuery));
+
+var container = AndAssignManyBoolQueries(q);
+
+DefaultMustAssert(container);
+
+container.Bool.Must.Cast<IQueryContainer>().Should()
+    .OnlyContain(s => s.Bool != null && s.Bool.Should != null && s.Bool.Must != null);
+
+AssertBoolQuery(q, b => {
+    b.Should.Should().NotBeNullOrEmpty();
+    b.Must.Should().NotBeNullOrEmpty();
+});
+
+b.Should.Should().NotBeNullOrEmpty();
+
+b.Must.Should().NotBeNullOrEmpty();
+----
+
+[source,csharp]
+----
+var q = Query<Project>.Bool(b => b.Should(ATermQuery).MustNot(ATermQuery));
+
+var container = AndAssignManyBoolQueries(q);
+
+DefaultMustAssert(container);
+
+container.Bool.Must.Cast<IQueryContainer>().Should()
+    .OnlyContain(s => s.Bool != null && s.Bool.Should != null && s.Bool.MustNot != null);
+
+AssertBoolQuery(q, b => {
+    b.Should.Should().NotBeNullOrEmpty();
+    b.MustNot.Should().NotBeNullOrEmpty();
+});
+
+b.Should.Should().NotBeNullOrEmpty();
+
+b.MustNot.Should().NotBeNullOrEmpty();
+----
+
+[source,csharp]
+----
+var q = Query<Project>.Bool(b => b.Must(ATermQuery).MustNot(ATermQuery));
+
+var container = AndAssignManyBoolQueries(q);
+
+DefaultMustAssert(container);
+
+container.Bool.MustNot.Should().NotBeEmpty().And.HaveCount(Iterations);
+
+container.Bool.Must.Cast<IQueryContainer>().Should().OnlyContain(s => s.Term != null);
+
+AssertBoolQuery(q, b => {
+    b.Must.Should().NotBeNullOrEmpty();
+    b.MustNot.Should().NotBeNullOrEmpty();
+});
+
+b.Must.Should().NotBeNullOrEmpty();
+
+b.MustNot.Should().NotBeNullOrEmpty();
+----
+
+[source,csharp]
+----
+var q = Query<Project>.Bool(b => b.Must(ATermQuery).Name("name"));
+
+var container = AndAssignManyBoolQueries(q);
+
+DefaultMustAssert(container);
+
+container.Bool.Must.Cast<IQueryContainer>().Should()
+    .OnlyContain(s => s.Bool != null && s.Bool.Must != null && s.Bool.Name == "name");
+
+AssertBoolQuery(q, b => {
+    b.Must.Should().NotBeNullOrEmpty();
+    b.Name.Should().NotBeNullOrEmpty();
+});
+
+b.Must.Should().NotBeNullOrEmpty();
+
+b.Name.Should().NotBeNullOrEmpty();
+----
+
+[source,csharp]
+----
+var container = new QueryContainer();
+
+Action act = () =>
+{
+    for (int i = 0; i < Iterations; i++) container &= q;
+};
+
+container &= q;
+
+act.ShouldNotThrow();
+----
+
+[source,csharp]
+----
+lotsOfAnds.Should().NotBeNull();
+
+lotsOfAnds.Bool.Should().NotBeNull();
+
+lotsOfAnds.Bool.Must.Should().NotBeEmpty().And.HaveCount(Iterations);
+----
+
+[source,csharp]
+----
+var q = Query<Project>.Bool(b => b.Should(ATermQuery));
+var container = AndAssignManyBoolQueries(q);
+DefaultMustAssert(container);
+container.Bool.Must.Cast<IQueryContainer>().Should().OnlyContain(s => s.Bool != null && s.Bool.Should != null);
+AssertBoolQuery(q, b => b.Should.Should().NotBeNullOrEmpty());
+----
+