Merge pull request #518 from wprzytula/downgrading-policy

Added DowngradingConsistencyPolicy

Author: psarna

docs/source/SUMMARY.md

@@ -48,6 +48,7 @@
 - [Retry policy configuration](retry-policy/retry-policy.md)
     - [Fallthrough retry policy](retry-policy/fallthrough.md)
     - [Default retry policy](retry-policy/default.md)
+    - [Downgrading consistency policy](retry-policy/downgrading_consistency.md)
 
 - [Speculative execution](speculative-execution/speculative.md)
     - [Simple](speculative-execution/simple.md)

docs/source/retry-policy/downgrading_consistency.md

@@ -0,0 +1,107 @@
+# Downgrading consistency retry policy
+
+A retry policy that sometimes retries with a lower consistency level than the one initially
+requested.
+**BEWARE**: this policy may retry queries using a lower consistency level than the one
+initially requested. By doing so, it may break consistency guarantees. In other words, if you use
+this retry policy, there are cases (documented below) where a read at `Consistency::Quorum` **may
+not** see a preceding write at `Consistency::Quorum`. Do not use this policy unless you have
+understood the cases where this can happen and are ok with that. It is also highly recommended to
+always log the occurrences of such consistency breaks.
+This policy implements the same retries than the [DefaultRetryPolicy](default.md) policy. But on top
+of that, it also retries in the following cases:
+  - On a read timeout: if the number of replicas that responded is greater than one, but lower
+    than is required by the requested consistency level, the operation is retried at a lower
+    consistency level.
+  - On a write timeout: if the operation is a `WriteType::UnloggedBatch` and at least one
+    replica acknowledged the write, the operation is retried at a lower consistency level.
+    Furthermore, for other operations, if at least one replica acknowledged the write, the
+    timeout is ignored.
+  - On an unavailable exception: if at least one replica is alive, the operation is retried at
+    a lower consistency level.
+
+The lower consistency level to use for retries is determined by the following rules:
+  - if more than 3 replicas responded, use `Consistency::Three`.
+  - if 1, 2 or 3 replicas responded, use the corresponding level `Consistency::One`, `Consistency::Two` or
+      `Consistency::Three`.
+
+Note that if the initial consistency level was `Consistency::EachQuorum`, Scylla returns the number
+of live replicas _in the datacenter that failed to reach consistency_, not the overall
+number in the cluster. Therefore if this number is 0, we still retry at `Consistency::One`, on the
+assumption that a host may still be up in another datacenter.
+The reasoning being this retry policy is the following one. If, based on the information the
+Scylla coordinator node returns, retrying the operation with the initially requested
+consistency has a chance to succeed, do it. Otherwise, if based on this information we know
+**the initially requested consistency level cannot be achieved currently**, then:
+  - For writes, ignore the exception (thus silently failing the consistency requirement) if we
+    know the write has been persisted on at least one replica.
+  - For reads, try reading at a lower consistency level (thus silently failing the consistency
+    requirement).
+In other words, this policy implements the idea that if the requested consistency level cannot be
+achieved, the next best thing for writes is to make sure the data is persisted, and that reading
+something is better than reading nothing, even if there is a risk of reading stale data.
+
+This policy is based on the one in [DataStax Java Driver](https://docs.datastax.com/en/drivers/java/3.11/com/datastax/driver/core/policies/DowngradingConsistencyRetryPolicy.html).
+The behaviour is the same.
+
+### Examples
+To use in `Session`:
+```rust
+# extern crate scylla;
+# use scylla::Session;
+# use std::error::Error;
+# async fn check_only_compiles() -> Result<(), Box<dyn Error>> {
+use scylla::{Session, SessionBuilder};
+use scylla::transport::downgrading_consistency_retry_policy::DowngradingConsistencyRetryPolicy;
+
+let session: Session = SessionBuilder::new()
+    .known_node("127.0.0.1:9042")
+    .retry_policy(Box::new(DowngradingConsistencyRetryPolicy::new()))
+    .build()
+    .await?;
+# Ok(())
+# }
+```
+
+To use in a [simple query](../queries/simple.md):
+```rust
+# extern crate scylla;
+# use scylla::Session;
+# use std::error::Error;
+# async fn check_only_compiles(session: &Session) -> Result<(), Box<dyn Error>> {
+use scylla::query::Query;
+use scylla::transport::downgrading_consistency_retry_policy::DowngradingConsistencyRetryPolicy;
+
+// Create a Query manually and set the retry policy
+let mut my_query: Query = Query::new("INSERT INTO ks.tab (a) VALUES(?)");
+my_query.set_retry_policy(Box::new(DowngradingConsistencyRetryPolicy::new()));
+
+// Run the query using this retry policy
+let to_insert: i32 = 12345;
+session.query(my_query, (to_insert,)).await?;
+# Ok(())
+# }
+```
+
+To use in a [prepared query](../queries/prepared.md):
+```rust
+# extern crate scylla;
+# use scylla::Session;
+# use std::error::Error;
+# async fn check_only_compiles(session: &Session) -> Result<(), Box<dyn Error>> {
+use scylla::prepared_statement::PreparedStatement;
+use scylla::transport::downgrading_consistency_retry_policy::DowngradingConsistencyRetryPolicy;
+
+// Create PreparedStatement manually and set the retry policy
+let mut prepared: PreparedStatement = session
+    .prepare("INSERT INTO ks.tab (a) VALUES(?)")
+    .await?;
+
+prepared.set_retry_policy(Box::new(DowngradingConsistencyRetryPolicy::new()));
+
+// Run the query using this retry policy
+let to_insert: i32 = 12345;
+session.execute(&prepared, (to_insert,)).await?;
+# Ok(())
+# }
+```

docs/source/retry-policy/retry-policy.md

@@ -4,9 +4,11 @@ After a query fails the driver might decide to retry it based on its `Retry Poli
 Retry policy can be configured for `Session` or just for a single query.
 
 ### Retry policies
-By default there are two retry policies:
+By default there are three retry policies:
 * [Fallthrough Retry Policy](fallthrough.md) - never retries, returns all errors straight to the user
 * [Default Retry Policy](default.md) - used by default, might retry if there is a high chance of success
+* [Downgrading Consistency Retry Policy](downgrading_consistency.md) - behaves as [Default Retry Policy](default.md), but also,
+    in some more cases, it retries **with lower `Consistency`**.
 
 It's possible to implement a custom `Retry Policy` by implementing the traits `RetryPolicy` and `RetrySession`.
 
@@ -47,5 +49,6 @@ prepared.set_is_idempotent(true);
 
    fallthrough
    default
+   downgrading_consistency
 
 ```

examples/Cargo.toml

@@ -13,7 +13,7 @@ rustyline-derive = "0.6"
 scylla = {path = "../scylla", features = ["ssl"]}
 tokio = {version = "1.1.0", features = ["full"]}
 tracing = "0.1.25"
-tracing-subscriber = "0.2.16"
+tracing-subscriber = "0.3.14"
 chrono = "0.4"
 uuid = "1.0"
 tower = "0.4"

scylla-cql/src/errors.rs

@@ -54,8 +54,8 @@ pub enum DbError {
     #[error("The submitted query has a syntax error")]
     SyntaxError,
 
-    /// The query is syntatically correct but invalid
-    #[error("The query is syntatically correct but invalid")]
+    /// The query is syntactically correct but invalid
+    #[error("The query is syntactically correct but invalid")]
     Invalid,
 
     /// Attempted to create a keyspace or a table that was already existing
@@ -178,7 +178,7 @@ pub enum DbError {
     WriteFailure {
         /// Consistency level of the query
         consistency: LegacyConsistency,
-        /// Number of ndoes that responded to the read request
+        /// Number of nodes that responded to the read request
         received: i32,
         /// Number of nodes required to respond to satisfy required consistency level
         required: i32,
@@ -253,6 +253,10 @@ pub enum BadQuery {
     /// Passed invalid keyspace name to use
     #[error("Passed invalid keyspace name to use: {0}")]
     BadKeyspaceName(#[from] BadKeyspaceName),
+
+    /// Other reasons of bad query
+    #[error("{0}")]
+    Other(String),
 }
 
 /// Error that occurred during session creation
@@ -447,7 +451,7 @@ mod tests {
     // - displays error description
     // - displays error parameters
     // - displays error message
-    // - indented multiline strings dont cause whitespace gaps
+    // - indented multiline strings don't cause whitespace gaps
     #[test]
     fn dberror_full_info() {
         // Test that DbError::Unavailable is displayed correctly