Skip to content

Commit 6a2a994

Browse files
author
Grzegorz Kołakowski
committed
Refactor error retry configuration parameters
1 parent beefad1 commit 6a2a994

File tree

5 files changed

+58
-20
lines changed

5 files changed

+58
-20
lines changed

‎README.md

Lines changed: 20 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -379,18 +379,26 @@ is provided.
379379

380380

381381
## HTTP status code handler
382-
Http Sink and Lookup Source connectors allow defining list of HTTP status codes that should be treated as errors.
383-
By default all 400s and 500s response codes will be interpreted as error code.
382+
383+
Http Sink and Lookup Source connectors allow defining list of HTTP status codes for which connector either retry
384+
request according to defined strategy, or returns empty result.
385+
By default, all 400s are interpreted as non-retryable errors, while 500s response codes as retryable errors.
384386

385387
This behavior can be changed by using below properties in table definition (DDL) for Sink and Lookup Source or passing it via
386388
`setProperty' method from Sink's builder. The property names are:
387-
- `gid.connector.http.sink.error.code` and `gid.connector.http.source.lookup.error.code` used to defined HTTP status code value that should be treated as error for example 404.
389+
- `gid.connector.http.sink.error.non-retryable.code` and `gid.connector.http.source.lookup.error.non-retryable.code`, allow to define HTTP status code values that should be treated as non-retryable error.
390+
- `gid.connector.http.sink.error.retryable.code` and `gid.connector.http.source.lookup.error.retryable.code`, allow to define HTTP status code values that should be treated as retryable error.
391+
388392
Many status codes can be defined in one value, where each code should be separated with comma, for example:
389393
`401, 402, 403`. User can use this property also to define a type code mask. In that case, all codes from given HTTP response type will be treated as errors.
390394
An example of such a mask would be `3XX, 4XX, 5XX`. In this case, all 300s, 400s and 500s status codes will be treated as errors.
391-
- `gid.connector.http.sink.error.code.exclude` and `gid.connector.http.source.lookup.error.code.exclude` used to exclude a HTTP code from error list.
392-
Many status codes can be defined in one value, where each code should be separated with comma, for example:
393-
`401, 402, 403`. In this example, codes 401, 402 and 403 would not be interpreted as error codes.
395+
396+
Another set of properties are:
397+
- `gid.connector.http.sink.error.non-retryable.code.exclude` and `gid.connector.http.source.lookup.error.non-retryable.code.exclude` used to exclude an HTTP code from non-retryable error list.
398+
- `gid.connector.http.sink.error.retryable.code.exclude` and `gid.connector.http.source.lookup.error.retryable.code.exclude` used to exclude an HTTP code from retryable error list.
399+
400+
Many status codes can be defined in one value, where each code should be separated with comma, for example:
401+
`401, 402, 403`. In this example, codes 401, 402 and 403 would not be interpreted as error codes.
394402

395403
## TLS and mTLS support
396404
Both Http Sink and Lookup Source connectors supports Https communication using TLS 1.2 and mTLS.
@@ -455,8 +463,12 @@ be requested if the current time is later than the cached token expiry time minu
455463
| lookup.retry-strategy.exponential-delay.attempts | optional | The number of times that connector retries lookup execution before connector returns empty result. |
456464
| lookup.retry-strategy.exponential-delay.initial-delay | optional | Initial delay between two consecutive retry attempts. |
457465
| lookup.retry-strategy.exponential-delay.max-delay | optional | The highest possible duration between two consecutive retry attempts. |
458-
| gid.connector.http.lookup.error.code | optional | List of HTTP status codes that should be treated as errors by HTTP Source, separated with comma. |
459-
| gid.connector.http.lookup.error.code.exclude | optional | List of HTTP status codes that should be excluded from the `gid.connector.http.lookup.error.code` list, separated with comma. |
466+
| gid.connector.http.lookup.error.code | optional | (Deprecated) List of HTTP status codes that should be treated as errors by HTTP Source, separated with comma. |
467+
| gid.connector.http.lookup.error.code.exclude | optional | (Deprecated) List of HTTP status codes that should be excluded from the `gid.connector.http.lookup.error.code` list, separated with comma. |
468+
| gid.connector.http.lookup.error.non-retryable.code | optional | List of HTTP status codes that should be treated as errors for which HTTP Source should not retry request, separated with comma. |
469+
| gid.connector.http.lookup.error.non-retryable.code.exclude | optional | List of HTTP status codes that should be excluded from the `gid.connector.http.lookup.error.code` list, separated with comma. |
470+
| gid.connector.http.lookup.error.retryable.code | optional | List of HTTP status codes that should be treated as errors for which HTTP Source should retry request, separated with comma. |
471+
| gid.connector.http.lookup.error.retryable.code.exclude | optional | List of HTTP status codes that should be excluded from the `gid.connector.http.lookup-retryable.error.code` list, separated with comma. |
460472
| gid.connector.http.security.cert.server | optional | Path to trusted HTTP server certificate that should be add to connectors key store. More than one path can be specified using `,` as path delimiter. |
461473
| gid.connector.http.security.cert.client | optional | Path to trusted certificate that should be used by connector's HTTP client for mTLS communication. |
462474
| gid.connector.http.security.key.client | optional | Path to trusted private key that should be used by connector's HTTP client for mTLS communication. |

‎src/main/java/com/getindata/connectors/http/internal/config/HttpConnectorConfigConstants.java

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -49,17 +49,25 @@ public final class HttpConnectorConfigConstants {
4949

5050
public static final String HTTP_ERROR_SINK_CODES_LIST = GID_CONNECTOR_HTTP + "sink.error.code";
5151

52+
@Deprecated
5253
public static final String HTTP_ERROR_SOURCE_LOOKUP_CODE_WHITE_LIST =
5354
GID_CONNECTOR_HTTP + "source.lookup.error.code.exclude";
5455

56+
@Deprecated
5557
public static final String HTTP_ERROR_SOURCE_LOOKUP_CODES_LIST =
5658
GID_CONNECTOR_HTTP + "source.lookup.error.code";
5759

60+
public static final String HTTP_ERROR_NON_RETRYABLE_SOURCE_LOOKUP_CODE_WHITE_LIST =
61+
GID_CONNECTOR_HTTP + "source.lookup.error.non-retryable.code.exclude";
62+
63+
public static final String HTTP_ERROR_NON_RETRYABLE_SOURCE_LOOKUP_CODES_LIST =
64+
GID_CONNECTOR_HTTP + "source.lookup.error.non-retryable.code";
65+
5866
public static final String HTTP_ERROR_RETRYABLE_SOURCE_LOOKUP_CODE_WHITE_LIST =
59-
GID_CONNECTOR_HTTP + "source.lookup.error-retryable.code.exclude";
67+
GID_CONNECTOR_HTTP + "source.lookup.error.retryable.code.exclude";
6068

6169
public static final String HTTP_ERROR_RETRYABLE_SOURCE_LOOKUP_CODES_LIST =
62-
GID_CONNECTOR_HTTP + "source.lookup.error-retryable.code";
70+
GID_CONNECTOR_HTTP + "source.lookup.error.retryable.code";
6371
// -----------------------------------------------------
6472

6573
public static final String SOURCE_LOOKUP_REQUEST_CALLBACK_IDENTIFIER =

‎src/main/java/com/getindata/connectors/http/internal/status/ComposeHttpStatusCodeChecker.java

Lines changed: 18 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -27,15 +27,25 @@ public class ComposeHttpStatusCodeChecker implements HttpStatusCodeChecker {
2727
new TypeStatusCodeCheckerPredicate(HttpResponseCodeType.CLIENT_ERROR);
2828
private static final Predicate<Integer> DEFAULT_RETRYABLE_ERROR_CODES =
2929
new TypeStatusCodeCheckerPredicate(HttpResponseCodeType.SERVER_ERROR);
30+
private static final Predicate<Integer> DEFAULT_DEPRECATED_ERROR_CODES =
31+
DEFAULT_ERROR_CODES.or(DEFAULT_RETRYABLE_ERROR_CODES);
3032

3133
private final Predicate<Integer> retryableErrorStatusCodes;
3234
private final Predicate<Integer> notRetryableErrorStatusCodes;
3335

3436
public ComposeHttpStatusCodeChecker(ComposeHttpStatusCodeCheckerConfig config) {
35-
retryableErrorStatusCodes = buildPredicate(config, config.getRetryableCodePrefix(),
36-
config.getRetryableWhiteListPrefix(), DEFAULT_RETRYABLE_ERROR_CODES);
37-
notRetryableErrorStatusCodes = buildPredicate(config, config.getErrorCodePrefix(),
38-
config.getErrorWhiteListPrefix(), DEFAULT_ERROR_CODES);
37+
// Handle deprecated configuration for backward compatibility.
38+
if (!StringUtils.isNullOrWhitespaceOnly(config.getDeprecatedCodePrefix()) ||
39+
!StringUtils.isNullOrWhitespaceOnly(config.getDeprecatedErrorWhiteListPrefix())) {
40+
notRetryableErrorStatusCodes = buildPredicate(config, config.getDeprecatedCodePrefix(),
41+
config.getDeprecatedErrorWhiteListPrefix(), DEFAULT_DEPRECATED_ERROR_CODES);
42+
retryableErrorStatusCodes = integer -> false;
43+
} else {
44+
retryableErrorStatusCodes = buildPredicate(config, config.getRetryableCodePrefix(),
45+
config.getRetryableWhiteListPrefix(), DEFAULT_RETRYABLE_ERROR_CODES);
46+
notRetryableErrorStatusCodes = buildPredicate(config, config.getErrorCodePrefix(),
47+
config.getErrorWhiteListPrefix(), DEFAULT_ERROR_CODES);
48+
}
3949
}
4050

4151
private Predicate<Integer> buildPredicate(
@@ -135,6 +145,10 @@ public HttpResponseStatus checkStatus(int statusCode) {
135145
@RequiredArgsConstructor(access = AccessLevel.PRIVATE)
136146
public static class ComposeHttpStatusCodeCheckerConfig {
137147

148+
private final String deprecatedErrorWhiteListPrefix;
149+
150+
private final String deprecatedCodePrefix;
151+
138152
private final String errorWhiteListPrefix;
139153

140154
private final String errorCodePrefix;

‎src/main/java/com/getindata/connectors/http/internal/table/lookup/JavaNetHttpPollingClient.java

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,10 @@
77
import java.time.Duration;
88
import java.util.Collections;
99
import java.util.Optional;
10+
import static com.getindata.connectors.http.internal.config.HttpConnectorConfigConstants.HTTP_ERROR_NON_RETRYABLE_SOURCE_LOOKUP_CODES_LIST;
11+
import static com.getindata.connectors.http.internal.config.HttpConnectorConfigConstants.HTTP_ERROR_NON_RETRYABLE_SOURCE_LOOKUP_CODE_WHITE_LIST;
12+
import static com.getindata.connectors.http.internal.config.HttpConnectorConfigConstants.HTTP_ERROR_SOURCE_LOOKUP_CODES_LIST;
13+
import static com.getindata.connectors.http.internal.config.HttpConnectorConfigConstants.HTTP_ERROR_SOURCE_LOOKUP_CODE_WHITE_LIST;
1014
import static java.lang.String.format;
1115

1216
import lombok.AllArgsConstructor;
@@ -30,8 +34,6 @@
3034
import com.getindata.connectors.http.internal.status.HttpStatusCodeChecker;
3135
import static com.getindata.connectors.http.internal.config.HttpConnectorConfigConstants.HTTP_ERROR_RETRYABLE_SOURCE_LOOKUP_CODES_LIST;
3236
import static com.getindata.connectors.http.internal.config.HttpConnectorConfigConstants.HTTP_ERROR_RETRYABLE_SOURCE_LOOKUP_CODE_WHITE_LIST;
33-
import static com.getindata.connectors.http.internal.config.HttpConnectorConfigConstants.HTTP_ERROR_SOURCE_LOOKUP_CODES_LIST;
34-
import static com.getindata.connectors.http.internal.config.HttpConnectorConfigConstants.HTTP_ERROR_SOURCE_LOOKUP_CODE_WHITE_LIST;
3537
import static com.getindata.connectors.http.internal.table.lookup.HttpLookupConnectorOptions.LOOKUP_RESTART_STRATEGY_EXPONENTIAL_DELAY_ATTEMPTS;
3638
import static com.getindata.connectors.http.internal.table.lookup.HttpLookupConnectorOptions.LOOKUP_RESTART_STRATEGY_EXPONENTIAL_DELAY_INITIAL_DELAY;
3739
import static com.getindata.connectors.http.internal.table.lookup.HttpLookupConnectorOptions.LOOKUP_RESTART_STRATEGY_EXPONENTIAL_DELAY_MAX_DELAY;
@@ -77,8 +79,10 @@ public JavaNetHttpPollingClient(
7779
ComposeHttpStatusCodeCheckerConfig checkerConfig =
7880
ComposeHttpStatusCodeCheckerConfig.builder()
7981
.properties(options.getProperties())
80-
.errorWhiteListPrefix(HTTP_ERROR_SOURCE_LOOKUP_CODE_WHITE_LIST)
81-
.errorCodePrefix(HTTP_ERROR_SOURCE_LOOKUP_CODES_LIST)
82+
.deprecatedErrorWhiteListPrefix(HTTP_ERROR_SOURCE_LOOKUP_CODE_WHITE_LIST)
83+
.deprecatedCodePrefix(HTTP_ERROR_SOURCE_LOOKUP_CODES_LIST)
84+
.errorWhiteListPrefix(HTTP_ERROR_NON_RETRYABLE_SOURCE_LOOKUP_CODE_WHITE_LIST)
85+
.errorCodePrefix(HTTP_ERROR_NON_RETRYABLE_SOURCE_LOOKUP_CODES_LIST)
8286
.retryableWhiteListPrefix(HTTP_ERROR_RETRYABLE_SOURCE_LOOKUP_CODE_WHITE_LIST)
8387
.retryableCodePrefix(HTTP_ERROR_RETRYABLE_SOURCE_LOOKUP_CODES_LIST)
8488
.build();

‎src/test/java/com/getindata/connectors/http/internal/table/lookup/JavaNetHttpPollingClientConnectionTest.java

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -436,11 +436,11 @@ private StubMapping setupServerStubForBasicAuth() {
436436
private static Properties prepareErrorCodeProperties(String errorCodeList, String whiteList) {
437437
Properties properties = new Properties();
438438
properties.setProperty(
439-
HttpConnectorConfigConstants.HTTP_ERROR_SOURCE_LOOKUP_CODE_WHITE_LIST,
439+
HttpConnectorConfigConstants.HTTP_ERROR_NON_RETRYABLE_SOURCE_LOOKUP_CODE_WHITE_LIST,
440440
whiteList
441441
);
442442
properties.setProperty(
443-
HttpConnectorConfigConstants.HTTP_ERROR_SOURCE_LOOKUP_CODES_LIST,
443+
HttpConnectorConfigConstants.HTTP_ERROR_NON_RETRYABLE_SOURCE_LOOKUP_CODES_LIST,
444444
errorCodeList
445445
);
446446

0 commit comments

Comments
 (0)