Add option in NIO for re-opening the channel to retry for some errors by jean-philippe-martin · Pull Request #1715 · googleapis/google-cloud-java

jean-philippe-martin · 2017-03-08T18:14:03Z

While exercising the NIO code on large inputs (~600GB) I noticed occasional errors that crashed the program yet could probably have been retried. Those were:

503 Service Unavailable
500 Server error
Connection closed prematurely

The program was using the default retry options (3 retries). Looking at the specific exceptions I see that those StorageException had retryable=false, so the normal retry mechanism isn't triggered. This makes perfect sense for "connection closed prematurely".

This PR adds an option that allows NIO to re-open the channel in case of "connection closed prematurely" error. The same option will also retry on 503 and 500 errors (without re-opening the channel in that case). The PR also includes the corresponding test cases.

Re-running the benchmarks with this updated code, I was able to see those errors again and observe that the NIO retry worked: the final output was correct.

It also retries a few times on exceptions that are not normally considered retriable (500, 503). Also add corresponding test code.

coveralls · 2017-03-08T21:01:32Z

Coverage decreased (-0.04%) to 81.45% when pulling 919e67b on jean-philippe-martin:jp_retries into 59459fc on GoogleCloudPlatform:master.

coveralls · 2017-03-08T21:18:05Z

Coverage decreased (-0.03%) to 81.464% when pulling 919e67b on jean-philippe-martin:jp_retries into 59459fc on GoogleCloudPlatform:master.

coveralls · 2017-03-11T00:18:12Z

Changes Unknown when pulling 52606c2 on jean-philippe-martin:jp_retries into ** on GoogleCloudPlatform:master**.

Sign in to view

+  /**
+   * Returns the number of times we try re-opening a channel if it's closed unexpectedly
+   * while reading.
+   */


Sign in to view

  private long size;

+  /**
+   * @param maxReopen max. number of times to try re-opening the channel if it closes on us unexpectedly.


Sign in to view

  @CheckReturnValue
  @SuppressWarnings("resource")
-  static CloudStorageReadChannel create(Storage gcsStorage, BlobId file, long position)
+  static CloudStorageReadChannel create(Storage gcsStorage, BlobId file, long position, int maxReopen)


Sign in to view

+            innerOpen();
+            continue;
+          } else if (exs.getCode() == 500 && retries < maxRetries) {
+            // server error. Not retryable yet retrying works in my experience.


Sign in to view

+      int maxRetries = 3;
+      dst.mark();
+      while (true) {
+        try {


Sign in to view

    }
  }

+  private void retryDelay(int attempt) {


Sign in to view

+    return new AutoValue_OptionChannelReopen(retryCount);
+  }
+
+  abstract int retry();


coveralls · 2017-03-13T23:52:03Z

Changes Unknown when pulling d67872a on jean-philippe-martin:jp_retries into ** on GoogleCloudPlatform:master**.

coveralls · 2017-03-14T00:00:46Z

Changes Unknown when pulling d67872a on jean-philippe-martin:jp_retries into ** on GoogleCloudPlatform:master**.

Sign in to view

+import com.google.auto.value.AutoValue;
+
+@AutoValue
+abstract class OptionMaxChannelReopen implements CloudStorageOption.OpenCopy {


Sign in to view

  private SeekableByteChannel newReadChannel(Path path, Set<? extends OpenOption> options)
      throws IOException {
    initStorage();
+    int maxChannelReopens = ((CloudStorageFileSystem)path.getFileSystem()).config().maxChannelReopens();


Sign in to view

            throw new UnsupportedOperationException(option.toString());
        }
+      } else if (option instanceof OptionMaxChannelReopen) {
+        maxChannelReopens = ((OptionMaxChannelReopen)option).maxChannelReopen();


Sign in to view

+            innerOpen();
+            continue;
+          } else if ((exs.getCode() == 500 || exs.getCode() == 503)  && retries < maxRetries) {
+            // Retrying works in practice.


coveralls · 2017-03-15T22:07:53Z

Changes Unknown when pulling bcabfbc on jean-philippe-martin:jp_retries into ** on GoogleCloudPlatform:master**.

jean-philippe-martin · 2017-03-15T22:34:09Z

Thank you!

Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>

…o v2.17.2 (googleapis#1715) [![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [com.fasterxml.jackson.core:jackson-databind](https://togithub.com/FasterXML/jackson) ([source](https://togithub.com/FasterXML/jackson-databind)) | `2.13.4.2` -> `2.17.2` | [![age](https://developer.mend.io/api/mc/badges/age/maven/com.fasterxml.jackson.core:jackson-databind/2.17.2?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://developer.mend.io/api/mc/badges/adoption/maven/com.fasterxml.jackson.core:jackson-databind/2.17.2?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://developer.mend.io/api/mc/badges/compatibility/maven/com.fasterxml.jackson.core:jackson-databind/2.13.4.2/2.17.2?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://developer.mend.io/api/mc/badges/confidence/maven/com.fasterxml.jackson.core:jackson-databind/2.13.4.2/2.17.2?slim=true)](https://docs.renovatebot.com/merge-confidence/) | --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR was generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View the [repository job log](https://developer.mend.io/github/googleapis/java-spanner-jdbc).

…#1715) * Add retry code in NIO that re-opens the channel if necessary It also retries a few times on exceptions that are not normally considered retriable (500, 503). Also add corresponding test code.

…to v0.9.26 (#1715)

…to v0.9.26 (#1715)

Add retry code in NIO that re-opens the channel if necessary

9e48440

It also retries a few times on exceptions that are not normally considered retriable (500, 503). Also add corresponding test code.

googlebot added the cla: yes This human has signed the Contributor License Agreement. label Mar 8, 2017

jean-philippe-martin changed the title ~~Add optional in NIO for re-opening the channel to retry for some errors~~ Add option in NIO for re-opening the channel to retry for some errors Mar 8, 2017

address linter

919e67b

jean-philippe-martin mentioned this pull request Mar 10, 2017

Update google-cloud-java dependency to a snapshot containing newly added NIO API methods broadinstitute/gatk#2441

Closed

Allow per-filesystem specification of channel reopen count

52606c2

garrettjonesgoogle reviewed Mar 13, 2017

View reviewed changes

Clean up names

d67872a

jean-philippe-martin force-pushed the jp_retries branch from e10e72a to d67872a Compare March 13, 2017 22:41

garrettjonesgoogle reviewed Mar 15, 2017

View reviewed changes

Reviewer comments

bcabfbc

garrettjonesgoogle merged commit f72746c into googleapis:master Mar 15, 2017

chingor13 pushed a commit that referenced this pull request Jan 22, 2026

chore(main): release 2.25.4-SNAPSHOT (#1715)

2a7a9a1

Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>

meltsufin pushed a commit that referenced this pull request Apr 29, 2026

deps: update dependency org.graalvm.buildtools:junit-platform-native …

f2cc75f

…to v0.9.26 (#1715)

meltsufin pushed a commit that referenced this pull request May 1, 2026

deps: update dependency org.graalvm.buildtools:junit-platform-native …

f1df0e3

…to v0.9.26 (#1715)

Conversation

jean-philippe-martin commented Mar 8, 2017

Uh oh!

coveralls commented Mar 8, 2017

Uh oh!

coveralls commented Mar 8, 2017

Uh oh!

coveralls commented Mar 11, 2017

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

coveralls commented Mar 13, 2017

Uh oh!

coveralls commented Mar 14, 2017

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

coveralls commented Mar 15, 2017

Uh oh!

jean-philippe-martin commented Mar 15, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants