Skip to content
This repository was archived by the owner on Jan 9, 2020. It is now read-only.

Commit f160267

Browse files
dhruveMarcelo Vanzin
authored andcommitted
[SPARK-21181] Release byteBuffers to suppress netty error messages
## What changes were proposed in this pull request? We are explicitly calling release on the byteBuf's used to encode the string to Base64 to suppress the memory leak error message reported by netty. This is to make it less confusing for the user. ### Changes proposed in this fix By explicitly invoking release on the byteBuf's we are decrement the internal reference counts for the wrappedByteBuf's. Now, when the GC kicks in, these would be reclaimed as before, just that netty wouldn't report any memory leak error messages as the internal ref. counts are now 0. ## How was this patch tested? Ran a few spark-applications and examined the logs. The error message no longer appears. Original PR was opened against branch-2.1 => apache#18392 Author: Dhruve Ashar <[email protected]> Closes apache#18407 from dhruve/master. (cherry picked from commit 1ebe7ff) Signed-off-by: Marcelo Vanzin <[email protected]>
1 parent 9d29808 commit f160267

File tree

1 file changed

+22
-4
lines changed

1 file changed

+22
-4
lines changed

common/network-common/src/main/java/org/apache/spark/network/sasl/SparkSaslServer.java

Lines changed: 22 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@
3434
import com.google.common.base.Preconditions;
3535
import com.google.common.base.Throwables;
3636
import com.google.common.collect.ImmutableMap;
37+
import io.netty.buffer.ByteBuf;
3738
import io.netty.buffer.Unpooled;
3839
import io.netty.handler.codec.base64.Base64;
3940
import org.slf4j.Logger;
@@ -187,14 +188,31 @@ public void handle(Callback[] callbacks) throws IOException, UnsupportedCallback
187188
/* Encode a byte[] identifier as a Base64-encoded string. */
188189
public static String encodeIdentifier(String identifier) {
189190
Preconditions.checkNotNull(identifier, "User cannot be null if SASL is enabled");
190-
return Base64.encode(Unpooled.wrappedBuffer(identifier.getBytes(StandardCharsets.UTF_8)))
191-
.toString(StandardCharsets.UTF_8);
191+
return getBase64EncodedString(identifier);
192192
}
193193

194194
/** Encode a password as a base64-encoded char[] array. */
195195
public static char[] encodePassword(String password) {
196196
Preconditions.checkNotNull(password, "Password cannot be null if SASL is enabled");
197-
return Base64.encode(Unpooled.wrappedBuffer(password.getBytes(StandardCharsets.UTF_8)))
198-
.toString(StandardCharsets.UTF_8).toCharArray();
197+
return getBase64EncodedString(password).toCharArray();
198+
}
199+
200+
/** Return a Base64-encoded string. */
201+
private static String getBase64EncodedString(String str) {
202+
ByteBuf byteBuf = null;
203+
ByteBuf encodedByteBuf = null;
204+
try {
205+
byteBuf = Unpooled.wrappedBuffer(str.getBytes(StandardCharsets.UTF_8));
206+
encodedByteBuf = Base64.encode(byteBuf);
207+
return encodedByteBuf.toString(StandardCharsets.UTF_8);
208+
} finally {
209+
// The release is called to suppress the memory leak error messages raised by netty.
210+
if (byteBuf != null) {
211+
byteBuf.release();
212+
if (encodedByteBuf != null) {
213+
encodedByteBuf.release();
214+
}
215+
}
216+
}
199217
}
200218
}

0 commit comments

Comments
 (0)