Skip to content

Commit

Permalink
[SPARK-48595][CORE] Cleanup deprecated api usage related to `commons-…
Browse files Browse the repository at this point in the history
…compress`

### What changes were proposed in this pull request?
This pr use `org.apache.commons.io.output.CountingOutputStream` instead of `org.apache.commons.compress.utils.CountingOutputStream` to fix the following compilation warnings related to 'commons-compress':

```
[WARNING] [Warn] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/deploy/history/EventLogFileWriters.scala:308: class CountingOutputStream in package utils is deprecated
Applicable -Wconf / nowarn filters for this warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.deploy.history.RollingEventLogFilesWriter.countingOutputStream, origin=org.apache.commons.compress.utils.CountingOutputStream
[WARNING] [Warn] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/deploy/history/EventLogFileWriters.scala:351: class CountingOutputStream in package utils is deprecated
Applicable -Wconf / nowarn filters for this warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.deploy.history.RollingEventLogFilesWriter.rollEventLogFile.$anonfun, origin=org.apache.commons.compress.utils.CountingOutputStream
```

The fix refers to:

https://github.com/apache/commons-compress/blob/95727006cac0892c654951c4e7f1db142462f22a/src/main/java/org/apache/commons/compress/utils/CountingOutputStream.java#L25-L33

```
/**
 * Stream that tracks the number of bytes read.
 *
 * since 1.3
 * NotThreadSafe
 * deprecated Use {link org.apache.commons.io.output.CountingOutputStream}.
 */
Deprecated
public class CountingOutputStream extends FilterOutputStream {
```

### Why are the changes needed?
Cleanup deprecated api usage related to `commons-compress`

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Pass GitHub Actions

### Was this patch authored or co-authored using generative AI tooling?
No

Closes #46950 from LuciferYang/SPARK-48595.

Authored-by: yangjie01 <[email protected]>
Signed-off-by: Kent Yao <[email protected]>
  • Loading branch information
LuciferYang authored and yaooqinn committed Jun 12, 2024
1 parent da81d8e commit a3625a9
Showing 1 changed file with 2 additions and 2 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ import java.io._
import java.net.URI
import java.nio.charset.StandardCharsets

import org.apache.commons.compress.utils.CountingOutputStream
import org.apache.commons.io.output.CountingOutputStream
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.{FileStatus, FileSystem, FSDataOutputStream, Path}
import org.apache.hadoop.fs.permission.FsPermission
Expand Down Expand Up @@ -330,7 +330,7 @@ class RollingEventLogFilesWriter(

override def writeEvent(eventJson: String, flushLogger: Boolean = false): Unit = {
writer.foreach { w =>
val currentLen = countingOutputStream.get.getBytesWritten
val currentLen = countingOutputStream.get.getByteCount
if (currentLen + eventJson.length > eventFileMaxLength) {
rollEventLogFile()
}
Expand Down

0 comments on commit a3625a9

Please sign in to comment.