Skip to content

Log Stage and Task ID to understand I/O bottlenecks. #268

Log Stage and Task ID to understand I/O bottlenecks.

Log Stage and Task ID to understand I/O bottlenecks. #268

Workflow file for this run

#
# Copyright 2022- IBM Inc. All rights reserved
# SPDX-License-Identifier: Apache2.0
#
name: CI
on:
pull_request:
push:
branches:
- main
tags:
- v*
jobs:
Build:
strategy:
fail-fast: true
matrix:
include:
- spark: 3.2.0
scala: 2.12.15
- spark: 3.2.0
scala: 2.13.5
- spark: 3.2.1
scala: 2.12.15
- spark: 3.2.1
scala: 2.13.5
- spark: 3.2.3
scala: 2.12.15
- spark: 3.2.3
scala: 2.13.5
- spark: 3.2.4
scala: 2.12.15
- spark: 3.2.4
scala: 2.13.5
- spark: 3.3.0
scala: 2.12.15
- spark: 3.3.0
scala: 2.13.8
- spark: 3.3.1
scala: 2.12.15
- spark: 3.3.1
scala: 2.13.8
- spark: 3.3.2
scala: 2.12.15
- spark: 3.3.2
scala: 2.13.8
- spark: 3.4.0
scala: 2.12.17
- spark: 3.4.0
scala: 2.13.8
runs-on: ubuntu-latest
env:
SPARK_VERSION: ${{ matrix.spark }}
SCALA_VERSION: ${{ matrix.scala }}
steps:
- name: Checkout
uses: actions/checkout@v2
- name: Setup JDK
uses: actions/setup-java@v3
with:
distribution: temurin
java-version: 11
cache: sbt
- name: Test Default Shuffle Fetch
shell: bash
if: startsWith(matrix.scala, '2.12.')
run: |
sbt test
- name: Test Spark Shuffle Fetch
shell: bash
if: startsWith(matrix.scala, '2.12.') && !startsWith(matrix.spark, '3.2.')
env:
USE_SPARK_SHUFFLE_FETCH: "true"
run: |
sbt test
- name: Package
shell: bash
run: |
rm -rf target
sbt package
sbt makePom
- name: Upload to maven
if: startsWith(github.ref, 'refs/tags/v')
shell: bash
run: ./.github/publish_maven.sh
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Upload release
if: startsWith(github.ref, 'refs/tags/v')
uses: softprops/action-gh-release@v1
with:
files: target/scala*/*.jar