Skip to content

Commit

Permalink
Refactor 'excludeJobIds' Serialization for Spark Job Compatibility
Browse files Browse the repository at this point in the history
This commit changes the serialization format of the 'excludeJobIds' field in the session document. Previously serialized as an array (e.g., ["id1", "id2"]), this field is now formatted as a comma-separated string ("id1,id2"). This adjustment is made to ensure compatibility with plugins that currently do not support array parsing. This simplification aids in seamless integration and easier data handling in downstream processes.

Testing done:
* modified unit test
Signed-off-by: Kaituo Li <[email protected]>
  • Loading branch information
kaituo committed Nov 14, 2023
1 parent 594534e commit 351614d
Show file tree
Hide file tree
Showing 2 changed files with 9 additions and 2 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,12 @@ object FlintInstance {
"state" -> job.state,
// update last update time
"lastUpdateTime" -> currentTime,
"excludeJobIds" -> job.excludedJobIds,
// Convert a Seq[String] into a comma-separated string, such as "id1,id2".
// This approach is chosen over serializing to an array format (e.g., ["id1", "id2"])
// because it simplifies client-side processing. With a comma-separated string,
// clients can easily ignore this field if it's not in use, avoiding the need
// for array parsing logic. This makes the serialized data more straightforward to handle.
"excludeJobIds" -> job.excludedJobIds.mkString(","),
"jobStartTime" -> job.jobStartTime))
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,9 @@ class FlintInstanceTest extends SparkFunSuite with Matchers {
json should include(""""sessionId":"session-789"""")
json should include(""""state":"RUNNING"""")
json should include(s""""lastUpdateTime":$currentTime""")
json should include(""""excludeJobIds":["job-101","job-202"]""")
json should include(
""""excludeJobIds":"job-101,job-202""""
) // Check for comma-separated string
json should include(""""jobStartTime":1620000001000""")
json should include(""""error":""""")
}
Expand Down

0 comments on commit 351614d

Please sign in to comment.