You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently the uploadResults function is limited in terms of allowing extendability that could allow packages to easily implement customizable upload functions.
The idea here is that these classes could allow users to modify data before upload, validate data or perform other tasks in the upload pipeline in a way that works well in the generic case and allows customisable complexity in a consistent way.
The initial usecase is to support a complicated example from the requirements of PLP here that works but could be implemented in a more consistent way.
Some requirements to gradually implement:
Define what generic classes are needed
Implement default behaviour that works as now (upload data and validate or overwrite according to specifications)
Support complex modifications that allow modifications of uploaded data
Improve upon current implementation by supporting improved loading concepts (load tables, table partition for large tables, cross platform indexes)
Allow loading from object storage like AWS S3
Fast loading with multiprocess operations
Support transactional loading of data - on error rollback certain tables or entire result sets if desired
The text was updated successfully, but these errors were encountered:
Currently the
uploadResults
function is limited in terms of allowing extendability that could allow packages to easily implement customizable upload functions.The idea here is that these classes could allow users to modify data before upload, validate data or perform other tasks in the upload pipeline in a way that works well in the generic case and allows customisable complexity in a consistent way.
The initial usecase is to support a complicated example from the requirements of PLP here that works but could be implemented in a more consistent way.
Some requirements to gradually implement:
The text was updated successfully, but these errors were encountered: