-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CSV File Uploading Practices between Frontend and Backend #152
Comments
Sample solutions for above solution directions1. Chunking a large CSV file
function processCSVChunk(chunk) {
// Process the CSV chunk here
// you can split the chunk into lines
// or covert the chunk into a plain text using chunk.text
}
async function processCSVFile(file) {
const chunkSize = 1024 * 1024;
let offset = 0;
// Read the file slice
while (offset < file.size) {
const chunk = file.slice(offset, offset + chunkSize);
const textChunk = await chunk.text();
await processCSVChunk(textChunk);
offset += chunkSize;
}
console.log('CSV processing completed');
}
2. Using streaming solutionVery similar to the solution of using chunk, and also potentially has the same issue of breaking a row into two different chunks and result in parsing error. |
Suggested NPM Packages for Addressing the above ChallengesThe package solution I gravitate towards is PapaParse, owing to its adept handling of the challenges mentioned above. Moreover, it also supports the functionality of aborting CSV file parsing after producing a set number of results. |
Crafting the WorkflowWhat are expectedWhen orchestrating the exchange of data between distinct systems using CSV as the medium, user expects:
In response to these user expectations, the design of CSV uploading systems demands consideration in these aspects:
The WorkflowPerformance considerations
The UI considerationsProvide user instruction and guidelines using step workflows
|
Introduction to CSV and its applications, challenges
what is CSV and what it used for
CSV is an acronym for Comma-Separated Values. CSV is commonly used for data exchange between different applications, importing and exporting data from spreadsheets, etc.
Structure of CSV
,
). Alternative delimiter values can also be a semicolon and tab, or any other character based on the requirements.Consideration and Challenges
General Challenges
Challenges on parsing large csv file
Performance:
Solutions directions
The text was updated successfully, but these errors were encountered: