Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rs hotfix/fix corrupt image uploads #7781

Closed
wants to merge 2 commits into from
Closed

Rs hotfix/fix corrupt image uploads #7781

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
483fce2
fix corruption issue
RileySeaburg Jul 11, 2024
c2bb85a
fix folder delete issue
RileySeaburg Jul 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 22 additions & 34 deletions config/gulp/file-upload.js
View file
Open in desktop
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How I tested:

  • copied test files to content/uploads/_inbox
  • ran npx gulp upload for each file type listed below
file result
test-upload-7-12-2024.png cannot find file error
test-upload-7-12-2024.jpg cannot find file error
test-upload-7-12-2024.doc uploads, but file is broken
test-upload-7-12-2024.pdf uploads, but file is broken
test-upload-7-12-2024.xlsx uploads, but file is broken

static files are uploaded, accessible on aws but the file is corrupt. Test files are located at static/test-upload-7-12-2024.*

image files, both png and jpg are not found:

[13:56:33] Using gulpfile ~/Documents/gsa/digitalgov.gov/gulpfile.js
[13:56:33] Starting 'upload'...
[13:56:33] Starting 'fileTidy'...
Moving file from content/uploads/_inbox/test-upload-7-12-2024.png to content/uploads/_working-images/to-process/test-upload-7-12-2024.png
[13:56:33] Finished 'fileTidy' after 2.58 ms
[13:56:33] Starting 'cleanInbox'...
[13:56:33] Finished 'cleanInbox' after 7.56 ms
[13:56:33] Starting 'writeDataFile'...
[13:56:33] Finished 'writeDataFile' after 20 ms
[13:56:33] Starting 'processImages'...
[13:56:33] Finished 'processImages' after 4.43 ms
[13:56:33] Starting 'removeProcessedImage'...
file is written
[13:56:33] Finished 'removeProcessedImage' after 20 ms
[13:56:33] Starting 'determineWhichToUpload'...
[13:56:33] Finished 'determineWhichToUpload' after 6.8 ms
[13:56:33] Starting 'cleanup'...
[13:56:33] Finished 'cleanup' after 418 μs
[13:56:33] Finished 'upload' after 67 ms
node:events:498
      throw er; // Unhandled 'error' event
      ^

<ref *2> Error: ENOENT: no such file or directory, open '/Users/nlyons/Documents/gsa/digitalgov.gov/content/uploads/_working-images/processed/test-upload-7-12-2024_w1200.png'
Emitted 'error' event on Domain instance at:
    at Transform.emit (node:domain:539:12)
    at ReadableState.afterDestroy (/Users/nlyons/Documents/gsa/digitalgov.gov/node_modules/streamx/index.js:493:19)
    at Transform._destroy (/Users/nlyons/Documents/gsa/digitalgov.gov/node_modules/streamx/index.js:629:5)
    at ReadableState.updateNonPrimary (/Users/nlyons/Documents/gsa/digitalgov.gov/node_modules/streamx/index.js:386:16)
    at ReadableState.update (/Users/nlyons/Documents/gsa/digitalgov.gov/node_modules/streamx/index.js:367:71)
    at ReadableState.updateReadNT (/Users/nlyons/Documents/gsa/digitalgov.gov/node_modules/streamx/index.js:536:10)
    at process.processTicksAndRejections (node:internal/process/task_queues:77:11) {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '/Users/nlyons/Documents/gsa/digitalgov.gov/content/uploads/_working-images/processed/test-upload-7-12-2024_w1200.png',
  domainEmitter: <ref *1> Transform {

Note

I removed the file.upload.do from the gulpfile.js and can confirm that the content/uploads/_working-images/processed/ is created.

digitalgov.gov/gulpfile.js

Line 40 in 4375f5f

exports.upload = series(file.prep.do, file.process.do, file.upload.do);

Screenshot 2024-07-12 at 1 29 54 PM

I don't see those folders when I use the new code changes:

Screenshot 2024-07-12 at 2 04 11 PM

Copy link
Member Author

@RileySeaburg RileySeaburg Jul 12, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nick-mon1 very strange, when I run upload on a .xlsx file I see this

image

It opens as spreadsheet as well at this link

https://s3.amazonaws.com/digitalgov/static/test-sheet-riley-v1.xlsx

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm assuming you're testing all the files uploading at once. I'll do that as well.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@RileySeaburg I tested each file one at a time, did not try all of them at once since they won't typically happen

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is weird.

I'm stepping off the happy path and am going to debug a multiple upload scenario.

That's where I'm able to get errors.

Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,9 @@ const headers = {

function uploadImage() {
return gulp
.src("content/uploads/_working-images/processed/*")
.src("content/uploads/_working-images/processed/*", {
encoding: false,
})
.pipe(publisher.publish(headers))
.pipe(
awspublish.reporter({
Expand All @@ -44,7 +46,9 @@ function uploadFile() {
});

return gulp
.src("content/uploads/_working-files/to-process/*")
.src("content/uploads/_working-files/to-process/*", {
removeBOM: true,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

We'll have to test slides and/or spreadsheet for this.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you log in to S3 you will see the PowerPoints I was using to test. Have not deleted them yet.

I tested personally,

  • image and slideshow,
  • just slideshow
  • just image.

})
.pipe(staticRename)
.pipe(publisher.publish(headers))
.pipe(
Expand All @@ -61,20 +65,18 @@ function uploadFile() {
function cleanup() {
return new Promise((resolve, reject) => {
let imageDir = "content/uploads/_working-images/processed";
let fileDir = "content/uploads/_working-files/to-process";
let fileDir = "content/uploads/_working-files";

if (fs.existsSync(imageDir)) {
if (fs.readdirSync(imageDir).length > 0) {
console.log(`Images have number of files ${fs.readdirSync(imageDir).length}`);
// delete tht folder
del([imageDir]);
resolve();
} else {
resolve();
}
} else {
resolve();
// remove the folder
del([imageDir]);
}

if (fs.existsSync(fileDir)) {
// remove the folder
del([fileDir]);
}
resolve();
});
}

Expand Down Expand Up @@ -114,33 +116,19 @@ function determineWhichToUpload() {

if (imageFiles.length > 0) {
uploadsToComplete += 1;
const imageUploadStream = uploadImage();
imageUploadStream.on('finish', () => {
uploadsCompleted += 1;
checkCompletion();
});
imageUploadStream.on('error', (err) => {
console.error("Error uploading images:", err);
reject(err);
});
uploadImage();
uploadsCompleted += 1;
checkCompletion();
}

if (fileFiles.length > 0) {
uploadsToComplete += 1;
const fileUploadStream = uploadFile();
fileUploadStream.on('finish', () => {
uploadsCompleted += 1;
checkCompletion();
});
fileUploadStream.on('error', (err) => {
console.error("Error uploading files:", err);
reject(err);
});
uploadFile();
uploadsCompleted += 1;
checkCompletion();
}

if (uploadsToComplete === 0) {
resolve(); // If no uploads are initiated, resolve immediately.
}
return resolve();
});
}

Expand Down