HPCC-31968 Increase ECLWatch upload buffer size to 1MB

[asselitx]

Get a quick reasonable fix deployed. Former 1K size likely cause of slow uploads to cloud, if not increased cost in some cases. Subsequent ticket will use configured preferred size per landing zone.

Type of change:

• This change is a bug fix (non-breaking change which fixes an issue).
• This change is a new feature (non-breaking change which adds functionality).
• This change improves the code (refactor or other change that does not change the functionality)
• This change fixes warnings (the fix does not alter the functionality or the generated code)
• This change is a breaking change (fix or feature that will cause existing behavior to change).
• This change alters the query API (existing queries will have to be recompiled)

Checklist:

• My code follows the code style of this project.
  • My code does not create any new warnings from compiler, build system, or lint.
• The commit message is properly formatted and free of typos.
  • The commit message title makes sense in a changelog, by itself.
  • The commit is signed.
• My change requires a change to the documentation.
  • I have updated the documentation accordingly, or...
  • I have created a JIRA ticket to update the documentation.
  • Any new interfaces or exported functions are appropriately commented.
• I have read the CONTRIBUTORS document.
• The change has been fully tested:
  • I have added tests to cover my changes.
  • All new and existing tests passed.
  • I have checked that this change does not introduce memory leaks.
  • I have used Valgrind or similar tools to check for potential issues.
• I have given due consideration to all of the following potential concerns:
  • Scalability
  • Performance
  • Security
  • Thread-safety
  • Cloud-compatibility
  • Premature optimization
  • Existing deployed queries will not be broken
  • This change fixes the problem, not just the symptom
  • The target branch of this pull request is appropriate for such a change.
• There are no similar instances of the same problem that should be addressed
  • I have addressed them here
  • I have raised JIRA issues to address them separately
• This is a user interface / front-end modification
  • I have tested my changes in multiple modern browsers
  • The component(s) render as expected

Smoketest:

• Send notifications about my Pull Request position in Smoketest queue.
• Test my draft Pull Request.

Testing:

Tested locally with instrumentation to confirm upload completes in fewer read/write cycles

[asselitx]

@ghalliday let me know if someone else is better suited to review. Tim is out or I would have asked him.

[github-actions]

Jira Issue: https://hpccsystems.atlassian.net//browse/HPCC-31968

Jirabot Action Result:
Workflow Transition: Merge Pending
Updated PR

[ghalliday]

@asselitx see comments and suggestion. Please ask me if you have any questions.

[ghalliday]

Allocating 1MB on the stack could cause problems.

The function also has a couple of other problems:

• It is unnecessarily memcpy-ing the data - which adds up if the file is large.
• It is unnecessarily adding a null terminator onto the string that was read (which makes the code confusing).

Better is something like:

    constexpr size32_t readChunkSize = 0x100000;
    size32_t sizeToRead = bytesNotRead > readChunkSize ? readChunkSize: (size32_t)readChunkSize;
    size32_t prevLen = buffer.length();
    char * target = (char *)buffer.reserve(sizeToRead);
    int readlen = m_bufferedsocket->read(target, sizeToRead);
    if(readlen <= 0)
    {
        if(readlen < 0)
            DBGLOG("Failed to read from socket");
        buffer.setLength(prevLen);
        return false;
    }

    buffer.setLength(prevLen + readlen);
    bytesNotRead -= readlen;
    return true;

(Untested). It reads the data directly into the buffer rather than copying it.

[asselitx]

This makes sense, it is a much better solution. I'd completely missed the consideration of stack size and the reserve on the buffer is slick. I'm implementing this and testing.

[ghalliday]

Well spotted (and that is a terrible interface!)