Package creation takes long time when there are many files to be packaged.

[chelarua]

Hello

we (together with @ringods) are using swaddle to create a package from many (~ 2000) of files and it seems its taking a bit long(18-20 mins). I checked a bit and it seems that most of the time is spent in _swaddle_touch_setMtimeAndAtimeAllRecursivelyUnderSimulatedRoot when it's calling fakeroot, which also seems to be doing a sleep on each call. I tried disabling _swaddle_touch_setMtimeAndAtimeAllRecursivelyUnderSimulatedRoot and the packaging time was much shorter, about a minute.

Any suggestions/ advice on this?
Thank you

[raphaelcohn]

Hmmm... The touch logic exists in order to ensure that files are packaged
with a consistent timestamp when included in a tarball or package. One of
the things I've always been keen on is making sure metadata is consistent
and useful. It would probably be possible to speed this code up by using
find to generate a list of suitable files and then pass these via xargs
or find's -exec option, eg something like

swaddle_touch_setMtimeAndAtimeAllRecursivelyUnderSimulatedRoot()
{

    local timestampInEpochSeconds="$(swaddle_configure_timestamp)"
    local mtime="$(date -d @$timestampInEpochSeconds +'%Y%m%d%H%M.%S')"
    local filePath
    for filePath in "$@"
    do
        swaddle_simulateroot_execute find "$filePath" -type f   PASS TO
touch -m -a -c -t "$mtime" "$filePath"
    done
}

_swaddle_touch_setMtimeAndAtimeAllRecursivelyUnderSimulatedRoot()
{
  NOW DEAD
}

Do you want to submit a patch?

Raphael Cohn
Chief Architect, StormMQ
Co-Chair, OASIS MQTT Standard for the Internet of Things
Advisor, UK Government Technical Standards Board
https://uk.linkedin.com/in/raphcohn
+44 7590 675 756

UK Office:
Hamblethorpe Farm, Crag Lane, Bradley BD20 9DB, North Yorkshire, United
Kingdom
+44 845 3712 567

Registered office:
16 Anchor Street, Chelmsford, Essex, CM2 0JY, United Kingdom
StormMQ IT Limited is Registered in England and Wales under Company Number
07175657

On 13 October 2015 at 11:29, chelarua [email protected] wrote:

Hello

we (together with @ringods https://github.com/ringods) are using
swaddle to create a package from many (~ 2000) of files and it seems its
taking a bit long(18-20 mins). I checked a bit and it seems that most of
the time is spent in
_swaddle_touch_setMtimeAndAtimeAllRecursivelyUnderSimulatedRoot when it's
calling fakeroot, which also seems to be doing a sleep on each call. I
tried disabling
_swaddle_touch_setMtimeAndAtimeAllRecursivelyUnderSimulatedRoot and the
packaging time was much shorter, about a minute.

Any suggestions/ advice on this?
Thank you

—
Reply to this email directly or view it on GitHub
#33.

[chelarua]

Hi, thanks for the reply.
I tried the suggested implementation and the speed was good, 55 secs instead of 18 minutes with the old implementation.

Made a pull request for this #34.

Thanks

[stormmq]

Right, I'll review the pull req today. Do you want to retain copyright on
it? I can add you to the COPYRIGHT file after integrating the pull, or we
can make it part of the PR itself.

Raphael Cohn
Chief Architect, StormMQ
Co-Chair, OASIS MQTT Standard for the Internet of Things
Advisor, UK Government Technical Standards Board
https://uk.linkedin.com/in/raphcohn
+44 7590 675 756

UK Office:
Hamblethorpe Farm, Crag Lane, Bradley BD20 9DB, North Yorkshire, United
Kingdom
+44 845 3712 567

Registered office:
16 Anchor Street, Chelmsford, Essex, CM2 0JY, United Kingdom
StormMQ IT Limited is Registered in England and Wales under Company Number
07175657

On 28 October 2015 at 10:24, chelarua [email protected] wrote:

Hi, thanks for the reply.
I tried the suggested implementation and the speed was good, 55 secs
instead of 18 minutes with the old implementation.

Made a pull request for this #34
#34.

Thanks

—
Reply to this email directly or view it on GitHub
#33 (comment).

[chelarua]

Thanks, don't think that is needed