Release in wrong directory after tar extract

[sondrekje]

Hi!

I noticed that --strip-components has been removed as an argument to the tar command upon extraction. It seems to have been removed in this commit:
3cf7454

This broke shipit-deploy deployments (in my case) as the leading directory is no longer being stripped. What was previously extracted to src/... is now being extracted to tmp-12../src/..

Log after 5.3.0:

 Running "cd /home/.../releases/20200323213903 && tar -xzf tmp-42Xp73RZdZvYq3.tar.gz" on host "ssh.example.com".
 @ssh.example.com-err tar: tmp-42ulD5Bg9qK4yf/.gitignore: time stamp 2020-03-23 22:39:00 is 40.005396223 s in the future
...
 @ssh.example.com-err npm WARN saveError ENOENT: no such file or directory, open '/home/..../releases/20200323213903/package.json'

It's looking for package.json in the correct directory, but package.json is now located in tmp-242../package.json.

Log pre 5.3.0:

 Running "cd /home/.../releases/20200317222927 && tar --strip-components=1 -xzf tmp-41PrwQtteALZ5p.tar.gz" on host "ssh.example.com".
 @ssh.example.com-err tar: .gitignore: time stamp 2020-03-17 23:29:24 is 36.715125466 s in the future
 @ssh.example.com-err tar: src/index.ts: time stamp 2020-03-17 23:29:24 is 36.713136291 s in the future

shipitfile.js

module.exports = (shipit) => {
    require('shipit-deploy')(shipit);

    shipit.initConfig({
        default: {
            deployTo: '/home/...',
            repositoryUrl: '[email protected]:...',
            ignores: ['.git', 'node_modules'],
            keepReleases: 3,
            keepWorkspace: false,
            deleteOnRollback: false,
            shallowClone: true,
        },
        production: {
            servers: '[email protected]:2222',
            deployTo: '/home/...',
            branch: 'master',
        },
    });

    shipit.blTask('install', () =>
       shipit.remote(`cd ${shipit.releasePath} && npm install`));

   shipit.on('updated', () => shipit.start('install'));
};

My apologies if this is an issue of misconfiguration from my side! My configuration was working as of the previous release.

[schwarmco]

I do have the exact same problem - my shipit.releasePath now looks like this:

-rw-rw-r-- 1 usr usr   27 Apr  1 16:15 package-lock.json
-rw-rw-r-- 1 usr usr   41 Apr  1 16:15 REVISION
drwxr-xr-x 4 usr usr 4096 Apr  1 16:14 tmp-31663xXIxG9Ls1APQ

[gregberge]

Hello, could you submit a fix for that?

[schwarmco]

Sure, i could do that. Though, there's probably a reason, that @oleg-andreyev removed the --strip-components=1 from each tar command!?

[oleg-andreyev]

@schwarmco I had some issues with unit tests, they simply were not working with --strip-components=1

I'll try to review my PR and remember the exact reason.

[oleg-andreyev]

I've returned --strip-components=1 and now single test is failing

Summary of all failing tests
 FAIL  packages/ssh-pool/tests/integration.test.js (15.512s)
  ● ssh-pool › should copy to remote

    Command failed: ssh -i /Users/oandreyev/Development/shipit/ssh/id_rsa [email protected] "cd ./ && cat test.1587159277564.txt"
    cat: test.1587159277564.txt: No such file or directory

it seems when is running this test it does not extract files (see below)

deploy@shipit-test:~$ tar --strip-components=1 -xzf foo.tar.gz
deploy@shipit-test:~$ ls -la
total 48
drwxr-xr-x 5 deploy deploy 4096 Apr 17 22:08 .
drwxr-xr-x 3 root   root   4096 Sep  3  2017 ..
-rw------- 1 deploy deploy 1540 Mar 10 21:44 .bash_history
-rw-r--r-- 1 deploy deploy  220 Sep  3  2017 .bash_logout
-rw-r--r-- 1 deploy deploy 3771 Sep  3  2017 .bashrc
drwx------ 2 deploy deploy 4096 Sep  3  2017 .cache
-rw-r--r-- 1 deploy deploy    0 Sep  3  2017 .cloud-locale-test.skip
-rw------- 1 deploy deploy   41 Mar 10 21:18 .lesshst
-rw-r--r-- 1 deploy deploy  655 Sep  3  2017 .profile
drwxrwxr-x 2 deploy deploy 4096 Sep  3  2017 .ssh
-rw------- 1 deploy deploy  642 Sep  3  2017 .viminfo
-rw-r--r-- 1 deploy deploy  141 Apr 17 22:05 foo.tar.gz
drwxrwxr-x 2 deploy deploy 4096 Apr 17 21:49 tmp
deploy@shipit-test:~$ 

but --strip-components=1 is remove

deploy@shipit-test:~$ tar -xzf foo.tar.gz
deploy@shipit-test:~$ ls
foo.tar.gz  test.1587161041444.txt  tmp
deploy@shipit-test:~$

[oleg-andreyev]

It seems that scpCopyToRemote works okay performing a copy of a folder/, but it does not work okay with a single file.

[trashhalo]

I had this same problem. I was on 5.3.0. I downgraded and deploy one version at a time until it went away. The first version that correctly un tared the contents was 4.1.4. I wasn't able to install 4.2.0 because it wasn't available in npm.

https://github.com/shipitjs/shipit/compare/v4.1.4..v5.3.0

[rifqiazam]

I do have the exact same problem - my shipit.releasePath now looks like this:

-rw-rw-r-- 1 usr usr   27 Apr  1 16:15 package-lock.json
-rw-rw-r-- 1 usr usr   41 Apr  1 16:15 REVISION
drwxr-xr-x 4 usr usr 4096 Apr  1 16:14 tmp-31663xXIxG9Ls1APQ

me too, my releasepath look like that also. how do you fix that ? i have downgrade to 4.1.4 but doesnt solve my problem

[Bodom78]

I have the same issue with 5.3.0

In the meantime I just edited my local node_modules/ssh-pool/lib/commands/tar.js file and added back args = [...args, '--strip-components=1'] on line 39 so deployments continue to work.

[gregberge]

Hello, open to a PR to fix it.

[srowe0091]

Has anyone found a fix for this?

[oleg-andreyev]

I've created #279, which is showing that --strip-components=1 was initially wrong.

When you create a tar from a folder, this folder should be located in tar, and later when you extract it this folder must be present.

---

I and @gregberge made a mistake (B/C) by removing --strip-components=1 in the minor version.
As a possible solution to return --strip-components=1 in 5.3.1, and remove it in 6.0.0

[jeanbmar]

I've monkey patched like this:

shipit.blTask('fix-tar', () => {
    shipit.pool.connections.forEach((connection) => {
        const { runLocally } = connection;
        connection.runLocally = function(cmd, options) {
            if (cmd.includes('tar -xzf')) {
                cmd = cmd.replace('.tar.gz', '.tar.gz --strip-components=1');
            }
            return runLocally.call(this, cmd, options);
        }
    });
});

shipit.on('deploy', () => {
    return shipit.start(['fix-tar']);
});

But I have a bad feeling about Shipit when I see such a critical bug unfixed for so long :(