Restoring Cache : Failling to unarchive cache #66

josh-deprogram · 2020-05-21T06:23:03Z

Hitting problems when trying to rerun a build. the pre Yarn install step of Restoring the Cache is failing, is there a way around skipping this step,

Orb version

4.4.2

What happened

Fails to unarchive cache

Size: 324 MiB
Cached paths:
  * /tmp/yarn

Downloading cache archive...
Validating cache...

Unarchiving cache...

Failed to unarchive cache

Error untarring cache: Error extracting tarball /var/folders/1b/gl7yt7ds26vcyr1pkgld6l040000gn/T/cache984330157 : tmp/yarn/: Cannot extract through symlink tmp/yarn tmp/yarn/v4/: Cannot extract through symlink tmp/yarn/v4 tmp/yarn/v4/.tmp/: Cannot extract through symlink tmp/yarn/v4/.tmp tmp/yarn/v4/npm-@babel-code-frame-7.8.3-33e25903d7481181534e12ec0a25f16b6fcf419e/: Cannot extract through symlink tmp/yarn/v4/npm-@babel-code-frame-7.8.3-33e25903d7481181534e12ec0a25f16b6fcf419e tmp/yarn/v4/npm-@babel-code-frame-7.8.3-33e25903d7481181534e12ec0a25f16b6fcf419e/node_modules/: Cannot extract through symlink tmp/yarn/v4/npm-@babel-code-frame-7.8.3-33e25903d7481181534e12ec0a25f16b6fcf419e/node_modules tmp/yarn/v4/npm-@babel-code-frame-7.8.3-33e25903d7481181534e12ec0a25f16b6fcf419e/node_modules/@babel/: Cannot extract through symlink tmp/yarn/v4/npm-@babel-code-frame-7.8.3-33e25903d7481181534e12ec0a25f16b6fcf419e/node_modules/@babel tmp/yarn/v4/npm-@babel-code-frame-7.8.3-33e25903d7481181534e12ec0a25f16b6fcf419e/node_modules/@babel/code-frame/: Cannot extract through symlink tmp/yarn/v4/[email protected]: exit status 1

Expected behavior

Use previous cache and carry on with the Yarn install

The text was updated successfully, but these errors were encountered:

vonovak · 2020-05-23T21:07:24Z

@Naturalclar I see you recently touched this in #59. On what type of runner were you experiencing the problems (mac/linux)?. Any idea what can be wrong? I'm also seeing this, on a mac runner. Thanks.

azizhk · 2020-05-25T05:59:30Z

Started getting this error as well.
Orb version: 4.4.2

mtt87 · 2020-06-02T15:26:39Z

Confirm it's happening on 4.4.2 only on Mac environment for me

mtt87 · 2020-06-03T09:18:28Z

Some more info, in my case it says

Found a cache from build 559 at yarn-cache-arch1-darwin-amd64-6_58-...

But I'm now at build 592, is there a way to say if restoring the cache goes wrong then ignore it and install everything from scratch?

mtt87 · 2020-06-03T09:39:56Z

After reading this
https://support.circleci.com/hc/en-us/articles/115015426888-Clear-project-dependency-cache
I set an environment variable CACHE_VERSION with value v2 and it seems like it solved the issue.

chrisbruford · 2020-06-03T09:53:23Z

@mtt87 when you first set it, you have changed the comparitor and so it won't be able to find a valid cache to use...thus it creates a new one and everything works.

I found that on the next attempt to build it will use your value v2 and find a matching cache, try to unarchive it and you'll get the same error as before.

mtt87 · 2020-06-03T09:55:41Z

Damn, that's a good point I'll try again soon to verify if it still happening.

Have you tried to downgrade to 3.2.2 to see if it works?

chrisbruford · 2020-06-03T09:59:58Z

Yes - we downgraded and await a fix to this

MateusAndrade · 2020-06-04T14:27:12Z

Facing the same issue on xcode: 11.5.0 orb 4.4.2, solved running a simple yarn install --frozen-lockfile --non-interactive since I don't need to cache this folder at a specific step.

troZee · 2020-07-23T12:38:02Z

Facing the same issue here: https://app.circleci.com/pipelines/github/react-native-community/react-native-viewpager/296/workflows/f47a2400-fc43-4aec-aedd-a3cafb80ed3e/jobs/972

in this PR: callstack/react-native-pager-view#201

mtt87 · 2020-07-23T12:44:32Z

Latest working version for me it's 4.2.2 FYI in case you wanna downgrade

troZee · 2020-07-23T12:52:59Z

The same for 4.2.2 https://app.circleci.com/pipelines/github/react-native-community/react-native-viewpager/302/workflows/f9554c1e-5dd8-4f27-ac8c-9745aae7e63c/jobs/993

Android works fine, but iOS does not work.

mtt87 · 2020-07-23T12:57:49Z

Add an env variable to your project if you don't already have it CACHE_VERSION and use 2, or if you already have it, increase it.
I remember this fixed it for me with 4.2.2 and it still works today

jdewells · 2020-08-27T07:50:06Z

We're finding that we need to change the CACHE_VERSION for every run, and so far we've only found a way to do it manually by changing the project settings.

m2mathew · 2020-09-04T17:24:25Z

We're finding that we need to change the CACHE_VERSION for every run, and so far we've only found a way to do it manually by changing the project settings.

With version 4.4.2 I have the same experience with the rn/ios_build job. I set all of this up with Android weeks ago, and the save/restore from cache has been flawless. Setting up the iOS side has been more of a pain.

Here is how we are using the rn orb's job in our workflow. It mostly matches the examples in the documentation (some names withheld to protect the innocent):

- rn/ios_build:
    build_configuration: Debug
    detox_configuration: ios.sim.staging
    device: iPhone 11 Pro
    name: build_ios_debug
    node_version: '12.18'
    project_path: ios/ProjectName.xcworkspace
    project_type: workspace
    scheme: ProjectName-staging
    start_metro: true
    requires:
      - analyze_js
      # ↑ We changed that to the American spelling :) 🇺🇸

The build fails with an error like this:

Found a cache from build 2500 at yarn-cache-arch1-darwin-amd64-6_58-CtErNzrQbrt0boxXcoHOSPYkhQO77C0mVdN+cOWwdlQ=-pkNLpKB43oV0NLrI2IjZ4BpTEqiMriIdA7xqTvUeC04=-v3
Size: 378 MiB
Cached paths:
  * /tmp/yarn

Downloading cache archive...
Validating cache...

Unarchiving cache...

Failed to unarchive cache

Error untarring cache: Error extracting tarball /var/folders/1b/gl7yt7dS26vcyr1pkgld6l050000gn/T/cache789244379 : tmp/yarn/: Cannot extract through symlink tmp/yarn tmp/yarn/v4/: Cannot extract through symlink tmp/yarn/v4 tmp/yarn/v4/.tmp/: Cannot extract through symlink tmp/yarn/v4/.tmp tmp/yarn/v4/.tmp/ae76De6c779b0c17fc78aa8d370ed403/: Cannot extract through symlink tmp/yarn/v4/.tmp/ae76De6c779b0c17fc78aa8d370ed403 tmp/yarn/v4/.tmp/ae76De6c779b0c17fc78aa8d370ed403/.buckconfig: Cannot extract through symlink tmp/yarn/v4/.tmp/ae76De6c779b0c17fc78aa8d370ed403/.buckconfig tmp/yarn/v4/.tmp/ae76De6c779b0c17fc78aa8d370ed403/.editorconfig: Cannot extract through symlink tmp/yarn/v4/.tmp/ae76De6c779b0c17fc78aa8d370ed403/.editorconfig tmp/yarn/v4/.tmp/ae76De6c779b0c17fc78aa8d370ed403/.eslintignore: Cannot extract through symlink tmp/yarn/v4/.tmp/ae76De6c779b0c17fc78aa8d370ed403/.eslintignore tmp/yarn/v4/.tmp/ae76De6c779b0c17fc78aa8d370ed403/.eslintrc.js: Cannot extract through symlink tmp/yarn/v4/.tmp/ae76De6c779b0c17fc78aa8d370ed403/.eslintrc.js tmp/yarn/v4/.tmp/ae76De6c779b0c17fc78aa8d370ed403/.flowconfig: Cannot extract thro: exit status 1

We go to the Circle CI environment variables, remove the CACHE_VERSION, and then add new variable with the same name but an incremented value.
- e.g., remove CACHE_VERSION with value of v2 and add CACHE_VERSION with a value of v3
- example url for Circle Environment Variables – https://app.circleci.com/settings/project/github/your-org-name/your-repo-name/environment-variables

The cache is successfully restored on the next run, but will fail on subsequent runs. It seems like the restore from cache has somehow changed on a macos executor.

Next steps for me

My next steps are to use the individual commands from the rn orb mixed in with my own save/restore cache commands, though I would love to use the orb here instead!

donatoaguirre24 · 2020-12-09T21:42:20Z

Hi guys! Did any found a solution or workaround to this issue?

dedan · 2021-01-18T12:22:13Z

I'm also still running into the same issue

airtonix · 2021-01-27T06:16:38Z

Looking into this.

We had to make the swissknife and react-native orbs local to our setup (we have unoffical orbs disabled).

But this means I can make changes!

So, investigating leads me to these links:

https://circleci.com/docs/2.0/caching/#yarn-node
https://circleci.com/docs/2.0/yarn/
- outcome: don't use /tmp/yarn for the cache, it might be on a different FS that doesn't treat symlinks as you might expect.

The other thing I noticed was that for monorepos, caching will need to be more nuanced:

lukebrandonfarrell · 2021-01-27T12:33:05Z

We're finding that we need to change the CACHE_VERSION for every run, and so far we've only found a way to do it manually by changing the project settings.

With version 4.4.2 I have the same experience with the rn/ios_build job. I set all of this up with Android weeks ago, and the save/restore from cache has been flawless. Setting up the iOS side has been more of a pain.

Here is how we are using the rn orb's job in our workflow. It mostly matches the examples in the documentation (some names withheld to protect the innocent):
- rn/ios_build:
    build_configuration: Debug
    detox_configuration: ios.sim.staging
    device: iPhone 11 Pro
    name: build_ios_debug
    node_version: '12.18'
    project_path: ios/ProjectName.xcworkspace
    project_type: workspace
    scheme: ProjectName-staging
    start_metro: true
    requires:
      - analyze_js
      # ↑ We changed that to the American spelling :) 🇺🇸
The build fails with an error like this:
Found a cache from build 2500 at yarn-cache-arch1-darwin-amd64-6_58-CtErNzrQbrt0boxXcoHOSPYkhQO77C0mVdN+cOWwdlQ=-pkNLpKB43oV0NLrI2IjZ4BpTEqiMriIdA7xqTvUeC04=-v3
Size: 378 MiB
Cached paths:
  * /tmp/yarn

Downloading cache archive...
Validating cache...

Unarchiving cache...

Failed to unarchive cache

Error untarring cache: Error extracting tarball /var/folders/1b/gl7yt7dS26vcyr1pkgld6l050000gn/T/cache789244379 : tmp/yarn/: Cannot extract through symlink tmp/yarn tmp/yarn/v4/: Cannot extract through symlink tmp/yarn/v4 tmp/yarn/v4/.tmp/: Cannot extract through symlink tmp/yarn/v4/.tmp tmp/yarn/v4/.tmp/ae76De6c779b0c17fc78aa8d370ed403/: Cannot extract through symlink tmp/yarn/v4/.tmp/ae76De6c779b0c17fc78aa8d370ed403 tmp/yarn/v4/.tmp/ae76De6c779b0c17fc78aa8d370ed403/.buckconfig: Cannot extract through symlink tmp/yarn/v4/.tmp/ae76De6c779b0c17fc78aa8d370ed403/.buckconfig tmp/yarn/v4/.tmp/ae76De6c779b0c17fc78aa8d370ed403/.editorconfig: Cannot extract through symlink tmp/yarn/v4/.tmp/ae76De6c779b0c17fc78aa8d370ed403/.editorconfig tmp/yarn/v4/.tmp/ae76De6c779b0c17fc78aa8d370ed403/.eslintignore: Cannot extract through symlink tmp/yarn/v4/.tmp/ae76De6c779b0c17fc78aa8d370ed403/.eslintignore tmp/yarn/v4/.tmp/ae76De6c779b0c17fc78aa8d370ed403/.eslintrc.js: Cannot extract through symlink tmp/yarn/v4/.tmp/ae76De6c779b0c17fc78aa8d370ed403/.eslintrc.js tmp/yarn/v4/.tmp/ae76De6c779b0c17fc78aa8d370ed403/.flowconfig: Cannot extract thro: exit status 1
We go to the Circle CI environment variables, remove the CACHE_VERSION, and then add new variable with the same name but an incremented value.

e.g., remove CACHE_VERSION with value of v2 and add CACHE_VERSION with a value of v3

example url for Circle Environment Variables – https://app.circleci.com/settings/project/github/your-org-name/your-repo-name/environment-variables

The cache is successfully restored on the next run, but will fail on subsequent runs. It seems like the restore from cache has somehow changed on a macos executor.

Next steps for me

My next steps are to use the individual commands from the rn orb mixed in with my own save/restore cache commands, though I would love to use the orb here instead!

@m2mathew Appreciate the detailed explanation. I am very new to Circle CI and I couldn't for the life of me figure out how to set environment variables, for those having the same issue as me, i.e. where the hell do you set CACHE_VERSION, as Mathew suggested, it is in the Circle CI UI and here is the link:

https://app.circleci.com/settings/project/github/your-org-name/your-repo-name/environment-variables

lukebrandonfarrell · 2021-02-01T12:39:06Z

@airtonix Would you mind sharing your "swissknife and react-native orbs local to our setup" approach ?

Jose4gg · 2021-02-19T14:24:06Z

Hi, I'm dealing with this issue, and I have to change the CACHE_VERSION almost every new PR, do we have a better workaround to fix this bug in the meantime?

haruelrovix · 2021-03-04T01:36:16Z

Hi, I'm dealing with this issue, and I have to change the CACHE_VERSION almost every new PR, do we have a better workaround to fix this bug in the meantime?

Have you checked this post? @Jose4gg
https://support.circleci.com/hc/en-us/articles/360056920051--Error-untarring-cache-when-running-a-macOS-job

vineyardbovines · 2021-03-24T11:46:00Z

This is still happening, v5.0.0 on the rn/ios_build step.

My config at this step:

- rn/ios_build:
      name: build_ios_release
      project_path: ios/MyApp.xcodeproj
      device: "iPhone 12"
      build_configuration: Release
      scheme: MyApp
      requires:
        - analyze_js

Here's the entire error:

Found a cache from build 846 at yarn-cache-arch1-darwin-amd64-6_58-D0_AuAkbuS7gt7h+QkuuifdD4ggj_6o0mKXP8aewiEQ=-XUzc4oblUuD6VhRoBYEv7OCtEibCDWiNuAbxRwyDlRk=-v4
Size: 275 MiB
Cached paths:
  * /tmp/yarn

Downloading cache archive...
Validating cache...

Unarchiving cache...

Failed to unarchive cache

Error untarring cache: Error extracting tarball /var/folders/6y/gy9gggt14379c_k39vwb50lc0000gn/T/cache725626886 : tmp/yarn/: Cannot extract through symlink tmp/yarn tmp/yarn/v6/: Cannot extract through symlink tmp/yarn/v6 tmp/yarn/v6/.tmp/: Cannot extract through symlink tmp/yarn/v6/.tmp tmp/yarn/v6/[email protected]/: Cannot extract through symlink tmp/yarn/v6/[email protected] tmp/yarn/v6/[email protected]/node_modules/: Cannot extract through symlink tmp/yarn/v6/[email protected]/node_modules tmp/yarn/v6/[email protected]/node_modules/@babel/: Cannot extract through symlink tmp/yarn/v6/[email protected]/node_modules/@babel tmp/yarn/v6/[email protected]/node_modules/@babel/code-frame/: Cannot extract through symlink tmp/yarn/v6/[email protected]/node_modules/@babel/code-frame tmp/yarn/v6/[email protected]/node_modules/@babel/code-frame/.yarn-metadata.json: Cannot extract through symlink tmp/yarn/v6/[email protected]: exit status 1

I've opened a potential PR for this.

vonovak · 2021-03-24T12:37:26Z

Hello, I talked with Matt and he does not have time for OSS so he likely won't be responsive in this repo going forward.
While I'm also reducing my free time that I spend with open source, I will be available here to review and merge pull requests.

regarding this issue, I believe it was caused by #59

The solution is that someone who is bothered by this issue should investigate the proper fix and contribute a PR. Thank you for your understanding! 🙂

matt-oakes · 2021-03-24T12:54:07Z

Thanks for the message @vonovak! It's true that I'll likely be stepping back for this a bit more and I'm happy for any help that anyone can provide on this.

Seeing as this is a common issue, and I had a spare half hour, I've implemented a workaround to allow you to disable caching using parameters on the commands and jobs. Details are in this commit and it's released in v5.1.0.

This is only a workaround to allow you to move forward with no cache, but I agree that someone who's experiencing this issue should look into a proper fix which allows the cache to be kept there as it can vastly speed up build times.

vineyardbovines · 2021-03-24T14:50:42Z

Thanks @matt-oakes and @vonovak for the quick response. Being able to disable the cache will definitely help for now. I've submitted a fix (#88), however I'm not entirely sure this a full fix, but it does allow the caches to be used.

ronwsmith · 2021-07-13T18:16:31Z

I submitted PR #103 which made it into orb version 5.4.0. This allows you to undo the pathing change that seemed to have introduced this issue. You can achieve that by using yarn_cache_folder: ~/.cache/yarn in the ios_build or ios_build_and_test jobs.

I've run my build several times with this config change and haven't run into the permissions issue that drove that breaking change initially.

kieran-osgood · 2021-07-14T07:10:03Z

I submitted PR #103 which made it into orb version 5.4.0. This allows you to undo the pathing change that seemed to have introduced this issue. You can achieve that by using yarn_cache_folder: ~/.cache/yarn in the ios_build or ios_build_and_test jobs.

I've run my build several times with this config change and haven't run into the permissions issue that drove that breaking change initially.

As a note for anyone planning on utilising this new yarn_cache_folder, just be aware that changing to the orb v5.4.0 and adding the new cache param above may still match the old location /tmp/yarn if there exists a matching checksum (I just had this) - I just chose to upgrade a package I was already planning on upgrading anyway, so they stopped matching and now the restore goes through perfectly!

Thanks @ronwsmith - this was a real pain point when i got the orb setup a few months ago lol

bionara · 2021-11-24T15:49:09Z

Any update on this? getting the same error now.

dtran-ellevest · 2022-02-17T19:41:49Z

@kieran-osgood is there a way to get Circle to stop matching the old checksum without upgrading/removing a package?

kieran-osgood · 2022-02-17T19:47:09Z

@kieran-osgood is there a way to get Circle to stop matching the old checksum without upgrading/removing a package?

Not a clue im afraid!

josh-deprogram added the bug Something isn't working label May 21, 2020

vonovak mentioned this issue May 23, 2020

improve CI build times react-native-datetimepicker/datetimepicker#180

Closed

This comment has been minimized.

Sign in to view

vegaro mentioned this issue Sep 21, 2020

Version 3.4.0. Attribution v2 and nonSubscriptionTransactions RevenueCat/react-native-purchases#174

Merged

vineyardbovines mentioned this issue Mar 24, 2021

Replace cache version with branch in yarn_install command #88

Closed

vonovak added the help wanted Extra attention is needed label Mar 24, 2021

ronwsmith mentioned this issue Jul 9, 2021

Make yarn cache folder configurable #103

Merged

cryptonemo mentioned this issue Oct 29, 2021

Darwin CI Generated Parameters filecoin-project/rust-fil-proofs#1534

Closed

vegaro mentioned this issue Dec 29, 2021

New Example - PurchaseTester TypeScript RevenueCat/react-native-purchases#320

Merged

11 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Restoring Cache : Failling to unarchive cache #66

Restoring Cache : Failling to unarchive cache #66

josh-deprogram commented May 21, 2020 •

edited

Loading

vonovak commented May 23, 2020

azizhk commented May 25, 2020

mtt87 commented Jun 2, 2020

mtt87 commented Jun 3, 2020

mtt87 commented Jun 3, 2020

chrisbruford commented Jun 3, 2020

mtt87 commented Jun 3, 2020

chrisbruford commented Jun 3, 2020

MateusAndrade commented Jun 4, 2020

This comment has been minimized.

troZee commented Jul 23, 2020 •

edited

Loading

mtt87 commented Jul 23, 2020

troZee commented Jul 23, 2020

mtt87 commented Jul 23, 2020

jdewells commented Aug 27, 2020

m2mathew commented Sep 4, 2020 •

edited

Loading

donatoaguirre24 commented Dec 9, 2020

dedan commented Jan 18, 2021

airtonix commented Jan 27, 2021

lukebrandonfarrell commented Jan 27, 2021 •

edited

Loading

Next steps for me

lukebrandonfarrell commented Feb 1, 2021

Jose4gg commented Feb 19, 2021

haruelrovix commented Mar 4, 2021

vineyardbovines commented Mar 24, 2021 •

edited

Loading

vonovak commented Mar 24, 2021

matt-oakes commented Mar 24, 2021

vineyardbovines commented Mar 24, 2021

ronwsmith commented Jul 13, 2021

kieran-osgood commented Jul 14, 2021

bionara commented Nov 24, 2021

dtran-ellevest commented Feb 17, 2022

kieran-osgood commented Feb 17, 2022

Restoring Cache : Failling to unarchive cache #66

Restoring Cache : Failling to unarchive cache #66

Comments

josh-deprogram commented May 21, 2020 • edited Loading

Orb version

What happened

Expected behavior

vonovak commented May 23, 2020

azizhk commented May 25, 2020

mtt87 commented Jun 2, 2020

mtt87 commented Jun 3, 2020

mtt87 commented Jun 3, 2020

chrisbruford commented Jun 3, 2020

mtt87 commented Jun 3, 2020

chrisbruford commented Jun 3, 2020

MateusAndrade commented Jun 4, 2020

This comment has been minimized.

troZee commented Jul 23, 2020 • edited Loading

mtt87 commented Jul 23, 2020

troZee commented Jul 23, 2020

mtt87 commented Jul 23, 2020

jdewells commented Aug 27, 2020

m2mathew commented Sep 4, 2020 • edited Loading

Next steps for me

donatoaguirre24 commented Dec 9, 2020

dedan commented Jan 18, 2021

airtonix commented Jan 27, 2021

lukebrandonfarrell commented Jan 27, 2021 • edited Loading

Next steps for me

lukebrandonfarrell commented Feb 1, 2021

Jose4gg commented Feb 19, 2021

haruelrovix commented Mar 4, 2021

vineyardbovines commented Mar 24, 2021 • edited Loading

vonovak commented Mar 24, 2021

matt-oakes commented Mar 24, 2021

vineyardbovines commented Mar 24, 2021

ronwsmith commented Jul 13, 2021

kieran-osgood commented Jul 14, 2021

bionara commented Nov 24, 2021

dtran-ellevest commented Feb 17, 2022

kieran-osgood commented Feb 17, 2022

josh-deprogram commented May 21, 2020 •

edited

Loading

troZee commented Jul 23, 2020 •

edited

Loading

m2mathew commented Sep 4, 2020 •

edited

Loading

lukebrandonfarrell commented Jan 27, 2021 •

edited

Loading

vineyardbovines commented Mar 24, 2021 •

edited

Loading