Batali cache: git sources problems building working tree and uncommitted changes #93

webframp · 2017-02-14T21:50:55Z

Filing an issue for tracking this one since it's kind of strange. Using latest code from #92 in an attempt to fix #88

Install fails for what appears to be a dirty git working tree for a git source, but the tree is actually clean.

[ERROR]: Reason - git "--git-dir=C:/Users/sescriva/AppData/Local/.batali/cache/git/Z2l0QGFwcHNnaXQuYmV0aGVsLmp3Lm9yZzpjaGVmL2Nvb2tib29rLWFwcHNlcnZlci5naXQ=/.git" "--work-tree=C:/Users/sescriva/AppData/Local/.batali/cache/git/Z2l0QGFwcHNnaXQuYmV0aGVsLmp3Lm9yZzpjaGVmL2Nvb2tib29rLWFwcHNlcnZlci5naXQ=" pull "origin" "master"  2>&1:error: cannot pull with rebase:
You have unstaged changes.
error: additionally, your index contains uncommitted changes.
error: please commit or stash them.
ERROR: Git::GitExecuteError: git "--git-dir=C:/Users/sescriva/AppData/Local/.batali/cache/git/Z2l0QGFwcHNnaXQuYmV0aGVsLmp3Lm9yZzpjaGVmL2Nvb2tib29rLWFwcHNlcnZlci5naXQ=/.git" "--work-tree=C:/Users/sescriva/AppData/Local/.batali/cache/git/Z2l0QGFwcHNnaXQuYmV0aGVsLmp3Lm9yZzpjaGVmL2Nvb2tib29rLWFwcHNlcnZlci5naXQ=" pull "origin" "master"  2>&1:error: cannot pull with rebase: You have unstaged changes.
error: additionally, your index contains uncommitted changes.
error: please commit or stash them.

C:\src\chef\infra-repo [batali/debug +2 ~1 -0 !]> cd "C:/Users/sescriva/AppData/Local/.batali/cache/git/Z2l0QGFwcHNnaXQuYmV0aGVsLmp3Lm9yZzpjaGVmL2Nvb2tib29rLWFwcHNlcnZlci5naXQ="

C:\Users\sescriva\AppData\Local\.batali\cache\git\Z2l0QGFwcHNnaXQuYmV0aGVsLmp3Lm9yZzpjaGVmL2Nvb2tib29rLWFwcHNlcnZlci5naXQ= [master ≡]> git status
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working tree clean

Scrubbing cache and re-installing results in same error as #88 (which was pre-existing):

C:\src\chef\infra-repo [batali/debug +2 ~1 -0 !]> be batali cache -s
[Batali]: Remove all contents from local cache (C:\Users\sescriva\AppData\Local/.batali/cache) (Y/N): Y
[Batali]: Scrubbing local cache... complete!
Batali cache information:
  Path: C:\Users\sescriva\AppData\Local/.batali/cache
  Size: 0.00M
C:\src\chef\infra-repo [batali/debug +2 ~1 -0 !]> be batali install
[Batali]: Readying installation destination... complete!
[Batali]: Installing cookbooks... error!
[ERROR]: Reason - No such file or directory - Failed to locate asset within `C:\Users\sescriva\AppData\Local/.batali/cache/site/remote_site/aHR0cHM6Ly9zdXBlcm1hcmtldC5jaGVmLmlvOjQ0My9hcGkvdjEvY29va2Jvb2tzL3dpbmRvd3MvdmVyc2lvbnMvMi4xLjEvZG93bmxvYWQ=`
ERROR: Errno::ENOENT: No such file or directory - Failed to locate asset within `C:\Users\sescriva\AppData\Local/.batali/cache/site/remote_site/aHR0cHM6Ly9zdXBlcm1hcmtldC5jaGVmLmlvOjQ0My9hcGkvdjEvY29va2Jvb2tzL3dpbmRvd3MvdmVyc2lvbnMvMi4xLjEvZG93bmxvYWQ=`

webframp · 2017-03-03T21:11:14Z

Oddly ran into a similar thing again, in our CI environment:

$ bundle exec batali install
[Batali]: Readying installation destination... complete!
[Batali]: Installing cookbooks... ERROR: Git::GitExecuteError: git '--git-dir=/home/gitlab-runner/.batali/cache/git/Z2l0QGFwcHNnaXQuYmV0aGVsLmp3Lm9yZzpjaGVmL2Nvb2tib29rLWJldGhlbF96YWJiaXguZ2l0/.git' '--work-tree=/home/gitlab-runner/.batali/cache/git/Z2l0QGFwcHNnaXQuYmV0aGVsLmp3Lm9yZzpjaGVmL2Nvb2tib29rLWJldGhlbF96YWJiaXguZ2l0' checkout 'ed03aee3721ff69158ed40fb10dd10ecdb01aff9'  2>&1:error: Your local changes to the following files would be overwritten by checkout:
	metadata.rb
Please, commit your changes or stash them before you can switch branches.
Aborting
error!
[ERROR]: Reason - git '--git-dir=/home/gitlab-runner/.batali/cache/git/Z2l0QGFwcHNnaXQuYmV0aGVsLmp3Lm9yZzpjaGVmL2Nvb2tib29rLWJldGhlbF96YWJiaXguZ2l0/.git' '--work-tree=/home/gitlab-runner/.batali/cache/git/Z2l0QGFwcHNnaXQuYmV0aGVsLmp3Lm9yZzpjaGVmL2Nvb2tib29rLWJldGhlbF96YWJiaXguZ2l0' checkout 'ed03aee3721ff69158ed40fb10dd10ecdb01aff9'  2>&1:error: Your local changes to the following files would be overwritten by checkout:
	metadata.rb
Please, commit your changes or stash them before you can switch branches.
Aborting

webframp · 2017-03-06T22:23:08Z

I don't know why the initial checkout of a repo would result in a git index out of sync with master but I've ran into two different scenarios. One were files are reported as deleted, and another where files are modifed.

Maybe something as dumb as this would be enough in the right place in: https://github.com/spox/batali/blob/master/lib/batali/git.rb#L19

if repo.status.changed.any?
  # whatever 'git checkout -- .' looks like using git gem
end

if repo.status.deleted.any?
  # whatever 'git reset HEAD' looks like using the git gem
end

webframp · 2017-03-08T20:20:29Z

Happening pretty consistently for me in our CI environment, let me know what sort of good debug info I could provide if needed

webframp · 2017-03-09T14:08:45Z

Here's a little more strangeness, same basic issue. First the error:

ERROR: Git::GitExecuteError: git '--git-dir=/home/gitlab-runner/.batali/cache/git/Z2l0QGFwcHNnaXQuYmV0aGVsLmp3Lm9yZzpjaGVmL2Nvb2tib29rLWJldGhlbF9wYWNrYWdpbmcuZ2l0/.git' '--work-tree=/home/gitlab-runner/.batali/cache/git/Z2l0QGFwcHNnaXQuYmV0aGVsLmp3Lm9yZzpjaGVmL2Nvb2tib29rLWJldGhlbF9wYWNrYWdpbmcuZ2l0' checkout '7682c1df1c070f5c3ff12654bae479b7f4913150'  2>&1:error: Your local changes to the following files would be overwritten by checkout:
	README.md
	attributes/default.rb
	files/artifactory.system.properties
	metadata.rb
	recipes/artifactory.rb
	spec/unit/recipes/artifactory_spec.rb
	test/integration/artifactory/inspec/artifactory_spec.rb
	test/integration/default/inspec/default_spec.rb
Please, commit your changes or stash them before you can switch branches.
error: The following untracked working tree files would be removed by checkout:
	files/artifactory.base.config.xml
	recipes/artifactory_config.rb
	recipes/artifactory_monitoring.rb
	spec/unit/recipes/artifactory_config_spec.rb
	spec/unit/recipes/artifactory_monitoring_spec.rb
Please move or remove them before you can switch branches.
Aborting

But then if I look at the actual repo in the cache it's fine:

$ cd /home/gitlab-runner/.batali/cache/git/Z2l0QGFwcHNnaXQuYmV0aGVsLmp3Lm9yZzpjaGVmL2Nvb2tib29rLWJldGhlbF9wYWNrYWdpbmcuZ2l0/
$ git status
On branch master
Your branch is up-to-date with 'origin/master'.

nothing to commit, working directory clean

Then if I simply rm -fr /home/gitlab-runner/.batali the next attempt gives this error:

ERROR: Git::GitExecuteError: git  clone '--' '[email protected]:chef/deploy_key_vault.git' '/home/gitlab-runner/.batali/cache/git/Z2l0QGFwcHNnaXQuYmV0aGVsLmp3Lm9yZzpjaGVmL2RlcGxveV9rZXlfdmF1bHQuZ2l0'  2>&1:fatal: working tree '/home/gitlab-runner/.batali/cache/git/Z2l0QGFwcHNnaXQuYmV0aGVsLmp3Lm9yZzpjaGVmL2Nvb2tib29rLWJldGhlbF9iYXNlLmdpdA==' already exists.

A bit stumped I guess.

webframp · 2017-03-31T04:25:46Z

Best guess at this point from a little testing: git repos where files have been committed with windows line endings are used as git sources in the Batali file and then batali is run on a linux system. Somewhere along the line the resulting git repo in the batali cache shows changes due to line ending differences.

If there was an option to always clobber the cache, or a --no-cache behavior, it might be enough to workaround it at the cost of being a little slower.

webframp · 2017-03-31T15:46:21Z

I forced core.eol = lf globally in a few places where I was seeing this but it didn't help

chrisroberts · 2017-04-02T14:15:18Z

@webframp After some git digging, it looks like this might fix things up:

git config core.autocrlf false

webframp · 2017-04-02T21:47:46Z

Unfortunately I tried that and still get the issues. Line endings with git are the most annoying cross platform issue. This page has more info too: http://adaptivepatchwork.com/2012/03/01/mind-the-end-of-your-line/

I still either get 'Your local changes to the following files would be overwritten by checkout' error, or I get the following, which seems related but looks like a different error:

$ rm -fr ~/.batali/
$ bundle exec batali install
[Batali]: Readying installation destination... complete!
[Batali]: Installing cookbooks... error!
[ERROR]: Reason - git  clone '--' '[email protected]:chef/cookbook-build_machine.git' '/home/sescriva/.batali/cache/git/Z2l0QGFwcHNnaXQuYmV0aGVsLmp3Lm9yZzpjaGVmL2Nvb2tib29rLWJ1aWxkX21hY2hpbmUuZ2l0'  2>&1:fatal: working tree '/home/sescriva/.batali/cache/git/Z2l0QGFwcHNnaXQuYmV0aGVsLmp3Lm9yZzpjaGVmL2Nvb2tib29rLWJldGhlbF9jaGVmX2NsaWVudC5naXQ=' already exists.
ERROR: Git::GitExecuteError: git  clone '--' '[email protected]:chef/cookbook-build_machine.git' '/home/sescriva/.batali/cache/git/Z2l0QGFwcHNnaXQuYmV0aGVsLmp3Lm9yZzpjaGVmL2Nvb2tib29rLWJ1aWxkX21hY2hpbmUuZ2l0'  2>&1:fatal: working tree '/home/sescriva/.batali/cache/git/Z2l0QGFwcHNnaXQuYmV0aGVsLmp3Lm9yZzpjaGVmL2Nvb2tib29rLWJldGhlbF9jaGVmX2NsaWVudC5naXQ=' already exists.

I'm probably going to have to stop using a git source to actually get past this which is ok. If I can help debug or step through things better let me know. I'm happy to monkey patch the installed batali with debug code even if you think it will help pinpoint the issue

webframp · 2017-04-04T14:38:12Z

@chrisroberts I might have a scenario to recreate this, will contact you on irc to discuss

webframp · 2017-04-10T18:02:06Z

Annoyingly, this does seem to be a line ending issue. Add a .gitattributes file to every git repo referenced by the Batali file "fixed" the issue. I just used a simple one with the following contents:

* text eol=native

webframp changed the title ~~"index contains uncommitted changes" with git source~~ Batali cache: git sources problems building working tree and uncommitted changes Mar 9, 2017

webframp mentioned this issue Mar 31, 2017

Force UNC style to support win long paths #95

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Batali cache: git sources problems building working tree and uncommitted changes #93

Batali cache: git sources problems building working tree and uncommitted changes #93

webframp commented Feb 14, 2017 •

edited

Loading

webframp commented Mar 3, 2017

webframp commented Mar 6, 2017 •

edited

Loading

webframp commented Mar 8, 2017

webframp commented Mar 9, 2017 •

edited

Loading

webframp commented Mar 31, 2017

webframp commented Mar 31, 2017

chrisroberts commented Apr 2, 2017

webframp commented Apr 2, 2017 •

edited

Loading

webframp commented Apr 4, 2017

webframp commented Apr 10, 2017

Batali cache: git sources problems building working tree and uncommitted changes #93

Batali cache: git sources problems building working tree and uncommitted changes #93

Comments

webframp commented Feb 14, 2017 • edited Loading

webframp commented Mar 3, 2017

webframp commented Mar 6, 2017 • edited Loading

webframp commented Mar 8, 2017

webframp commented Mar 9, 2017 • edited Loading

webframp commented Mar 31, 2017

webframp commented Mar 31, 2017

chrisroberts commented Apr 2, 2017

webframp commented Apr 2, 2017 • edited Loading

webframp commented Apr 4, 2017

webframp commented Apr 10, 2017

webframp commented Feb 14, 2017 •

edited

Loading

webframp commented Mar 6, 2017 •

edited

Loading

webframp commented Mar 9, 2017 •

edited

Loading

webframp commented Apr 2, 2017 •

edited

Loading