(beta4) Restore issue #58
Comments
|
Thanks for the report, Do you mind running the upload again with the |
|
@BarthV Thanks for this. Unfortunately, it looks like you only used a single |
|
ooops sry. |
|
It's quite long, hope it will help you to see what's going wrong :) |
|
@BarthV Hrm, if you attempt the restore with --concurrency 1 set on the command line, does it work? (It will take a longggg time). |
|
So, I've got a couple of comments - @BarthV, @stevendanna is asking you to run with concurrency 1 to see if you've hit a race condition on the Chef Server when uploading cookbooks. We've found a few of these when doing backup restores, and while we've fixed one there still seems to be one nagging one still outstanding that we haven't found a reproducible case for. By setting the concurrency to 1 you force cookbook uploads to be serial and avoid the race condition, but as a result it takes a long time. I just wanted to make the reasoning for this clear. @stevendanna My question for you is, based on the output given above, are we even in the cookbook upload code? The last statement is about clients. I realize this code isn't the best about it's output however, so we could be in the cookbook upload code. Or am I wrong on another front and the concurrency setting applies to more than just cookbook uploads? |
|
@mmzyk The last thing I see in the debug log is a sandbox upload. I don't actually see the error but sandbox uploads happen as part of cookbook uploads which is what got me thinking about the concurrency bug. We definitely need to level up the logging in this tool. I'm not sure if the default Chef logger is actually thread-safe, so hopefully setting concurrency to 1 will also make the output a bit more reasonable. |
|
no changes with --concurrency 1 : |
|
@BarthV Sorry to be a pain, but do you have any more of that log message? A bit further up it should have the URL that you were trying to hit with that request. Also, if you have access to the server logs, it might be useful to run the |
|
god ... it just works after a cleanse and an ec restore :/ Destroying my VM and trying it againt "from scratch" |
|
ok after a complete rebuild it fails again : I'll try to cleanse and reproduce the "good & complete" restore operation that I reach to make earlier. |
|
after a cleanse + reconfigure, ec restore worked well (it was the same backup) : |
|
@BarthV my hunch is that this is some sort of transient server-side failure occuring when opscode-erchef (the main API service) is talking to bookshelf (an internal service that provides an S3-compatible API for cookbook storage) rather than a bug in knife-ec-backup itself. That said, I'm happy to help look into it further if you would like. I think the next step from here would be to get the full server log dump from directly after the failure. Since the restore was working for you as of your last update, I also understand if you'd like to drop this for now. In the worst case we have the information you provided here as a reference if we see this pop up again. |
|
I'll try to do more tests and get chef server log if needed when i'll got some time :). |
|
erchef log: |
|
Hello, I just encountered the same symptoms with chef-server 12.14.0 and knife ec backup 2.2.3 EDIT: even after a cleanse + reconfigure |
|
As it turned out, I was uploading /<orgname>/containers/<objects>.json Not sure it's 100% related to the inital issue then |
|
@sixela I have the same problem with chef-server 12.15.0. How you solved this problem? |
|
Sorry devopstales but after 4y and not working with Chef in a while I don't recall |
Steps :
acmeorg,adminorg-admin andjdoeuser (withchef-server-ctlcli)Restore :
Result :
The text was updated successfully, but these errors were encountered: