Post: How to use CocoaPods with Xcode CI Bots

[kylef]

No description provided.

[kylef]

See https://twitter.com/SebThiebaud/status/397499004331642880

[mtitolo]

There's also a lot of info here: https://groups.google.com/d/msg/cocoapods/eYL8QB3XjyQ/10nmCRN8YxoJ

[Dimillian]

So now we have at least 2 technique that works (and I have 1 complicated one). What is the best way to do this ? Should we write a blog post with all these techniques or prepare the gem we have discussed on the community chat?

[alloy]

@kylef That tweet is unavailable :/

[mtitolo]

Looks like tweets were removed, but here's a screenshot from Tweetbot.
http://cl.ly/image/1y091n2l3n10

[amccarri]

I just finished setting up builds using Xcode bots this morning for our app which uses cocoa pods, however I will say that we are going to stick with Jenkins. So far my impression of bots is that they are meant for people who have never done CI before. They are EXTREMELY limited in capability (can’t monitor a build as it’s happening, build env can’t pass parameters to the build such as build numbers, can’t run scripts pre/post build etc, can’t administer a list of people allowed to download your test app, Xcode build list not auto-refreshed when new build is run, etc).

Looks like the only thing you can do is what you can add to your build scheme, and the only feature benefit is Xcode integration (which sucks so far, was not even able to create a bot in Xcode, had to use web page) so you can link from automated test runs to source code. Neither of these warrant moving from Jenkinsish/Testflight solutions.

All that is why we are NOT converting to bots, but if you want to use cocoa pods with bots here’s what I had to do:

1 - .go through the voodoo of setting up os x server and Xcode automation
- create home folder /var/teamsserver for _teamsserver user if it doesn’t already exist
- make sure to set proper ownerships on /var/teamsserver to _teamsserver account

2 - add .ssh with keys that can access your local pod repo to /var/teamsserver

3 - Set up new schemes specifically for Xcode bots
- edit scheme / build / pre-action add a bash script (or other script, your choice) that does something like
if ~/.cocoapods/repos/localrepo does not exist, pod repo add your local repo
if Pods folder does not exist pod install, else pod update
- share and check in the scheme

4 - Go http://yourserver/xcode and create the bot.
- create bots via xcode server web page… Xcode never let’s you get past authentication settings even when it shows them correct
(this may be because we don’t allow HTTP urls on our github server, only ssh, so our url looks like [email protected]:Team/App)

Fairly short list of steps, but OS X server docs were no help at all.

I’m really hoping this is just the starting point for bots, and that they will get a lot better in the future. Currently, even though jenkins is a much more complex CI server, I found myself pulling my hair out less setting up Jenkins than I did Xcode bots. Jenkins also supports something called a config matrix so you can run multiple build configs on a single checkout.

On Nov 7, 2013, at 5:05 PM, Michele [email protected] wrote:

Looks like tweets were removed, but here's a screenshot from Tweetbot.
http://cl.ly/image/1y091n2l3n10

—
Reply to this email directly or view it on GitHub.

[qnoid]

Thanks @amccarri for the detailed report! Much appreciated.
What about server's ability to run your tests on multiple devices? Isn't that enough to justify it over Jenkins?

[blakewatters]

I also took the Xcode bots for a test spin. I went a slightly different direction than @amccarri in my setup:

Rather than creating a home directory at /var/teamsserver, I overrode the CP_HOME_DIR and CP_REPOS_DIR environment variables to point to a location inside the build path based off of PROJECT_ROOT (there are a bunch of environment variables configured by the build so there are options about where to put things).

If your project contains private pods (I have a private pod repository), then you wind up needing to do double configuration of your SSH access. This is because the Xcode bots walk you through setting up an RSA key to clone your code, but that configuration is never exposed to the build once it begins. So you have to configure secured Git access separately from secured Pod access. This also applies for public Pods if you pull from Github via SSH instead of via HTTPS.

Another irritating wrinkle is that you need to use a globally accessible Ruby installation for CocoaPods. If you are working off of the system Ruby and have done sudo installations of CocoaPods then it will work. But it’s pretty common (and often advisable) to see RVM or rbenv used to install parallel copies of Ruby. If you have these installed into your home directory and expect to use the same bits, then you are SOL. I went with an rbenv installation of Ruby into /usr/local/var/rbenv

I also have Bundler managing some additional dependencies necessary for my build, so I wound up having to install the bundler dependencies into a path under the build directory.

All of this required setting up pre build scripts on a Shared scheme, then pushing it remotely and testing, which was extremely slow and painful to work through.

The build output also did not update until the build had failed, so there’s a lot of dead time waiting. Then once it was all set up, the feedback mechanisms were limited and it felt very much like a bunch of trouble for very little payoff when compared to Travis CI Pro or Jenkins.

As to the ability to run tests on multiple devices:

I am not convinced that there's actually any benefit in this over running your test suite across multiple versions of the Simulator. I've done tons of automated testing and CI on iOS over the last couple of years and the types of issues that we uncover on manual on-device testing are problems that are related to concurrency, unexpected user behaviors, and performance. The concurrency and user behavior problems won't become visible just because you run your suite on a device instead of the simulator and performance issues are typically perceputal -- a view "feels sluggish" or laggy. These kinds of conditions aren't testable in an automated way. There is also a major performance penalty to running your tests on real hardware. They are going to run much slower as you have slower processors, less memory, and less I/O throughput. The slower they are to execute, the lower your iteration throughput and the less you can lean on CI. So what's the benefit over the simulator?

After giving CI bots a thorough tire-kicking, I ultimately went back to Jenkins and haven't looked back. My Jenkins setup is simpler and much more powerful/flexible. Hope this saves someone some wasted hours.

[amccarri]

Not unique to bots, you can do this with jenkins or any other CI tool that can run a command line build. From the xcodebuild man page:

xcodebuild -workspace MyWorkspace.xcworkspace -scheme MyScheme -destination 'platform=iOS Simulator,name=iPhone' -destination
'platform=iOS,name=My iPad' test

I created a blog post out of my original email that includes a sample script for my bots config, can find it here:

http://alexlikescode.blogspot.com

On Nov 10, 2013, at 4:14 AM, Markos Charatzas [email protected] wrote:

Thanks @amccarri for the detailed report! Much appreciated.
What about server's ability to run your tests on multiple devices? Isn't that enough to justify it over Jenkins?

—
Reply to this email directly or view it on GitHub.

[mtitolo]

@blakewatters thanks for the info! What kind of advantage do you see with the environment variables instead of the folder in /var/teamsserver? Do you think this approach would be more robust and possibly future-proof for updates?

@amccarri We already have instructions for other CI systems. This post will be specifically on Bots because so many people have asked about them. This isn't saying we recommend using Bots, but we should provide instructions on how to use cocoapods with them since Apple will likely push people to use them.

[amccarri]

@mtitolo sorry should have directed my comment, @qnoid was asking about bots' ability to run tests on multiple devices as an advantage over other solutions. I was just pointing out that running tests on multiple devices is not necessarily a feature of bots, but rather a feature of xcodebuild which can be used anywhere.

[amccarri]

Oh btw, here's my build pre-action script for using cocoa pods with bots:

cd $SRCROOT

if [ ! -e "$HOME/.cocoapods/repos/OurRepo" ]
then
    pod repo add OurRepo git@ourgithub:OurApps/CocoaPodSpecs.git
fi

if [ -e "Pods" ]
then
    pod update
else
    pod install
fi

[orta]

So is this nixed for the minute?

[mtitolo]

It still needs to happen. I haven't had the time to actually write the post and double check all of the solutions.

[swizzlr]

BUMP.

[mtitolo]

@swizzlr if you want to be useful, you could try one of the solutions listed above.

[swizzlr]

Thanks, I will!

[xanderdunn]

Thanks @amccarri, your experience really helped me.

Mac OS X 10.9.2, Xcode 5.1b5, CocoaPods 0.29.0

I created /var/teamsserver and set it's owner to _teamsserver.

I created a scheme pre-action script:

export PATH=/usr/local/opt/ruby/bin:/usr/local/bin:$PATH
brew update
brew upgrade
gem update --system
gem update cocoa pods
cd ${SRCROOT}
if [ -e "Pods" ]
then
pod update
else
pod install
fi

It's working flawlessly. As a two-step set-up procedure, I'd say this is really not a problem.

[swizzlr]

Alex, thank you for summarizing that. I'll see if I can try wrapping it into a cp plugin.

[mtitolo]

Starting to work on the blog post for this. Comments welcome - https://github.com/CocoaPods/blog.cocoapods.org/blob/master/_drafts/CocoaPods-Bots.markdown

This won't use any of the plugins (which may or may not work with Server anyway).

[egueiros]

That's great work everyone. I have all my builds compiling and my tests passing. I have, however, one error in my 'Analyze' phase.
=== ANALYZE TARGET CI_TestProject OF PROJECT CI_TestProject WITH CONFIGURATION Debug ===
ld: library not found for -lPods
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Everything else is great. Pre-script is run, pods are installed and linked. But Analyze seems to be the very first thing that runs, and I get that error. Any pointers? Thanks in advance!

[moayes]

@egueiros not sure if this applies to your case. Check Build Active Architecture Only. In my case Pods project had Build Active Architecture Only set to YES for Debug builds, but in the main project it was set to NO. Changing Pods project flag to NO fixed the problem.

[mtitolo]

And another blog post for reference: http://chris.cm/setting-up-xcode-bots-with-cocoapods/

[m1entus]

@egueiros I have the same issue, ld: library not found for -lPods

[alloy]

@moayes You should normally change it on your project to YES, not set it to NO on the Pods project.

[m1entus]

I hacked something but right now i'm getting error when building some of static lib, locally everything is working fine :/:

   The following build commands failed:
CompileC /Library/Server/Xcode/Data/BotRuns/Cache/2ab22af0-2cb0-4b18-ac23-4e5edb40d808/DerivedData/Build/Intermediates/Pods.build/Debug-iphonesimulator/Pods-SDWebImage.build/Objects-normal/i386/NSData+ImageContentType.o SDWebImage/SDWebImage/NSData+ImageContentType.m normal i386 objective-c com.apple.compilers.llvm.clang.1_0.compiler
(1 failure)

Additionaly if i uncheck find implicit dependencies on shared target i'm getting

ld: library not found for -lPods
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Update #1:
Everything working on Debug (XCode), but when i changed target to release, it says that: ld: library not found for -lPods

[MatejBalantic]

@egueiros & @m1entus: I have the same problem:
ld: library not found for -lPods

Did you find any solution?

[thebarndog]

Ah I was so close, I set the git remote to incorrectly and now having set it back correctly, I'm back at this error:

Cloning into 'myproject'...
  fatal: could not read Username for 'https://github.com': Device not configured

Oh well, it figures apple would make this such a laborious process.

[joeblau]

What is the URI of your remote?

[thebarndog]

https://github.com/busycm/ios and there's three private repos, one is the private specs repo and the other two are just pods/

[joeblau]

When I setup private repos, I use the ssh URI ([email protected]:busycm/ios) and then add the public and private keys to the _xcsbuildd user. You shouldn't use https unless you have a really good reason to put your username/password in the clear.

[thebarndog]

Do you mean you set the git remote to be [email protected]/busycm/ios by doing git remote add origin [email protected]/busy/ios? And not [email protected]:busycm/ios?

[joeblau]

The version with the semicolon before your project/username.

[thebarndog]

Yeah not a clue. I went through and changed the remote of all my cocoapods to use ssh and re-added those into xcode using my public key authentication but it seems to be a lost cause.

[czechboy0]

Also there needs to be .git at the end, like
http://[email protected]/busycm/ios.git
On Thu, Oct 1, 2015 at 8:27 PM Brendan Conron [email protected]
wrote:

Yeah not a clue. I went through and changed the remote of all my cocoapods
to use ssh and re-added those into xcode using my public key authentication
but it seems to be a lost cause.

—
Reply to this email directly or view it on GitHub
#21 (comment)
.

[thebarndog]

Not http:// though, doesn't that defeat the whole purpose of using ssh?

[joeblau]

@startupthekid You can copy the correct URL directly from GitHub's web UI.

You also need to add the public/private key to _xcsbuildd user so that you can do the post setup clone via Cocoapods.

[thebarndog]

I have a public/private key for the _xcsbuildd user but does that key pair have to be the same as my regular user for github?

[czechboy0]

No, as long as you copied that public key to GitHub.

And yes, the http was not supposed to be there, rather it should be
[email protected]/busycm/ios.git
On Thu, Oct 1, 2015 at 8:57 PM Brendan Conron [email protected]
wrote:

I have a public/private key for the _xcsbuildd user but does that key
pair have to be the same as my regular user for github?

—
Reply to this email directly or view it on GitHub
#21 (comment)
.

[thebarndog]

Ok cool. I ended up doing a complete clean of all my repos and set everything back the way it was and now I'm getting this:

[!] Error installing BZYBinders
[!] /usr/local/bin/git clone https://github.com/busycm/BZYBinders.git /var/folders/82/xnzs001n09dccsqt27tn7lsw000087/T/d20151001-59388-kvmypg --single-branch --depth 1 --branch 0.5.4

Cloning into '/var/folders/82/xnzs001n09dccsqt27tn7lsw000087/T/d20151001-59388-kvmypg'...
fatal: could not read Username for 'https://github.com': Device not configured

(BZYBinders is one of my private pods)

I think the solution here is to make all my private pods public and see if that works.

[czechboy0]

Before you do that, can you log in as _xcsbuildd and just try git clone [email protected]/busycm/ios.git? If that works, then the issue is not the SSH keys. If it doesn't, please make sure to go through the tutorial again to make sure that you got it all setup. Should hopefully make it easier to spot the issue.

Making Pods public is really just a workaround, but I've been using private pods for years without a problem, so I know it's possible :)

[thebarndog]

Cloning works super super duper!

Do you think that possibly it's because my project has a private pod that uses another private pod as a dependency? I don't know how that would affect it but it seems to be unique from what most people are trying.

And yeah! I'd love to be able to use private pods and Xcode bots. I've tried jenkins but that was a pain because you can only run in on a mac server and whatnot.

[czechboy0]

Then your SSH keys are setup correctly! So it must be something with CocoaPods. Did you specify both private spec-repos with the source command in your Podfile? I'm sure other people here can advice you better on how to debug this - but when it comes to Xcode Server & SSH keys, you're all set.

[thebarndog]

Yup I put

source '[email protected]:CocoaPods/Specs.git'
source '[email protected]:busycm/CocoapodsPrivateSpecs.git'

at the top of all my Podfiles, not just my project but all my private pods as well to be on the safe side.

[czechboy0]

my project has a private pod that uses another private pod as a dependency

AFAIK that shouldn't be a problem as long as both pods have their specs in one of the specified repositories.

Anyway - I actually think you should create a new ticket with your specific setup, I'm sure you'll get help there quicker (this thread is otherwise getting off-topic).

[thebarndog]

👍

[kaosdg]

If @startupthekid is still having issues with this, I had a similar problem:

My private Pods also brought in other private Pods. Part of the problem is how the sources for the .podspecs are configured. Based on your error message, your podspec's s.source is https://, so it won't use the SSH Keys here.

What's going to happen is the process is going to prompt for the username and password to be used.
On a normal user, you would type that in, then be asked if you want to add it to your keychain (so you're not prompted again)
Since _xcsbuildd is headless, you get the "device not configured" issue. (fatal: could not read Username for 'https://github.com': Device not configured)

The way I worked around it was to copy my internet password from the login Keychain into the System keychain (making sure the ACL's were set correctly). Kind of a hack workaround, until I can figure out how to add internet and app passwords to _xcsbuildd's login keychain.
(PSA: DON'T CHANGE _xcsbuildd's password.)

[thebarndog]

@kaosdg Yeah the issue for me was that my private pod included other private pods of mine as dependencies so including them via https was throwing the error. Using git@ worked just fine.

[joeblau]

The new version of Xcode Server fixes this build issue with an actual user:

https://developer.apple.com/videos/play/wwdc2016/409/

[Vadimkomis]

Something that worked for me:
Software: MacOS Sierra, Server 5.2, Xcode 8
export PATH=$PATH:/usr/local/bin
cd
pod install

[JessieHu]

I used Sever & Bots in Xcode 9, but It showed an error when integrating.
The error was "Trigger Error Trigger exited with non-zero status 134."

Below was my Pre-Integration Scripts:

export LANG=en_US.UTF-8
export PATH=/usr/local/bin:$PATH
cd /Users/jessie/proj_bsoft/hcn-ios-master-pub/hcn-ios-public-dev
pod update --verbose

Below was part of the Log:

Generating Pods project
  - Creating Pods project
  - Adding source files to Pods project
  - Adding frameworks to Pods project
  - Adding libraries to Pods project
  - Adding resources to Pods project
  - Linking headers
  - Installing targets
    - Installing target `AFNetworking` iOS 7.0
    - Installing target `AJKBusiness` iOS 7.0
    - Installing target `AJKCommon` iOS 7.0
    - Installing target `AJKFrame` iOS 7.0
    - Installing target `FMDB` iOS 4.3
    - Installing target `FMDBMigrationManager` iOS 7.0
    - Installing target `FSCalendar` iOS 7.0
    - Installing target `HealthMonitor` iOS 7.0
    - Installing target `IQKeyboardManager` iOS 8.0
    - Installing target `MBProgressHUD` iOS 6.0
    - Installing target `MJExtension` iOS 6.0
    - Installing target `MJRefresh` iOS 6.0
    - Installing target `Masonry` iOS 6.0
    - Installing target `SDCycleScrollView` iOS 7.0
    - Installing target `SDWebImage` iOS 7.0
    - Installing target `TZImagePickerController` iOS 6.0
    - Installing target `YTKNetwork` iOS 7.0
/var/folders/nw/jbl3gsns345gpj2h9ykksdzw0000gn/T/14AD2EE0-8562-4F39-AAD4-B2E2826B9FDA-489-000018E09C80D691: line 6: 90050 Abort trap: 6           pod update --verbose

I don't know why it abort? How can I fix this error?

[lolgear]

Any updates?

I would like to know complete steps to setup bot with cocoa pods dependencies.
It would be nice if parts of these steps are compiled into Wiki page.

[nick3389]

@lolgear I agree. I still get the error for podfile.lock not found and the sandbox is not in sync....
Is there any complete guide?

[Apocryphon]

@nick3389 I think that issue is happening in Build Phase [CP] Check Pods Manifest.lock:

diff "${PODS_PODFILE_DIR_PATH}/Podfile.lock" "${PODS_ROOT}/Manifest.lock" > /dev/null
if [ $? != 0 ] ; then
    # print error to STDERR
    echo "error: The sandbox is not in sync with the Podfile.lock. Run 'pod install' or update your CocoaPods installation." >&2
    exit 1
fi
# This output is used by Xcode 'outputs' to avoid re-running this script phase.
echo "SUCCESS" > "${SCRIPT_OUTPUT_FILE_0}"

Don't know how to resolve this, did you get past it?

[nick3389]

@Apocryphon Yes everything is ok now!. Thanx!

[secretpromise]

I use Xcode 9.4.1, I found a solution:
1- a different account (like xcodeserver above) need to log in, then
2- the account need to install cocoapods, then
3- run pod setup

log in as who create Bot if needed,
3. edit Bot, add a Pre-Integration Script:

#!/bin/sh
# use env to print all environment var
cd $(basename "${XCS_PRIMARY_REPO_DIR}")
# the cocoapods exec's path, I use rbenv, you should change to your own
~/.rbenv/shims/pod install

[pmvcaqr]

@nick3389
I have the same issue with Build Phase [CP] Check Pods Manifest.lock
How do you get it pass?