iOS Release pipelines taking an hour

[IeuanWalker]

Description

iOS Azure CI pipeline taking an hour and causing it too fail -

All the steps are completed but takes almost an hour to produce a build -

Majority of the time it's stuck on -
Optimizing assemblies for size. This process might take a while.

Our XF iOS pipeline only takes 6m

Steps to Reproduce

trigger: none
variables:
- name: 'BUILD_CONFIGURATION'
  value: 'Release'
- name: 'DOTNET_VERSION'
  value: '8.0.x'
- name: 'FRAMEWORK_IOS'
  value: 'net8.0-ios'
- name: 'PROJECT_PATH'
  value: 'src/App/App.csproj'
- name: 'ARTIFACT_IOS_DROP_NAME'
  value: 'drop-ios'
- group: IosVariables

pool:
  vmImage: 'macos-14'
  demands: xcode

steps:
- script: |
    echo Mac OS version:
    sw_vers -productVersion
    echo
    echo Installed Xcode versions:
    ls /Applications | grep 'Xcode'
    echo
    echo currently selected xcode:
    xcrun xcode-select --print-path
    echo
    echo selecting latest xcode...
    sudo xcode-select -s /Applications/Xcode_16.0.app
    sudo xcode-select -switch /Applications/Xcode_16.app/Contents/Developer
    xcrun xcode-select --print-path
    xcodebuild -version
  displayName: Select Xcode Version
- task: UseDotNet@2
  displayName: .NET Version
  inputs:
    packageType: 'sdk'
    version: '$(DOTNET_VERSION)'
- task: CmdLine@2
  displayName: Install MAUI
  inputs:
    script: |
      dotnet nuget locals all --clear 
      dotnet workload install maui
- task: InstallAppleCertificate@2
  displayName: 'Get apple p12 certificate (Distribution)'
  inputs:
    certSecureFile: 'DistributionCertificate.p12'
    certPwd: '{PASSWORD}'
    deleteCert: true
- task: InstallAppleProvisioningProfile@1
  displayName: 'Get provisioning profile (CardiffGov-AdHoc for appcenter builds)'
  inputs:
    provisioningProfileLocation: 'secureFiles'
    provProfileSecureFile: 'CardiffGovAdHoc.mobileprovision'
    removeProfile: true
- task: DotNetCoreCLI@2
  displayName: 'Build the iOS Binaries'
  inputs:
    command: 'publish'
    publishWebProjects: false
    projects: 'src/App/App.csproj'
    arguments: '-f:net8.0-ios -c:Release /p:ArchiveOnBuild=true /p:EnableAssemblyILStripping=false'
    zipAfterPublish: false
- task: CopyFiles@2
  displayName: Copy IPA to Artifact Staging Directory
  inputs:
    Contents: '**/*.ipa'
    TargetFolder: '$(Build.ArtifactStagingDirectory)'
    flattenFolders: true
- task: PublishBuildArtifacts@1
  displayName: Publish IPA
  inputs:
    PathtoPublish: '$(Build.ArtifactStagingDirectory)'
    ArtifactName: '$(ARTIFACT_IOS_DROP_NAME)'
    publishLocation: 'Container'  

Link to public reproduction project repository

No response

Version with bug

8.0.91 SR9.1

Is this a regression from previous behavior?

Yes, this used to work in Xamarin.Forms

Last version that worked well

Unknown/Other

Affected platforms

iOS

Affected platform versions

No response

Did you find any workaround?

No response

Relevant log output

No response

[jfversluis]

I know the hosted build agents are a bit underpowered so it definitely might run for a while there, if you run it locally, how long does it run there? Do you know?

Not a great solution, but you can make the timeout longer with timeoutInMinutes on your task, or there might be a setting for your whole pipeline.

Looks like there is some additional info here too: actions/runner-images#10200

@rolfbjarne is there anything people can do here to speed this up?

[IeuanWalker]

@jfversluis thanks, looks like i found that issue a few months back when I first started to look at the pipelines. Its increased by 12m since then - actions/runner-images#10200 (comment)

Locally I think it only takes about 30m (still a massive jump compared to XF)

[IeuanWalker]

@jfversluis I presume its this setting that's taking up all the time?

if it is, do you know if there is an argument I can add to the publish command to turn it off so a build can be tested quicker when needed?

- task: DotNetCoreCLI@2
  displayName: 'Build the iOS Binaries'
  inputs:
    command: 'publish'
    publishWebProjects: false
    projects: 'src/App/App.csproj'
    arguments: '-f:net8.0-ios -c:Release /p:ArchiveOnBuild=true /p:EnableAssemblyILStripping=false'
    zipAfterPublish: false

[rolfbjarne]

You can disable llvm by passing /p:MtouchUseLlvm=false to dotnet publish, that should speed things up considerably.

[IeuanWalker]

thanks @rolfbjarne, that takes it down to about the same as the android pipeline -

[gabsamples6]

We are experienciing the same issue - no change in our pipeline

few months ago

Now

something has changed that increased this drammatically.

[rolfbjarne]

The best way to diagnose any differences would be if we could get an MSBuild binlog from an older (fast) build + newer (slow) build, that way we can compare the exact differences in the build (if any). The binlogs will also have timings, to see where the time is spend, which would be really useful in this scenario.

[gabsamples6]

@rolfbjarne I have a feeling this is not a bug but a ADO agent issue , bcse depending on the time of the day I get different timings

our nightly build took 23 mins

during the day it took 40-50 mins and also I no longer have a way to get the "old" binlog etc

but something is for sure its consistently slower and takes longer, I know this does not help you as you need "facts" binlogs to compare..

[IeuanWalker]

@rolfbjarne sorry for late reply, was on holiday 👌

Here is the .binlog file from a CI that took about 1h10m
msbuild.zip

Tried looking at the logs myself using (https://msbuildlog.com) but couldn't see how to use it/ search it.

---

Anyone else looking to generate the binlog file, I updated the above pipeline with -

 arguments: -f:net8.0-ios -c:Release /p:ArchiveOnBuild=true /p:MtouchUseLlvm=$(OptimiseBuild) /bl:msbuild.binlog -verbosity:diagnostic

  - task: CopyFiles@2
    displayName: Copy .binlog files
    inputs:
      Contents: '**/*.binlog'
      TargetFolder: '$(Build.ArtifactStagingDirectory)'
      flattenFolders: true

[rolfbjarne]

AOTCompile = 01:00:17.5566990, 1 calls.

The AOT compilation took an hour, which is pretty much all of it.

MtouchUseLlvm = true

Disabling LLVM should make it much faster. The LLVM part of the compilation is really taxes the CPU, so if the bots are already underpowered / under load, things might unravel rather drastically if the load becomes too much for them.

We automatically AOTCompile in parallel according to how many CPUs the machine has, which may cause problem if the machine is already under load. If this is the case, you can adjust the parallelism manually by setting the AotCompileMaxDegreeOfParallelism property to some number (how many parallel AOT compilation tasks should be run in parallel - there will be on AOT compilation task for every assembly in the app):

<PropertyGroup>
    <AotCompileMaxDegreeOfParallelism>4</AotCompileMaxDegreeOfParallelism>
</PropertyGroup>

Note: I'm assuming the before and after builds AOT-compile the same things and in the same way, but I can't confirm this without getting both a before and an after binlog.

[jfversluis]

No response, assuming this is fixed!