Test Apple Major OS releases 2024

[kahest]

Description

Description

iOS 18, tvOS 18, watchOS 11, macOS 15 are about to GA in September 2024. We should test whether our SDK works appropriately on these new OS versions.

Other tasks

Must-have - pre release

Beta Give feedback

1. Test plan manual iOS+macOS/MacCatalyst
2. Test plan manual tvOS
3. Test plan manual watchOS

Must-have - post release

Beta Give feedback

1. Test plan check with customer data

Optional / TBD

Beta Give feedback

1. Add SwiftData operations to iOS-Swift Sample #3105
   Platform: Cocoa +1
2. Enable "mergeable library" feature #3719
   Platform: Cocoa +1
   
3. Add sample app using mergeable libraries #3103
   Platform: Cocoa +1
   
4. Add viewIsAppearing to UIViewController transactions #3100
   Platform: Cocoa Type: Enhancement +2
5. Swift Macro for Instrumenting SwiftUI #3104
   Platform: Cocoa +1

Test plan manual test

Feature	iOS	macOS	tvOS	MacCatalyst
Crashes with symbolication	✅	✅		✅
Error events with symbolication	✅	✅		✅
OOMs	✅	🚫		🚫
App hangs	✅	🚫		❓
Release health	✅	✅		✅
View hierarchy	✅	🚫		✅
Screenshots	✅	🚫		✅
Auto-generated breadcrumbs	✅	✅		✅
Attachments	✅	✅		✅
App start	✅	🚫	☑️	✅
Slow & frozen frames	✅	🚫		✅
UIViewController tracking	✅	🚫		✅
TTID/TTFD	✅	🚫		✅
Network tracking	✅	🚫		✅
File I/O tracking	✅	🚫		✅
Core data tracking	✅	🚫	☑️	✅
User interaction tracking	✅	🚫		✅
Swift UI tracking	✅	✅		✅
MetricKit	✅	❓	☑️	✅
Profiling	❓	🚫	☑️	✅

🚫 = not available
☑️ = works on another platform. Not needed to put in the effort to test on this platform, because we are going to validate the feature with customer data.
✅ = tested successfully

Test plan check with customer data

Check features with customer data once iOS 18 is released. We only check iOS with customer data as things breaking there would have the most significant negative impact because we have, by far, the most projects on iOS. For the other platforms, the manual test above is sufficient.

Feature	iOS
Crashes with symbolication
Error events with symbolication
OOMs
App hangs
Release health
View hierarchy
Screenshots
Auto-generated breadcrumbs
Attachments
App start
Slow & frozen frames
UIViewController tracking
TTID/TTFD
Network tracking
File I/O tracking
Core data tracking
User interaction tracking
Swift UI tracking
MetricKit
Profiling

---

Similar to #3129

[denrase]

iOS

Setup

iPhone 13 Pro - iOS 18.0 (22A5350a) Public Beta
iPhone 16 Pro Simulator - iOS 18.0

Sentry 8.36.0 - TestFlight & Sentry 8.35.1 Debug build with Xcode 16.0

Results

Feature	Status	Notes
Crashes with symbolication	✅
Error events with symbolication	✅
OOMs	✅
App hangs	✅
Release Health	✅
View hierarchy	✅	No view hierarchy for crashes?
Auto-generated breadcrumbs	✅
Attachments	✅
App start	✅
Slow & frozen frames	✅
UIViewController tracking	✅
TTID/TTFD	✅
Network tracking	✅
File I/O tracking	✅
Core data tracking	✅	Are there supposed to be spans for record creation/deletion?
User interaction tracking	✅
Swift UI tracking	✅
MetricKit	✅
Profiling	❌	See comments below.

Issues / Questions

View hierarchy

No view hierarchy in crash, is this expected behaviour? ❓

https://sentry-sdks.sentry.io/issues/3607568922/?project=5428557&query=is%3Aunresolved+issue.priority%3A%5Bhigh%2C+medium%5D&referrer=issue-stream&statsPeriod=1h&stream_index=4

Core data tracking ❓

Are there supposed to be spans for record creation/deletion? ❓

Profiling ❌

Could not check, as the sample app crashes when trying to view profile data.

https://sentry-sdks.sentry.io/issues/5795748619/?project=5428557&query=is%3Aunresolved+issue.priority%3A%5Bhigh%2C+medium%5D&referrer=issue-stream&statsPeriod=1h&stream_index=0

ProfilesViewController shown with count 3, but there is no Flamegraph/Call Tree
https://sentry-sdks.sentry.io/performance/summary/profiles/?project=5428557&statsPeriod=1h&transaction=ProfilingViewController

Tested again, flame graphs don't seem useful to me, am i doing something wrong?

Other

• Device orientation seems to be incorrect
• https://sentry-sdks.sentry.io/issues/2862440042/events/d8050b41d5864f05af59d974be6ec7b4/?project=5428557&query=is%3Aunresolved+issue.priority%3A%5Bhigh%2C+medium%5D&referrer=next-event&statsPeriod=1h&stream_index=0

[denrase]

MacOS (Catalyst)

Setup

macOS Sequoia 15.0 Beta (24A5331b)
MacBook Pro 14-inch 2019 (M1)

Xcode Version 16.0 (16A242d)
Runing in debug mode.

Results

Feature	Status	Notes
Crashes with symbolication	✅	--
Error events with symbolication	✅	--
App hangs	❓	--
Release health	✅	--
View hierarchy	✅	--
Screenshots	✅	--
Auto-generated breadcrumbs	✅	--
Attachments	✅	--
App start	✅	--
Slow & frozen frames	✅	--
UIViewController tracking	✅	--
TTID/TTFD	✅	--
Network tracking	✅	--
File I/O tracking	✅	--
Core data tracking	✅	--
User interaction tracking	✅	--
Swift UI tracking	✅	--
MetricKit	✅	--
Profiling	✅	--

Issues / Questions

App hangs

• Could not provoke app hang errors for mac catalyst or simulator from the iOS13-Swift sample app. The wift sample app works in the simulator, so I'd assume it also works for catalyst.

Other

Cannot run iOS13-Swift sample application because Sentry is not found. Used the iOS13-Swift sample app instead.

[denrase]

watchOS

Did basic tests on watchOS, as the available features are part of the list above.

Setup

watchOS 11 (22R5348a)
Apple Watch Ultra 2
Version 16.0 beta 6 (16A5230g)
Runing in debug mode.

Results

Test	Result	Notes
captureMessage	✅	--
captureError	✅	--
captureException	✅	--
crash	❌	Not showing up on sentry.io. According to logs, envelope sent successfully.
auto breadcrumbs	✅	--
attachments	✅	--

Issues / Questions

crash

Could not get SentrySDK.crash() to show up on sentry.io .

[denrase]

macOS

Setup

macOS Sequoia 15.0 Beta (24A5331b)
MacBook Pro 15-inch 2019 (Intel)

Xcode Version 16.0 beta 6 (16A5230g)
Runing in debug mode.

Results

Feature	Status	Notes
Crashes with symbolication	✅	--
Error events with symbolication	✅	--
Release health	✅	--
Auto-generated breadcrumbs	✅	--
Attachments	✅	--
Swift UI tracking	✅	--
MetricKit	?	Could not provoke MetricKit issue. Debug option is disabled

Issues / Questions

MetricKit

Could not provoke MetricKit issue. Debug option is disabled

[philipphofmann]

All major OS versions have already been released. If there would be major issues, people would have reported these already. As almost everything was tested we can close this.