Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Refactor loading of snapshots #20632

Merged
merged 32 commits into from
Apr 16, 2024
Merged
Show file tree
Hide file tree
Changes from 25 commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
a633392
feat: Refactor loading of snapshots
benjackwhite Feb 29, 2024
ab797d2
Update UI snapshots for `chromium` (2)
github-actions[bot] Feb 29, 2024
4d1c7a7
Update query snapshots
github-actions[bot] Feb 29, 2024
6543e07
Update UI snapshots for `chromium` (2)
github-actions[bot] Feb 29, 2024
3a4c063
Update query snapshots
github-actions[bot] Feb 29, 2024
93353b5
Update UI snapshots for `chromium` (2)
github-actions[bot] Feb 29, 2024
1a7ed41
Update UI snapshots for `chromium` (2)
github-actions[bot] Feb 29, 2024
ce82531
Update UI snapshots for `webkit` (2)
github-actions[bot] Feb 29, 2024
f53b69c
Update UI snapshots for `webkit` (2)
github-actions[bot] Feb 29, 2024
2e95887
Fixes
benjackwhite Feb 29, 2024
1c33af8
Fix tests
benjackwhite Feb 29, 2024
d0cfc9b
Fixes
benjackwhite Feb 29, 2024
aa37022
Update UI snapshots for `chromium` (2)
github-actions[bot] Feb 29, 2024
bdb7775
Fix
benjackwhite Feb 29, 2024
76d4e5b
Fixes
benjackwhite Feb 29, 2024
a73dc1b
Update UI snapshots for `chromium` (2)
github-actions[bot] Feb 29, 2024
b8583de
Updated logs
benjackwhite Feb 29, 2024
6623d03
Fix file size check
benjackwhite Feb 29, 2024
d76809a
Merge branch 'master' into fix/v3-snapshots
benjackwhite Mar 5, 2024
cce7b1e
Update UI snapshots for `webkit` (2)
github-actions[bot] Mar 5, 2024
78f9067
Update UI snapshots for `webkit` (2)
github-actions[bot] Mar 5, 2024
079a6cb
Update UI snapshots for `chromium` (1)
github-actions[bot] Mar 5, 2024
4fd09e5
Update UI snapshots for `webkit` (2)
github-actions[bot] Mar 5, 2024
9c8b11e
Update UI snapshots for `chromium` (1)
github-actions[bot] Mar 5, 2024
6d126ff
Merge branch 'master' into fix/v3-snapshots
daibhin Mar 28, 2024
7d75dc8
minor fixes from review
daibhin Apr 2, 2024
64f0e01
Merge branch 'master' into fix/v3-snapshots
daibhin Apr 4, 2024
60447d4
Merge branch 'master' into fix/v3-snapshots
daibhin Apr 5, 2024
242fdd6
remove etag changes
daibhin Apr 5, 2024
e3307f4
remove unnecessary line break
daibhin Apr 5, 2024
713574c
fix comment
daibhin Apr 5, 2024
8dc8f36
Merge branch 'master' into fix/v3-snapshots
daibhin Apr 16, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've shipped some smaller parts of this change in previous PRs and did a thorough review today. It's pretty difficult to review 100% but I'm fairly confident it does what it's intended to. My plan is to test it locally but think it's probably worth shipping at this point and following up with any fixes forward we need.

@benjackwhite is there anything you'd like to do before we ship? There's still a few TODOs knocking about

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just don't have any spare capacity for it so its fully in your hands

Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ export const convertSnapshotsResponse = (
snapshotsByWindowId: { [key: string]: eventWithTime[] },
existingSnapshots?: RecordingSnapshot[]
): RecordingSnapshot[] => {
return deduplicateSnapshots(convertSnapshotsByWindowId(snapshotsByWindowId), existingSnapshots)
return deduplicateSnapshots([...convertSnapshotsByWindowId(snapshotsByWindowId), ...(existingSnapshots ?? [])])
pauldambra marked this conversation as resolved.
Show resolved Hide resolved
}

export const sortedRecordingSnapshots = (): { snapshot_data_by_window_id: Record<string, RecordingSnapshot[]> } => {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,29 +16,10 @@ import {
parseEncodedSnapshots,
sessionRecordingDataLogic,
} from '../player/sessionRecordingDataLogic'
import type { sessionRecordingDataLogicType } from '../player/sessionRecordingDataLogicType'
import { sessionRecordingDataLogicType } from '../player/sessionRecordingDataLogicType'
import type { sessionRecordingFilePlaybackLogicType } from './sessionRecordingFilePlaybackLogicType'
import { ExportedSessionRecordingFileV1, ExportedSessionRecordingFileV2 } from './types'

export const createExportedSessionRecording = (
logic: BuiltLogic<sessionRecordingDataLogicType>,
// DEBUG signal only, to be removed before release
exportUntransformedMobileSnapshotData: boolean
): ExportedSessionRecordingFileV2 => {
const { sessionPlayerMetaData, sessionPlayerSnapshotData } = logic.values

return {
version: '2023-04-28',
data: {
id: sessionPlayerMetaData?.id ?? '',
person: sessionPlayerMetaData?.person,
snapshots: exportUntransformedMobileSnapshotData
? sessionPlayerSnapshotData?.untransformed_snapshots || []
: sessionPlayerSnapshotData?.snapshots || [],
},
}
}

export const parseExportedSessionRecording = (fileData: string): ExportedSessionRecordingFileV2 => {
const data = JSON.parse(fileData) as ExportedSessionRecordingFileV1 | ExportedSessionRecordingFileV2

Expand Down Expand Up @@ -163,9 +144,13 @@ export const sessionRecordingFilePlaybackLogic = kea<sessionRecordingFilePlaybac
await parseEncodedSnapshots(values.sessionRecording.snapshots, values.sessionRecording.id)
)

dataLogic.actions.loadRecordingSnapshotsSuccess({
snapshots,
// Simulate a loaded source and sources so that nothing extra gets loaded
dataLogic.actions.loadSnapshotsForSourceSuccess({
snapshots: snapshots,
untransformed_snapshots: snapshots,
source: { source: 'file' },
})
dataLogic.actions.loadSnapshotSourcesSuccess([{ source: 'file' }])

dataLogic.actions.loadRecordingMetaSuccess({
id: values.sessionRecording.id,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ import { SimilarRecordings } from './SimilarRecordings'

const PlayerFrameOverlayContent = (): JSX.Element | null => {
const { currentPlayerState } = useValues(sessionRecordingPlayerLogic)

let content = null
const pausedState =
currentPlayerState === SessionPlayerState.PAUSE || currentPlayerState === SessionPlayerState.READY
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -172,7 +172,7 @@ export const playerInspectorLogic = kea<playerInspectorLogicType>([
[
'sessionPlayerData',
'sessionPlayerMetaDataLoading',
'sessionPlayerSnapshotDataLoading',
'snapshotsLoading',
'sessionEventsData',
'sessionEventsDataLoading',
'windowIds',
Expand Down Expand Up @@ -856,7 +856,7 @@ export const playerInspectorLogic = kea<playerInspectorLogicType>([
(s) => [
s.sessionEventsDataLoading,
s.sessionPlayerMetaDataLoading,
s.sessionPlayerSnapshotDataLoading,
s.snapshotsLoading,
s.sessionEventsData,
s.consoleLogs,
s.allPerformanceEvents,
Expand All @@ -865,27 +865,27 @@ export const playerInspectorLogic = kea<playerInspectorLogicType>([
(
sessionEventsDataLoading,
sessionPlayerMetaDataLoading,
sessionPlayerSnapshotDataLoading,
snapshotsLoading,
events,
logs,
performanceEvents,
doctorEvents
): Record<SessionRecordingPlayerTab, 'loading' | 'ready' | 'empty'> => {
const tabEventsState = sessionEventsDataLoading ? 'loading' : events?.length ? 'ready' : 'empty'
const tabConsoleState =
sessionPlayerMetaDataLoading || sessionPlayerSnapshotDataLoading || !logs
sessionPlayerMetaDataLoading || snapshotsLoading || !logs
? 'loading'
: logs.length
? 'ready'
: 'empty'
const tabNetworkState =
sessionPlayerMetaDataLoading || sessionPlayerSnapshotDataLoading || !performanceEvents
sessionPlayerMetaDataLoading || snapshotsLoading || !performanceEvents
? 'loading'
: performanceEvents.length
? 'ready'
: 'empty'
const tabDoctorState =
sessionPlayerMetaDataLoading || sessionPlayerSnapshotDataLoading || !performanceEvents
sessionPlayerMetaDataLoading || snapshotsLoading || !performanceEvents
? 'loading'
: doctorEvents.length
? 'ready'
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,14 +30,12 @@ const BLOB_SOURCE: SessionRecordingSnapshotSource = {
start_timestamp: '2023-08-11T12:03:36.097000Z',
end_timestamp: '2023-08-11T12:04:52.268000Z',
blob_key: '1691755416097-1691755492268',
loaded: false,
}
const REALTIME_SOURCE: SessionRecordingSnapshotSource = {
source: 'realtime',
start_timestamp: '2024-01-28T21:19:49.217000Z',
end_timestamp: undefined,
blob_key: undefined,
loaded: false,
}

describe('sessionRecordingDataLogic', () => {
Expand Down Expand Up @@ -115,11 +113,14 @@ describe('sessionRecordingDataLogic', () => {
it('loads all data', async () => {
await expectLogic(logic, () => {
logic.actions.loadRecordingMeta()
logic.actions.loadRecordingSnapshots()
logic.actions.loadSnapshots()
})
.toDispatchActions([
'loadSnapshots',
'loadSnapshotSources',
'loadRecordingMetaSuccess',
'loadRecordingSnapshotsSuccess',
'loadSnapshotSourcesSuccess',
'loadSnapshotsForSourceSuccess',
'reportUsageIfFullyLoaded',
])
.toFinishAllListeners()
Expand Down Expand Up @@ -165,7 +166,7 @@ describe('sessionRecordingDataLogic', () => {

it('fetch metadata success and snapshots error', async () => {
silenceKeaLoadersErrors()
// Unmount and remount the logic to trigger fetching the data again after the mock change
// Unmount and r emount the logic to trigger fetching the data again after the mock change
logic.unmount()
useMocks({
get: {
Expand All @@ -174,9 +175,9 @@ describe('sessionRecordingDataLogic', () => {
})
logic.mount()
logic.actions.loadRecordingMeta()
logic.actions.loadRecordingSnapshots()
logic.actions.loadSnapshots()

await expectLogic(logic).toDispatchActions(['loadRecordingMetaSuccess', 'loadRecordingSnapshotsFailure'])
await expectLogic(logic).toDispatchActions(['loadRecordingMetaSuccess', 'loadSnapshotSourcesFailure'])
expect(logic.values.sessionPlayerData).toMatchObject({
person: recordingMetaJson.person,
durationMs: 11868,
Expand Down Expand Up @@ -219,7 +220,7 @@ describe('sessionRecordingDataLogic', () => {
})

await expectLogic(logic, () => {
logic.actions.loadRecordingSnapshots()
logic.actions.loadSnapshots()
}).toDispatchActions(['loadEvents', 'loadEventsSuccess'])

expect(api.create).toHaveBeenCalledWith(
Expand Down Expand Up @@ -255,21 +256,21 @@ describe('sessionRecordingDataLogic', () => {
describe('report usage', () => {
it('sends `recording loaded` event only when entire recording has loaded', async () => {
await expectLogic(logic, () => {
logic.actions.loadRecordingSnapshots()
logic.actions.loadSnapshots()
})
.toDispatchActionsInAnyOrder([
'loadRecordingSnapshots',
'loadRecordingSnapshotsSuccess',
'loadSnapshots',
'loadSnapshotsForSourceSuccess',
'loadEvents',
'loadEventsSuccess',
])
.toDispatchActions([eventUsageLogic.actionTypes.reportRecording])
})
it('sends `recording viewed` and `recording analyzed` event on first contentful paint', async () => {
await expectLogic(logic, () => {
logic.actions.loadRecordingSnapshots()
logic.actions.loadSnapshots()
})
.toDispatchActions(['loadRecordingSnapshotsSuccess'])
.toDispatchActions(['loadSnapshotsForSourceSuccess'])
.toDispatchActionsInAnyOrder([
eventUsageLogic.actionTypes.reportRecording, // loaded
eventUsageLogic.actionTypes.reportRecording, // viewed
Expand All @@ -278,7 +279,7 @@ describe('sessionRecordingDataLogic', () => {
})
it('clears the cache after unmounting', async () => {
await expectLogic(logic, () => {
logic.actions.loadRecordingSnapshots()
logic.actions.loadSnapshots()
})
expect(Object.keys(logic.cache)).toEqual(
expect.arrayContaining(['metaStartTime', 'snapshotsStartTime', 'eventsStartTime'])
Expand Down Expand Up @@ -326,7 +327,9 @@ describe('sessionRecordingDataLogic', () => {
},
]
// we call this multiple times and pass existing data in, so we need to make sure it doesn't change
expect(deduplicateSnapshots(verySimilarSnapshots, verySimilarSnapshots)).toEqual(verySimilarSnapshots)
expect(deduplicateSnapshots([...verySimilarSnapshots, ...verySimilarSnapshots])).toEqual(
verySimilarSnapshots
)
})

it('should match snapshot', () => {
Expand All @@ -351,66 +354,68 @@ describe('sessionRecordingDataLogic', () => {

it('loads each source, and on success reports recording viewed', async () => {
await expectLogic(logic, () => {
logic.actions.loadRecordingSnapshots()
// loading the snapshots will trigger a loadRecordingSnapshotsSuccess
logic.actions.loadSnapshots()
// loading the snapshots will trigger a loadSnapshotsForSourceSuccess
// that will have the blob source
// that triggers loadRecordingSnapshots
// that triggers loadNextSnapshotSource
}).toDispatchActions([
// the action we triggered
logic.actionCreators.loadRecordingSnapshots(),
'loadRecordingSnapshotsSuccess',
'loadSnapshots',
// the response to that triggers loading of the first item which is the blob source
(action) =>
action.type === logic.actionTypes.loadRecordingSnapshots &&
action.type === logic.actionTypes.loadSnapshotsForSource &&
action.payload.source?.source === 'blob',
'loadRecordingSnapshotsSuccess',
'loadSnapshotsForSourceSuccess',
// and then we report having viewed the recording
'reportViewed',
// the response to the success action triggers loading of the second item which is the realtime source
(action) =>
action.type === logic.actionTypes.loadRecordingSnapshots &&
action.type === logic.actionTypes.loadSnapshotsForSource &&
action.payload.source?.source === 'realtime',
'loadRecordingSnapshotsSuccess',
'loadSnapshotsForSourceSuccess',
// having loaded any real time data we start polling to check for more
'startRealTimePolling',
'pollRealtimeSnapshots',
// which in turn triggers another load
(action) =>
action.type === logic.actionTypes.loadSnapshotsForSource &&
action.payload.source?.source === 'realtime',
'loadSnapshotsForSourceSuccess',
])
})

it('can start polling for snapshots', async () => {
it('polls up to a max threshold', async () => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤯 I've just realised that the polling shouldn't trigger loading state in the logic - we don't want the loading affordances in the UI when we poll.

Totally separate to this, reading the code back made me realise

await expectLogic(logic, () => {
logic.actions.startRealTimePolling()
logic.actions.loadSnapshots()
})
.toDispatchActions([
// the action we triggered
'startRealTimePolling',
'pollRecordingSnapshots', // 0
'pollRecordingSnapshotsSuccess',
'loadSnapshotsForSource', // blob
'loadSnapshotsForSourceSuccess',
// the returned data isn't changing from our mock,
// so we'll not keep polling indefinitely
'pollRecordingSnapshots', // 1
'pollRecordingSnapshotsSuccess',
'pollRecordingSnapshots', // 2
'pollRecordingSnapshotsSuccess',
'pollRecordingSnapshots', // 3
'pollRecordingSnapshotsSuccess',
'pollRecordingSnapshots', // 4
'pollRecordingSnapshotsSuccess',
'pollRecordingSnapshots', // 5
'pollRecordingSnapshotsSuccess',
'pollRecordingSnapshots', // 6
'pollRecordingSnapshotsSuccess',
'pollRecordingSnapshots', // 7
'pollRecordingSnapshotsSuccess',
'pollRecordingSnapshots', // 8
'pollRecordingSnapshotsSuccess',
'pollRecordingSnapshots', // 9
'pollRecordingSnapshotsSuccess',
'pollRecordingSnapshots', // 10
'pollRecordingSnapshotsSuccess',
'loadSnapshotsForSource', // 1
'loadSnapshotsForSourceSuccess',
'loadSnapshotsForSource', // 2
'loadSnapshotsForSourceSuccess',
'loadSnapshotsForSource', // 3
'loadSnapshotsForSourceSuccess',
'loadSnapshotsForSource', // 4
'loadSnapshotsForSourceSuccess',
'loadSnapshotsForSource', // 5
'loadSnapshotsForSourceSuccess',
'loadSnapshotsForSource', // 6
'loadSnapshotsForSourceSuccess',
'loadSnapshotsForSource', // 7
'loadSnapshotsForSourceSuccess',
'loadSnapshotsForSource', // 8
'loadSnapshotsForSourceSuccess',
'loadSnapshotsForSource', // 9
'loadSnapshotsForSourceSuccess',
'loadSnapshotsForSource', // 10
'loadSnapshotsForSourceSuccess',
])
.toNotHaveDispatchedActions([
// this isn't called again
'pollRecordingSnapshots',
'loadSnapshotsForSource',
])

await waitForExpect(() => {
Expand All @@ -433,12 +438,14 @@ describe('sessionRecordingDataLogic', () => {

it('should start polling even though realtime is empty', async () => {
await expectLogic(logic, () => {
logic.actions.loadRecordingSnapshots()
logic.actions.loadSnapshots()
}).toDispatchActions([
'loadRecordingSnapshotsSuccess',
'startRealTimePolling',
'pollRecordingSnapshots',
'pollRecordingSnapshotsSuccess',
'loadSnapshots',
'loadSnapshotSourcesSuccess',
'loadNextSnapshotSource',
'pollRealtimeSnapshots',
'loadSnapshotsForSource',
'loadSnapshotsForSourceSuccess',
])
})
})
Expand Down
Loading
Loading