Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat: Streaming video elements via webrtc #1046

Draft
wants to merge 63 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
63 commits
Select commit Hold shift + click to select a range
1cabfbf
Add `recordCrossOriginIframe` setting
Juice10 Oct 24, 2022
bd0d633
Set up messaging between iframes
Juice10 Oct 25, 2022
017bd3e
should emit full snapshot event from iframe as mutation event
Juice10 Oct 25, 2022
f1c8d0b
this.mirror was dropped on attachIframe
Juice10 Oct 25, 2022
580ed6a
should use unique id for child of iframe
Juice10 Oct 25, 2022
f248392
Cross origin iframe recording in `yarn live-stream`
Juice10 Oct 25, 2022
e34e856
Root iframe check thats supported by firefox
Juice10 Oct 26, 2022
7e6a5ee
Live stream: Inject script in all frames
Juice10 Oct 26, 2022
a8542b8
Record same origin and cross origin iframes differently
Juice10 Oct 26, 2022
6b5a9c6
Should map Input events correctly
Juice10 Oct 26, 2022
fee185a
Turn on other tests
Juice10 Oct 26, 2022
cb1e065
Fix compatibility with newer puppeteer
Juice10 Oct 26, 2022
c54b981
puppeteer vs 12 seems stable without to many changes needed
Juice10 Oct 26, 2022
505ba2c
normalize port numbers in snapshots
Juice10 Oct 26, 2022
15c2cc0
Handle scroll and ViewportResize events in cross origin iframe
Juice10 Oct 26, 2022
9febf1c
Correctly map cross origin mutations
Juice10 Oct 26, 2022
5fa4634
Map selection events for cross origin iframes
Juice10 Oct 26, 2022
05a1fbc
Map canvas mutations for cross origin iframes
Juice10 Oct 26, 2022
8ea496c
Update snapshot to include canvas events
Juice10 Oct 26, 2022
8ac677b
Skip all meta events
Juice10 Oct 26, 2022
92b9594
Support custom events as best we can in cross origin iframes
Juice10 Oct 26, 2022
20a39e6
Use earliest version of puppeteer that works with cross origin live-s…
Juice10 Oct 26, 2022
738a1fc
Map mouse/touch interaction events
Juice10 Oct 27, 2022
6635390
Update snapshots for correctly mapped click events
Juice10 Oct 27, 2022
f884878
Tweak tests for new puppeteer version
Juice10 Oct 27, 2022
a598429
Map MediaInteraction correctly for cross origin iframes
Juice10 Oct 27, 2022
0f91e98
Make tests consistent between high and low dpi devices
Juice10 Oct 27, 2022
784c2da
Make test less flaky
Juice10 Oct 27, 2022
50c907e
Make test less flaky
Juice10 Oct 27, 2022
97b15ed
Make test less flaky
Juice10 Oct 27, 2022
89b14bd
Make test less flaky
Juice10 Oct 27, 2022
cb26153
Add support for styles in cross origin iframes
Juice10 Oct 27, 2022
fc2ad8c
Map traditional stylesheet mutations on cross origin iframes
Juice10 Oct 27, 2022
a348d33
Add todo
Juice10 Oct 27, 2022
bbdea9b
Add iframe mirror
Juice10 Oct 28, 2022
4bca6bb
Get iframe manager to use iframe mirrors internally
Juice10 Oct 31, 2022
8147d7f
Rename `IframeMirror` to `CrossOriginIframeMirror`
Juice10 Oct 31, 2022
7f23d34
Setup basic cross origin canvas webrtc streaming
Juice10 Oct 31, 2022
da4c53b
Clean up removed canvas elements
Juice10 Oct 31, 2022
e0e1a9b
reset style mirror on new full snapshot
Juice10 Nov 1, 2022
2e64e5e
Fix cross origin canvas webrtc streaming
Juice10 Nov 2, 2022
7d9bcb7
Make emit optional
Juice10 Nov 2, 2022
8bc678f
Run tests on github actions
Juice10 Nov 2, 2022
7c985d0
Upload image artifacts from failed tests
Juice10 Nov 2, 2022
48b8c1e
Use newer github actions
Juice10 Nov 2, 2022
dc02163
Test: hopefully adding more wait will fix it
Juice10 Nov 2, 2022
1a9c164
add extra wait
Juice10 Nov 2, 2022
c51e84f
Fix image snapshot tests
Juice10 Nov 2, 2022
6eb1d2e
Make tests run with new puppeteer version
Juice10 Nov 2, 2022
728e779
upgrade eslint-plugin-jest
Juice10 Nov 2, 2022
7b5c3cf
Chore: Remove travis ci as ci's running on github actions
Juice10 Nov 2, 2022
99ab6d7
Chore: Support recording cross origin iframe in repl
Juice10 Nov 3, 2022
2963357
Force developers to update the cross origin iframe mapping when addin…
Juice10 Nov 3, 2022
243530d
Document cross origin iframe recording
Juice10 Nov 3, 2022
f3705b3
Docs: cross origin iframes recording methods
Juice10 Nov 3, 2022
4a5e4cf
Docs: AI translated, cross origin iframe recording
Juice10 Nov 3, 2022
05632d2
Merge branch 'master' of https://github.com/rrweb-io/rrweb into cross…
Juice10 Nov 4, 2022
522cadb
rename getParentId to getId
Juice10 Nov 4, 2022
1608da3
Migrate to @rrweb/types
Juice10 Nov 4, 2022
1d92dcc
Run on pull request
Juice10 Nov 4, 2022
3ddfeed
Setup video streaming via webrtc
Juice10 Nov 7, 2022
5264328
Make viewport match window size
Juice10 Nov 7, 2022
e571935
Stream video elements with blob: based source urls
Juice10 Nov 7, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci-cd.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
name: Tests

on: [push, pull_request_target]
on: [push, pull_request]

concurrency: ${{ github.workflow }}-${{ github.ref }}

Expand Down
116 changes: 116 additions & 0 deletions docs/recipes/cross-origin-iframes.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
# Cross origin iframes

By default browsers make it difficult to access the contents of an iframe that is hosted on a different domain. This is a security feature to prevent malicious sites from accessing sensitive information on other sites. It is possible to work around this security feature, but it is not recommended unless you [are very strict](https://stackoverflow.com/a/21629575) about allowing only the sites you trust to embed your website inside of an iframe.
Since if you allow recording cross origin iframes, any malicious website can embed your website and as long as they have rrweb running they can record all the contents of your website.

## How to record cross origin iframes

Enable recording cross-origin iframes in your parent page:

```js
rrweb.record({
emit(event) {}, // all events will be emitted here, including events from cross origin iframes
recordCrossOriginIframes: true,
});
```

Enable replaying cross-origin iframes in your child page:

```js
rrweb.record({
emit(event) {}, // this is required for rrweb, but the child page will not emit any events
recordCrossOriginIframes: true,
});
```

## Considerations

When cross origin iframe recording is turned on rrweb will check to see if it is being run in a top level window.
If it isn't it'll send the events to the parent window via `postMessage`.

If you don't have rrweb running in the top level window, the events will be lost when `recordCrossOriginIframes` is turned on.

If the top level window is a malicious website it can listen to the events and send them to a server of its choosing.

Or if a malicious script is running in on your page they can listen in on `postMessage` and as communication between the child and parent window is not encrypted. And they can see the events.

## Options for injecting rrweb into cross origin iframes

### 1. Website owners, add rrweb in the iframes

If you own the website that with the iframe and the website that is being embedded in an iframe, you can add rrweb to both pages via a script tag.

### 2. Browser extension

See https://developer.chrome.com/docs/extensions/mv3/content_scripts/#functionality

### 3. Puppeteer script

```js
import puppeteer from 'puppeteer';

async function injectRecording(frame) {
await frame.evaluate((rrwebCode) => {
if (window.__IS_RECORDING__) return;
window.__IS_RECORDING__ = true;

(async () => {
function loadScript(code) {
const s = document.createElement('script');
s.type = 'text/javascript';
s.innerHTML = code;
if (document.head) {
document.head.append(s);
} else {
requestAnimationFrame(() => {
document.head.append(s);
});
}
}
loadScript(rrwebCode);

window.rrweb.record({
emit: (event) => {
window._captureEvent(event);
},
recordCrossOriginIframes: true,
});
})();
}, code);
}

const browser = await puppeteer.launch();
const page = (await browser.pages())[0];

const events = []; // contains all events from all frames

await page.exposeFunction('_captureEvent', (event) => {
events.push(event);
});

page.on('framenavigated', async (frame) => {
await injectRecording(frame); // injects rrweb into the iframe
});

await page.goto('https://example.com');

// your events are in the events array
```

### 4. Electron

```ts
const win = new BrowserWindow({
width: 800,
height: 600,
webPreferences: {
preload: path.join(__dirname, 'rrweb-recording-script.js'),
// this turns on preload inside iframes, but disables node integration
nodeIntegrationInSubFrames: true,
nodeIntegration: false,
},
});
```

See https://www.electronjs.org/docs/latest/tutorial/tutorial-preload
And https://www.electronjs.org/docs/latest/tutorial/process-model#preload-scripts
117 changes: 117 additions & 0 deletions docs/recipes/cross-origin-iframes.zh_CN.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
# Cross origin iframes

默认情况下,浏览器很难访问托管在不同域上的 iframe 的内容。 这是一项安全功能,可防止恶意站点访问其他站点上的敏感信息。 可以解决此安全功能。
但不建议这样做,除非您非常严格 https://stackoverflow.com/a/21629575 只允许您信任的网站将您的网站嵌入 iframe 中。
因为如果您允许记录跨源 iframe,任何恶意网站都可以嵌入您的网站,并且只要它们运行 rrweb,它们就可以记录您网站的所有内容。

## 如何记录跨源 iframe

在父页面中启用录制跨域 iframe:

```js
rrweb.record({
emit(event) {}, // 所有事件都将在此处发出,包括来自跨源 iframe 的事件
recordCrossOriginIframes: true,
});
```

在您的子页面中启用重放跨域 iframe:

```js
rrweb.record({
emit(event) {}, // 这是 rrweb 所必需的,但子页面不会发出任何事件
recordCrossOriginIframes: true,
});
```

## 注意事项

当跨源 iframe 录制打开时,rrweb 将检查它是否正在顶级窗口中运行。
如果不是,它将通过 `postMessage` 将事件发送到父窗口。

如果您没有在顶层窗口中运行 rrweb,则打开 `recordCrossOriginIframes` 时事件将丢失。

如果顶层窗口是一个恶意网站,它可以监听事件并将它们发送到它选择的服务器。

或者,如果您的页面上正在运行恶意脚本,他们可以监听“postMessage”,并且子窗口和父窗口之间的通信未加密。 他们可以看到事件。

## 将 rrweb 注入跨源 iframe 的选项

### 1. 网站所有者,在 iframe 中添加 rrweb

如果您拥有使用 iframe 的网站和嵌入在 iframe 中的网站,您可以通过脚本标签将 rrweb 添加到这两个页面。

### 2. 浏览器扩展

See https://developer.chrome.com/docs/extensions/mv3/content_scripts/#functionality

### 3. Puppeteer script

```js
import puppeteer from 'puppeteer';

async function injectRecording(frame) {
await frame.evaluate((rrwebCode) => {
if (window.__IS_RECORDING__) return;
window.__IS_RECORDING__ = true;

(async () => {
function loadScript(code) {
const s = document.createElement('script');
s.type = 'text/javascript';
s.innerHTML = code;
if (document.head) {
document.head.append(s);
} else {
requestAnimationFrame(() => {
document.head.append(s);
});
}
}
loadScript(rrwebCode);

window.rrweb.record({
emit: (event) => {
window._captureEvent(event);
},
recordCrossOriginIframes: true,
});
})();
}, code);
}

const browser = await puppeteer.launch();
const page = (await browser.pages())[0];

const events = []; // 包含来自所有帧的所有事件

await page.exposeFunction('_captureEvent', (event) => {
events.push(event);
});

page.on('framenavigated', async (frame) => {
await injectRecording(frame); // 将 rrweb 注入 iframe
});

await page.goto('https://example.com');

// 您的事件在事件数组中
```

### 4. Electron

```ts
const win = new BrowserWindow({
width: 800,
height: 600,
webPreferences: {
preload: path.join(__dirname, 'rrweb-recording-script.js'),
// 这会打开 iframe 内的预加载,但会禁用节点集成
nodeIntegrationInSubFrames: true,
nodeIntegration: false,
},
});
```

See https://www.electronjs.org/docs/latest/tutorial/tutorial-preload
And https://www.electronjs.org/docs/latest/tutorial/process-model#preload-scripts
2 changes: 1 addition & 1 deletion docs/recipes/dive-into-event.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,4 +67,4 @@ source -> IncrementalSource.Font
data -> fontData
```

enum IncrementalSource's definition can be found in this [list](https://github.com/rrweb-io/rrweb/blob/master/packages/rrweb/typings/types.d.ts#L62).
enum IncrementalSource's definition can be found in this [list](https://github.com/rrweb-io/rrweb/blob/98e71cd0d23628cd1fbdbe47664a65748084c4a4/packages/types/src/index.ts#L69).
Loading