Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Supports audio sharing #181

Open
wants to merge 7 commits into
base: master
Choose a base branch
from
+110 −27
Open

Supports audio sharing #181

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
677a441
Add audio support
strvert Jun 9, 2024
7972214
fix: Fixed icon when audio is disabled
strvert Jun 9, 2024
cfd5f29
feat: audio button is hidden when there is no audioStream.
strvert Jun 16, 2024
5b6cf15
feat: Explicitly disable effects for screen audio
strvert Jun 16, 2024
3b73f6d
refactor: MediaStream into one.
strvert Jun 16, 2024
72a2d59
fix: Change to useRef, which is appropriate for element references.
strvert Jun 16, 2024
b65a42f
feat: Added hotkeys for audio
strvert Jun 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 70 additions & 10 deletions ui/src/Room.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ import CancelPresentationIcon from '@mui/icons-material/CancelPresentation';
import PresentToAllIcon from '@mui/icons-material/PresentToAll';
import FullScreenIcon from '@mui/icons-material/Fullscreen';
import PeopleIcon from '@mui/icons-material/People';
import HeadsetIcon from '@mui/icons-material/Headset';
import HeadsetOff from '@mui/icons-material/HeadsetOff';
import SettingsIcon from '@mui/icons-material/Settings';
import {useHotkeys} from 'react-hotkeys-hook';
import {Video} from './Video';
Expand Down Expand Up @@ -70,6 +72,8 @@ export const Room = ({
const [hoverControl, setHoverControl] = React.useState(false);
const [selectedStream, setSelectedStream] = React.useState<string | typeof HostStream>();
const [videoElement, setVideoElement] = React.useState<FullScreenHTMLVideoElement | null>(null);
const audioElementRef = React.useRef<HTMLAudioElement | null>(null);
const [playingAudio, setPlayingAudio] = React.useState(false);

useShowOnMouseMovement(setShowControl);

Expand All @@ -89,17 +93,30 @@ export const Room = ({
setSelectedStream(state.clientStreams[0]?.id);
}, [state.clientStreams, selectedStream, state.hostStream]);

const stream =
const videoStream =
selectedStream === HostStream
? state.hostStream
: state.clientStreams.find(({id}) => selectedStream === id)?.stream;

const audioStream = state.clientStreams.find(({id, stream}) => selectedStream === id && stream.getAudioTracks().length != 0 )?.stream;

React.useEffect(() => {
if (videoElement && stream) {
videoElement.srcObject = stream;
if (videoElement && videoStream) {
videoElement.srcObject = videoStream;
videoElement.play().catch((e) => console.log('Could not play main video', e));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The video audio can be muted when in fullscreen via native controls, the playingAudio state in screego should reflect that. PlayingAudio should be updated via volume events from the video e.g.:

            videoElement.onvolumechange = () => setPlayingAudio(videoElement.muted)

}
}, [videoElement, stream]);
}, [videoElement, videoStream]);

React.useEffect(() => {
if (audioElementRef.current && audioStream) {
audioElementRef.current.srcObject = audioStream;
}
if (playingAudio) {
playAudio();
} else {
pauseAudio();
}
}, [audioElementRef, audioStream]);

const copyLink = () => {
navigator?.clipboard?.writeText(window.location.href)?.then(
Expand All @@ -116,6 +133,31 @@ export const Room = ({
[setHoverControl]
);

const playAudio = () => {
if (audioElementRef.current) {
audioElementRef.current.play().then(() => {
setPlayingAudio(true);
}).catch((e) => {
console.log('Could not play main audio', e);
});
}
}
const pauseAudio = () => {
if (audioElementRef.current) {
audioElementRef.current.pause();
setPlayingAudio(false);
}
}

const toggleAudio = () => {
if (playingAudio) {
pauseAudio();
} else {
playAudio();
}
}

const audioButtonVisible = audioStream && selectedStream !== HostStream;
const controlVisible = showControl || open || hoverControl;

useHotkeys('s', () => (state.hostStream ? stopShare() : share()), [state.hostStream]);
Expand Down Expand Up @@ -161,6 +203,7 @@ export const Room = ({
},
[state.clientStreams, selectedStream]
);
useHotkeys('a', toggleAudio, [playingAudio]);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the hotkey should be m for mute.


const videoClasses = () => {
switch (settings.displayMode) {
Expand Down Expand Up @@ -192,7 +235,7 @@ export const Room = ({
</Paper>
)}

{stream ? (
{videoStream ? (
<video
muted
ref={setVideoElement}
Expand All @@ -215,6 +258,13 @@ export const Room = ({
</Typography>
)}

{audioStream && (
<audio
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The <video> element should be able to play audio, we should use this instead of creating another audio element. the muted property should be used to control if the audio should be played or not.

ref={audioElementRef}
style={{ display: 'none' }}
/>
)}

{controlVisible && (
<Paper className={classes.control} elevation={10} {...setHoverState}>
{state.hostStream ? (
Expand Down Expand Up @@ -249,6 +299,14 @@ export const Room = ({
<PeopleIcon fontSize="large" />
</Badge>
</Tooltip>
{audioButtonVisible && <Tooltip title={playingAudio ? "Mute Audio" : "Hear Audio"} arrow>
<IconButton
onClick={toggleAudio}
size="large"
>
{playingAudio ? <HeadsetIcon fontSize="large" /> : <HeadsetOff fontSize="large" />}
</IconButton>
</Tooltip>}
<Tooltip title="Fullscreen" arrow>
<IconButton
onClick={() => handleFullscreen()}
Expand Down Expand Up @@ -278,11 +336,13 @@ export const Room = ({
className={classes.smallVideoContainer}
onClick={() => setSelectedStream(client.id)}
>
<Video
key={client.id}
src={client.stream}
className={classes.smallVideo}
/>
{
client.stream && <Video
key={client.id}
src={client.stream}
className={classes.smallVideo}
/>
}
<Typography
variant="subtitle1"
component="div"
Expand Down
57 changes: 40 additions & 17 deletions ui/src/useRoom.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,8 @@ const clientSession = async ({
}): Promise<RTCPeerConnection> => {
console.log('ice', ice);
const peer = new RTCPeerConnection({...relayConfig, iceServers: ice});
const stream = new MediaStream();

peer.onicecandidate = (event) => {
if (!event.candidate) {
return;
Expand All @@ -144,8 +146,17 @@ const clientSession = async ({
}
};
peer.ontrack = (event) => {
const stream = new MediaStream();
stream.addTrack(event.track);
if (event.track.kind === 'video') {
if (stream.getVideoTracks().length === 0) {
stream.addTrack(event.track);
}
}
if (event.track.kind === 'audio') {
if (stream.getAudioTracks().length === 0) {
stream.addTrack(event.track);
}
}

onTrack(stream);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be moved inside the if (event.track.kind === 'video), then we don't have to adjust the onTrack setState implementation at all.

};

Expand Down Expand Up @@ -230,21 +241,29 @@ export const useRoom = (config: UIConfig): UseRoom => {
);
},
onTrack: (stream) =>
setState((current) =>
current
? {
...current,
clientStreams: [
...current.clientStreams,
{
id: sid,
stream,
peer_id: peer,
},
],
}
: current
),
setState((current) => {
if (!current) {
return current;
}

const existingStream = current.clientStreams.find(({id}) => id === sid);
if (existingStream) {
existingStream.stream = stream;
return current;
}

return {
...current,
clientStreams: [
...current.clientStreams,
{
id: sid,
peer_id: peer,
stream,
}
]
}
}),
}).then((peer) => (client.current[event.payload.id] = peer));
return;
case 'clientice':
Expand Down Expand Up @@ -327,6 +346,10 @@ export const useRoom = (config: UIConfig): UseRoom => {
}
stream.current = await navigator.mediaDevices.getDisplayMedia({
video: {frameRate: loadSettings().framerate},
audio: {
echoCancellation: false,
noiseSuppression: false
}
});
stream.current?.getVideoTracks()[0].addEventListener('ended', () => stopShare());
setState((current) => (current ? {...current, hostStream: stream.current} : current));
Expand Down
Loading