Scale out Origin servers for more incoming streams #1077
Replies: 45 comments
-
Could you please provide a link to where you did read this? As far as I understand the docs, you should be able to just set up a bunch of Origin servers in different regions close to your streamers to avoid RTMP streaming issues, and on the edge servers in the config, just add a bunch of origins and an application for each of those origins, differentiating them by url. |
Beta Was this translation helpful? Give feedback.
-
|
In the Clustering docs, the diagram shows Primary and Secondary, and I didn't realize I could have more than that pair. If I have a bunch of similarly located people streaming to one Origin server and it starts being overwhelmed, can I add more Origin servers in that same area and load balance the incoming RTMP traffic across them? If so, what's the best method to do that — round robin? IP hash? |
Beta Was this translation helpful? Give feedback.
-
|
You could probably use HAProxy or similar TCP-based load-balancer to round-robin balance different incoming RTMP streams to different origin servers running OME. I'd suggest avoiding just using IP address hash because thanks to carrier grade NAT, many users could show up having the same source IP address. The question however would be, how does your user get the stream-url which points to the correct origin server (as part of the path which the client requests from the edge server)? An alternative approach would be to assign different customers to different origin servers (thus they always stream to the same one, you don't necessarily need a tcp-load-balancer in that case and the stream-url for the webclient would stay the same. |
Beta Was this translation helpful? Give feedback.
-
|
We also solve this problem by assigning a separate domain to each Origin server when operating a commercial service and then providing a different input URL to the sender. Of course, this process is automated in the web service. When the sender clicks Create Channel, it provides the URL of the Origin server that is being used the least at the moment. |
Beta Was this translation helpful? Give feedback.
-
It would be great to be able to scale origin servers as @command-tab mentioned. We currently have a few origin OMEs on different domain names in different regions (one origin per region) and use the same approach you mentioned (let the user choose their preferred ingest region when they create a room) and then show them the relevant Ingest URL. But the concern is that at a later point (days / weeks later) the traffic changes and you may run into a situation where too many users are streaming to origin-1.ome and overloading the server. We could make it so that we spin up more origin servers as required and set our web app to not give the same origin url to more than X users, but then we run into a situation where we have many origins that become largely unused in the future, but we can't shut them down because some users are still streaming to them. I think an ideal implementation for us would be if we could route incoming streams to origin servers dynamically, based on the current usage of the server. For example: in our web app the user selects "US West" as the ingest region, which always gives the same url "us-west.ome". And then within the US-West region we have multiple Origin servers. When a user streams to us-west.ome, the stream is automatically routed to the least-used origin server within that region. |
Beta Was this translation helpful? Give feedback.
-
|
I’d write that straight into your own application.
The flow would be something like:
- GET /myApp/newStream?somecredential=123
- App stores a list of OME origin URLs
- App gets a list of active streams from each OME origin using OME API
- Whichever origin has the least streams (or meets whatever condition you need), app signs stream key and returns response like:
{server_id: “eu-5” server_url: “wss://eu5.com/live” signed_key: “abc123”}
Does that make sense?
… On Jun 2, 2021, at 8:17 AM, llspalex ***@***.***> wrote:
We also solve this problem by assigning a separate domain to each Origin server when operating a commercial service and then providing a different input URL to the sender.
Of course, this process is automated in the web service. When the sender clicks Create Channel, it provides the URL of the Origin server that is being used the least at the moment.
It would be great to be able to scale origin servers as @command-tab <https://github.com/command-tab> mentioned. We currently have a few origin OMEs on different domain names in different regions (one origin per region) and use the same approach you mentioned (let the user choose their preferred ingest region when they create a room) and then show them the relevant Ingest URL.
But the concern is that at a later point (days / weeks later) the traffic changes and you may run into a situation where too many users are streaming to origin-1.ome and overloading the server. We could make it so that we spin up more origin servers as required and set our web app to not give the same origin url to more than X users, but then we run into a situation where we have many origins that become largely unused in the future, but we can't shut them down because some users are still streaming to them.
I think an ideal implementation for us would be if we could route incoming streams to origin servers dynamically, based on the current usage of the server.
For example, user selects "US West" as the ingest region, which always gives the same url "us-west.ome". And then within the US-West region we have multiple Origin servers. When a user streams to us-west.ome, the stream is automatically routed to the least-used origin server within that region.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub <#343 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIAMIQTA6C22Q34Q4QVZXZTTQYOL5ANCNFSM43D2JSNA>.
|
Beta Was this translation helpful? Give feedback.
-
|
for that to work, the streaming protocols like RTMP, RTSP or SRT would have to support something like following HTTP 302 redirects or similar. Does this exist in the mentioned protocols? Then a load distributor service could be written, which distributes the streams to a server with enough capacity in the geographic region of the streamer. The discovery of the correct stream edge server and endpoint/app can already be done similarly with HTTP 302 forwarding and that scales well to thousands of concurrent users (for example last weekend I had an event with ~35000 viewers over HLS using a custom load balancer like this).. |
Beta Was this translation helpful? Give feedback.
-
|
Oh, I meant these checks could take place before a stream URL is issued - not as a redirect, you are correct.
… On Jun 2, 2021, at 8:45 AM, basisbit ***@***.***> wrote:
for that to work, the streaming protocols like RTMP, RTSP or SRT would have to support something like following HTTP 302 redirects or similar. Does this exist in the mentioned protocols? Then a load distributor service could be written, which distributes the streams to a server with enough capacity in the geographic region of the streamer.
The discovery of the correct stream edge server and endpoint/app can already be done similarly with HTTP 302 forwarding and that scales well to thousands of concurrent users.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub <#343 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIAMIQR7BQZK3OLKB7KAVLTTQYRXNANCNFSM43D2JSNA>.
|
Beta Was this translation helpful? Give feedback.
-
|
Thanks @bchah - I actually forgot that we could leverage the OME API to check current usage. Though as you said, it wouldn't take care of dynamically routing streams after the stream URL is issued. |
Beta Was this translation helpful? Give feedback.
-
|
This is a very good discussion thread. Since redirect is not provided by all protocols, it is difficult to use other than HTTP-based protocols. Also the implementation between Encoder - Provider is more difficult. So as a way to automatically scale the origin I have the following idea. To scale Origin automatically, Edge needs to be a bit smarter. If origin is bundled into one domain, when a request to play “/app/stream” comes to Edge, Edge does not know which origin has “app/stream”. There are several ways to make Edge know this.
Please let me know if you have any better ideas or have similar examples of other media servers that have addressed this. |
Beta Was this translation helpful? Give feedback.
-
|
Another thing to consider: If you go the route where you arrange to provide users with a different Origin URL every time they stream (based on current usage stats), they will have to adjust the settings of their encoder every time they stream (to point to the new Origin URL you give them), and that adds some friction. By contrast, if I stream to YouTube in OBS, I don't have to change anything every time I start streaming. I really like the ideas about making Edge smarter and being able to figure out which Origin has the requested stream 😍 |
Beta Was this translation helpful? Give feedback.
-
|
Bundling origin servers into one domain (or a few different domains based on region) would be amazing. I agree that to the streamer, the process should be invisible and the Stream URL and Stream Key used should not have to change between streaming sessions. @getroot, I'm not sure which of your two proposed ideas would be better, though in our current implementation we do not use Edge servers (viewers connect to our Origins) because a) we have a low number of viewers per room (our platform is designed for many rooms with <20 viewers per room rather than a few rooms with many viewers per room) and b) we are trying to keep latency as low as possible - every ms counts. That said, we are not opposed to using Edge servers, and we will probably eventually need Edge servers. Auto-scaling / load-balancing of OME ingest in a way that we can make it invisible to the streamer would be a very welcome feature. |
Beta Was this translation helpful? Give feedback.
-
|
YouTube uses Anycast DNS for load-balancing based on regions, as well as 10Gb/s connected servers. If a region needs more capacity, they can add more servers to the domain name entry at the resolver. You can also partially solve the ingest congestion issue by using much more subdomain names than you have servers, and on demand changing the subdomains to different servers, to distribute the load evenly. However, this requires the "smart" edge server solution mentioned above, which knows where to get the stream from. For scaling this nicely, this should probably be implemented using event streams (server-sent events), so each edge only holds one http request to each configured origin server. The origin servers then should send stream-started and stream-stopped events over that event channel - as well as initially send a list of all the currently already running streams to the edge at initial connection. This ways, you could dynamically load-balance the ingest side to the origin servers and the egress side to the edge servers. This should scale well up to the point where you would run out of bandwidth of the origin server sending video streams to the edge servers, and be rather easy to implement. |
Beta Was this translation helpful? Give feedback.
-
|
Uh, the ingress side could even be changed to requiring much less subdomains than mentioned above for smaller setups:
You give the streamers/customers only the region-specific origin address. Then you dynamically change the address of the region specific subdomain to the IP address of the origin server which currently has the least amount of load. You could also write your own small dns server for that, which returns the IP address of the least used origin server to dns resoution requests. |
Beta Was this translation helpful? Give feedback.
-
|
@command-tab @bchah @llspalex @basisbit |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
I committed a new feature today called OriginMapStore. This is a function where the origin server updates its app/stream information to the Redis server, and the Edge retrieves the app/stream information from the Redis server and connects to the origin server. With this feature, Edge no longer needs to know in advance the URL of the origin server that has a specific app/stream. In other words, even if you create a stream in any origin server, Edge can pull the app/stream in the cluster. So you are free to expand your Origin server. Note that the app/stream name is used as a key, so it must be unique within the cluster. I tested this with Redis Server 5.0.7. Set And also in Edge, set it like this: Edge first looks for |
Beta Was this translation helpful? Give feedback.
-
|
You must rerun prerequisite.sh for redis client library to test OringinMapStore feature. |
Beta Was this translation helpful? Give feedback.
-
|
Oh, this is very cool! I look forward to experimenting with this 😍 |
Beta Was this translation helpful? Give feedback.
-
|
@command-tab You can experiment with the master branch. Many comments are welcome! :-) |
Beta Was this translation helpful? Give feedback.
-
|
Very nice improvement! Congratulations! |
Beta Was this translation helpful? Give feedback.
-
|
I'm not sure that securing communication with Redis needs to be in scope for OME. Redis's own docs say:
Historically, securing that network path is up to the admin, not the particular software, as Redis is best used not exposed directly to the internet. Maybe a VPN, Wireguard tunnel, or other private networking system would be in order. This is all good, since it helps OME retain focus. |
Beta Was this translation helpful? Give feedback.
-
|
@basisbit I haven't implemented it yet, but Redis also supports TLS. But in our case I am wondering if it is necessary. This is because the only information OME sends and receives with Redis is the app/stream name and OVT URL of Origin. Origin can exist in a secure group that only Edge can access, so even if it is exposed, there will be no problem. |
Beta Was this translation helpful? Give feedback.
-
|
Hello @getroot, I’m currently trying OME and OriginMapStore feature. Origin server publishes well the stream on Redis and I can found it browsing keys:
However, when I try to connect through Edge also using OriginMapStore feature, it’s unable to find stream: [2022-10-14 15:35:08.530] I [SPRtcSig-T4334:11] Publisher | publisher.cpp:186 | Try to pull stream from local origin map: [#default#app/azerty] I don’t explain it. I didn’t have issue with a static Origin config. Origin part seems to works well, it looks like issue is only on Edge side but I’ve no error message and so I don’t know how to debug issue. I did monitor Redis query, and I found GET is similar to SET so it should find it: Thanks |
Beta Was this translation helpful? Give feedback.
-
|
@guyguy333 plz upload your server.xml files of origin and edge |
Beta Was this translation helpful? Give feedback.
-
|
@getroot Yes sure. I'm using Redis 7. Edge: <?xml version="1.0" encoding="UTF-8"?>
<Server version="8">
<Name>edge</Name>
<IP>*</IP>
<Type>edge</Type>
<PrivacyProtection>false</PrivacyProtection>
<StunServer>stun.l.google.com:19302</StunServer>
<Modules>
<!--
Currently OME only supports h2 like all browsers do. Therefore, HTTP/2 only works on TLS ports.
-->
<HTTP2>
<Enable>true</Enable>
</HTTP2>
<!-- P2P works only in WebRTC and is experiment feature -->
<!--
<P2P>
<Enable>false</Enable>
<MaxClientPeersPerHostPeer>2</MaxClientPeersPerHostPeer>
</P2P>
-->
</Modules>
<Bind>
<Providers />
<Publishers>
<WebRTC>
<Signalling>
<Port>4333</Port>
<TLSPort>4334</TLSPort>
<WorkerCount>1</WorkerCount>
</Signalling>
<IceCandidates>
<IceCandidate>*:11000-11005/udp</IceCandidate>
<TcpRelay>*:4478</TcpRelay>
<TcpForce>true</TcpForce>
<TcpRelayWorkerCount>1</TcpRelayWorkerCount>
</IceCandidates>
</WebRTC>
</Publishers>
</Bind>
<VirtualHosts>
<VirtualHost>
<Name>default</Name>
<Host>
<Names>
<Name>efoo.example.com</Name>
</Names>
<TLS>
<CertPath>/opt/ovenmediaengine/bin/edge_certs/tls.crt</CertPath>
<KeyPath>/opt/ovenmediaengine/bin/edge_certs/tls.key</KeyPath>
<ChainCertPath>/opt/ovenmediaengine/bin/edge_certs/tls.crt</ChainCertPath>
</TLS>
</Host>
<Distribution>example.com</Distribution>
<OriginMapStore>
<RedisServer>
<Host>ovenmediaengine-redis-master:6379</Host>
<Auth>test</Auth>
</RedisServer>
</OriginMapStore>
</VirtualHost>
</VirtualHosts>
</Server>Origin: <?xml version="1.0" encoding="UTF-8"?>
<Server version="8">
<Name>Origin</Name>
<IP>*</IP>
<Type>origin</Type>
<PrivacyProtection>false</PrivacyProtection>
<StunServer>stun.l.google.com:19302</StunServer>
<Managers>
<Host>
<Names>
<Name>*</Name>
</Names>
</Host>
<API>
<AccessToken>test</AccessToken>
</API>
</Managers>
<Bind>
<Managers>
<API>
<Port>8081</Port>
<WorkerCount>1</WorkerCount>
</API>
</Managers>
<Providers>
<RTMP>
<Port>1935</Port>
<WorkerCount>1</WorkerCount>
</RTMP>
<WebRTC>
<Signalling>
<Port>3333</Port>
<TLSPort>3334</TLSPort>
<WorkerCount>1</WorkerCount>
</Signalling>
<IceCandidates>
<IceCandidate>*:10000-10005/udp</IceCandidate>
<!--
If you want to stream WebRTC over TCP, specify IP:Port for TURN server.
This uses the TURN protocol, which delivers the stream from the built-in TURN server to the player's TURN client over TCP.
For detailed information, refer https://airensoft.gitbook.io/ovenmediaengine/streaming/webrtc-publishing#webrtc-over-tcp
-->
<TcpRelay>*:3478</TcpRelay>
<!-- TcpForce is an option to force the use of TCP rather than UDP in WebRTC streaming. (You can omit ?transport=tcp accordingly.) If <TcpRelay> is not set, playback may fail. -->
<TcpForce>true</TcpForce>
<TcpRelayWorkerCount>1</TcpRelayWorkerCount>
</IceCandidates>
</WebRTC>
</Providers>
<Publishers>
<OVT>
<Port>9000</Port>
</OVT>
</Publishers>
</Bind>
<VirtualHosts>
<VirtualHost>
<Name>default</Name>
<Host>
<Names>
<Name>foo.example.com</Name>
<Name>ovenmediaengine-origin</Name>
</Names>
<TLS>
<CertPath>/opt/ovenmediaengine/bin/origin_certs/tls.crt</CertPath>
<KeyPath>/opt/ovenmediaengine/bin/origin_certs/tls.key</KeyPath>
<ChainCertPath>/opt/ovenmediaengine/bin/origin_certs/tls.crt</ChainCertPath>
</TLS>
</Host>
<OriginMapStore>
<!-- In order to use OriginMap, you must enable OVT Publisher in Origin and OVT Provider in Edge. -->
<RedisServer>
<Host>ovenmediaengine-redis-master:6379</Host>
<Auth>test</Auth>
</RedisServer>
<!-- This is only needed for the origin server and used to register the ovt address of the stream. -->
<OriginHostName>${env:POD_IP}</OriginHostName>
</OriginMapStore>
<Applications>
<Application>
<Name>app</Name>
<Type>live</Type>
<OutputProfiles>
<HardwareAcceleration>false</HardwareAcceleration>
<OutputProfile>
<Name>abr</Name>
<OutputStreamName>${OriginStreamName}</OutputStreamName>
<!--LLHLS URL : https://domain/app/stream/abr.m3u8 -->
<Playlist>
<Name>For LLHLS</Name>
<FileName>abr</FileName>
<Options> <!-- Optinal -->
<!--
Automatically switch rendition in WebRTC ABR
[Default] : true
-->
<WebRtcAutoAbr>true</WebRtcAutoAbr>
</Options>
<Rendition>
<Name>Bypass</Name>
<Video>bypass_video</Video>
<Audio>bypass_audio</Audio>
</Rendition>
<Rendition>
<Name>FHD</Name>
<Video>video_1280</Video>
<Audio>bypass_audio</Audio>
</Rendition>
<Rendition>
<Name>HD</Name>
<Video>video_720</Video>
<Audio>bypass_audio</Audio>
</Rendition>
<Rendition>
<Name>SD</Name>
<Video>video_480</Video>
<Audio>bypass_audio</Audio>
</Rendition>
</Playlist>
<!--LLHLS URL : https://domain/app/stream/llhls.m3u8 -->
<Playlist>
<Name>Change Default</Name>
<FileName>llhls</FileName>
<Rendition>
<Name>HD</Name>
<Video>video_720</Video>
<Audio>bypass_audio</Audio>
</Rendition>
</Playlist>
<Encodes>
<Audio>
<Name>bypass_audio</Name>
<Bypass>true</Bypass>
</Audio>
<Video>
<Name>bypass_video</Name>
<Bypass>true</Bypass>
</Video>
<Audio>
<Codec>opus</Codec>
<Bitrate>128000</Bitrate>
<Samplerate>48000</Samplerate>
<Channel>2</Channel>
</Audio>
<Video>
<Name>video_1280</Name>
<Codec>h264</Codec>
<Bitrate>5024000</Bitrate>
<Framerate>30</Framerate>
<Height>1280</Height>
<Preset>faster</Preset>
</Video>
<Video>
<Name>video_720</Name>
<Codec>h264</Codec>
<Bitrate>2024000</Bitrate>
<Framerate>30</Framerate>
<Height>720</Height>
<Preset>faster</Preset>
</Video>
<Video>
<Name>video_480</Name>
<Codec>h264</Codec>
<Bitrate>506000</Bitrate>
<Framerate>30</Framerate>
<Height>480</Height>
<Preset>faster</Preset>
</Video>
</Encodes>
</OutputProfile>
</OutputProfiles>
<Providers>
<WebRTC />
<RTMP />
</Providers>
<Publishers>
<AppWorkerCount>1</AppWorkerCount>
<StreamWorkerCount>8</StreamWorkerCount>
<OVT />
</Publishers>
</Application>
</Applications>
</VirtualHost>
</VirtualHosts>
</Server> |
Beta Was this translation helpful? Give feedback.
-
|
@guyguy333 Would you like to add Application to Edge and enable OVT Provider? I don't know if OVT is enabled by default because the default application policy is currently unclear. Or post your entire edge log so I can check it out. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @getroot. I just added section and enabled OVT provider. Result is the same. Here is full log for Edge and full Edge config. ovenmediaengine-edge-5446998599-xbbd8.log Edge: <?xml version="1.0" encoding="UTF-8"?>
<Server version="8">
<Name>edge</Name>
<IP>*</IP>
<Type>edge</Type>
<PrivacyProtection>false</PrivacyProtection>
<StunServer>stun.l.google.com:19302</StunServer>
<Modules>
<!--
Currently OME only supports h2 like all browsers do. Therefore, HTTP/2 only works on TLS ports.
-->
<HTTP2>
<Enable>true</Enable>
</HTTP2>
<!-- P2P works only in WebRTC and is experiment feature -->
<!--
<P2P>
<Enable>false</Enable>
<MaxClientPeersPerHostPeer>2</MaxClientPeersPerHostPeer>
</P2P>
-->
</Modules>
<Bind>
<Providers />
<Publishers>
<WebRTC>
<Signalling>
<Port>4333</Port>
<TLSPort>4334</TLSPort>
<WorkerCount>1</WorkerCount>
</Signalling>
<IceCandidates>
<IceCandidate>*:11000-11005/udp</IceCandidate>
<TcpRelay>*:4478</TcpRelay>
<TcpForce>true</TcpForce>
<TcpRelayWorkerCount>1</TcpRelayWorkerCount>
</IceCandidates>
</WebRTC>
</Publishers>
</Bind>
<VirtualHosts>
<VirtualHost>
<Name>default</Name>
<Host>
<Names>
<Name>efoo.example.com</Name>
</Names>
<TLS>
<CertPath>/opt/ovenmediaengine/bin/edge_certs/tls.crt</CertPath>
<KeyPath>/opt/ovenmediaengine/bin/edge_certs/tls.key</KeyPath>
<ChainCertPath>/opt/ovenmediaengine/bin/edge_certs/tls.crt</ChainCertPath>
</TLS>
</Host>
<Distribution>example.com</Distribution>
<OriginMapStore>
<RedisServer>
<Host>ovenmediaengine-redis-master:6379</Host>
<Auth>test</Auth>
</RedisServer>
</OriginMapStore>
<Applications>
<Application>
<Name>app</Name>
<Type>live</Type>
<OutputProfiles>
<OutputProfile>
<Name>passthrough</Name>
<OutputStreamName>${OriginStreamName}</OutputStreamName>
<Encodes>
<Video>
<Bypass>true</Bypass>
</Video>
<Audio>
<Bypass>true</Bypass>
</Audio>
</Encodes>
</OutputProfile>
</OutputProfiles>
<Providers>
<OVT />
</Providers>
<Publishers>
<AppWorkerCount>1</AppWorkerCount>
<StreamWorkerCount>8</StreamWorkerCount>
<WebRTC>
<Timeout>30000</Timeout>
<Rtx>false</Rtx>
<Ulpfec>false</Ulpfec>
<JitterBuffer>false</JitterBuffer>
</WebRTC>
<LLHLS>
<ChunkDuration>0.5</ChunkDuration>
<SegmentDuration>6</SegmentDuration>
<SegmentCount>10</SegmentCount>
<CrossDomains>
<Url>*</Url>
</CrossDomains>
</LLHLS>
</Publishers>
</Application>
</Applications>
</VirtualHost>
</VirtualHosts>
</Server> |
Beta Was this translation helpful? Give feedback.
-
v0.14.10 == ok this commit make edge OriginMapStore fail/crash. [11-02 03:41:29.883] I [SPRtcSig-T3333:152246] Publisher | publisher.cpp:186 | Try to pull stream from local origin map: [#edge#app/test] |
Beta Was this translation helpful? Give feedback.
-
|
@chakphanu Thanks for the detailed report. Could you please make this a new issue? |
Beta Was this translation helpful? Give feedback.
-
I have a question about Origin-Edge mode: If I understand the architecture correctly, the Origin server performs any configured transcoding, and the Edge servers fetch that content over OVT and scale out to handle all the necessary viewer connections. Let's say I wanted to support dozens of incoming RTMP streams. According to the docs, there can be only one primary Origin and one failover. So how would you recommend scaling of Origin servers to accommodate lots of incoming streams?
Thank you!
Beta Was this translation helpful? Give feedback.
All reactions