Advanced Configuration

Ready to dive in with some more advanced features? Here are some quick start tutorials. Heads-up: assistance with advanced configurations is not covered by regular maintenance support and would constitute professional services if you require it. Remember you can always experiment, then reset the engine config and it will go back to the way it was when you started up Streamwell.

Performance Tuning

Larger-scale deployments will require some performance tuning on the server itself once viewership or stream counts reach into higher numbers.

These optimizations are all to do with OvenMediaEngine, the streaming engine that powers Streamwell. They are borrowed with gratitude from these OME guides (Troubleshooting and Performance Tuning) which any sysadmins should dive into deeper to realize the best performance for your server.

Choosing the Right Host OS

Streamwell works great on any modern Mac or Windows computer running Docker. However, there are network performance limitations in these operating systems (especially when factoring in the emulation layer they run for Docker) that limit them to serving a certain amount of concurrent viewers - in our testing we haven't met a modern machine that couldn't easily handle 25 viewers at 1080p / 4Mbps. But for anything more than that (25 simultaneous viewers or 100Mbps throughput) we recommend using a native Linux-based host OS instead of macOS or Windows. Read on for more tuning and testing advice.

Adjusting Thread Counts Across Cores

The 'StreamWorker' process is one of the hardest-working components of OME. In your engine configuration (under Administration -> Server -> Engine Control), increase the various "worker" thread counts, especially StreamWorker count to match the number of CPU cores you have available:

<StreamWorkerCount>8</StreamWorkerCount>

If you are streaming primarily to HLS viewers, you should also set the HLS Worker Counts appropriately.

It is assumed that the CPU cores provided are powerful enough to sustain single-threaded operations like decoding. Mobile CPUs, efficiency cores or virtualized processors should only be used if you are certain they are not choking at the per-thread level.

A great way to check this is to run the top command on your host, get the PID of "OvenMediaEngine" then run top -H -p 12345 where 12345 is the PID. It will show you the individual threads running within the application. If any of these are hitting 100% that means the individual threads are overloaded and your CPU cores are not fast enough for the task at hand.

Tuning the Linux Kernel

By default, the Linux Kernel cannot handle high-throughput. There are some system-level configuration files you can edit with more optimized values:

/etc/sysctl.conf

fs.file-max = 100000
net.core.somaxconn = 65535
net.ipv4.tcp_max_tw_buckets = 1440000
net.ipv4.ip_local_port_range = 1024 65000
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_max_syn_backlog = 3240000
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.rmem_default = 16777216
net.core.wmem_default = 16777216
net.core.optmem_max = 40960
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.core.netdev_max_backlog = 50000
net.ipv4.tcp_max_syn_backlog = 30000
net.ipv4.tcp_max_tw_buckets = 2000000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 10
net.ipv4.tcp_slow_start_after_idle = 0
​
/etc/security/limits.conf

* soft nofile 1048576
* hard nofile 1048576

Performance Testing

Here is a great tool for testing multiple connections to the server: http://files.streamwell.net/tester.zip

To run the test:

  • Install or confirm you are running GO on your test client: https://go.dev

  • You will need to disable SignedPolicy in the Engine config temporarily for the test to run. Under Administration -> Server -> Engine Control -> Configure Engine, comment out or cut the <SignedPolicy> section of the config.

  • Choose an open channel and copy the stream key only (minus the signature, everything including "?=" forward)

  • Start your test stream and run OvenRTCTester like so (use wss: if you are connecting to a hostname with HTTPS enabled):

$ go run OvenRtcTester.go -url "ws(s)://<SERVER-IP-OR-HOSTNAME>/live/streamKey" -n "25"

  • Adjust the -n value up bit by bit to add more simulated viewers as your tests complete successfully. Please note you will need to conduct these tests on a fast machine with fast networking as it is playing the role of potentially hundreds of viewers. A good way to conduct this test is to spin up a VPS on a reliable service like DigitalOcean and run the test from there. This ensures that your local network / computer is not acting as a bottleneck.

SRT Encryption

SRT passphrase encryption is available, along with 50+ other native SRT socket options.

To enable SRT encryption, simply uncomment this section in the engine configuration and change the passphrase to something you are familiar with:

<!-- <Options>
<Option>
<Key>SRTO_PASSPHRASE</Key>
<Value>thisismypassphrase</Value>
</Option>
</Options> -->

N.B. SRT Passphrases must be between 10-80 characters in length or the Engine may not start!

You can also add additional SRT socket options in the same manner as shown above. Only the SRTO_PASSPHRASE option has been tested with Streamwell but you can really get into fine-tuning if you are brave!

Different encoders have different ways of setting the SRT encryption. As an example using our old friend OBS, you can enter the passphrase as shown (the stream key itself stays the same):

Proxy Streaming / ABR

To enable Proxy Streaming, first check the engine configuration and uncomment the sections related to ABR:

<!-- <Rendition>
<Name>Proxy</Name>
<Video>proxy_video</Video>
<Audio>opus_audio</Audio>
</Rendition> -->
 <!-- <Video><Name>proxy_video</Name>
 <Codec>h264</Codec>
<Bitrate>500000</Bitrate>
<Profile>baseline</Profile>
<Preset>slow</Preset>
<Width>640</Width>
</Video> -->

For a quick-start, simply uncomment these and commit the new configuration, then restart the engine. To use your new Proxy Streaming config in the web client, enable Proxy Streaming under Administration -> Server -> Features.

You can edit the encoding settings and add additional <Rendition> entries, each linked to a <Video> component by name. Just remember that for each entry in the <Rendition> list your server will need to transcode another video stream for each channel that is live. This can quickly consume CPU resources at an exponential level, so please be aware of this as you enable transcoding functionality.

See more on this via the OvenMediaEngine documentation.

Timezone Configuration

By default, the server uses an internal clock set to UTC ('Coordinated Universal Time'). You can change this value with an environment variable ("TZ") when you run the Streamwell Docker container.

Besides the default value of 'UTC', the following values are supported:

America:

  • America/Argentina/Buenos_Aires

  • America/Chicago

  • America/Los_Angeles

  • America/New_York

  • America/Toronto

  • America/Vancouver

Africa:

  • Africa/Abidjan

  • Africa/Accra

  • Africa/Algiers

  • Africa/Cairo

  • Africa/Casablanca

  • Africa/Johannesburg

  • Africa/Lagos

Asia:

  • Asia/Dubai

  • Asia/Kolkata

  • Asia/Shanghai

  • Asia/Tokyo

Europe:

  • Europe/Amsterdam

  • Europe/London

  • Europe/Moscow

  • Europe/Paris

  • Europe/Rome

Pacific:

  • Pacific/Auckland

  • Pacific/Honolulu

  • Pacific/Sydney

Firewall Config

Windows and macOS offer built-in firewalls in their system settings, where you can allow connections to the Docker application which should allow Streamwell to function if you have these firewalls enabled.

On Ubuntu and many flavours of Linux, the 'ufw' firewall is built-in. Here is an example of configuring ufw for Streamwell (including SSH access on port 22 - if you enable ufw without this you may lock yourself out of your server!):

ufw allow 22,80,443,1935,1936,3333,3334,8000,8001,8080,8081,10010/tcp
ufw allow 9998,9999,10010/udp
ufw enable -y

Once your system is in production, you should disable / block any unused ports as a best security practice.

Enabling Legacy HLS

Certain external players and devices (like ROKU) require a legacy version of the HLS standard when playing the stream back on those devices. To support this we have enabled a deprecated version of the HLS standard. However this module is not actively supported or maintained and may become incompatible over time, so you should use the modern LL-HLS standard whenever possible.

By default, Legacy HLS runs on port 8080 (HTTP) and 8081 (HTTPS) but can be customized on container start with environment variables HLS_LEGACY_PORT and HLS_LEGACY_TLS_PORT.

Here are the steps to enable Legacy HLS in the engine/interface:

1) Under Administration -> Server -> Engine Control -> Configure Engine, there are two sections marked <!-- Legacy HLS --> with <HLS></HLS> configurations that are commented out. Uncomment these, commit the configuration, then restart the engine.

2) Toggle the legacy HLS links in the interface with cheat codes legacyHlsLinksEnabled or legacyHlsLinksDisabled (enter them under Administration -> Customization -> Set Custom Title).

3) A new 'Legacy HLS Link' will appear in the stream info panel on each channel. Ta-da!

Note: HLS streams need to fill their playlist before they are available to stream. For legacy HLS this takes up to 1 minute. You can change the segment duration and count in the engine configuration but it is not recommended to lower it beneath a duration of 3 and a count of 5.

Cheat Codes 🤓

Some advanced, very specific features are activated via CHEAT CODES. These can be entered in Administration -> Customization -> Set Custom Title. Use at your own risk!

encoderControlEnabled / encoderControlDisabled Lets you connect Uray UHE-series encoders directly to channels.

legacyHlsLinksEnabled / legacyHlsLinksDisabled Show links for legacy HLS streams (Old spec, TS chunked instead of fMP4). This is specifically for Roku/Amazon Fire/Android compatibility.

legacyHlsAddRokuSuffixEnabled / legacyHlsAddRokuSuffixDisabled Changes the label on HLS links to refer to them specifically as 'Roku' links.

adminWarningsEnabled / adminWarningsDisabled Supress popup warnings in the admin interface for low space / expiring TLS cert, etc.

h265FirefoxEnabled / h265FirefoxDisabled By default, H265 is blocked in FireFox because it does not work. If you have a workaround in place, use this to unblock it.

hideSsoSyncButtons / showSsoSyncButtons Hides or shows the SSO user sync buttons, preventing admins from syncing with the remote directory (which deletes any users that no longer exist).

Use External OvenMediaEngine

If you are running your own OvenMediaEngine instance or cluster, you can use these environment variables in your Docker run command to tell Streamwell to connect to the external instance (Requires Streamwell v1.7 or later):

OME_EXTERNAL_HOST (e.g. www.yourOMEinstance.com) OME_EXTERNAL_KEY (your OME signature key, used to calculate the signed URLs) OME_EXTERNAL_TLS (true / false depending on whether your OME instance uses TLS, default false) OME_EXTERNAL_API_HOST (Same as host, OR another path to the API endpoint if it differs) OME_EXTERNAL_API_KEY (your OME API access key, could be same or different from signature) OME_EXTERNAL_API_PORT (your OME API port. If unset, uses regular API_PORT value)

There are a few considerations when using an external OME instance:

  • The "Engine Control" dialog will not be available, as it is only able to manage the internal instance of OME in Streamwell.

  • Recordings can be started and stopped, however the resultant files themselves will be unavailable in Streamwell unless you map the output of your OME instance recordings to the same configuration as is used in the built-in OME instance. Check /opt/ovenmediaengine/bin/conf/Server.xml in the Docker container for details, or contact Pro Services for assistance configuring this.

  • By default, HLS and WebSocket traffic are assumed to be proxied to port 80/443. This is because many corporate or otherwise secure networks won't allow this traffic on non-standard ports. To accommodate this you must either proxy the traffic to your OME server in the same fashion, or to use the default ports (WS 3333, WSS 3334, HLS 8000, HLS TLS 8001, Legacy HLS 8080, Legacy HLS TLS 8081, Stream Data 10010 tcp+udp) you should set your environment variables accordingly with WS_PROXY, HLS_PROXY and HLS_LEGACY_PROXY equal to "false". Check /start.sh in the Docker container for details, or contact Pro Services for assistance configuring this.

  • API communication is strictly between Streamwell and OME so proxies do not apply here. The default API port used in Streamwell is 42069.

Use External Database

Streamwell uses a built-in MariaDB database server, however it can also be connected to an external / cloud hosted SQL-compatible database by specifying these environment variables (Requires Streamwell v1.7 or later):

DB_HOSTNAME (e.g. www.yourmariadbinstance.com) DB_USERNAME (typically streamwell) DB_PASSWORD DB_PORT (default is 3306) DB_SSL (do not set this if the external DB does not require SSL)

In all cases, Streamwell will look for a schema called... wait for it... streamwell

To initialize your remote database and create the streamwell schema, first run the "/create_db.sql" script against your MariaDB/MySQL instance. You'll notice this script also creates a streamwell user and grants all privileges on the streamwell schema as well as SYSTEM_VARIABLES_ADMIN to allow setting up event schedulers, etc... If you wish to use a different user for the remote DB please ensure it has these same permissions or the world will implode.

PreviousOvenMediaEngineNextTerms and Conditions

Last updated