How to run Jitsi video conference server behind NAT and internal Reverse Proxy

Written by - 0 comments

Published on - Listed in Internet Nginx Linux Network Office


To be independent of externally hosted video conferencing tools, such as MS Teams, Google Meet, Zoom and others, I've installed a Jitsi video conference server a few years ago. Besides having full control over the server and its resources, I and my clients (I do offer consulting services, too) can be sure that the conversation is not registered on any third party (cloud) server and all (temporary) data is stored in Switzerland on my own environment.

This setup, behind a NAT and an internal Reverse Proxy (Nginx), has been working very well in the past - until Jitsi was updated a few months ago. Since then I ran into weird connectivity and video issues previously not seen.

External users not able to join 

One of the new issues was that external users were not able to join the video call. While devices in the internal LAN were able to correctly join the conference call, external users coming from the Internet were not able to join. To make this even more interesting: When a remote participant wanted to join the call, everybody was kicked off the meeting (disconnected), even the internal devices. 

The logs of the jitsi-videobridge (jvb) showed such entries:

root@jitsi:/var/log/jitsi# tail -f jvb.log
JVB 2023-10-31 08:16:42.382 SEVERE: [30] HealthChecker.run#181: Health check failed in PT0.000014S: Result(success=false, hardFailure=true, responseCode=null, sticky=false, message=Address discovery through STUN failed)
JVB 2023-10-31 08:17:42.382 SEVERE: [30] HealthChecker.run#181: Health check failed in PT0.000012S: Result(success=false, hardFailure=true, responseCode=null, sticky=false, message=Address discovery through STUN failed)

At the same time Kernel log entries showed the firewall rejected an outgoing connection from Jitsi (SRC) to the remote participant (DST) on port UDP/443.

According to a comment in issue #1165 on GitHub, UDP/443 outgoing (to any) is now required for Jitsi, followed by a restart of jitsi-videobridge.

But after this additional outgoing firewall rule was added, yet another annoying error showed up.

Video quality impaired / conference call not working

Once the problem with the remote participant join was fixed, the video conference still did not work. Instead a warning pop up showed up in the bottom left part of the screen, indicating a problem with the communication to the video bridge:

Jitsi conference call not working, warning showing video quality impaired

The information showed the following warning:

Video quality impaired
The bridge channel has been disconnected and thus video quality is limited to its lowest setting.

However video (and audio!) did not work at all - it was not just limited to lower settings.

The Jitsi log in this situation was pointing to a failed connection back to the client:

JVB 2024-03-28 13:10:38.048 INFO: [99] [confId=77ad92d64b80315 conf_name=test@conference.meet.example.com meeting_id=977a6475 epId=4de9e2aa stats_id=Oleta-sYL local_ufrag=2tnje1hq2itsb9 ufrag=2tnje1hq2itsb9] ConnectivityCheckClient.processTimeout#881: timeout for pair: 192.168.1.190:10000/udp/host -> 172.16.238.1:32887/udp/prflx (stream-4de9e2aa.RTP), failing.
JVB 2024-03-28 13:10:38.070 INFO: [99] [confId=77ad92d64b80315 conf_name=test@conference.meet.example.com meeting_id=977a6475 epId=4de9e2aa stats_id=Oleta-sYL local_ufrag=2tnje1hq2itsb9 ufrag=2tnje1hq2itsb9] ConnectivityCheckClient.processTimeout#881: timeout for pair: 192.168.1.190:10000/udp/host -> 192.168.187.1:49514/udp/prflx (stream-4de9e2aa.RTP), failing.
JVB 2024-03-28 13:10:44.680 INFO: [97] [confId=77ad92d64b80315 conf_name=test@conference.meet.example.com meeting_id=977a6475] Conference.recentSpeakersChanged#475: Recent speakers changed: [4de9e2aa, 2853e836], dominant speaker changed: true silence:false

I tried all kinds of hints, from reinstalling Jitsi to using a newer prosody version to reconfigure Jitsi from scratch, but nothing helped.

At last I came across a very important hint, that recent Jitsi versions now use web socket connections, which require changes in the reverse proxy setup. 

Rebuild the (network) architecture

With all these problems and also the fact that there seems to be updated firewall requirements, I decided to re-create the firewall rules (port forwardings) and reverse proxy config from scratch.

Jitsi behind NAT and internal Reverse Proxy

So let's start from scratch - network wise.

I did not touch the existing Jitsi configuration, which I adjusted to use prosody authentication. The installed Jitsi packages on the Jitsi server (Debian Bookworm) are currently at the newest version anyway:

root@jitsi:/# dpkg -l |grep jitsi
ii  jitsi-meet              2.0.9364-1          all          WebRTC JavaScript video conferences
ii  jitsi-meet-prosody      1.0.7874-1          all          Prosody configuration for Jitsi Meet
ii  jitsi-meet-turnserver   1.0.7874-1          all          Configures coturn to be used with Jitsi Meet
ii  jitsi-meet-web          1.0.7874-1          all          WebRTC JavaScript video conferences
ii  jitsi-meet-web-config   1.0.7874-1          all          Configuration for web serving of Jitsi Meet
ii  jitsi-videobridge2      2.3-92-g64f9f34f-1  all          WebRTC compatible Selective Forwarding Unit (SFU)
ii  lua-basexx              0.4.1-jitsi1        all          baseXX encoding/decoding library for Lua
ii  lua-cjson:amd64         2.1.0.10-jitsi1     amd64        JSON parser/encoder for Lua

The following system architecture drawing should be helpful information how my Jitsi setup looks like:

Jitsi server behind NAT and internal reverse proxy

 There are two very important parts of this setup:

Port forwarding (DNAT) on firewall

The firewall (either on your router or on a separate device) handles the incoming connections and must forward specific ports to the Jitsi server in the internal LAN. The following ports need to be forwarded:

  • UDP/10000 is used for the connection to the Jitsi video bridge => to Jitsi server
  • UDP/3478 is used for connecting to the stun server (coturn) => to Jitsi server
  • TCP/5349 is used as a fall-back connection for video/audio communication (in case the UDP connection did not work) => to Jitsi server
  • TCP/443 is used for accessing Jitsi (UI) itself, in this case an internal Reverse Proxy (Nginx) is handling these connections and is also responsible for using a valid TLS certificate => to Reverse Proxy server

Reverse Proxy setup including Web Sockets

As mentioned above, Jitsi now seems to use web sockets for the video call (since when exactly I was not able to find out). The Nginx configuration on the reverse proxy was adjusted accordingly:

root@reverseproxy ~ # cat /etc/nginx/sites-enabled/meet.example.com.conf
###########################
# meet.example.com
###########################

server {
  listen 443 ssl http2;
  server_name meet.example.com;
  access_log /var/log/nginx/meet.example.com.access.log;
  error_log /var/log/nginx/meet.example.com.error.log;

  # [...] ssl settings and stuff [...]

  location / {
    include /etc/nginx/proxy.conf;
    proxy_set_header X-Forwarded-Proto https;
    proxy_pass https://192.168.1.190;  # internal IP of Jitsi server
  }

  location /xmpp-websocket {
    proxy_pass https://192.168.1.190;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
  }

  location /colibri-ws {
    proxy_pass https://192.168.1.190;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
  }

}

After applying the new reverse proxy configuration, the error with the "impaired video quality" disappeared!

Jitsi conference calls working again!

To sum this up, the following network configurations were adjusted or verified:

  • Incoming port forwarding directly to Jitsi server
  • Outgoing connections from Jitsi server allowed
  • Reverse proxy configuration adjusted for web sockets (Connection Upgrade)

Once these configs were corrected/verified and Jitsi restarted, video conference calls started working again with both internal and remote devices (tested here with desktop computer and smartphone connected to 5G). Hurray!

Jitsi conference call working again


Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.

RSS feed

Blog Tags:

  AWS   Android   Ansible   Apache   Apple   Atlassian   BSD   Backup   Bash   Bluecoat   CMS   Chef   Cloud   Coding   Consul   Containers   CouchDB   DB   DNS   Database   Databases   Docker   ELK   Elasticsearch   Filebeat   FreeBSD   Galera   Git   GlusterFS   Grafana   Graphics   HAProxy   HTML   Hacks   Hardware   Icinga   Influx   Internet   Java   KVM   Kibana   Kodi   Kubernetes   LVM   LXC   Linux   Logstash   Mac   Macintosh   Mail   MariaDB   Minio   MongoDB   Monitoring   Multimedia   MySQL   NFS   Nagios   Network   Nginx   OSSEC   OTRS   Office   PGSQL   PHP   Perl   Personal   PostgreSQL   Postgres   PowerDNS   Proxmox   Proxy   Python   Rancher   Rant   Redis   Roundcube   SSL   Samba   Seafile   Security   Shell   SmartOS   Solaris   Surveillance   Systemd   TLS   Tomcat   Ubuntu   Unix   VMWare   VMware   Varnish   Virtualization   Windows   Wireless   Wordpress   Wyse   ZFS   Zoneminder