Overriding Tailscale rules in Aliyun ECS
Overview
In the previous post, we explained that Tailscale’s default firewall rules were blocking access to our DNS servers. To fix this, we added custom DNS rules that would take precedence over Tailscale's rules. However, when Tailscale or the instance restarts, the default Tailscale rules are reapplied, overriding our custom DNS rules and making them ineffective. In this post, we will configure the firewall to ensure our settings persist across restarts.
Outline of approach
Save and reload firewall rules
First, we will create basic firewall rules. Then, we will set up a process to save and reload these rules automatically upon reboot.
Add Tailscale and DNS rules
After verifying the basic setup, we will add the Tailscale and DNS rules in the correct order and save this updated set of rules.
Enable Tailscale NAT traversal (optional)
Tailscale's default behaviour prevents us from using NAT traversal to establish direct connections with other devices on the Tailscale network. We will apply additional settings to enable this. Without these settings, devices are still able to connect, but via DERP servers instead, which are slower.
Part 1: save and reload firewall rules
We will use this guide to set up basic firewall rules. The following commands are taken from this guide and assume we are connected to the ECS instance via SSH.
Start by checking the current set of rules on the instance:
$ iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
Next, apply the following set of rules to create a simple stateful firewall that allows incoming SSH connections:
iptables -N TCP
iptables -N UDP
iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -m conntrack --ctstate INVALID -j DROP
iptables -A INPUT -p icmp --icmp-type 8 -m conntrack --ctstate NEW -j ACCEPT
iptables -A INPUT -p udp -m conntrack --ctstate NEW -j UDP
iptables -A INPUT -p tcp --syn -m conntrack --ctstate NEW -j TCP
iptables -A INPUT -p udp -j REJECT --reject-with icmp-port-unreachable
iptables -A INPUT -p tcp -j REJECT --reject-with tcp-reset
iptables -A INPUT -j REJECT --reject-with icmp-proto-unreachable
# SSH
iptables -A TCP -p tcp --dport 22 -j ACCEPT
# as mentioned in the guide, do this after SSH rules to prevent the connection from dropping
iptables -P INPUT DROP
Once these rules are applied, create a new SSH connection to the machine to verify they are working.
Part 2: set up firewall rule save and load
We will install iptables-persistent to save the firewall rules and ensure they persist after a reboot.
apt install -y iptables-persistent
After installation, save and restore the rules. Then, create a new SSH connection to the machine to verify everything is working correctly.
iptables-save -f /etc/iptables/rules.v4
iptables-restore < /etc/iptables/rules.v4
Finally, reboot the machine and check that the rules were successfully reloaded.
reboot
iptables -S
Part 3: install Tailscale and configure the rules
Install Tailscale. After the setup, we notice that DNS queries no longer work.
curl -fsSL https://tailscale.com/install.sh | sh
tailscale up
dig google.com # does not work!
Checking the firewall rules, we see that Tailscale has added new rules. In particular, in Chain INPUT, we see that ts-input is at the top, and that Chain ts-input contains DROP 100.64.0.0/10. This is the rule responsible for blocking our DNS responses.
iptables --list
Save the new rules so we can use them later.
iptables-save -f /etc/iptables/rules.v4
Modify /etc/iptables/rules.v4 by moving the addition of the ts-input chain -A INPUT -j ts-input to after the TCP chain.
# diff -c rules.v4.bak rules.v4
***************
*** 7,13 ****
:UDP - [0:0]
:ts-forward - [0:0]
:ts-input - [0:0]
- -A INPUT -j ts-input
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -m conntrack --ctstate INVALID -j DROP
--- 7,12 ----
***************
*** 17,22 ****
--- 16,22 ----
-A INPUT -p udp -j REJECT --reject-with icmp-port-unreachable
-A INPUT -p tcp -j REJECT --reject-with tcp-reset
-A INPUT -j REJECT --reject-with icmp-proto-unreachable
+ -A INPUT -j ts-input
-A FORWARD -j ts-forward
-A TCP -p tcp -m tcp --dport 22 -j ACCEPT
-A ts-forward -i tailscale0 -j MARK --set-xmark 0x40000/0xff0000
Then, rename all instances of ts-input, ts-forward, ts-postrouting to something else, such as ts-input-manual, ts-forward-manual, ts-postrouting-manual, as Tailscale will remove rules with those names after reboot.
Apply the changes and verify that DNS now works as intended.
iptables-restore < /etc/iptables/rules.v4
dig google.com # works!
Configure Tailscale to avoid using setting up firewall rules now that we’ve done so manually.
tailscale up --netfilter-mode=off
Part 4: disable randomize port
When we ping another device on the Tailscale network, we notice that a direct connection cannot be established.
$ tailscale ping my-laptop
pong from my-laptop (100.64.0.1) via DERP(sin) in 9ms
pong from my-laptop (100.64.0.1) via DERP(sin) in 10ms
pong from my-laptop (100.64.0.1) via DERP(sin) in 9ms
pong from my-laptop (100.64.0.1) via DERP(sin) in 8ms
pong from my-laptop (100.64.0.1) via DERP(sin) in 8ms
pong from my-laptop (100.64.0.1) via DERP(sin) in 8ms
pong from my-laptop (100.64.0.1) via DERP(sin) in 9ms
pong from my-laptop (100.64.0.1) via DERP(sin) in 9ms
pong from my-laptop (100.64.0.1) via DERP(sin) in 10ms
pong from my-laptop (100.64.0.1) via DERP(sin) in 9ms
direct connection not established
For Tailscale to establish a direct connection with another machine, it needs to use a process called NAT traversal. A detailed explanation can be found in Tailscale’s blog, but in a nutshell, Tailscale uses a specific port for NAT traversal, and our firewall needs to allow this port.
However, Tailscale uses a random port, which we cannot predict in advance. The firewall rules we created earlier do not allow this port, causing issues with direct connections. To work around this, we need to disable randomizeClientPort in the Tailscale ACLs. Tailscale will then start to use the default port of 41641, and we can modify the iptables rule to allow this port:
# /etc/iptables/rules.v4
-A ts-input-manual -p udp -m udp --dport 41641 -j ACCEPT
Apply the new rules, restart Tailscale to run with the new settings, and try again.
iptables-restore /etc/iptables/rules.v4
tailscale down
tailscale up
Success!
$ tailscale ping my-laptop
pong from my-laptop (100.64.0.1) via 192.168.0.1:41641 in 3ms