Working with a different hosting option that uses FTPS instead of SFTP, I found that my ASUS RT-AC56R was preventing passive, explicit FTPS connections. It took some hours of troubleshooting and patient support of the hosting company but I finally narrowed down the problem when I bypassed the router and was able to connect to the server securely when directly plugged into my cable modem. I was also able to connect using plain FTP but that is not a secure option. The FTPES problem also persisted across trying different clients.
I tried disabling many features of the ASUS modem but it was not until I got to the AiProtection Vulnerability Protection integrated from Trend Micro that I was able to connect to the server. Once I disabled that setting, I was able to connect properly whereas before it would hang at the step of ‘Initializing TLS’. Using packet monitoring software Wireshark, the TLS Ack packets were being dropped by the router when Vulnerability Protection was enabled.
I did not necessarily want to just disable vulnerability protection but I checked for firmware updates through the router admin panel and it said I was running the latest version, version 3.0.0.4.378_4585. I wanted to confirm that it was correct, so I headed to the ASUS website after finding similar reports of the problem on other router models like the RT-AC66U. I could not find a support page for the RT-AC56R modem (the ending ‘R’ stands for Retail, meaning it is found in Best Buy, Wal-Mart, and other physical retailers) but I did find a page for the RT-AC56U. Essentially, the identical modem, it had several newer versions of the firmware up to 3.0.0.4.378_9459 with lots of bug fixes in each release.
I downloaded the firmware and uploaded it to the router through the admin panel. As soon as it was done applying the update, I logged back in and everything seemed to be working normally. I headed to the AiProtection – Network Protection page and enabled Vulnerability Protection. Now, with the Protection enabled, I was still able to connect to my server over FTPES. I was also able to get the OpenVPN configuration to work, which always failed before.
I guess the moral of the story is to check a second source when trying to keep your devices up-to-date. One of the bugfixes might just be for the auto-update system, which worked before.