We have an entry level enterprise wireless router TP-Link TL-WVR1300G (Chinese page) in city B, and we have two physical servers located in city A and city B, and we setup a VoIP (SIP) trunk through internet between these two servers.
Everything was fine until recent days: The voice quality on this VoIP trunk become so bad that two side of a call can't hear each other (or one side can hear another side, but not vice versa).
At first, I thought it was caused by poor internet or some one consuming large bandwidth, but after the following check and diagnoses, it's not the reason.
Checking and diagnosing
When poor voice quality happened,
- The total internet traffic in city A did not reached full bandwidth of ISP provided, not even reached half of the bandwidth.
- I can ping IP address of the router in city B from server in city A, but I will got large packet loss percentage (>10%, I even got 63% once, ping 100 times in every 5 minutes). Although I will not got packet loss percentage greater than 10% every 5 minutes, but still, lots.
- Employees in city B may experienced low internet speed at the same time.
- I can't the access web administration page of the router using a popular browser -- there's no response.
Well, actually, if I use
curlcommand to access the homepage of the web administration page of the router in the morning, I can get response. Then I tried using
wget -rto fetch the whole site of router, it hangs when accessing one .png file:
--2018-11-18 13:43:03-- http://172.18.1.1/webpages/themes/neoteric/img/password_unshow.png 正在连接 172.18.1.1:80... 已连接。 已发出 HTTP 请求，正在等待回应... 200 OK 长度：3365 (3.3K) [image/png] 正在保存至: “172.18.1.1/webpages/themes/neoteric/img/password_unshow.png” 172.18.1.1/webpages/themes/neoteric/img/password_unshow.png 100%[======================================================================>] 3.29K 660 B/s 用时 5.6s 2018-11-18 13:43:08 (600 B/s) - 在 3372/3365 字节处发生读取错误 ((null))。重试中。 --2018-11-18 13:43:09-- (尝试次数： 2) http://172.18.1.1/webpages/themes/neoteric/img/password_unshow.png 正在连接 172.18.1.1:80... 已连接。 已发出 HTTP 请求，正在等待回应... ^C
Now, the web administration page of the router is not accessible (no response) even if using command line to access it until 08:00am next day.
But look at the byte range that
2018-11-18 13:43:08 (600 B/s) - 在 3372/3365 字节处发生读取错误 ((null))。重试中. The size of the png file is 3365 bytes, yet
wgettry to read at 3372 byte which is larger than the size of file (and it got null), why?
- So I disabled loading images option (change
2in about:config page) in Firefox, and visit the web administration page of the router next day, no trouble. And I found there's a config named 'Auto cleanup', and it was scheduled at 08:00am each day (That's why the router automatically self healed).
- In the weekend, when there're no employees on duty in city B, the packet loss percentage of ping is very low even if the web administration page of router is not accessible.
- Is it a phenomenon of hardware failure (flash memory failed?) that the router does not respond the HTTP request when accessing the
- Possible reasons that caused large packet loss percentage?
I tried the python script from https://github.com/coincoin7/Wireless-Router-Vulnerability/blob/master/TplinkDiagnosticAuthenticatedRCE.txt, and logged into the router using
telnet, it is running an OpenWRT variant, and I found there're IO errors when copy some image files:
root@TP-LINK:/etc/config# uname -a Linux TP-LINK 3.3.8 #1 Wed Oct 19 19:14:51 CST 2016 mips GNU/Linux root@TP-LINK:/etc/config# cat /etc/openwrt_version 14.07 root@TP-LINK:/etc/config# cat /etc/openwrt_release DISTRIB_ID="OpenWrt" DISTRIB_RELEASE="Barrier Breaker" DISTRIB_REVISION="r38802" DISTRIB_CODENAME="barrier_breaker" DISTRIB_TARGET="ar71xx/generic" DISTRIB_DESCRIPTION="OpenWrt Barrier Breaker 14.07" DISTRIB_TAINTS="no-all no-ipv6 busybox" root@TP-LINK:~# ll /www/webpages/themes/neoteric/img/password_unshow.png -rw-rw-r-- 1 root root 3365 Oct 8 2016 /www/webpages/themes/neoteric/img/password_unshow.png root@TP-LINK:~# cp -p /www/webpages/themes/neoteric/img/password_unshow.png /tmp cp: read error: Input/output error root@TP-LINK:~# cp -p /www/webpages/themes/neoteric/img/password_shown.png /tmp cp: read error: Input/output error root@TP-LINK:~# cp -p /www/webpages/themes/neoteric/img/tp-link.png /tmp cp: read error: Input/output error root@TP-LINK:~# cp -p /www/webpages/themes/neoteric/img/icons-02.png /tmp/ root@TP-LINK:~# cp -p /www/webpages/themes/neoteric/img/icons-01.png /tmp/ root@TP-LINK:~# cp -p /www/webpages/themes/neoteric/img/loading.gif /tmp/ root@TP-LINK:~# cp -p /www/webpages/themes/neoteric/img/refresh.png /tmp/ cp: read error: Input/output error
So, it seems that the flash memory is failed.