How to selectively filter or block non-http/s traffic?

Question

I know that it is possible to filter web content, say, with Squid+ICAP Server.

But, is it conceptually as well as practically possible to monitor, filter, and programatically transform arbitrary (outgoing + incoming) traffic crossing your organization's routers?

For example, if a programmer in your organization decides to employ a client-server program purposely built to send sensitive data out of the organization (with his server listening on standard http/s port or any other known/arbitrary port somewhere on the Internet), then using what techniques and software can this malicious attempt be monitored and controlled?

I'm interested in getting some pointers on both the concepts/techniques involved, as well as some Linux-based FOSS suggestions that I could explore further. Note that, DLP products such as MyDLP talk only about web content, and not the above scenario, namely, of data theft via a purpose-built program using standard or non-standard data transfer protocol.

Answer 1

Is it possible to monitor/filter/transform arbitrary traffic?

Yes. It's certainly conceptually possible. Before routers were readily available and cheap, it was common to find a cheap old computer, install Linux and share the network connection (IP Masquerading, etc.). You could watch everything just with tcpdump, if nothing else. And that is everything - you'll see every SYN-ACK handshake, every SSL certificate request, every DNS lookup, etc.

What techniques could help monitor/control?

The obvious one is to watch for what hosts you're connecting too. Lots of tools to help with that, same stuff that keeps kids of adult sites and employees off facebook. See this unix.se question, notably ntop.

Restricting ports certainly reduces the space. A port is just an arbitrary number, but I've consulted for paranoid organizations that locked out everything except port 80. That forced us to do things like tunnel ssh over https or more elaborate schemes (two-headed ssh tunnels) when we needed to get stuff from back home.

But that still leaves a scary upload-company-secrets tunnel that just looks like HTTPS. I've been playing with Fiddler a lot recently. If you were really hell-bent on catching everything, you'd situate an https logging proxy in the middle, and just declare that everyone inside your shop has to accept your certificate, which means you watch everything. So much for privacy, of course -- you'll see people gmail passwords in plain text (really! try it and see!) -- but it's gonna get awfully tricky for your black hat to get anything out without you noticing.

Useful thought experiment, anyway.