Telnet can be used to manually do things that programs do automatically. When I connect to a server I noticed that it automatically does the 3-way handshake to establish a TCP connection. Can it be done manually in telnet? If not, then what program can do it?
If by "manually" you mean "instruct
It would be possible for a program to use raw IP sockets and manage the TCP layer all by itself. But it's generally an incredibly pointless thing to do – it would copy a lot of code from kernel.
If you are trying to learn how TCP works, try Scapy. It can create and send packets using given parameters, including TCP. (You'll probably have to use a packet sniffer to watch for such things as SYN+ACK though.)
The book "TCP/IP Illustrated, Volume 1" by Stevens explains what goes on in creating and using a TCP connection. Yes, it has been around a while ( (c) 1994 ) but sometimes the explanations given when a technology is new have more depth than anything that comes after.
To see the details of what is happening on the wire a copy of Wireshark (http://www.wireshark.org) is what you need. When you understand how a header and message are structured and packed wireshark shows you the structure and you can drill down to the bit fields.