What is it? ...
Wikipedia has a decent explanation. From that article:
/bin: "Essential command binaries that need to be available in single user mode; for all users, e.g., cat, ls, cp."
/usr/bin: "Non-essential command binaries (not needed in single user mode); for all users."
In the old days of small disks, it was common to have /bin on the boot disk, and /usr/bin on a separate disk (sometimes even a shared network drive). /bin would contain anything needed to either boot the system, or fix it if /usr/bin couldn't be mounted for some reason. Its another one of those historical accidents that doesn't make as much sense today.
The distinction is simple:
/bin is where system binaries are stored (mostly these binaries are owned by root) whereas
/usr/bin is the usual spot for binaries that are used by ordinary non-root users. It is also a common place to deposit binary programs that were built by the compiler to be used by other users.
A lot of times (even today, for various reasons), people create /usr as a separate filesystem. In these circumstances, / is mounted first, then programs on / are used to mount /bin. So there's a requirement that anything needed to get to the point of mounting /usr needs to live in /bin or /sbin. Additionally, anything useful in troubleshooting problems (if you can't mount /usr) is also best stored in /bin or /sbin.