Yes. Every piece of information on your computer is a comprised of bits (on PCs it's normally 8 bits per byte, and some systems define bytes as having a different number of bits). Although, source code, technically, is stored in binary form, a variety of approaches and methods can be used to identify it as "text."
Binary is the notation used to describe the base-2 representation of the data (1s and 0s). This document explains this notation in great detail, and will probably be of interest to you:
After compilation, the resulting binary data usually represents a sequence of instructions (a.k.a., sequences of byte codes) that the processor understands and acts on accordingly. There are a large number of instructions that perform mathematical operations, modify memory contents, communicate with peripherals, and deal with a number of other things.
A high-level language like C, Pascal, Perl, etc., can be compiled into binary code which is then run directly by the CPU, and this binary code is most commonly referred to "machine language."
Assembler source code (which is not considered a "high-level language") is the form of source code that is closest to machine language, but also needs to be compiled to binary code for the CPU to process it correctly.
When interpreted code is run (such as with a script), it is being compiled on-the-fly by an interpreter as each line or section is encountered, and tends to run much slower than binary code as a result. BASIC (such as GW-BASIC from the DOS days in the 1980s and early 1990s), although not known as a scripting language, was also interpreted (including the loops -- there was no caching, every line was continually re-interpreted).