The initial key exchange is structured in such a way that it is not be possible for your ISP to determine the key, even if they have intercepted the entire key exchange process. This is possible because of asymmetric public-key cryptography.
In assymetric cryptography, data is encrypted and decrypted using different keys (unlike symmetric cryptography, where the same key is used for both). The encrypting and decrypting keys are related in such a way that it is not practical with present computers to calculate one from the other.
Public-key cryptography is the typical case of asymmetric cryptography where one key is the "Public" key and the other key is the "Private" key. When you connect to a computer on the internet using SSL, the server has a private key and your computer has a private key (generated by your browser for the connection). Your computer and the server exchange public keys, but do not need to send each other private keys. Your computer then encrypts data using the server's public key, and the data can be decrypted only with the server's private key. The reverse works as well: the server encrypts data with your public key, and only your private key can be used to decrypt it.
Because the private keys are never exchanged on the wire, your ISP cannot see them. Your ISP does see the public keys, but it is not possible to decrypt information using the public keys (only to encrypt it), so your ISP is unable to decrypt the information exchanged.
As an interesting note, public-key systems generally also work in reverse - the private key can be used to encrypt something, and it can then only be decrypted using the public key. This is how 'digital signatures' work - if the public key of a person or device successfully decrypts information, then you know that it was created by someone or something in possession of the private key.
Note that asymmetric encryption is much more processor intensive than symmetric encryption. To maximize performance, SSL uses asymmetric encryption only for the key exchange stage - your computer and the server create a temporary connection protected by asymmetric encryption in order to exchange a key that will be used to encrypt the actual connection symmetrically. The symmetric key is never revealed in plaintext though, so your ISP never sees it.
Asymmetric encryption is a very interesting system that is vital to security on the internet: it has the powerful property of allowing two parties to communicate securely without having to trust the messenger.