The function in Excel for doing a hyperlink is =Hyperlink("http://www.techonthenet.com","Tech on the Net")
where "http://www.techonthenet.com"
is the internet address and "Tech on the Net"
is the title that appears in the Excel cell.
Thus when you are writing the urls into the Excel file just wrap this function around each url. If you don't want to come up with a dynamic name you can always put the url as the name too.
If you aren't inserting the values programmatically then this site mentions using the HYPERLINK worksheet function. Though a even better reference is this which walks you through how to add a macro to excel and they supply the code for the macro. Thus when after you add this macro you can select the column of urls and run the macro and it converts the whole column into hyperlinks