In OS X Leopard, when using Finder, somebody released some sort of script which was added to the Finder toolbar/panel as an icon, so if you clicked on it, a terminal window would appear directly at that PATH. I wonder if similar thing exists for Lion, since the old one is broken.
According to this answer, since OS X Lion, the feature is even available as a Service from Terminal.
To assign it to a keyboard shortcut, go to System Preferences » Keyboard » Keyboard Shortcuts » Services.
Here, you need to enable "New Terminal Tab at Folder" and assign a shortcut. As a service, this will not only work from Finder.
Alfred has an "Open Terminal here" action for files and folders too.
When you buy the powerpack, you can invoke Alfred for any file or folder currently selected in Finder, too, and then call the appropriate action. This is what I normally do.
If you do not get what you want, you can always drag the icon in the title bar of the Finder window onto the dock tile for Terminal. Er, at least you can do so under Lion. Dragging is not as easy as clicking, but it's something.