My trial of Red Hat Enterprise Linux 5.8 will expire tomorrow.
What behaviour should I expect? It will not boot?
It will boot and continue working normally, but will not receive any updates, which is quite dangerous.
If you want/need an OS based on the RHEL source code, consider CentOS or Scientific Linux.
Seems like you'll (at least) no longer receive any updates:
I obtained my copy of RHEL 6 using the free trial offer from Red Hat. For a month, you could continue getting Yum updates for free. At the end of that period, Yum complains you have no access to the Red Hat Network (RHN) repository.
You could even change the /etc/yum.repos.d/* to something in the likes of CentOS and thus get updates. RedHat is really all about the support and updates.