Should I change my hostname on the machine to mysubdomain.mydomain.com?
Normally, SPAM filter take apart of the domain name, the IP originating the spam (in some cases even the entire ip block or the ASN completly, check UCEPROTECT and Whatsmyip.com blacklist check)
What other things do I have to do with the DNS to prevent my email from going into the spam folder?
This has been said several times already but, do not send SPAM, that should be the first method. After that, use SSL/TLS for all the connections from/to the server. Suspend accounts that use your server to send SPAM. Don't configure your server as Open Relay. Harden your server against hackers. Etc. Etc. There is just too many tools and method you can employ to do this that it will not end.
I just updated the PTR to the main domain name. How can I diagnose why my server's mail isn't reaching its destination?
Are you sure that your ISP allows you to send E-mails? Have you configured your server ports correctly? Is the server able to mail itself? Is difficult to diagnose since there are just too many things could have go wrong from local, to middle to end point and without logs, pings, test and more test is impossible to know. If you want you can check http://whatismyipaddress.com/blacklist-check and input your server IP. If it's listed is probable that that could be the reason you don't receive emails from the server, if it's not listed then you must provide more information and test that you have done.
Resources:
sendmail
from the command line would give you a better understanding of what is going on.