I don't believe that formula language is translated by the Domino router during SMTP/MIME conversion the way it is by Domino's HTTP server. So I don't think you can use formula language.
And I don't think there's an equivalent to the "mailto:" URL that causes the receiving mail client to generate a reply. It's not a Notes restriction. It's just not part of the standards. I checked out the standard for mailto syntax, and it doesn't seem to be covered.
The only thing I think you can do is set the Reply-To header in the outbound message, and give the recipients instructions in the text telling them to use the Reply or Reply with Attachments feature of their mail program. To actually set the Reply-To header, just set the ReplyTo field in the Notes message.