Only if the source port of the original outgoing datagram was also port N, and if the NAT didn't choose to float the source port.
That is, the first UDP datagram from Machine A looks like this on your LAN:
Source IP: MachineAPrivate
Source Port: PortA <-- note this is typically different than the destination port
Destination IP: MachineBPublic
Destination Port: PortN
Then, after it is translated by the NAT in the outbound direction, it looks like this:
Source IP: NATPublic
Source Port: PortC <-- note this may or may not be the same as "PortA" above
Destination IP: MachineBPublic
Destination Port: PortN
Now, when Machine B replies, the reply typically looks like this:
Source IP: MachineBPublic
Source Port: PortN
Destination IP: NATPublic
Destination Port: PortC
Then, after it goes through the inbound NAT translation process:
Source IP: MachineBPublic
Source Port: PortN
Destination IP: MachineAPrivate
Destination Port: PortA
So, IF Machine A sends the frame from the same source port as the destination port ("Port N"), and IF the NAT is able to preserve that source port (i.e. it's configured to preserve source ports when possible, and that source port is not in use), THEN you can expect a reply to "Port N" to get back to Machine A.
Here's the authoritative reference on proper NAT UDP behavior:
RFC 4787 / BCP 127: Network Address Translation (NAT) Behavioral Requirements for Unicast UDP