What network filesystem do I need to set up? NFS? AFS? SMB?
SMB has traditionally been used in Windows environments but these days it also has good support on other platforms. Enabling the service will allow you to access files on the NAS from both your Mac and your Windows laptop. You'll need to configure users/groups first though.
When you say AFS I think you mean AFP since the Synology products do not support AFS. AFP is Apple's proprietary standard loosely similar to SMB but it doesn't have widespread support. You will probably have to enable it anyway though since you are using Time Machine.
Don't bother with NFS as it is slower (due to the overheads involved and the limited resources of the NAS) and has no real advantages for your situation.
The box does DNLA. Should I turn this on for media sharing, or will the shared folders suffice?
If you want to be able to play content on your XBOX you can use the built in UPnP/DLNA media server, or there are several third party applications (known as "packages") available that do the same thing. Bear in mind that this type of device cannot transcode video on demand, so you need to make sure that everything is in an appropriate format.
Is there a Time Machine equivalent for the Windows laptop I should be using?
It would be very difficult to recommend a direct alternative to Time Machine. You are best to research this yourself so that you can be sure that you get the exact functionality you need. Most commercial options out there offer a trial of some sort. I would like to point out that Windows has some backup features built in, particularly in the Ultimate, Business and Enterprise versions, although the end result is not as sophisticated as Time Machine.
Should I set the box to DHCP, or fix an IP address. What do I need to watch out for?
I'm not sure what advantage you would get from using a fixed IP address. Normally you would access the NAS by hostname, e.g. smb://diskstation in OS X or \\diskstation in Windows. In some cases it may not be possible to use the friendly name in which case choosing a static IP might make more sense.
What else do I need to know?
Ultimately you might find some useful information within the Tutorial section of the Synology web site: http://www.synology.com/tutorials/tutorials.php?lang=us. The same site also has some forums where people are generally very helpful.
Also Synology release new firmware quite frequently so if you encounter any compatibility issues, e.g. relating to newer versions of OS X, then you may need to upgrade it.