It isn't very clear from the question but I have to assume that some other program is using the port. Which stops you from opening the port, required to get the settings. Serial ports cannot be shared. You can do so after the program closes the port, but there are non-zero odds that it restores the settings.
The best way is to use the free SysInternals' Portmon utility. It installs a filter driver that snoops on the serial port driver. You'll see everything the program does, including initializing the port.