I am using VMware and I want to connect to Ubuntu VMs via serial port. I am connecting both using pipe as referred in this website https://www.vmware.com/support/ws3/doc/ws32_devices3.html How do I proceed to connecting and using them?
You should probably try this in three steps: 1. setup a terminal emulator program on the Host side. 2. test the login program on the VM side. 3. setup Ubuntu to boot with a serial console.
Step #2 will verify that you have a working virtual serial link before attempting the more complex step #3. Otherwise step #2 is just a test, and not a required step in setting up a serial console. The first half of step #3 may suffice for your requirements (which you have not described) and then you won't have to do any boot parameter modifications.
Step 1: setup a terminal emulator program on the Host side.
Follow the VMware guide that you mention for "Connecting an Application on the Host to a Virtual Machine". The "application on the Host" will be a terminal emulator program such a HyperTerminal, Putty or TeraTerm (the later two are preferred and multi-platform). You are going to use only the bare essentials of the terminal emulator program; you do not need any SSH or telnet or even modem capability.
Configure the Host's terminal emulator for VT102 emulation and the port to 115200 baud, 8 data bits, no parity, 1 stop bit. Use the lowest numbered serial port (e.g. COM1 or ttyS0) from VMware.
Step 2: test the login program on the VM (Ubuntu) side.
Use the lowest numbered serial port, /dev/ttyS0. Make sure your username is a member of group "dialout" (which should own /dev/ttyS0).
Configure the virtual serial port at the Ubuntu shell prompt:
Inspect the terminal emulator program for output.
Execute the login program for the serial connection.
Try to login at the terminal emulator program.
Step 3: setup Ubuntu to boot with a serial console.
If you get this far, then you should use the Ubuntu howto for serial console.
The "Configuring the console login process" will use the same getty that was tested in the previous step. This configuration is to ensure that getty is re-executed (or "respawned") at the appropriate runlevels and after each logoff. Note that there are different startup procedures depending on the version of Ubuntu, so I won't bother trying to reiterate that here.