To build on Paul's answer, if you want to enter incognito mode by default and you're using Windows, you can modify your Chrome shortcut's target adding --incognito.
For example:
C:\<your path>\Google\Chrome\Application\chrome.exe --incognito
Just update <your path> with wherever Chrome is installed locally and You'll be incognito whenever you start up your Chrome browser with the modified shortcut.
Manually cear History:
As a follow-on, you can also manually clear Chrome's history with any of the following options:
You can read more about this at the support page: Delete your browsing history