In this example the sample data is in Cells D2:E7.
Now you have already applied Conditional Formatting to Column E based on Duplicate Values. Make a note of the color or select a color from Custom Format.
Now in D2 put the following Conditional Formatting Rule. You have to select Formula this time.
Apply the same color that you opted earlier for Column E. Click OK and the using Format Painter apply the Formatting to rest of the cells in Column D.
Now your matching values shall be highlighted.
Now put a Filter on First Row. If you have a header select it else the first row of the table. Better to have a Header row to your data.
Now Filter by Color and No Fill. So all non duplicates will be shown and Duplicates shall be filtered. Select the rows and Delete.
Update: How to use Format Painter
After you have applied the Conditional Formatting Formula Rule to A1, click on A1 and go to Format Painter on Home Tab
Click the format painter. Now it will be attached to the cursor.
Now Click and Hold on A2 and just Drag it down up to the intended rows. The Formatting shall get applied to all applicable cells.
See the below sample animation on how to use Format Painter.