Both hosts will consume some memory to keep track of the half-open connection as soon as the handshake starts. However, the client will additionally consume some memory as soon as the program initializes the "socket" representing the connection, even before any packet is sent.
Likewise, as soon as the server receives the SYN packet, it'll allocate some memory to keep track of the connection's parameters.
Sending a large number of SYNs to exhaust the server's resources is actually a common attack called a SYN flood. In response to that, some servers have a "SYN-cookie" mode which doesn't keep resources allocated after sending the SYN+ACK response; it can discard the information until the client's ACK arrives, at which point the server begins tracking the connection normally.
From a programming point of view, the entire handshake happens in one step. However, when you're writing a client, the variable representing that connection is assigned before asking the OS to perform the handshake.
conn = socket(AF_INET, SOCK_STREAM)
# The variable representing the "socket" is initialized and some memory is
# allocated, but no connection is done yet
# The entire TCP handshake is performed during connect()
When writing a server, it's the opposite. The "listening" socket is set up first, but each individual received connection is first accepted by the OS (performing the handshake) and only then provided to the program.
conn = accept(listener)