File level permisions will be tied to owner, group, and world. Unless you want to change the owner of the file (
chown), you can't do anything user wise.
The u flag for
chmod is actually the owner of the file. You can use g instead to change the group permissions, e.g.
chmod g-r file
Alternatively, you can use the numeric mode to change permissions for owner, group, and world all at once.
Relatedly, if you want to only have certain users access the file, you may need to create a new group (
groupadd) and then change the group on the file using