No, it can't be done using iptables. The answer is in the name of the command and what you're trying to do.
In the OSI model, MAC addresses belong to the data link layer. This means that protocols such as IEEE 802.3 (Ethernet) have a concept of MAC addresses, but protocols such as IPv4 don't. Data link layer protocols can be thought of as transmitting "frames".
iptables contains "IP", which means Internet Protocol. This is a protocol at the Network layer, which is a higher layer than the data link layer. Network layer protocols can be thought of as transmitting "packets".
Since the protocols are built on top of one another like a stack, it is only possible for lower level protocols to directly affect the behavior of higher level protocols, not the other way around. So, it is not possible for a tool which operates on the IP layer to directly affect the behavior of a hardware device operating on, say, the Ethernet (802.3) layer.
If you know about HTTP, you can think of it this way. How would you answer if someone asked you, "How can I send arbitrary data over TCP port 6790 by just using HTTP?" The answer is that you can't -- for one, the TCP port is already decided by the time the HTTP protocol is in play. Second, the HTTP protocol has mandatory headers and structural elements that can't be removed without violating the standard. So as soon as you say "HTTP" you're talking about operating within an existing TCP socket on a pre-determined port and you're stuck adhering to the protocol rules of HTTP.
The same is true of IP. Once you're dealing with logical addresses in the IP stack, you can't "reach down" into the physical layers and do low-level stuff. As a simple explanation why, it's possible for a device to have a valid IPv4 address, but to operate using a data link layer that does not use MAC addresses at all. For example, a USB protocol could be designed to transmit IP packets, but it would have no concept of "Ethernet" or "frames", so it wouldn't have a MAC address or implement the IEEE 802.3 standard.
What you need is
ebtables will allow you to perform similar processing at the ethernet frame layer. You will need to be certain that all the devices you want to firewall have a direct ethernet connection to the computer that is doing the firewalling, otherwise this won't work. And if any of your devices don't use ethernet at all, you're out of luck.