Post-Solving Edit

The reason this was so hard to solve at the firewall level was that it wasn't a firewall problem. Something @tom-yan said in chat made me revisit the script which pulls out the IP Addresses. It seems that sed was pumping out the entire access_log file (rather than what I was telling it - as it did in testing). I replaced it with grep and the script is now exporting what I would expect.

Which makes this yet another example of check your assumptions.

Thank you to Tom Yan & @pillsbury-it-doughboy for your efforts. I appreciate it.

(Since Tom's is the only surviving answer, I'm going to mark his as the correct one.)


I've got a small development server at my office with port 80 and 443 port forwarded from the modem.

To restate the title in the form of a question:

Why doesn't the firewall drop the incoming traffic from the IPv4 addresses that are listed in the rich rules.

Background: As you would expect, Bots & Baddies™ are looking for various things that a) don't exist, and b) would be bad if they they got into if they did exist. So I have a script that pulls IP addresses from Apache logs, which then end up in the firewall after they've been curated by me.

However, the firewall isn't dropping the connections. Addresses that are added on previous days are present in the new rules to be added and the logs again on subsequent days.

Details: The command that puts in the rich rules is this:

firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='' reject"

However some rules have extra info:

rule family="ipv4" source address="" log prefix="Shodan.io" reject

While some are simply:

rule family="ipv4" source address="" reject

(those last two are rules copied from the output of firewall-cmd --list-all)

As shown below, the active zone is public, which isn't explicitly noted in the rule as I read that without it specified, it applies to the active zone.

While researching this, I realised that I might be in an unusual situation with the Modem port forwarding to the machine, rather than having it in a DMZ or hosted externally. The Apache logs show the internet facing IP addresses for the http/s client machines, and I've been assuming that these IPs are the address that presented to the firewall.

(netstat -tn does show a current connection to an external IPv4 address, but I can't establish if that's inbound or outbound.)

Warning: ALREADY_ENABLED: rule family='ipv4' source address='' reject was a frequent response until I added logging notes into the rule, now it just happily accepts multiple reject rules for the same subnet.

Since the first time I wrote this question on serverfault.com (and was then told it wasn't ok to ask this there), I've had time to add the aforementioned logging, and have now been been to establish this:

[user@server ~]# firewall-cmd --list-rich-rules | grep 194.195.251.
rule family="ipv4" source address="" log prefix="Sun May 23 21:23:50 UTC 2021" reject
rule family="ipv4" source address="" log prefix=" - Mon May 24 23:58:51 UTC 2021" reject
rule family="ipv4" source address="" log prefix=" - Tue May 25 21:25:27 UTC 2021" reject
rule family="ipv4" source address="" log prefix=" - Wed May 26 21:25:57 UTC 2021" reject

And this:

[user@server ~]# firewall-cmd --list-rich-rules | grep 192.241.196.
rule family="ipv4" source address="" log prefix=" - Mon May 24 23:59:25 UTC 2021" reject
rule family="ipv4" source address="" log prefix=" - Tue May 25 21:26:02 UTC 2021" reject
rule family="ipv4" source address="" log prefix=" - Wed May 26 21:26:31 UTC 2021" reject

So clearly, this isn't dropping traffic. (The date/times are the dates added to the firewall.)

In case it's helpful, here's the top part of firewall-cmd --list-all

public (active)
  target: default
  icmp-block-inversion: no
  interfaces: eno1
  services: cockpit dhcpv6-client http https ssh
  ports: [redacted integer]/tcp [redacted integer]/tcp [redacted integer]-[redacted integer]/udp [redacted integer]-[redacted integer]/tcp
  masquerade: no
  rich rules:

The rich rules then follow. And because why not, here's the active zone query:

[user@server ~]# firewall-cmd --get-active-zone
  interfaces: virbr0
  interfaces: eno1

Other things to note:

  • There is only one Ethernet port on the box.
  • The firewall is reloaded after adding new rules (firewall-cmd --reload)
  • The server reboots every night
  • This server was CentOS Stream, but I converted it to CentOS 8
  • The firewall is active, enabled and functional (firewall-cmd --state returns 'running')
  • I'm aware that firewall-cmd --complete-reload exists, but it terminates all connections and that's not what I need.
  • Some things from the conf file:
    • FirewallBackend=nftables
    • IPv6_rpfilter=yes
    • RFC3964_IPv4=yes

Any ideas?

( List of Rich Rules: https://pastebin.com/qXFMBvqh )

-- EDIT -- Under the banner of "This is ridiculous", I put another local IP in the firewall (without --permanent) to test, then ran curl from the CLI. The pre-adding test worked fine, the post-adding test gave me curl: (7) Failed connect to **[redacted]**:80; Connection refused. With that behaving as expected, I cannot work out WHY these other things are not.

Have now turned on official logging (firewall-cmd --set-log-denied=all)

-- EDIT 2 -- More info on the scripts in question:

  • The first script parses the Apache logs and establishes a list of IP addresses. Upon that, it puts the firewall-cmd adding commands into a .txt file. (It also looks up Geo-location information, but that's out of scope here.)
  • The second script reads the .txt file, and executes the firewall-cmd add commands, then runs the firewall reload (--reload)

This allows the .txt file to build up over the weekend if I'm AFK. I inspect the GEO location information prior to running the second script to ensure that any IPs in the same country are more closely inspected to see whether they should be removed from the list.

-- EDIT 3 -- Logs shows this:

Jun 21 09:47:42 <SERVER_NAME> kernel: STATE_INVALID_DROP: IN=eno1 OUT= MAC=a8:a1:59:3a:03:14:20:b0:01:c0:a5:26:08:00 SRC= DST=<SERVER_INTERNAL_IP> LEN=40 TOS=0x00 PREC=0x00 TTL=44 ID=38952 DF PROTO=TCP SPT=54750 DPT=80 WINDOW=0 RES=0x00 RST URGP=0

There are four of these entries, all at the same millisecond. Interestingly, that IP address is not in the firewall rules at all.

  • 1
    You know about the typo in the invalid IP address you put in the rule family="ipv4" source address="7162.159.244.38" portion of your question, right? I need to play with Centos 8 more but do you have reject rules logging, look into that and look at the logs to see what is actually being rejected. It seems like you are listing the rules and not logs but I may not know what I'm looking at. I'm not seeing logs indicating these rules are not working in your question? Am I missing it, my apologies if so, just curious if I could help you with this question/issue. Jun 17 at 4:07
  • 1
    The rules in the firewall should work in order of precedent too, so see if you can put the deny rules before any allow rules with firewall-cmd as it could be as simple as that. An allow rule might be allowing something through first before the deny rule even applies, ensure the deny rule runs first precedent wise. Jun 17 at 4:10
  • 1
    @Mark I wonder with it has something to do with the fact that, traffics of conntrack state established are accepted before they can be matched against the blocked source IP. It's common approach, but I think what you experiencing might be exactly one of the pitfalls: you need to make conntrack consider the connection to be dropped for the rules to be "in effect" (in the way as you desire/expect).
    – Tom Yan
    Jun 23 at 5:26
  • 1
    @Mark So you might neither need to flush the conntrack table after the new rules are loaded (which might have some serious side effect on those that are not supposed to be blocked), or see if you can cherry-pick those connections that should be dropped. Or, do early rejection in nftables (it can be easily done if you are using nftables directly; no idea what's the right way to make firewalld have that done instead).
    – Tom Yan
    Jun 23 at 5:28
  • 1
    @Mark it might be worth mentioning that, the "accept established traffics first" approach is sort of supposed to work with "only further whitelisting will be done later". IMHO any blacklisting rules should be inserted before it. Unfortunately, firewalld seems to be too "simple-minded", the it only precede the blacklisting rules before the whitelist rules in their own group.
    – Tom Yan
    Jun 23 at 6:06

See if the following workaround makes any difference. Run the following commands as root / with sudo (every time after reloading rules; shouldn't be necessary after adding new rules without --permanent afterwards though):

(Btw, make sure you bare in mind that rules added without --permanent will be gone after reloading, just like the following.)

nft add chain inet firewalld filter_IN_public_black
nft add rule inet firewalld filter_IN_public_black jump filter_IN_public_log
nft add rule inet firewalld filter_IN_public_black jump filter_IN_public_deny
nft insert rule inet firewalld filter_INPUT jump filter_IN_public_black

(Since typical table flush does NOT remove chains, but only rules, added to a table, you probably can't use create instead of add for the filter_IN_public_black chain to determine whether this needs to be applied again. So just check with e.g. nft list ruleset when in doubt. But of course, firewalld might enumerate all chains in its tables and delete them one by one upon reload. Do some tests if you want to know.)

P.S. This override will be applied to ALL interfaces / "zones" (with log and deny rules added, before or after the override is applied if without --permenant, for the public zone). If desired, further customize it to make it less "aggressive" (e.g. add an iifname match for the jump to filter_IN_public_black that is inserted to filter_INPUT).

  • Hi, Thank you for following up with an answer before I'd had a chance to work through your three comments on the question. Q: I do have one concern before commencing. How do I revert back if the above fails / breaks something? I'm currently looking for a file that looks remotely like the output from nft list ruleset, but haven't found one yet. I'm not well versed in NFT, so I may require extra info/clarification in the answers using that directly.
    – Mark
    Jun 24 at 1:18
  • 1
    @Mark What nft list ruleset shows are volatile, i.e. rules that are currently loaded into the firewall engine. Since the commands manipulate the table firewalld creates, reloading rules with firewalld should get rid of them (except the then empty and unused filter_IN_public_black chain). The commands themselves do not create anything persistent. (Although there can be a system service that dump the ruleset to a file upon shutdown and reload from it upon boot, such service shouldn't be enabled if you use a frontend like firewalld.)
    – Tom Yan
    Jun 24 at 2:06
  • 1
    @Mark what the commands do is essentially, create a "pathway" that leads to existing "pathways" (that consists of / in turns lead to you desired logging / rejecting rules), and make that pathway the first pathway the input traffics will go through. (Basically that means, match the traffics against those rules first, and apply what should be done if they are a match, of course.)
    – Tom Yan
    Jun 24 at 2:12
  • 1
    @Mark what might help you understand better is that, there are terminating and non-terminating rules (I think those are the wordings, IIRC). Like rules with accept, reject and drop are the former, those keywords will determine that fate/verdict for the matched traffics, so no further matching or action will be applied on them; logging rules are on the other hand the latter, therefore you see the jump to the log chain is added before the one to the deny chain.
    – Tom Yan
    Jun 24 at 2:31
  • 1
    @Mark The difference of jump and goto is not really your concern here, since neither firewalld or my commands take advantage of goto in their design. But knowing that "if there's no verdict after a jump, it will go back to where it jumped and continue" might help.
    – Tom Yan
    Jun 24 at 2:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.