4

I am using IPSet to managed tens of thousands of IPv4 CIDR/netmask ranges that then get linked to IPTables rules. This setup is working great, but I would like to get a good, high-level overview count of the IP host addresses IPSet acts on for client reporting purposes.

The IPSet entry formatting is consistently like this:

123.456.0.0/16 timeout 86400

So I can grep on the lines that have timeout to get values to act on the CIDR/netmask ranges that the entry contains.

For example, if I save the IPSet output (via ipset -L -n > ipset-20181228.txt) to a text file named ipset-20181228.txt and then run a combination of grep and wc -l like this:

grep  "timeout" ipset-20181228.txt  | wc -l

I get a count of 39,000+ items which equate to 39,000+ CIDR/netmask ranges. But that is (of course) only counting the CIDR/netmask ranges and not full counts of IP host addresses in that range.

I attempted to use prips (which expands CIDR/netmask values to actual IP addresses in Bash) with grep to cull out only items with CIDR/netmask ranges like this:

grep -oE '(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\/([0-9]{1,2})' ipset-20181228.txt | awk 'NF { system( "prips " $0)  }' | wc -l

And after a whopping 20 to 30 minutes (!!!) on my 2018 MacBook Air (with the fans kicking in), the count I got was 736,000,000+ which is what I am going for… But 20 to 30 minutes is way too long. I want this to be as scriptable and non-intrusive as possible, and can’t trust a command like that to run on a production server without eating up resources; I mean look at how it behaves on my local 2018 MacBook Air development setup.

Is there any way to just calculate the CIDR/netmask range count based on simply the CIDR/netmask value? I am hoping there is just some command line tool—or option in existing tools I am using—that I am unaware of that can help.

  • Which conversion do you want? Option A. 1.2.3.0/24 to 1.2.3.0-1.2.3.255 Option B. 1.2.3.0-1.2.3.255 to 1.2.3.0/24. Nmap, traceroute has some features. – Biswapriyo Dec 29 '18 at 7:19
  • 1
    @Biswapriyo Neither. Did you even read the question? I am simply asking to get a count of IP addresses based on CIDR/netmask notation. Not convert from one form of range notation to another. – JakeGould Dec 29 '18 at 12:30
  • 2
    If I understood correctly, you want to know how many hosts are in a given ip/mask pair. The hosts number is 2^(32-mask)-2. The mask represents the net bits, so the other part represents the hosts bits. -2 is for the net address and broadcast ips. If you have ipcalc installed run it to see those information. (Excuse-me if I misunderstood your question). – Paulo Dec 29 '18 at 12:57
  • 1
    @Paulo: Do not subtract 2; do not assume that each range directly corresponds to a subnet. Routes and filter lists have no correspondence to physical layout – an entry might match just part of a subnet, or cover multiple subnets at once, or even apply to networks which have no concept of "subnet" in the first place (e.g. point-to-point links). And really, whether an address represents a host or not isn't even relevant in filtering. Each prefix has exactly 2^(32-mask) addresses and that's it. – grawity Dec 29 '18 at 15:58
  • @grawity I see your point, I think I was stuck to the concept of a subnet range. That range from the question can be a range of ips, just like you said. Thanks for clarifying. I subscribe to 2^(32-mask). – Paulo Dec 29 '18 at 18:21
2

If your grep command outputs lines like 123.456.0.0/16, then you need to pipe them to

awk -F / '{ count[$2]++ } END { for (mask in count) total+=count[mask]*2^(32-mask); print total }'

The command only extracts masks (i.e. what is after /) and counts occurrences of each mask. At the end the number of hosts is calculated for each encountered mask (2^(32-mask)), multiplied by the number of occurrences and summed up.

Notes:

  • No sanity check is performed. E.g. input like 1.2.3.4/40 will be accepted, non-integer output will be calculated. Improve your preliminary grep filter if needed.
  • Each range independently contributes to the total number. If your ranges overlap then you will get an inflated result (I think your try with prips was no better in this).
  • This is excellent. Running it as follows works quite quickly and the final command I used this with is: grep -oE '(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\/([0-9]{1,2})' ipset-*.txt | awk -F / '{ count[$2]++ } END { for (mask in count) total+=count[mask]*2^(32-mask); print total }' – JakeGould Dec 30 '18 at 0:07
0

I thought something like this would work since the original poster is running grep to get the CIDR from lines with timeout:

awk -F'[ /]' '/timeout/ {hosts+=2^(32-$2)};ENDFILE{print "Hosts number in "FILENAME": "hosts;total+=hosts;hosts=0};END {print "Total: "total}' ipset*.txt

EDIT- The awk program above runs ok only with GNU awk. ENDFILE is a GNU extension.

I guess BSD awk ignores ENDFILE and runs that section if it was part of the main program section.

This is compatible with GNU and BSD awk.

awk -F'[ /]' '{if (filename != FILENAME) hosts=0};/timeout/ {hosts+=2^(32-$2)};{filename=FILENAME;file_total[filename]=hosts};END{for (i in file_total) {print "Hosts number in "i": "file_total[i];total+=file_total[i]};{print "Total: "total};}' ipset*.txt
  • Your revised script works in macOS, but as I explained previously it’s horribly over counting in macOS now as well as Linux: I am getting a hosts number of 8,500,000,000,000+ using your script instead of the expected 736,000,000+ I am using in a live sample test file with the accepted answer as well as prips. Sorry, thanks for the effort but this is not working to say the least. – JakeGould Dec 30 '18 at 4:33
  • 1
    @jakegould Ok, not a problem at all. Probably I didn't understand the output to filter it correctly. Besides I couldn't get an easy way to install prips and take a look how it works and it's output. Anyway it worth the exercise to get the awk report without any GNU extension. – Paulo Dec 30 '18 at 4:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.