Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I've this plain text file that I need to be automatically examined and results written in another plain text file. Note that I'll use plain text and CSV interchangeably, since CSVs are plain text with common separator between data (commas, tabs, etc).

Data examples


Process one - recognize the first two sections of the data (text-number) to repeat and count them in output CSV:


Process two - First part of the data gathered above has sort of a primary key assigned to them on a different file. I would like to get the primary keys mentioned in the newly generated files as well:

#primary key file

#newly generated file

So basically, what I'm asking her is if there's any software that can help me with this on either Windows or Linux-based operating systems. Bash can help me with simpler processing, but I don't think it can handle this much. Please advise.

share|improve this question
Can the same data occur multiple times in the same file? (for example, "ABD-01A, ABD-01B, AL-25A, ABD-01A"...) – grawity Jan 25 '12 at 15:22
@grawity No, all the data are unique. – Oxwivi Jan 25 '12 at 16:27
In that case, you could use the example scripts I provided in my other comment. (Both do the same, but written in different languages so you can compare.) – grawity Jan 25 '12 at 17:34
up vote 1 down vote accepted
  1. Choose your preferred language – Perl works well with text; Python, PHP and Ruby are okay, if a little slower.
  2. Read the key file first:
    • split each key,data line into key and data,
    • then store both in a dict/hash/array: keys[data] = key
    • also counts[data] = 0 if the language demands initialization
  3. Read all data files:
    1. use a regexp to find the "TEXT-NUMBER" data at the beginning of every line,
    2. increment counts[data] by one,
    3. and immediately output keys[data], the line, and counts[data].

I wrote an example script in both Perl and Python, so your choice.

share|improve this answer

I would do it in Python using regular expressions. Just type python in your shell to see if it is installed.

Otherwise you can use Perl. Just type perl in your shell to see if it is installed. Is has built-in support of regular expressions.

share|improve this answer
Can you provide some examples on how to use it? – Oxwivi Jan 25 '12 at 14:45
@Oxwivi: Here are a few, but a language cannot be learned from only "some examples"... "Learning Perl" and similar books might be a good start. – grawity Jan 25 '12 at 15:36
@grawity Sorry for getting back at you so late, I faced unexpected issues generating the data. I did this: perl */images products.csv > images - multiple data files, but no conflicts. However the results were without the keys - for example: ,ABD-47,2. The area reserved for the key value is empty. Also, if you will, please add an answer of your own, since it's likely I will choose yours. – Oxwivi Jan 27 '12 at 19:08
@grawity Only the first */images file encountered is processed. Referring directly to that file works as expected. – Oxwivi Jan 27 '12 at 19:11
@grawity What exactly is unmatched input supposed to mean? It found no match in the key file? – Oxwivi Jan 27 '12 at 19:23

Process 1

perl datafile …

where is something like

use strict;
use warnings;

my %headwordcount;

while (<>) {
  if (/^([A-Z]+-\d+)/) { $headwordcount{$1}++; }
  # else { warn "Bad data: $_"; } # uncomment line for data warnings

  foreach (sort keys %headwordcount) {
     print "$_,$headwordcount{$_}\n";

Untested, caveat emptor.

Process 2

Add something like

   my %key;

   BEGIN {
     my $keyfilename = 'primary.key';
     open my $fh, '<', $keyfilename or die "Can't read '$keyfilename' - $!\n";
     while (<$fh>) {
        my ($key,$headword) = split(/,/, $_, 2);
        $key{$headword} = $key;        
     close $fh;

and change the print line in END {} to

   print "$key{$_},$_,$headwordcount{$_}\n";

Again, untested.

You'll generate warnings if the data file contains headwords not in the key file. You can check for $key{$_} being undefined and if so print w/o key.

share|improve this answer
What is that script supposed to do? The first process I described? – Oxwivi Jan 25 '12 at 16:35
@Oxwivi: Yes. If you can't easily follow the code. It may be best to try a different answer. If you understand bash-scripting and/or awk, Perl should not be too difficult. – RedGrittyBrick Jan 25 '12 at 16:38
I did use basic bash scripts, though I can't say I followed them well (inline syntaxes, mainly). And do I add the new codes you edited into the answer indented? – Oxwivi Jan 25 '12 at 16:41
@Oxwivi, yes, add the new stuff immediately after my %headwordcount; (apart from the replacement print line of course, which replaces the one inside the ÈND … foreach loop). – RedGrittyBrick Jan 25 '12 at 16:56
Okay, thank you very much! One last thing - datafile is to be replaced with the input file, that is obvious, but is the ... following it some Perl specific things? – Oxwivi Jan 25 '12 at 17:05

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.