Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I've CSV file (around 10,000 rows ; each row having 300 columns) stored on LINUX server. I want to break this CSV file into 500 CSV files of 20 records each. (Each having same CSV header as present in original CSV)

Is there any linux command to help this conversion?

share|improve this question

4 Answers 4

This should do it without the CSV header:

tail -n +2 file.csv | split -l 20

You can then add the header to each of the files:

for file in x*
do
    (head -n 1 file.csv; cat "$file") > "$file".new
    mv "$file".new "$file" # Stolen from @PawanMude's answer
done
share|improve this answer

Try:

fn="infile" c=0
{ 
  read header
  split -a 3 -l 3 - "$fn"
  for f in "$fn"???; do
    c=$((c+1))
    printf "%s\n" "$header" | cat - "$f" > "${f%???}-$c" && rm "$f"
  done 
} < $fn

Or try with awk:

awk 'NR==1{h=$0; next} !((NR-2)%n){close(f); f=FILENAME "-" ++c; print h>f}{print>f}' n=3 infile

multi-line version:

awk '
  NR==1 {
    h=$0
    next
  }
  !((NR-2)%n) {
    close(f)
    f=FILENAME "-" ++c
    print h>f
  }
  {
    print>f
  }
' n=3 infile
share|improve this answer

Best Way to solve this using POST mentioned below :

Solution

    tail -n +2 file.txt | split -l 4 - split_
for file in split_*
do
    head -n 1 file.txt > tmp_file
    cat $file >> tmp_file
    mv -f tmp_file $file
done
share|improve this answer

For the sake of completeness, here are some minor improvements:

  • You could save the header once and reuse many times
  • You could insert the header in the split files using sed without temporary files

Like this:

header=$(head -n 1 file.csv)
tail -n +2 file.csv | split -l 20
for file in x??; do
    sed -i -e 1i$'\\\n'"$header" "$file"
done

The $'\\\n' there is a NEWLINE character escaped with a backslash. The sed expression means: insert $header before the 1st line.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.