So I have a massive file something like this...
1/20/2016,somerandomdata
1/20/2016,somerandomdata
1/20/2016,somerandomdata
1/20/2016,somerandomdata
1/21/2016,somerandomdata
1/21/2016,somerandomdata
1/21/2016,somerandomdata
1/21/2016,somerandomdata
1/22/2016,somerandomdata
1/22/2016,somerandomdata
1/22/2016,somerandomdata
1/22/2016,somerandomdata
And I want to split it into a bunch of smaller files based on the first column. Easy: use awk like this:
awk -F '[,/]' '{print > filename$1$2$3".dat"}'
Here's the catch: I want the output files to be compressed. So, I could go ahead and do this after the fact...
find . -name "filename*.dat" | xargs -l xz
The problem with that is that I want the xz to be in the pipeline instead of after the data is split. Something like this:
curl "url" | grep "blah" | xz -c > filename.dat.xz
Of course, this doesn't actually split the file.
The reason I want it in the pipeline is because I am downloading the data and want to run compression at the same time as downloading instead of after. (I'm pretty sure this would make things go faster, but if I'm wrong, correct me)
So, my goal is something like....
curl "url" | grep "blah" | awk -F '[,/]' '{print > filename$1$2$3".dat"}' | xz -c > filename.dat.xz
But not, because that will obviously not work
If you have a better solution to my problem or if you think I'm doing something completely stupid, I'm flexible.