83 reputation
7
bio website ken-far.com
location Colorado
age 74
visits member for 1 years, 10 months
seen Dec 9 '13 at 17:25

Enjoy python, "big data", data science, business intelligence, data warehousing, ETL, scoring, security and unix.


Jun
15
revised Unix sort keys cause performance problems
added 534 characters in body
Jun
14
comment Unix sort keys cause performance problems
I'm assuming the problem is algorithmic, but I'll try giving it more memory. I already tried it with & without numeric sorting and found no difference.
Jun
14
asked Unix sort keys cause performance problems
Jun
14
comment File Sync Solution for Batch Processing (ETL)
This looks great - except that I'd rather not delete the source files - I'd prefer to move them to another directory. The reason is that then if the downstream server craters it's much easier to rebuild it to a point in time.
Jun
12
comment File Sync Solution for Batch Processing (ETL)
I used to move files with a scheduler, but it became too difficult: if it's time-based then it won't automatically retry & fix without a lot of extra jobs. If it's event-driven, then it only handles the triggering of the process, not the process itself. My current solution has probably moved 30 million files over the past ten years and only failed on recovery about 20 times. I'm looking for something that can improve on that.
Jun
12
comment File Sync Solution for Batch Processing (ETL)
Except - I've written utilities to do this several times, and once we had them the organization used them extensively. I've also seen quite a few organizations used Rsync for the same purpose, but in ways that are prone to losing or duplicating files delivered.
Jun
12
comment File Sync Solution for Batch Processing (ETL)
I agree that this isn't exactly a "sync" operation. But there doesn't appear to be a better common term for it. Also, I think if I do as you suggest that I'd need separate scripts to rename/move both the source & target side. This would take 2-3 configs, be difficult to make recoverable - especially since new files might be arriving while the process is running. I think I'd be better off writing something from scratch.
Jun
12
comment File Sync Solution for Batch Processing (ETL)
Yes quite a bit, but since I'm not planning to use them for the transformations (need more flexibility) it would be overkill to get them just for file transfers.
Jun
11
comment File Sync Solution for Batch Processing (ETL)
I hadn't looked at those yet, but just reviewed them now. Those are all schedulers - and aren't file transfer programs. They could be part of the solution (as could cron), but only provide the easy part.
Jun
11
comment File Sync Solution for Batch Processing (ETL)
I've had to write this utility 3 times over the past fifteen years for different corporate employers or customers. And wouldn't mind writing an open source version if there's no good one yet (I've looked through alternativeto.net and can't find anything, but haven't combed through freshmeat, etc). I don't think you can really break this down much either: the process needs to look for qualifying files, move them, rename/move/etc source & dest after the move.
Jun
11
awarded  Student
Jun
9
awarded  Promoter
Jun
7
awarded  Editor
Jun
7
revised File Sync Solution for Batch Processing (ETL)
added 384 characters in body
Jun
7
asked File Sync Solution for Batch Processing (ETL)
Jun
7
awarded  Autobiographer