83 reputation
7
bio website ken-far.com
location Colorado
age 74
visits member for 2 years, 6 months
seen Nov 17 at 16:20

Enjoy python, "big data", data science, business intelligence, data warehousing, ETL, scoring, security and unix.


Aug
21
comment How to search/replace while including or excluding folded text?
Excellent - that's exactly what I needed and somehow couldn't find.
Aug
21
awarded  Scholar
Aug
21
accepted How to search/replace while including or excluding folded text?
Aug
20
asked How to search/replace while including or excluding folded text?
Jun
16
comment Unix sort keys cause performance problems
That makes sense - though the performance is far worse than just 1 pass per key. For example one key takes 30 seconds while three keys takes 10 times as long, or 300 seconds. These seems extremely repeatable and unaffected by using a buffer_size larger than the data.
Jun
16
revised Unix sort keys cause performance problems
changed title to make it more searchable
Jun
16
comment Unix sort keys cause performance problems
Since I'm not looking to access just 1% of the data, but rather the entire set, an index isn't helpful to me. The only thing another piece of software like SQLite would offer is potentially a better sorting algorithm. Does that make sense? Would you agree?
Jun
16
revised Unix sort keys cause performance problems
added key_consolidation work-around code example
Jun
16
comment Unix sort keys cause performance problems
I need to access 100% of the data - so indexes won't help me. And in fact, if there's just 1 key, the performance is reasonable.
Jun
16
revised Unix sort keys cause performance problems
updated benchmarks
Jun
16
awarded  Supporter
Jun
16
comment File Sync Solution for Batch Processing (ETL)
Unfortunately, it doesn't address the need to rename or move the source file rather than delete it.
Jun
16
comment File Sync Solution for Batch Processing (ETL)
It's not about professional vs non-professional - it's about requirements in this case: rysnc as you've presented it appears to be able to meet all my requirements except the ability to move the source file rather than delete it. This feature is important as it allows us to easily recreate state if we lose the downstream machine. Also, I've already implemented two solutions using SSH that rename the file. It just seems that I shouldn't have to write this again.
Jun
15
awarded  Commentator
Jun
15
revised Unix sort keys cause performance problems
updating buffer-size measurements
Jun
15
comment File Sync Solution for Batch Processing (ETL)
But at that point does rysnc do much for me beyond what I can already do with scp? That is, if I have to script that much of it, why not do the entire thing?
Jun
15
revised Unix sort keys cause performance problems
added 107 characters in body
Jun
15
comment Unix sort keys cause performance problems
The sort is performed within a tool that allows the user to configure which columns are keys. So, it must potentially support multiple, non-continuous keys. I suppose I could write the code to consolidate that when the keys are adjacent.
Jun
15
revised Unix sort keys cause performance problems
deleted 228 characters in body
Jun
15
revised Unix sort keys cause performance problems
added 274 characters in body