https://adamdrake.com/command-line-tools-can-be-235x-faster-th | The lazy guy channel

https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html

This is an old article that becomes even more significant in the age of 50-core machines with half-terabyte RAM for 7$/hour.

Modern computers are ridiculously powerful. We tend to forget that because all that power made them very forgiving to people using vastly ineffective algorithms. And it is so easy to reach out for “cloud” that people tend to do that without even thinking if it is possible to make their program a little more efficient. I see this every day at a BigCo where I work because cloud is essentially “free” for us. Until it is not.

Next time you think you need to make your data processing distributed, think again. If you are processing up to single terabytes of data, it is very likely that single server should be powerful enough to process it in reasonable time. If it isn’t, then you may be doing something wrong.