L̶u̵m̶i̵n̷o̴u̶s̶m̶e̵n̵B̶l̵o̵g̵

2021-06-06 19:19:00 Thundering Herd

Imagine that popular news has just appeared on the front page of a news site. The news has not been loaded into the cache yet. After it's published on the front page of the website, it is very likely that many users will click on the new article, and they will all do so at approximately the same time.

Because the application server processes all user requests at the same time and all user requests will evaluate the contents of the cache at approximately the same time, they will all come to the same conclusion — miss!

Next step, all user request threads will proceed to read data from the database. Thus, even though developers has been careful to implement caching in the application, the database is still prone to bursts of activity.

This is known generally as the "Thundering Herd" problem:

A large number of processes waiting for an event wake up when a certain event occurs, but only one process can continue running at a time. After processes wake up all of them require resources and it is necessary to decide which process may continue working. Once a decision has been made, the remaining processes are put to sleep again, and then wake up again to request access to the resource.

This is repeated until there are no more processes left to wake up. Since all processes use system resources when waking up, it is more efficient if only one process wakes up at a time.

Simply speaking, when the programmer controls something like a herd of threads (analogous to a herd of animals) with a semaphore (as an alternative), he wakes up the whole herd of threads for a single process (such as an incoming client connection). This problem is solved by writing a scheduler, which will decide which of the pool of slumbering threads should perform a certain task.

#big_data Open / Comment

2021-06-05 19:19:00 On reddit there was a post by an anonymous drunk senior developer who wrote about what he had learned as an engineer over the past ten years. According to the author, he will probably regret it, but his revelations have already been appreciated by eight thousand upvotes and unremitting discussions. Upvote the hero! Open / Comment

2021-06-05 16:16:00 AutoML with mathematical methods will save a lot of money Open / Comment

2021-06-04 16:16:00 Open / Comment

2021-06-02 19:19:00 Recommendation engines from the industry giants - YouTube, Netflix, Spotify, Airbnb and Uber. It's more of the system design then actual algorithms, which sometimes more interesting.

https://www.theinsaneapp.com/2021/03/system-design-and-recommendation-algorithms.html?m=1

#big_data Open / Comment

2021-06-01 19:19:00 In this blog post series, I will introduce you to the most interesting and frequently asked questions from Python developers

Python interview questions. Part I. Junior

Python interview questions. Part II. Middle

Python interview questions. Part III. Senior Open / Comment

2021-05-25 19:19:00 Coming into the interview, engineers have the idea that HDFS and S3 are fundamentally the same system. Even if S3 clients make it seem like they are, they are not.

https://luminousmen.com/post/hdfs-vs-cloud-based-object-storage-s3 Open / Comment

2021-05-25 16:16:00 Great PySpark style guide from Palantir team:

https://github.com/palantir/pyspark-style-guide

#spark Open / Comment

2021-05-19 19:19:00 A good conversation is like a miniskirt, short enough to retain interest but long enough to cover the subject — Celeste Headlee

When your job hinges on how well you talk to people, you learn a lot about how to have conversations -- and that most of us don't converse very well. I came across this Ted talk on how to have a better conversation.

Some of this may seem too obvious, but you have to admit there are some things here that you knew about, but you forgot to apply.

1. Don't do several things at once. Be in the moment, put your smartphone away.
2. You don't have to be important or clever. If you are not very interested in the opinion of others — blog as I do
3. Ask open-ended questions (that don't involve a "yes" or "no" answer).
4. Swim with the flow. If a thought comes up during a conversation, let it go and listen to the person you're talking to.
5. If you don't know something, admit it. This will save you from embarrassment and preserve your authority.
6. There is no need to draw parallels with the experience of your interlocutors. You don't need to tell others what an asshole your boss is in response to your companion's complaints about his boss.
7. Don't repeat what you've said.
8. Avoid unnecessary details. Names, dates, and other details will quickly slip his mind.
9. Listen. You can't learn anything with your mouth open.
10. Be brief. Brevity is the sister of you-know-what.

#soft_skills Open / Comment

2021-05-12 16:16:00 Caching... There is so much in that word - the pain of invalidation and the joy of reusing computation. In Spark, this is known as an optimization technique...

https://luminousmen.com/post/explaining-the-mechanics-of-spark-caching Open / Comment