L̶u̵m̶i̵n̷o̴u̶s̶m̶e̵n̵B̶l̵o̵g̵

2021-09-27 16:16:00 NiFi and Apache Airflow are the best ETL tools. What separates them, what are their greatest strengths and weaknesses, which is suitable for your needs?


https://www.astronomer.io/blog/apache-nifi-vs-airflow Open / Comment

2021-09-16 16:16:00 Apache Hudi pioneered the serverless transactional layer for event logs that significantly shape the data infrastructure. The article gives an in-depth overview of Apache Hudi's building blocks and future roadmap aligning with its founding principle. Open / Comment

2021-09-14 19:19:00 Don't go into management. Or go, it's up to you. My job is to show the "what if" picture.

https://luminousmen.com/post/management-is-not-a-promotion Open / Comment

2021-09-10 16:16:00 Interviews are thoroughly broken and I have no idea how to actually make them better. Open / Comment

2021-08-17 19:19:00 Caching, as trivial as it may seem, is a difficult task for engineers.

https://luminousmen.com/post/spark-tips-caching Open / Comment

2021-08-17 16:16:00 Hive meta store is a critical component in the interception of all query engines path provides a virtualization layer between the storage and compute. What makes Hive Metastore so special?

This article attempts to describe some of them and says that the Hive metastore is not going anywhere in the near future. Open / Comment

2021-08-10 16:16:00 I remember being 10 I was thinking the teachers knew what was going on.

Now I’m there I realize they were still figuring this shit out too. Open / Comment

2021-07-26 16:16:00 Apache Hadoop provides the following filesystem clients for reading from and writing to Amazon S3:

S3N (URI scheme: s3n) - A native filesystem for reading and writing regular files on S3. S3N allows Hadoop to access files on S3 that were written with other tools, and conversely, other tools can access files written to S3N using Hadoop. S3N is stable and widely used, but it is not being updated with any new features. S3N requires a suitable version of the jets3t JAR on the classpath.

S3A (URI scheme: s3a) - Hadoop’s successor to the S3N filesystem. S3A uses Amazon’s libraries to interact with S3. S3A supports accessing files larger than 5 GB, and it provides performance enhancements and other improvements. For Apache Hadoop, S3A is the successor to S3N and is backward compatible with S3N. Using Apache Hadoop, all objects accessible from s3n:// URLs should also be accessible from S3A by replacing the URL scheme.
Note
Amazon EMR does not currently support use of the Apache Hadoop S3A file system.

S3 (URI scheme: s3) - Apache Hadoop implementation of a block-based filesystem backed by S3. Apache Hadoop has deprecated use of this filesystem as of May 2016.

Just want to save it here. Original in web archive

#aws Open / Comment

2021-07-25 16:16:00 Accidental complexity is stuff you have to do, like builds and infrastructure configuration and so forth, whereas essential complexity is things like the code your customers want you to deliver. We as a developers want to do essential 100% of the time and throw everything else over the fence to other teams. Now with microservices it become a harder thing to do, but service mesh can solve some of the problems.

"Service mesh" is an umbrella term for products that seek to solve the problems that microservices’ architectures create. These challenges include security, network traffic control, and application telemetry.

Service mesh is typically implemented as a scalable set of network proxies deployed alongside application code (a pattern sometimes called a sidecar). These proxies handle the communication between the microservices and also act as a point at which the service mesh features can be introduced. They trying to decouple the platform-level features from the application itself.

For those who's interested this is a good article on the evolution and current problems that service mesh is solving.

But yeah, you got it — it's another layer of problems, probably more complex, but who cares? DevOps can handle that, right? Right?! Open / Comment

2021-07-20 16:16:00 Good digging into optimizing Apache Spark join operations with or conditions:

https://sujithjay.com/spark/or-within-joins

#spark Open / Comment