Something about deduplication with DISTINCT You can exclude du | Big Data Science

Something about deduplication with DISTINCT
You can exclude duplicates from the selection by simply adding the DISTINCT keyword to the SQL query. However, this simple solution will not always be correct. To ensure that there are no duplicates in a data set, the DBMS needs to compare all rows with each other, filtering out duplicates. This requires a lot of CPU and memory resources to store all the strings. they need to be compared with each other in memory, even if the hash is being worked on at a low level. In addition, DISTINCT reduces computational parallelism by slowing down query execution.
DISTINCT removes duplicates, but does not resolve incorrect joins and filters, which in practice most often lead to repetitions, for example, due to CROSS JOIN or using RANK instead of ROW_NUMBER, which leads to duplication due to a poorly defined section window. See here for details with code examples: https://jmarquesdatabeyond.medium.com/sql-like-a-pro-please-stop-using-distinct-31bdb6481256