Big Data Science

2022-01-19 06:23:47 Bayesian statistics with PyMC3: brief overview
Frequency statistics rely on long-term event rates (data points) to calculate the desired variable. The Bayesian method can also work without a lot of events, even with a single data point. Frequency analysis gives a point estimate, while Bayesian analysis gives a distribution that can be interpreted as the confidence that the mean of the distribution is a good estimate for the variable. However, there is an uncertainty in the form of the standard deviation.
The Bayesian approach is useful in ML problems where estimates and validity are important. For example, today it could rain with a 60% chance.” The main formula underlying the Bayesian approach is Bayes' theorem, which allows you to calculate the posterior probability P(A|B ) of event A depending on event B.
• P(B|A) is called the probability that if event A happened, how likely is event B to happen?
• P(A) – probability of event A, a prior (initial) assumption about the variable of interest.
• P(B) is the probability of event B (evidence), which is usually difficult to calculate when estimating the posterior probability.
You can quickly calculate the Bayesian probability using the PyMC3 Python library https://docs.pymc.io/en/v3/. It allows you to write models using an intuitive syntax to describe the data generation process. PyMC3 allows you to tune an ML model with gradient-based MCMC algorithms like NUTS, with ADVI for fast approximate inference, including a mini-batch ADVI for scaling to large datasets, or with Gaussian processes to build Bayesian non-parametric models. PyMC3 includes a complete set of predefined statistical distributions that can be used as the building blocks of a Bayesian model.
This probabilistic programming package for Python allows users to fit Bayesian models using various numerical methods, most notably Markov Chain Monte Carlo (MCMC) and Variational Inference (VI). Instead of providing a basic model specification and fitting functions, PyMC3 includes functions for summarizing output and diagnosing the model.
PyMC3 aims to make Bayesian modeling as simple and painless as possible by allowing users to focus on their scientific problem rather than the methods used to solve it. The package uses Theano as a computational backend to quickly evaluate an expression, compute the gradient automatically, and perform computations on the GPU.
PyMC3 also has built-in support for modeling Gaussian processes, allowing you to generalize models and build graphs. There's model validation and convergence detection, custom stepwise methods, and unusual probability distributions. Bayesian models obtained using PyMC3 can be embedded in larger programs, and the results can be analyzed using any Python tools.
https://medium.com/@akashkadel94/bayesian-statistics-overview-and-your-first-bayesian-linear-regression-model-ba566676c5a7 Open / Comment

2022-01-17 05:44:40 Компоненты технологической платформы цифрового правительства от Gartner Open / Comment

2022-01-17 05:44:20 5 Essential Components of Gartner's Digital Government Technology Platform
The Digital Government Technology Platform (DGTP) makes digital transformation a reality, but requires dedicated leadership. According to a Gartner study, by 2023, more than 80% of government digital implementations that are not based on a technology platform will fail.
DGTP is a set of end-to-end, integrated, horizontal capabilities that coordinate government services across multiple domains by integrating five platforms:
• Citizen Experience platform provides interfaces and technologies, implements policies and procedures for citizen-business interaction, and measures the experience of its users;
• Ecosystem platform – a set of digital interfaces that implement policies and procedures for governments and ecosystem partners to share data and services.
• Internet of Things (IoT) platform provides interfaces, data management and context, and implements policies and procedures for collecting and processing data from IoT sensors
• Information System Platform - Corporate information systems are at the heart of government IT efforts today. The information system platform provides the technologies, policies and procedures for integrating these back office systems into the DGTP
• Intelligence Platform provides advanced analytics, geospatial and location analytics, robotic process automation (RPA) and AI capabilities to process data collected or stored in any area of the platform.
The key reusable components in DGTP are applications and services that can provide a seamless mix of data, services, and capabilities that work together within DGTP and are accessible across networks and devices. DGTP is not a turnkey solution, but it gives government agencies the ability to innovate, reduce costs, and deliver new capabilities quickly and flexibly.
https://www.gartner.com/en/articles/government-cios-here-s-an-essential-piece-of-the-digital-transformation-puzzle Open / Comment

2022-01-14 06:02:00 Accelerating Big Data Analytics: Expedia Group Case Study with Apache Druid and DataSketches
When analyzing big data, problematic queries often arise that do not scale, since they require enormous computational resources and time to obtain accurate results. For example, counting individual items, quantiles, most frequent items, table joins in SQL queries, matrix calculations and graph analysis. If the approximate results for such calculations are acceptable, there are special streaming algorithms or sketches that run several orders of magnitude faster with acceptable errors. The sketches helped Yahoo successfully reduce processing time from days or hours to minutes or seconds. One such tool is the open-source library Apache DataSketches.
It is used by the large travel company Expedia Group to speed up time series analysis in Apache Druid, where table joins are limited, requiring a single dataset to be put into memory. DataSketches supports set operations, including join, intersection, and difference, with little loss in precision. This is useful when looking for and booking tickets. With DataSketches, each dataset can be queried independently of Druid to get the desired object for each dataset for preliminary and then final calculation. Since Druid did not initially support merging DataSketches objects, Expedia Group engineers had to write their own Java code. Moreover, the DataSketches object takes up very little memory space, despite the large size of the set. As a result, Apache Druid, a column-based DBMS for quickly receiving huge amounts of event data and submitting queries with low latency, became even faster.
https://datasketches.apache.org/
https://medium.com/expedia-group-tech/fast-approximate-counting-using-druid-and-datasketch-f5f163131acd Open / Comment

2022-01-12 05:35:09 SQL queries against csv file with csvkit
csvkit is a command line toolkit for converting and working with CSV files. This utility allows you to perform the following operations in plain Python:
• Convert Excel and JSON files to CSV
• Display only column names
• Slice data
• change the order of columns
• find rows with matching cells
• convert CSV to JSON
• generate summary statistics
• refer to CSV using SQL queries
• import data into databases and extract from them
• parse CSV data
• work with column delimiters
The pip package manager will help you install csvkit: pip install csvkit
And the syntax for accessing a CSV file via an SQL query on the command line will look like this:
csvsql --query "SQL Query Here - source file name as table name (without .CSV)" source_filename> target_filename
To use this in your Python script you should
1) first import CSVSQL from csvkit utility
from csvkit.utilities.csvsql import CSVSQL
2) further define the arguments as a list of values, for example:
args = ['--query', 'select distinct manufacturer from playground', 'payground.csv']
3) then call CSVSQL with arguments
result = CSVSQL (args)
3) finally, the results can be shown
print (result.main ())
https://csvkit.readthedocs.io/en/latest/index.html
https://medium.com/data-engineering-ramstkp/sql-queries-on-csv-using-python-24a472fe53b1 Open / Comment

2022-01-10 07:28:58 Top 5 Data Engineering Trends in 2022: Astronomer Research
Astronomer, which commercializes and promotes the popular batch automation tool for working with data, Apache AirFlow, conducted a series of interviews with experts in the field of data engineering to identify the most pressing trends in the IT field.
• Data lineage, Data provenance and Data Quality
• Decentralization of data across different contexts and teams, but within a single consistent infrastructure with centralization of resources
• Consolidation of data tools, including orchestration of processing pipelines
• Data Mesh, eliminating silos between processing teams through the connection of used platforms
• mutual integration of DataOps, MLOps, AIOps for more efficient and faster use of consistent data and tools for seamless work with them.
https://www.astronomer.io/blog/top-data-management-trends-2022 Open / Comment

2022-01-07 11:18:18 How to read tables from PDF: tabula-py
Sometimes the raw data for analysis is stored in pdf documents. To automatically extract data from this format straight into a dataframe, try tabula-py. It is a simple Python wrapper for tabula-java that can read PDF tables and convert to pandas dataframe as well as CSV / TSV / JSON files.
Just first install it through your pip package manager: pip install tabula-py
And then import into your Python script:
import tabula as tb
And you can use:
file = 'DataFile.pdf'
data = tb.read_pdf (file, pages = '12')
df = pd.DataFrame (data)
Examples: https://medium.com/codestorm/how-to-read-and-scrape-data-from-pdf-file-using-python-2f2a2fe73ae7
Documentation: https://tabula-py.readthedocs.io/en/latest/ Open / Comment

2022-01-05 09:47:01 How to choose a validation measure for ML models: Yandex approach
Every practical machine learning problem has a problem with measuring results. Different measures can lead to different assessment results and, therefore, to different chosen algorithms. Therefore, it is very important to find a suitable measure of quality. Researchers from Yandex compare various approaches to solving typical ML problems, from classification to clustering, in order to formulate a universal method for choosing the most optimal quality measure. Key messages and main results are presented in articles at conferences In recent articles published on ICML 2021 and NeurIPS 2021, and a short retelling is available directly on the Yandex website https://research.yandex.com/news/how-to-validate-validation-measures.
http://proceedings.mlr.press/v139/gosgens21a/gosgens21a.pdf
https://papers.nips.cc/paper/2021/file/8e489b4966fe8f703b5be647f1cbae63-Paper.pdf Open / Comment

2022-01-03 08:27:21 Speed up scikit learn: a new extension of the good old Python library for DS
The popular Sci-Kit Learn Python library is familiar to every Data Scientist. It has many advantages, but unlike the powerful ML frameworks PyTorch and TensorFlow, Scikit-learn does not allow for fast model training on GPUs. Sklearnex (Extension for Scikit-learn), a Sci-Kit Learn extension from Intel Corporation, addresses this issue. Sklearnex is a free AI software module that provides 10x to 100x acceleration for a variety of applications. It fully supports all Scikit-Learn APIs and algorithms, speeding up code by replacing standard algorithms with optimized versions. The extension supports Python 3.6 and newer, and you can install it using the typical pip or conda package managers:
pip install scikit-learn-intelex
conda install scikit-learn-intelex -c conda-forge
https://intel.github.io/scikit-learn-intelex/
https://medium.com/@vamsik23/boost-sklearn-using-intels-sklearnex-cf2669f425bd Open / Comment

2022-01-02 14:10:55 TOP-25 International Data Science events 2022:

1. WAICF - World Artificial Intelligence Cannes Festival https://worldaicannes.com/ February 10-12, Cannes, France
2. Deep and Reinforcement Learning Summit https://www.re-work.co/events/deep-learning-summit-2022 February 17-18, San Francisco, USA
3. Reinforce https://reinforceconf.com/ March 8-10, Budapest, Hungary
4. MLconf https://mlconf.com/event/mlconf-nyc/ March 31, New York City, USA
5. Open Data Science Conference EAST https://odsc.com/boston/ April 19-21, Boston, USA
6. ICLR - International Conference on Learning Representations https://iclr.cc/ April 25–29, online
7. SDM - SIAM International Conference on Data Mining https://www.siam.org/conferences/cm/conference/sdm22 April 28–30, Westin Alexandria Old Town, Virginia, USA
8. World Summit AI Americas https://americas.worldsummit.ai/ May 4-5, Montreal, Canada
9. The Data Science Conference https://www.thedatascienceconference.com/ May 12-13, Chicago, USA
10. World Data Summit https://worlddatasummit.com/ May 18-22, Amsterdam, The Netherlands
11. Machine Learning Prague https://mlprague.com/ May 27-29, Prague, Czech Republic
12. The AI Summit London https://london.theaisummit.com/ June 15-16, London, UK
13. Machine Learning Week https://www.predictiveanalyticsworld.com/machinelearningweek/ June 19-24, Las Vegas, USA
14. Enterprise AI Summit https://www.re-work.co/events/enterprise-ai-summit-berlin-2022 June 29–30, Berlin, Germany
15. DELTA - International Conference on Deep Learning Theory and Applications https://delta.scitevents.org/ July 12-14, Lisbon, Portugal
16. ICML - International Conference on Machine Learning https://icml.cc/ July 17-23, online
17. KDD - Knowledge Discovery and Data Mining https://kdd.org/kdd2022/ August 14-18, Washington, DC, USA
18. Open Data Science Conference APAC https://odsc.com/apac/ September 7-8, online
19. RecSys – ACM Conference on Recommender Systems https://recsys.acm.org/recsys22/ September 18-23, Seattle, USA
20. INTERSPEECH https://interspeech2022.org/ September 18-22, Incheon, Korea
21. BIG DATA CONFERENCE EUROPE https://bigdataconference.eu/ November 21-24, Vilnius, Lithuania
22. EMNLP - Conference on Empirical Methods in Natural Language Processing https://2021.emnlp.org/ November, TBA
23. Data Science Conference https://datasciconference.com/ November, Belgrade, Serbia
24. Data Science Summit http://dssconf.pl/ December, Warsaw, Poland
25. NeurIPS https://nips.cc/ December, TBA Open / Comment