Big Data Science

2022-05-18 07:04:52 Graph visualization with PyGraphistry
PyGraphistry is a Python AI library for visual graphs that allows you to extract, transform, analyze and visualize large graphs along with end-to-end Graphistry graphics server sessions. Graphics created specifically for large graphs. The WebGL custom rendering engine client renders up to 8 million nodes + a number at a time, and most client GPUs detect between 100,000 and 2 million elements. The GPU analytics engine on the server interface supports even larger graphics. Graphics smoothes graphics workflows in the PyData ecosystem, including Pandas/Spark/Dask dataframes, Nvidia RAPIDS GPUs, GPU graphics, DGL/PyTorch graphics neural networks, and various data connectors.
PyGraphistry is a streamlined and optimized native PyData interface for language independent Graphistry REST APIs. It is possible to use PyGraphistry using Python data sources such as CSV, SQL, Neo4j, Splunk and more.
The PyGraphistry Python client uses different categories of users:
• Data Explorer: Comprehensive data exploration to accelerated visual analysis in a couple of lines, share results on time phenomena, create complex predictions in Jupyter Notebook and Google Colab.
• Developer: Quickly prototype amazing Python solutions with PyGraphistry, embed in a language-independent way with the REST API, customize colors, icons, templates, JavaScript, and more.
• Analyst: Customize visual ashboards using interactive search, filters, timelines, bar charts, and more, embedding them in any framework.
https://github.com/graphistry/pygraphistry Open / Comment

2022-05-16 06:04:41 TOP 5 New Python Alpha 5
At the beginning of 2022, a new version of Python was released - Alpha 5 (3.11). Main features:
• Improved debugging with the exception chain of their messages. Python 2022 is showing features with trace detection pointing directly to where the error occurs. Python 2 had a similar feature, but required the addition of context to the code, which made things more difficult. Now context is done automatically.
• Variable exception handling - you can now reduce dependency in different ways depending on what other exceptions it is associated with. The ability to use multiple exception operations with an explicit random exception for all. Just create a big try/except block with all possible exception names, and then add more exclude statements to it. It is for this purpose that a group of exceptions, which can be associated with grouping many different exceptions together and applying a single handler function, is only called if something occurs internally due to exceptional probability.
• Variadic Generics - now you can create functions that take a variable number of arguments (up to 22). It was necessary to define a characteristic that could take into account the magnitude of the transferred value each time. Variadic Generics in Python 3.6 allows you to select any number of options at once, which is useful when iterating multiple operations.
• CPython performance optimization. Changes to functions applied to calls and word lookups should reduce overhead, call by the C stack, speed up everything from developing object-oriented code to accessing data dictionaries.
• Simplify work in other languages such as JavaScript on top of Python through high performance and parallel computing.
https://morioh.com/p/af7debd024e2
https://medium.com/@Sabrina-Carpenter/python-alpha-5-is-here-5-promising-features-that-will-blow-your-mind-a4abd406d0ad Open / Comment

2022-05-13 07:11:10 Open / Comment

2022-05-11 06:42:33 Loguru for logging Python scripts
This library is useful for ML specialists and data engineers who often write in Python. It automates the logging and debugging process. In addition, Loguru includes a number of useful features that ensure that standard logging facilities are retained.
Loguru works according to a plug-and-play recipe and has features such as collapsing multiple event logs, quickly compressing log files, and deleting them regularly. It also supports multi-threaded security and log highlighting. This open source library can be used in conjunction with email media to receive email emails or to send other types of messages.
Finally, Loguru supports correlation with a large Python research module, increases the transmission of all information, measures the initial concentration of the logger, in Loguru.
Source code: https://github.com/Delgan/loguru
Use case example: https://medium.com/geekculture/python-loguru-a-powerful-logging-module-5f4208f4f78c Open / Comment

2022-05-09 04:57:59 YDB: scalable fault-tolerant NewSQL DBMS from Yandex. Now open source
April 19, 2022. Yandex has published the source code of the distributed NewSQL DBMS YDB, which allows you to create scalable, fault-tolerant services that can support a large operational load. The code is available under the Apache 2.0 license.
YDB is an open-source Distributed SQL Database that combines high availability and scalability with strict consistency and ACID transactions. YDB observes the occurrence of occurrences and recovery in the event of occurrences from the formation or even the occurrence of the center of the day. The reliability of YDB has been tested on Yandex services (Alisa, Taxi, Market, Metrika and almost 500 more projects). You can deploy YDB both at will and on external servers, including Yandex Cloud or providers.
https://ydb.tech/
https://github.com/ydb-platform/ydb Open / Comment

2022-05-07 10:59:45 Open / Comment

2022-05-04 06:51:27 Continuous Machine Learning: CML for CI/CD
Need to introduce CI / CD in the development of ML systems? Try CML, an open source CLI tool from Iterative.ai for implementing CI/CD within MLOps. It is suitable for automating ML model development workflows, including provisioning, training and evaluation, comparison of experiments in the history of the project, and monitoring of changing datasets. CML is based on the following principles:
• GitLab or GitHub for managing ML experiments, monitoring model training and data changes using DVC;
• Automated reports for machine learning experiments with metrics and graphs on every Git pull to make informed decisions based on data.
• no additional services - only GitLab, Bitbucket or GitHub, Docker and DVC. Optionally, you can add cloud storage, as well as self-hosted or cloud workers such as AWS EC2 or MS Azure.
CML introduces CI/CD-style automation into the workflow: most of the configurations are defined in the cml.yaml file stored in the repository. This file specifies what actions should be taken when a new feature branch is ready to be merged into the main branch. When a pull request is created, GitHub Actions uses this workflow and performs the actions specified in the configuration file.
Source code: https://github.com/iterative/cml
Documentation: https://cml.dev/doc
Use case example: https://towardsdatascience.com/continuous-machine-learning-e1ffb847b8da Open / Comment

2022-05-01 07:35:07 TOP-15 Data Science and ML conferences all over the World in May 2022:
• 5-6 May - The #1 MLOps Conference on the planet - Marriott Marquis, New York, NY https://rev.dominodatalab.com/
• 5-6 May - Data Innovation Summit 2022 - KISTAMÄSSAN, STOCKHOLM https://datainnovationsummit.com/
• 10-12 May - Wrangle Summit 2022 Virtual https://www.trifacta.com/events/wrangle-summit-2022/
• 11-12 May - Big Data & AI World. Frankfurt, Germany. https://www.bigdataworldfrankfurt.de/
• 12-13 May - The Data Science Conference. Chicago, IL, USA https://www.thedatascienceconference.com/
• 12 - May 9AM ET, Ontotext Demo-Day. Virtual. https://event.gotowebinar.com/event/bfd3b6ef-828c-46a1-a644-b4e785cece6c
• 15-18 - May FLAIRS-35: Special Track on Neural Networks and Data Mining, Jensen Beach, FL, USA. https://sites.google.com/view/flairs-35-nn-dm-track/home
• 17 May - The data dividend: reimagining data strategies to deepen insight. San Francisco, CA, USA https://events.economist.com/custom-events/the-data-dividend-san-francisco/
• 18 May - Data Science Mini Salon | AI and ML in Retail & E-Commerce. Virtual. https://www.datascience.salon/retail-and-ecommerce
• 23-25 May - TDWI Visualization, Dashboards, and Analytics Adoption https://tdwi.org/events/seminars/may/dashboards-visualization-analytics-adoption/home.aspx
• 24-25 May - Graph + AI Summit. Virtual. https://www.tigergraph.com/graphaisummit
• 24-25 May - Chief Data & Analytics Officers, Insurance US. New York, NY, USA. https://cdaoi.coriniumintelligence.com/
• 25-26 May - Data Reliability Engineering Conference. Virtual https://drecon.org/
• 26 May - Zero Gravity: A Modern Cloud Data Pipeline Event. Virtual. https://www.incorta.com/zerogravity
• 30 May – HeyGrowth - Yerevan, Armenia https://heygrowth.com/yerevan Open / Comment

2022-04-29 07:55:55 Open / Comment

2022-04-27 08:00:15 DataSpell: A professional data science development environment from JetBrains
Lacking a comfortable development environment in a lightweight Jupyter notebook? Need to write Python code in a reliable IDE with all DS libraries? Try DataSpell by JetBrains, a professional IDE like PyCharm that combines many popular data analysis and machine learning libraries with a powerful set of developer tools.
Released in 2020, today DataSpell is in demand by machine learning developers and data analysts around the world.
https://www.jetbrains.com/ru-ru/dataspell/ Open / Comment