Big Data Science

2021-10-27 06:06:50 Introduction to feature engineering for time series forecasting - extract from the book by Dr. Francesca “Machine Learning for Time Series Forecasting with Python” published by Wiley in December 2020

Developing ML models is often time-consuming and requires many factors to be considered: algorithm iteration, hyperparameter tuning, and feature engineering. These options are additionally multiplied by time series data, since DS specialists still need to take into account trends, seasonality, holidays, and external economic variables. Each ML algorithm expects data as input to be formatted. Therefore, time series datasets require precleaning and feature definitions before running the simulation.
The peculiarity of time series analysis is that data points are linked to time. For example, you need to build the output of an ML model by defining the variable you want to predict (future sales next Monday) and then use the historical data and feature set to create the input variables. This is necessary to achieve the following goals:
• creation of the correct set of input data for the ML-algorithm in order to create input features from historical data and form a dataset as a supervised learning problem;
• improving the performance of ML-models - creating a valid relationship between the input features and the target variable that needs to be predicted.
The main categories of features useful for time series analysis are as follows:
• Date time features based on the timestamp value of each observation, such as the hour, month, and day of the week for each observation. This also includes weekends and holidays, seasons, etc.
• Lag features and window features - Values at previous time steps that are considered useful because they are created on the assumption that what happened in the past may influence or contain some kind of internal information about the future. For example, it might be useful to create functions for sales that occurred on previous days at 4:00 pm if you want to predict similar sales at the same time the next day.
• Sliding (Rolling) Window Statistics, to compute statistics on values from a given sample of data by defining a range that includes the sample itself, as well as a specified number of values before and after the sample used.
• Expandable feature window statistics that include all previous data.
Illustrations and examples of Python are available here:
https://medium.com/data-science-at-microsoft/introduction-to-feature-engineering-for-time-series-forecasting-620aa55fcab0 Open / Comment

2021-10-25 17:52:18 Open / Comment

2021-10-25 17:52:10 Math for the Data Scientist: Measuring Distance, Part 3
• The Jaccard index or Intersection over Union is a measure for calculating the similarity and diversity of multiple samples — the size of the intersection divided by the size of their union. In practice, this is the total number of similar objects between sets divided by the total number of objects. For example, if two sets have 1 common entity and only 5 different entities, then the Jaccard index will be 1/5 = 0.2. The main disadvantage of this measure is its dependence on the amount of data: the larger the sample, the larger the index value. The Jaccard index is often used in applications that use binary data. For example, the DL model predicts image segments. In this case, the Jaccard index can be used to calculate how closely the forecast matches reality. This exact measure can be applied to text similarity analysis to measure how often words are selected between documents and to compare sets of multiple patterns.
• The Sørensen-Dice index is very similar to the Jaccard index — it also measures the similarity and diversity of multiple samples. While they are calculated in a similar way, the Sorensen-Deiss index is a little more intuitive because it can be thought of as the percentage of overlap between two sets, which is a value between 0 and 1. Like the Jaccard index, the Sorensen-Deiss index exaggerates the importance of sets in which there are practically no reliable positive results, weighing each element in inverse proportion to the size of its sample. This measure is often used in image segmentation problems and in text similarity analysis.
• Haversine distance is the distance between two points on the sphere, taking into account their longitude and latitude. This is similar to Euclidean distance in that it calculates the shortest line between two points that are on a sphere. This is the main drawback of this measure - ideal spheres do not exist in reality. For example, due to the unevenness of the planet's landscape, calculations may be distorted. Instead, you can use the Vincenty distance, which works with an ellipsoid instead of a sphere. Unsurprisingly, Haversine distance is often used in navigation. For example, to calculate the distance between two countries when flying between them. It makes no sense to apply this measure at short distances, because a small radius of curvature has little effect. Open / Comment

2021-10-22 16:27:19 SimVLM by Google AI Open / Comment

2021-10-22 16:27:11 Google AI's SimVLM: Pre-Learning a Weakly Controlled Visual Language Model
Visual language modeling involves understanding the language on visual inputs that can be useful for developing products and tools. For example, the image caption model generates natural language descriptions based on understanding the essence of the image. Over the past few years, significant progress has been made in visual language modeling thanks to the introduction of VLP (Vision-Language Pre-training) technology.
This approach is aimed at studying a single functional space immediately from visual and language inputs. For this purpose, VLP often uses an object detector such as the Faster R-CNN, trained on datasets of tagged objects to highlight regions of interest, relying on task-specific approaches and collaboratively exploring the representation of images and texts. These approaches require annotated datasets or time to label them and are therefore less scalable.
To solve this problem, Google AI researchers propose a minimalistic and efficient VLP called SimVLM (Simple Visual Language Model). SimVLM is trained from start to finish with a single goal, similar to language modeling, on a huge number of poorly aligned image-text pairs, i.e. text paired with an image is not necessarily an accurate description of the image.
The simplicity of SimVLM enables efficient training on such a scalable dataset, helping the model achieve the highest level of performance across six tests in the visualization language. In addition, SimVLM includes a unified multimodal presentation that provides reliable cross-modal transmission with no fine-tuning or fine-tuning for text-only data, incl. visualization of answers to questions, captions for images and multimodal translation.
Unlike BERT and other VLP methods that apply pre-training procedures, SimVLM takes a sequence-by-sequence structure and is trained with a single prefix language target model (PrefixLM), which receives the leading part of the sequence (prefix) as input, then predicts its continuation. For example, for a dog chasing a yellow ball sequence, the sequence is randomly truncated to the chasing dog prefix, and the model predicts its continuation. The concept of a prefix is similarly applied to images, where the image is divided into a series of "slices", a subset of which are sequentially fed into the model as input. In SimVLM, for multimodal input data (images and their signatures), a prefix is a concatenation of a sequence of image fragments and a sequence of prefix text received by the encoder. The decoder then predicts the continuation of the text sequence.
Through this idea, SimVLM maximizes the flexibility and versatility in adapting the ML model to different task settings. And successfully tested in BERT and ViT, the transformer architecture allows models to directly accept raw images as input. It also applies a convolution step from the first three ResNet blocks to extract contextualized patches, which is more beneficial than the naive linear projection of the original ViT model.
The model is pretrained on large-scale datasets with images and texts. ALIGN was used as a training dataset, containing about 1.8 billion noisy image-text pairs. For the text data, the Colossal Clean Crawled Corpus (C4) dataset of 800G web documents was used. SimVLM testing has shown this ML model to be successful even without supervised fine tuning. SimVLM was able to achieve subtitle quality close to the results of controlled methods.
https://ai.googleblog.com/2021/10/simvlm-simple-visual-language-model-pre.html Open / Comment

2021-10-20 06:38:51 Analyzing Time Series Data: 5 Tips for a Data Scientist
One of the most common mistakes beginners make in analyzing time series data is the assumption that the data has regular points and does not contain gaps. In practice, this is usually not confirmed and leads to incorrect results. In real datasets, data points are often missing, and the available ones are located unevenly or inconsistently. Therefore, before analyzing time series data, a preliminary preparation stage should be carried out:
• Understand the time range and detail of the time series by data points using dataset visualization;
• Compare the actual number of ticks in each time series with the number of expected ticks depending on the interval between points and the total length of the time series. This ratio is sometimes referred to as the duty cycle, which is the difference between the maximum and minimum timestamp divided by the point spacing. If this value is much less than 1, then a lot of data is missing.
• Filter out batches with low duty cycle by setting a limit, for example, 40% or whatever is appropriate for a specific task.
• Standardize the spacing between time series cues by upsampling to finer resolution.
• Fill upsampled gaps using an appropriate interpolation method such as last known value or linear / quadratic interpolation. In Apache Spark, you can use the applyInPandas method in the PySpark grouped dataframe for this, under the hood of which is pandasUDF, the performance of which is much higher than simple UDF functions due to more efficient data transfer through Apache Arrow and calculations through Pandas vectorization.
https://towardsdatascience.com/a-common-mistake-to-avoid-when-working-with-time-series-data-eedf60a8b4c1 Open / Comment

2021-10-18 05:32:06 Open / Comment

2021-10-18 05:31:59 Math for the Data Scientist: another 3 Distance Measures, Part 2
• Manhattan Distance, also called a taxi or city block measure, calculates the distance between vectors with real values. Then Manhattan distance refers to the distance between two vectors on a uniform grid if they can only move at right angles. No diagonal movement is used when calculating the distance. While Manhattan distance seems to be acceptable for multidimensional data, it is a less intuitive measure than Euclidean distance. A measure is more likely to give a higher distance value than Euclidean distance, since it is not the shortest possible distance. However, if the dataset has discrete and / or binary attributes, the Manhattan distance works well because it takes into account real paths within the possible values.
• Chebyshev distance is defined as the greatest difference between two vectors in any coordinate dimension, it is simply the maximum distance along one axis. This measure is also often called the distance of the chessboard, since the minimum number of moves required for the king to move from one square to another is equal to the distance of Chebyshev. This distance is usually used in very specific use cases, making it difficult to use it as a universal measure of distance, as opposed to Euclidean distance or cosine similarity. Therefore, the Chebyshev distance is only recommended in certain cases. For example, to determine the minimum number of moves in games that allow unlimited 8-sided movement. Also, the Chebyshev distance is often used in warehouse logistics, for example, to determine the time required for an overhead crane to move an object.
• Minkowski distance is a more complex measure used in normalized vector space (n-dimensional real space), where distances can be represented as a vector with length. When using this measure, there is a zero vector that has zero length and all others are positive, the vector can be multiplied by a number (scalar coefficient), and the shortest distance between two points is a straight line. You can also use the p parameter here to control distance metrics similar to other measures, for example, p = 1 for Manhattan distance, p = 2 for Euclidean, and p = ∞ for Chebyshev distance. Therefore, in order to work with the Minkowski distance, you need to understand the purpose, advantages and disadvantages of the Manhattan, Euclidean and Chebyshev measures. Finding the correct value for p can be computationally inefficient, gives flexibility in the distance metric, and can be a huge advantage if correctly selected. Open / Comment

2021-10-15 06:11:39 News from DeepMind AI: Enformer Architecture for Genetic Research
The Enformer architecture, powered by Transformers, advances genetic research to accurately predict how DNA sequence affects gene expression. In early October 2021, Nature Methods published an article by DeepMind and Calico researchers about the new Enformer neural network architecture, which greatly improves the accuracy of predicting gene expression from a DNA sequence. The developers have made this model and its initial predictions of common genetic variants publicly available.
Enformer builds on transformers common in natural language processing to use self-attention mechanisms for greater coverage of the DNA context. By efficiently processing sequences to account for interactions at distances more than 5 times (i.e. 200,000 base pairs) longer than previous methods, the new architecture can simulate the influence of important regulatory elements on the expression of genes found in the DNA sequence.
AI can be used to explore new possibilities for finding patterns in the genome and to put forward mechanistic hypotheses about sequence changes. Like a spell checker, Enformer partially understands a DNA sequence dictionary and can highlight changes that could alter gene expression.
The main application of this new model is to predict which changes in DNA letters, also called genetic variants, will affect gene expression. Compared to previous models, Enformer is much more accurate in predicting the effect of variants on gene expression, both in the case of natural genetic variants and synthetic variants that alter important regulatory sequences. This property is useful for interpreting the growing number of disease-related variants derived from genome-wide associative studies. Variants associated with complex genetic diseases are predominantly located in the non-coding region of the genome, likely causing disease by altering gene expression. But because of the intrinsic correlation between options, many of these disease-related options are only falsely correlated and not causal. Computing tools help distinguish true associations from false positives.
https://deepmind.com/blog/article/enformer
https://www.nature.com/articles/s41592-021-01252-x
https://github.com/deepmind/deepmind-research/tree/master/enformer Open / Comment

2021-10-13 07:44:38 Сравнение FLAN с GPT-3 Open / Comment