Data Science by ODS.ai 🦜

2021-06-28 23:37:23 Recently I have found an Instagram of artist from Tomsk, Evgeny Schwenk – he redraws characters from Soviet cartoons as if they were real people. I have applied neural.love neural network which made his drawings even more realistic. Just a bit of Photoshop (mainly for hats) and here we go.

I guess Karlsson-on-the-Roof is my best result. Open / Comment

2021-06-27 22:15:40 Article on how to use #XGBoost for #timeseries forcasting

Link: https://machinelearningmastery.com/xgboost-for-time-series-forecasting/ Open / Comment

2021-06-26 10:57:05 Open / Comment

2021-06-24 19:20:23 DocNLI

Natural Language Inference (NLI) is the task of determining whether a “hypothesis” is true (entailment), false (contradiction), or undetermined (neutral) given a “premise”.

Previously, this task was solved for sentence-level texts. A new work "DOCNLI: A Large-scale Dataset for Document-level Natural Language Inference" to be appeared in ACL 2021 presenting the study for document/paragraph level NLI:
https://arxiv.org/abs/2106.09449v1

In Github repo you can find data and pretrained weights of RoBERTa:
https://github.com/salesforce/DocNLI
For release in HuggingFace we, probably, should wait...

P.S. I am already waiting to test this setup for fake news detection Open / Comment

2021-06-22 10:18:08 Transformer Module Optimization

Article on how to apply different methods to make your transformer network up to 10x smaller and faster:

- Plain model optimization and PyTorch tricks;
- How and why to use FFT instead of self-attention;
- Model Factorization and quantization;

https://habr.com/ru/post/563778/

#deep_learning Open / Comment

2021-06-18 20:53:14 ​​Semi-Autoregressive Transformer for Image Captioning

Current state-of-the-art image captioning models use autoregressive decoders - they generate one word after another, which leads to heavy latency during inference. Non-autoregressive models predict all the words in parallel; however, they suffer from quality degradation as they remove word dependence excessively.

The authors suggest a semi-autoregressive approach to image captioning to improve a trade-off between speed and quality: the model keeps the autoregressive property in global but generates words parallelly in local. Experiments on MSCOCO show that SATIC can achieve a better trade-off without bells and whistles.

Paper: https://arxiv.org/abs/2106.09436

A detailed unofficial overview of the paper: https://andlukyane.com/blog/paper-review-satic

#imagecaptioning #deeplearning #transformer Open / Comment

2021-06-16 11:53:25 Color2Style: Real-Time Exemplar-Based Image Colorization with Self-Reference Learning and Deep Feature Modulation

ArXiV: https://arxiv.org/pdf/2106.08017.pdf

#colorization #dl Open / Comment

2021-06-13 13:43:04 Chinese researchers are very fond of doing extensive surveys of a particular sub-field of machine learning, listing the main works and the major breakthrough ideas. There are so many articles published every day, and it is impossible to read everything. Therefore, such reviews are valuable (if they are well written, of course, which is quite rare).

Recently there was a very good paper reviewing various variants of Transformers with a focus on language modeling (NLP). This is a must-read for anyone getting into the world of NLP and interested in Transformers. The paper discusses the basic principles of self-attention and such details of modern variants of Transformers as architecture modifications, pre-training, and various applications.

Paper: A Survey of Transformers. Open / Comment

2021-06-13 12:44:59 Microsoft's FLAML - Fast and Lightweight AutoML

Github: https://github.com/microsoft/FLAML

Code: https://github.com/microsoft/FLAML/tree/main/notebook/

Paper: https://arxiv.org/abs/2106.04815v1

@a Open / Comment

2021-06-10 17:57:53 ​​CoAtNet: Marrying Convolution and Attention for All Data Sizes

This is a paper on combining CNN and attention for Computer Vision tasks by Google Research.

The authors unify depthwise convolutions and self-attention via relative attention and vertically stack attention and convolutional layers in a specific way.
Resulting CoAtNets have good generalization, capacity and efficiency.

CoAtNet achieves 86.0% ImageNet top-1 accuracy without extra data and 89.77% with extra JFT data, outperforming the prior state of the art of both convolutional networks and Transformers. Notably, when pre-trained with 13M images from ImageNet-21K, CoAtNet achieves 88.56% top-1 accuracy, matching ViT-huge pre-trained with 300M images from JFT while using 23x less data.

Paper: https://arxiv.org/abs/2106.04803

A detailed unofficial overview of the paper: https://andlukyane.com/blog/paper-review-coatnet

#cv #deeplearning #transformer #pretraining Open / Comment