​​Program Synthesis with Large Language Models Paper compares | Data Science by ODS.ai 🦜

​​Program Synthesis with Large Language Models

Paper compares models used for program synthesis in general purpose programming languages against two new benchmarks, MBPP (The Mostly Basic Programming Problems) and MathQA-Python, in both the few-shot and fine-tuning regimes.

MBPP contains 974 programming tasks, designed to be solvable by entry-level programmers. MathQA benchmark, contains 23914 problems that evaluate the ability of the models to synthesize code from more complex text.

Largest fine-tuned model achieves 83.8 percent accuracy on the latter benchmark.

Why this is interesting: better models for code / problem understanding means improved search for the coding tasks and the improvement of the coding-assistant projects like #TabNine or #Copilot

ArXiV: https://arxiv.org/abs/2108.07732

#DL #NLU #codewritingcode #benchmark