OpenAI Codex (next generation) - it looks like we are finally "there"

[dmm]

There was a live demo of the next generation of OpenAI Codex code-generating software on August 10.

My impression from it is that "we have finally arrived" - this is a programming tool which seems to be more useful than an extra entry-level software engineer on the team. This starts to address the key bottleneck of our times: our limited ability to create software.

This was always my threshold: can we create an AI software which can be "hired instead of a junior software engineer"? That was the main temporal uncertainty for me: how long would it take to reach that level? It looks like this has been accomplished.

We are rapidly approaching the situation when AI will actively participate in programming AI software, for better or for worse...

OpenAI Codex is now a part of OpenAI API (which is still a closed beta with a waitlist), and it will be possible to participate in an informal competition today from 10am Pacific time (1pm Eastern) till 1pm Pacific (4pm Eastern) and try it a bit.

Links are in the comments.

Comments

[dmm]

The video with the demo: https://www.youtube.com/watch?v=SGUCcjHTmGY

The competition: https://challenge.openai.com/

[dmm]

There is an unpleasant defect at the beginning of the demo, where their icons obscure the code which is generated, so the first few minutes look weird. But later it becomes really good (the length of that video is 30 min; there are also other related short videos on the OpenAI YouTube channel).

[dmm]

Originally, this was presented on https://www.twitch.tv/openai

The video is https://www.twitch.tv/videos/1114111652 (first 15 min or so is blank, twitch also muted some music)

Presenters:

Greg Brockman https://twitter.com/gdb

Ilya Sutskever https://twitter.com/ilyasut

Wojciech Zaremba https://twitter.com/woj_zaremba

[juan_gandhi]

Interesting. How does it know what to do, I wonder.

[dmm]

Like, is there an additional secret sauce, in addition to just training a large Transformer on a selected subset of GitHub?

There might be some things which they don't yet disclose or spell out.

An earlier version of Codex is described in a July paper which I discuss a bit in the comments here: https://dmm.dreamwidth.org/44860.html

So, the main principles underlying the functioning of this system are reasonably well known.

But I think there might be some further improvements which are not spelled out yet (this new version is really good).

[dmm]

The "challenge" page references this: https://openai.com/blog/openai-codex/

[dmm]

The competition is starting in a minute or so: https://challenge.openai.com/

[dmm]

I am performing miserably at the challenge, but I finally figured out the interface for Codex there, and it works, the bodies in these two examples are autogenerated:

def first_line(csv_contents: str):
    """read the first line of the buffer"""
    return csv_contents.split('\n')[0]

def find_index_of_Date_field(line: str):
    """
    line contains words separated by commas
    find the index of the word Date
    """
    return line.split(',').index('Date')   

[dmm]

The next one is super-impressive:

def compute_a_column(csv_contents: str, index: int)
    """
    csv_contents has lines
    lines contain words separated by commas
    form a list of one word from each line,
    where the selected word is situated at index position within its line
    """
    return [line.split(',')[index] for line in csv_contents.split('\n')[1:]] 

The system has even correctly figured out that I'd rather omit line 0.

[dmm]

Then it got less brilliant, generated wrong solution here

def compute_difference_between_two_dates(date1: str, date2: str)
    """
    compute difference between two dates in days
    """
    return abs((date1 - date2).days)

and then when I wanted to try this, the interface (Firefox, being quite flaky today with this challenge) just stopped working (tells me that I still have some unused assists, but does not work):

def convert_date_to_day(date: str):
    """
    date looks like 2014-05-20
    convert in to the number of days
    """

[dmm]

In some sense, one should remember that Codex is non-deterministic, and even if the first attempt did not produce a good result, one can always try again (100 repeated attempts might yield a pretty good performance, per Codex paper).

In the competition, the stand-alone Codex was not as fast as many of the participants, but it eventually solved all 5 problems on its own.

They did offer bonus problem number 6 to the leaders. I wonder, if they offered it to the stand-alone Codex as well...

[dmm]

Oh, they've now removed problem 6 related updates from the dashboard (I guess not everyone noticed it, and it seemed unfair to the winners of the originally defined run)

[dmm]

And it was not completely wrong, it just did not take the type into account.
Something like this is the correct solution, so it just ignored my type hint:

def compute_difference_between_two_dates(date1, date2):
    d1 = datetime.strptime(date1, "%Y-%m-%d")
    d2 = datetime.strptime(date2, "%Y-%m-%d")
    return abs((d1 - d2).days)

[dmm]

"The Codex Challenge servers are currently overloaded due to demand (Codex itself is fine though!). Team is fixing... please stand by."

https://twitter.com/OpenAI/status/1425872169280540672

So, it was not just me. But some people worked fine through these conditions...

[dmm]

Fixed the server problems (they say), extending the challenge by 3 hours:

https://twitter.com/OpenAI/status/1425898290785046529

[dmm]

OK, it's better now (solved a couple of problems, did not use Codex much; let's continue)

[dmm]

Another one. I used one assist, I was not sure about its result, but it was approximately right, and helped me to quickly edit this to the right answer (otherwise, I would be thinking about this for ages, and searching the net for things):

def decompress(compressed: str, tree: Tree) -> str:
    """
    initialize result as empty string
    for each symbol in compressed
       traverse the tree till a leaf is reached
       if the leaf is reached
          then add the leaf to the result
    return result
    """
    result = ""
    t = tree
    for char in compressed:
        t = t[char]
        if not isinstance(t, dict):
            result += t
            t = tree
    return result

[dmm]

Next task:

Problem 4

Parse the given Python source code and return the list of full-qualified paths for all imported symbols, sorted in ascending lexicographic order.
Constraints

The input will not contain any wildcard imports (from ... import *).

Ignore aliases (renamings): from x import y as z should be represented as x.y.
Library Suggestion

Consider using the ast module.
Examples

Input

import os
import concurrent.futures
from os import path as renamed_path
from typing import (
List, Tuple
)

Output

['concurrent.futures', 'os', 'os.path', 'typing.List', 'typing.Tuple']

I used Codex as follows, and it produced correct test results:

import ast
from typing import List


def parse_imports(code: str) -> List[str]:
    """
    use ast.parse to make a parse tree from code
    traverse the parse tree to find Import constructs
    and to generate a fully qualified imported symbol for each construct
    sort the resulting list in ascending lexicographic order
    """
    tree = ast.parse(code)
    result = []
    for node in ast.walk(tree):
        if isinstance(node, ast.Import):
            for name in node.names:
                result.append(name.name)
        elif isinstance(node, ast.ImportFrom):
            if node.module == '__future__':
                continue
            result.append(node.module)
            for name in node.names:
                result.append(f'{node.module}.{name.name}')
    return sorted(result)
    

# Examples
print(parse_imports('import os'))
print(parse_imports('import os\nfrom typing import List'))

I am actually not 100% sure they are correct:

['os']
['os', 'typing', 'typing.List']

Perhaps, 'typing' was not supposed to be there. But I can submit and see if it passes.

[dmm]

No, they also think what I suspected:

"Expected parse_imports('import os\nfrom typing import List') to equal ['os', 'typing.List'] but got ['os', 'typing', 'typing.List']"

I should also probably add their test case from the formulation of the problem to the code.

[dmm]

And after commenting out one line, I am indeed getting a correct result (or, at least, it passed all their tests and was accepted):

import ast
from typing import List


def parse_imports(code: str) -> List[str]:
    """
    use ast.parse to make a parse tree from code
    traverse the parse tree to find Import constructs
    and to generate a fully qualified imported symbol for each construct
    sort the resulting list in ascending lexicographic order
    """
    tree = ast.parse(code)
    result = []
    for node in ast.walk(tree):
        if isinstance(node, ast.Import):
            for name in node.names:
                result.append(name.name)
        elif isinstance(node, ast.ImportFrom):
            if node.module == '__future__':
                continue
            # result.append(node.module)
            for name in node.names:
                result.append(f'{node.module}.{name.name}')
    return sorted(result)
    

# Examples
print(parse_imports('import os'))
print(parse_imports('import os\nfrom typing import List'))
test = """
import os
import concurrent.futures
from os import path as renamed_path
from typing import (
    List, Tuple
)
"""
# print(parse_imports(test))
print("this works in my Anaconda terminal, but has alignment problems within test string here")

Rather remarkable!

[dmm]

The next problem, Codex just solved it for me (I did formulate the prompt well, I guess):

Problem 5

You have several buckets whose sizes are represented by the list sizes. Find the number of different ways to arrange the buckets such that the first bucket’s size is greater than the second bucket’s size.
Constraints

sizes is a list of positive integers.

If sizes has fewer than 2 buckets, return 0.

Some buckets may have the same size, but are nevertheless treated as unique buckets.
Library Suggestion

Consider using the itertools module.
Examples
Input [10]
Output 0
Explanation No arrangement is possible.
Input [1, 2]
Output 1
Explanation The possible arrangements are (1, 2) and (2, 1). Of these, only (2, 1) has a first bucket that is bigger than the second bucket.
Input [1, 3, 1]
Output 2
Explanation Only the arrangements (3, 1, 1) and (3, 1, 1) satisfy the property that the first bucket is bigger than the second bucket.

import itertools
from typing import List


def count_arrangements(sizes: List[int]) -> int:
    """
    if len(sizes) < 2 return 0
    count permutations of sizes such that the first element
    of the permuted array is greater than the second element
    """
    if len(sizes) < 2:
        return 0
    return len(
        [
            p
            for p in itertools.permutations(sizes)
            if p[0] > p[1]
        ]
    )
    

# Examples
print(count_arrangements([1, 3, 1]))
print(count_arrangements([1, 2]))

[dmm]

This is a functional solution!

It formed the list of good permutations, and returned its length!

It did not try to iterate and add the counter and all that imperative programming stupidity. Wow...

[dmm]

I tweeted the result: https://twitter.com/ComputingByArts/status/1425945168016969729

[dmm]

К последней задаче соревнования она вообще нашла мне решение в стиле функционального программирования, не побоялась просто сформировать список перестановок с желанным свойством через list comprehension и вернуть его длину в качестве результата:

https://dmm.dreamwidth.org/47121.html?thread=186385#cmt186385

[dmm]

Note that GitHub Copilot is awailable starting from Teams pricing plan ($4 per month per user) as part of in-browser version of VSCode https://github.com/features/codespaces

So, if one is in a hurry to play with it, this might be an option...

But one can just get on the OpenAI waiting list...

[dmm]

My June 29 post on GitHub Copilot:

https://dmm.dreamwidth.org/44860.html