This new system for end-user programming from MIT raises a question for me about users’ mental models, which I think is key for computing education (e.g., for figuring out how to do inquiry learning in computer science). Imagine that you use the system described below: You give the system some examples of text you want transformed, before-and-after. You run the system on some new inputs. It gets it wrong. What happens then? I do believe the authors that they could train the system in three examples, as described below. How hard is it for non-programmers to figure out the right three examples? More interesting: When the system gets it wrong, what do the non-programmers think that the computer is doing, and what examples do they add to clear up the bug?
Technically, the chief challenge in designing the system was handling the explosion of possible interpretations for any group of examples. Suppose that you had a list of times in military format that you wanted to convert to conventional hour-and-minute format. Your first example might be converting “1515” to “3:15.” But which 15 in the first string corresponds to the 15 in the second? It’s even possible that the string “3:15” takes its 1 from the first 1 in “1515” and its 5 from the second 5. Similarly, the first 15 may correspond to the 3, but it’s also possible that all the new strings are supposed to begin with 3’s.
“Typically, we have millions of expressions that actually conform to a single example,” Singh says. “Then we have multiple examples, and I’m going to intersect them to find common expressions that work for all of them.” The trick, Singh explains, was to find a way to represent features shared by many expressions only once each. In experiments, Singh and Gulwani found that they never needed more than three examples in order to train their system.