Learning by making something that pushes back

When I was a kid, I spent most of my time with Lego Technic. What I loved about it was that you could test things right away. You rearranged the cogs, pushed the button on the motor, and your creation either moved the way you wanted or it didn’t. There was nothing to argue about!

A sketch is different. A sketch can agree with you for a very long time, while a chair can’t, because eventually somebody sits on it.

That’s what I like about making stuff. The idea gets in contact with something that can resist it: a program receives input nobody expected, a piece of wood splits along the grain, a person looks at your interface and understands it in a way you never considered. Sometimes it’s embarrassing, and it’s also the most useful information you’ll get.

Józef Lubacz has a paper on epistemic and poietic processes, where he distinguishes knowing (epistemic) from bringing something about (poietic), and says that in real activity the two are intertwined. I’d put it simpler: building is often how I find out what the problem was.

Say you build an app for sharing tools between neighbors. The first version is probably a searchable catalog. Then people start using it and it turns out that the hard parts are trust, returning the drill on time and who repairs it when it breaks. The prototype worked, and it also showed that the technical problem was a small part of the whole thing.

You have to be careful about what you conclude, though. A prototype that works with five enthusiastic friends tells you little about five thousand strangers. Friends tolerate bugs because they want you to succeed.

Coding with AI makes this more important. Implementation got cheap, so I can run many more experiments. I can also pile up demos that never meet an unfriendly input, a real user or the need to maintain them. Having more prototypes doesn’t mean I learned more.

Sutton’s The Bitter Lesson says something related about AI research: give methods room to discover things instead of feeding them only the solutions we already understand. For prototypes I’d add that the result has to be able to change what I do next. An experiment that always confirms what I expected probably wasn’t testing anything.

So now I try to ask of every prototype which of my assumptions it can break. If I think a process is confusing, can someone finish it without my help? If I think an explanation teaches something, can the reader use it in a different situation?

There’s still a place for playing around without any hypothesis, and a lot of good directions start that way. The discipline comes later, when I start claiming what the experiment has shown.

The tools that speed up creation are great when they speed up this contact with reality. I want to spend some of the time they save me on things that can push back.