This was prompted by, but is not meant to be critical of, some thoughts by Steve Krouse, a friend and fellow programming futurist. I'm going to expand a bit on what he wrote, then branch off it.

Take an existing text-based programming language.

Swap out all the literals (numbers, colors, strings) for GUIs. Scrubbers for numbers. Pickers for colors. A nice i18n-aware editor for strings. Any literal in your program can be configured with a GUI.

Then make GUIs for simple operators, like a nice GUI for math expressions, with plotting and helpful constants. Then a GUI for if-expressions and switch-expressions, a nice little logic diagram or state machine.

Make a GUI for defining functions, something that helps you see the cyclomatic graph and the structure of input and output. Make a GUI for organizing function calls, revealing the topology of your whole program.

Let these GUIs be expressions, and let them nest.

Keep tugging this thread, and you eventually end up with a luscious visual programming language. A collection of nested purpose-specific interfaces rather than one large interface, it will look different from typical visual programming languages. (How, exactly, all these GUIs are brought together in a cohesive way is left as an exercise of wild imagination for the reader.)

This collection of interfaces isn't going to significantly transform the programming experience. It'll improve the experience of configuring a program, because you can use a color picker instead of piddling and puttering your way to the right HSL value. It'll improve debugging, because you can plainly see that your switch statement has an unreachable branch. It'll change the physical feeling of programming, since you'll be moving a mouse.

But none of these GUIs will help you with abstraction, indirection, polymorphism, recursion, concurrency, distributed systems, etc. Why not? Because we forgot to account for those things when we made our GUIs above! We made GUIs for colors and function calls and states, but not reentrancy or process calculi. Oops.

So let's add GUIs for those, too — sure. But what if the programmer doesn't like our GUI for the PI calculus, and would rather a GUI for CSP?

What's missing is something we've seen in Racket (et al.) — rather than giving the programmer a fixed set of tools, give them tools for making tools.

You write Racket and Java with the same keyboard. They both have functions and data. The difference between them is in what you say and what you accomplish by saying it.

With our GUIs, we've changed what you say when programming. You don't say "#FFF", you just pick a color. You don't say if-then-else, you build a little state chart.

But we haven't necessarily changed what you accomplish. To change that, you need to be able to get from Java to Racket. You can write Racket in Java. If we've just made Visual Java, can you use it to build Visual Racket?

To start, we need to give the programmer tools for building new GUIs.

How do you make a GUI? What is a GUI?

Whether procedurally generated, Photoshop, or paper, the image of a GUI obviously isn't enough. You also need to express what each part of the GUI does. How do you do that?

GUIs "do" at two levels, and we need to handle both.

(A) Less importantly, GUIs have an appearance that changes to reflect the state of the world, which means the visual programmer needs to imbue their GUIs with dynamic motion, color, visibility, etc. This can be a pure function of the state of the world, and we all know how to crack that nut. If we give the visual programmer tools for drawing, and allow those drawings to be parameterized, we're off to a good start.

(NB. that this is the extent to which tools like Jupyter and Observable and LightTable are visual. They get off the train at this stop.)

(B) More importantly, GUIs have an effect on the state of the world. This is the heart of the matter. Any good toolset for making GUIs needs a robust means of describing effects. More broadly, we need really, really good tools for imbuing our GUIs with meaning.

The programming we have now expresses effects and imbues meaning in ways that have evolved over the past 60+ years, all the while taking as given the all but total assumption that programming is inherently linguistic.

For instance, you can bind a value to a name using an assignment operator like "=". In a verbose language, that could look like, "My name is Ivan." That's a statement.

What does a visual statement look like?

That image has a meaning. The meaning is unambiguous.

Like the meaning of the word "is" or the operator "=", the meaning of that image was communicated out-of-band. Once the meaning is known, you can use the image whenever you need to convey the meaning.

The process of making a visual programming language by re-presenting the primitives of a textual programming language will take us to a local maximum, where the only meanings we have available were the meanings that allowed for great expression via text.

You can make a language like AppleScript or Brainfuck or Piet, but these languages are still concerned with creating meaning via naming.

Experience has thus far shown us that this local maximum is perhaps lower than the local maximum reached by refining textual languages as text, further leveraging diff and hashing and Resharper and the like.

I contend that to reach a higher point in the visual programming design space, we need to start with a different set of primitives.

In short: If we're going to fuck with the "text" part of programming, we need to fuck with the "language" part of it too.

I propose that we dispense with the notion of a "visual programming language", and start again from a notion of "art programming".

Why "Art" Programming?

A big part of art (all art) is about meaning. Some art is intended to have a specific meaning. Some art is intended to have no meaning (but you can't stop the audience from projecting a meaning of their own). My drawing teacher said, "We are meaning-making machines." We have little agency in the matter. We're quick to render a verdict: "This painting of colored squares is meaningless!" At best, we're often reduced to giving post-hoc rationalization for our feelings about a work.

The creation of a programming tool that features a visual interface, rather than a textual interface, requires us to create tools for working with meaning in a visual way. That's a different shade of meaning than we're used to.

It's not sufficient to say, "red means error, green means success." That's all well and good until you need to make use of red as part of your data.

What would it mean to say "art language"? Would that be a style, like impressionist or cubist? Would it be a technique, like pointillism or encaustic? Would it be a medium, like gouache or oil or watercolor? Would it be a form, like painting, drawing, lithography, photography, sculpture?

Arguably, any of those. The point is, it's not directly analogous.

We should not look for the heart of "art programming" by analogy with "linguistic programming". We should instead bootstrap art programming by any means necessary, and give it 60+ years to grow, all the while taking as given the all but total assumption that programming is artistic.

What MIGHT Art Programming look like?

You need the computer to understand some primitive notions upon which you can build. So what are some good primitive notions?

I learned some primitives at art school, in a design class. What's nice about design (as opposed to art) is that it attempts to be systematic. You want to be able to convey specific ideas unambiguously, without additional out-of-band communication. Through theories like Gestalt, it tries to formalize this.

Here are some of those primitives:

• Proximity: Things are more related when they're closer together.
• Similarity: Things are more related when they're the same shape, or the same color.
• Arrangement: Things are more related when they're arranged in an obvious pattern, and unrelated when they're not.

To get from here to a Turing machine, all we need is a notion of place, a notion of execution, and some executable operations like moving, reading, and writing.

We get the notion of place for free, because this is a visual language which inherently needs to be spatial. (A musical programming might substitute 2d space for 2d time). We also get the notion of movement for free. We can earn reading & writing using color, shape, proximity, similarity, arrangement, any other design primitive, or any combination thereof.

CUTS

The change we've made is akin to a change from coding in the Roman alphabet to coding in Kanji — or Emoji. Instead of typing if-then-else, you have a character palette from which you pick an inquisitive face for "if", a happy face for "then", and a sad face for "else". This is a live coding environment, so the faces change their skin color from yellow (inactive) to green (active). This makes the language more visual, and you have greater visibility into the current state of the world.

This is a dead end.