photocredit for the image above: Robin Zahler
I’m delighted to announce that this paper, Towards Holistic Prompt Craft has been conditionally accepted for presentation at the DIS2025 conference (Designing Interactive Systems) based on the work Roger Whitham and I have been doing over the last year with realtime AI image generation.
You can download a pre-print of the paper from here.
In this paper we reflect on the process of designing and deploying our PromptJ user interface, which was the software that allowed us to deliver several different versions of our Shadowplay installation (see here, here, and here). PromptJ is inspired by the design of DJ mixers, but rather than mixing records, you are mixing AI prompts. It looks like this:
Annotated PromptJ User Interface
The strength of this approach lies not in simultaneously manipulating all available parameters, but in understanding their relationships, potential, and making them available for users to exploit.
Based on everything we learned while we built and used PromptJ, in the paper we described several strong concepts. Strong concepts are like bits of advice or rules of thumb that arise from the process of designing something. We think these strong concepts will be useful insights for those designing realtime image-based generative AI systems and they include the following ideas:
- Pixel prompting is incredibly powerful
- Not all text prompts are equal
- A single interface can support performance and exploration
- Visual anchors make vast possibilities more manageable
These strong concepts then lead on to a bigger idea that brings all these together, we called this Holistic Prompt Craft. Holistic Prompt Craft tries to establish the idea that in any given generative AI system, all the parameters and variables that effect the output, are always in a multi-dimensional relationship.
Examples of how different parameters impact image outputs
Keeping all this in your head at the same time is a bit overwhelming, but our argument is that if interfaces are built that support people in understanding an exploiting those complex relationships, then, users will likely develop a greater understanding of the underlying technology, and in doing so become much more empowered. That is in contrast to the status quo for generative AI systems, which more frequently tries to hide that complexity from the user.
This small conceptual shift allows UI designs that can support users in understanding and using the significant parameters, and in turn to access creative juxtapositions and supportive reinforcements within models’ latent space that might otherwise remain inaccessible.
In the paper we also present a new design concept for a user interface that is built around this idea of Holistic Prompt Craft, this new design is called PromptTank, and it looks this:
Annotated PromptTank User Interface
At the time we wrote the paper that was concept design and wasn’t functional, but we have subsequently developed a full-functional prototype which we are using for future installations of Shadowplay, as well as other applications of realtime AI image generators.
PromptTank in use
The work builds on the ideas described in our earlier publication From Prompt Engineering to Prompt Craft, and is a really good example of how the process of designing something (in this case a user interface) can reveal no end of productive and useful insights.
