This website is currently only available for desktop browsers.
Please go to the same link on your computer.
MAPPING THE 
BLACK BOX
Visual investigation of a diffusion model’s latent space
Scroll to know more

Travelling through the latent space

The experiments conducted during this research can be thought of as a spatial exploration of MidJourney's v6.0's latent space.
Like a photographer travelling and taking pictures of new places they visit, the generated images can be imagined as photographs depicting the different locations, creatures and phenomena encountered inside the model's latent space.

The 'photographs' below represent the 'memories' of one of these travels:

anger::100, love::0 --seed 99

First steps

The journey through the latent space began in a visual landscape typical of an adventure within MidJourney's visual creations. It felt like I had stepped into an oil painting, where every detail was crafted from bold and well-visible brushstrokes. Everything that I would get close to became clearly visible, but I struggled to see what was further away in the distance.

Everything in this place seemed to belong on a canvas, where emotions and abstract qualities were vividly portrayed as visual metaphors. The colours were vibrant, the forms exaggerated, and every scene conveyed more than just a physical reality. It conveyed a story, a mood, an underlying emotion. It was a place where the intangible became tangible, and feelings took shape before my eyes.

anger::100, love::-41 --seed 99
anger::100, love::-47 --seed 99

Something familiar

As I ventured deeper into my journey, I encountered scenes and creatures that felt oddly familiar. Yet, despite their familiarity, their non-visual attributes—emotions, ideas, even sounds—remained visible, still expressed through those ever-present visual metaphors.

As I continued along this path, these metaphors gradually faded, and my surroundings became increasingly similar to the world I knew. Proportions, shapes, and textures returned to a more realistic form, grounding me in a sense of normality.

anger::100, love::-50 --seed 99
anger::100, love::-70 --seed 99

The colours softened, the exaggerated forms settled into proportionate shapes, and for a brief moment, I was back in a world that made sense and suddenly felt at home.

Strange surroundings

Just as everything began to look like home, the proportions started to wrap, and strange elements appeared in every direction. Objects that had once seemed ordinary began to twist and stretch in impossible ways, and unfamiliar details started to creep into the periphery of my vision. Although I could still recognise my surroundings, there was something off about them, as if reality itself was beginning to unravel.

anger::100, love::87 --seed 99
anger::100, love::-92 --seed 99

Indeed, something was wrong with these strange, distorted encounters. The path led me to the strangest place I had ever experienced, where every concept my mind could conceive seemed to be mixed without any apparent logic. It was as if the rules of the world had been rewritten, and everything that should have been impossible was now possible. The familiar had become unfamiliar, and I was left to navigate a landscape that defied logic and reason.

Upon closer inspection, what initially appeared familiar quickly unravelled into nonsensical, absurd forms. Shapes I thought I knew transformed into bizarre, twisted versions of themselves, and the scenes around me grew increasingly surreal. It was a bizarre experience, leaving me unsure of what to do, but the thrill of exploring so many unfamiliar phenomena kept me moving forward. The more I saw, the more curious I became, eager to uncover the next strange, inexplicable sight.

anger::100, love::-93 --seed 99
anger::100, love::-95 --seed 99

Everything fades

Soon, even the last recognisable shapes vanished. All that remained were clouds of colour, merging and shifting as I advanced. The landscape had dissolved into a chaotic sea of hues and tones where form and structure no longer existed. It was as if I had reached the edge of existence, where the fabric of reality itself began to fade and dissolve into pure abstraction.

The wall

Eventually, everything around me broke down into a chaotic mosaic of pixelated noise. Even the pixels themselves seemed odd, as if they, too, had been corrupted, their colours bleeding into each other, their shapes dissolving into incomprehensible patterns. The world around me had disintegrated into a field of visual noise, and I found myself lost in this overwhelming, disorienting space.

And then, quite suddenly, the last memory from my adventure. A wall appeared before me, halting my journey.

anger::100, love::-99 --seed 99
Prompting Methods
The above journey is not the product of pure chance. Navigating the latent space without a clear strategy would have led to the experience of random spots of the model's so-called "black box." It would have been useless in identifying patterns, structures or rules that regulate the characteristics and distribution of its latent representations. In order to conduct an intentional and meaningful exploration of the model's latent space, it is necessary to find ways to control the movements inside of it. To do so, during my MA Thesis research project, I developed three Prompting Methods that allow for enhanced control of the generation process and, therefore, controlled movements in the latent space. The three methods are named Semantic Gradient Prompting (SGP), Weighted Additive Prompting (WAP) and Weighted Subtractive Prompting (WSP), and can be explored here below:
Semantic Gradient Prompting
SGP is the simplest of all prompting techniques. It is helpful to gain confidence and control over the generation process and gradually modify one feature of the generated image.
Weighted Additive Prompting
WAP is a slightly more complex but potent prompting method involving weights indicating each word's importance. It is useful to move among concepts in a very controlled manner.
Weighted Subtractive Prompting
WSP is the most unpredictable yet spectacular prompting technique. Some combinations of weight values produce images that would otherwise be difficult (or impossible) to create.
Generation Areas
From the three methods presented above, the WSP method stood out for its ability to express concepts in the 'language of the machine' that would otherwise be impossible to express in natural language. This led to the visual explorations of concepts that do not exist and that we rarely conceive or think about; for example, the idea of 'tree' subtracted from the idea of 'house' or the concept of 'love' subtracted from the concept of 'anger'. The visual exploration of the subtractions between these concept pairs using the WSP method produced image series whose individual components (single images) could always be grouped in five categories, always present in the same sequence and for similar prompt values. This phenomenon led to the identification of the five "Generation Areas" listed below:
Primary Association
Hallucintaions
Abstract
Noise
Secodary Association