The simple questions cracking the hard problem of consciousness

A few years back, I took a zombie test in Madison, Wisconsin, with neuroscientist Giulio Tononi, exploring his controversial integrated information theory of consciousness. This theory resulted in a consciousness detector, a tool designed to determine if unresponsive patients are actually conscious.

During the test, I sat in a chair resembling one at the dentist, while two doctors attached sensors to my scalp and applied magnetic pulses to my head. A conscious brain should respond with electrical activity. If I were a philosophical zombie, appearing sentient without true consciousness, my brain would respond with a dull thud.

After a few hours of testing, the results confirmed I was conscious, something I was confident about before the test, but now I had proof.

Yet, a simple yes-or-no test doesn’t capture the depth of conscious experiences—known as qualia, like the sensation of walking on slush or the routine of a dog walk. This gap, referred to as the hard problem of consciousness, appears unsolvable. However, neuroscientists are developing methods to make more nuanced distinctions in sensations and brain signals that might solve this mystery.

This “structural” approach is beginning to show how different people perceive colors, emotions, and sensations, addressing the philosophical question: is my perception of red the same as yours? “We’re at the end of the first phase of consciousness science and the start of the second,” says physicist Johannes Kleiner from the University of Bamberg in Germany.

Philosophers have long been intrigued by the structure of experience, which refers to how sensations are interconnected. “Structure will be at the core of the science of consciousness,” states David Chalmers at New York University, who formulated the hard problem of consciousness in the 1990s. Structuralists believe that each experience is interconnected with others. For example, perceiving the color red involves contrasting it with other colors and categorizing it differently from feelings like pain or joy. “Any individual experience hinges on this totality,” explains Holger Lyre, a philosopher at the University of Magdeburg in Germany.

The concept of using objective methods to explore subjective experiences dates back to the late 19th century, with structural studies in psychology. The fundamental method has not changed much: researchers present multiple stimuli, like color pairs, and ask participants to rate their similarity. Modern advancements lie in the scale and scope, with participants classifying thousands of stimuli including color combinations, shapes, and emotions.

This surge of quantitative data has drawn many young, mathematically inclined researchers from both inside and outside neuroscience. Based on people’s assessments of these differences in qualia, they develop classification systems of experience, often described as abstract geometric shapes. “Our approach is to categorise all possible relationships between qualia,” notes psychologist Nao Tsuchiya at Monash University in Melbourne, Australia, and ATR Computational Neuroscience Laboratories in Kyoto, Japan, who leads the Qualia Structure Project.

According to Tsuchiya, people’s judgments are generally consistent across age and culture when assessing colors, sounds, and other sensory qualities. In a study published last year, he and his colleagues surveyed 247 children aged 3 to 12 in Japan, 29 children aged 6 to 8 in China, and 84 adults about colors. The responses varied little. “The effect of language, culture and development seems rather small,” Tsuchiya comments. Language, culture, and environment influence the color terms we use, which can differ greatly: for instance, the Tsimané forager-farmers in the Bolivian Amazon combine blue and green into one concept, while ancient Greeks had numerous words for the sea’s subtle shades. Although culturally specific labels shape our perception of the world, Tsuchiya notes, they don’t seem to change our immediate experiences.

However, people’s judgments often differ from how these qualities have traditionally been structured through musical scales and color mixing rules. Most study participants do not identify tones an octave apart as the same, as musicians in varied cultures do. The color comparisons they make often appear mutually inconsistent, not aligning neatly on the rainbow. In other words, their experienced color space has more than the standard three dimensions—redness, greenness, blueness—in the widely used RGB color system. Tsuchiya’s experiments suggest we experience at least seven dimensions of color.

The project has transformed some classic philosophical thought experiments into empirical ones. For instance, is my red the same as your red? Studies of atypical color vision have addressed this. Following an online survey last year, the project invited participants into the lab for rigorous testing. Four were flagged as color-blind on a standard screening test, four weren’t flagged but reported that their color vision seemed different, and three had typical color vision. Within each group, people made similar judgments about color resemblance, but these assessments diverged between groups. Through geometric analysis, researchers distilled the data into a shape. The second group, the self-reported color-blind individuals, exhibited features of the shapes of the other groups. “It’s an intermediate shape,” says Tsuchiya. In structural thinking, what we perceive as red is defined by its position within this shape. Therefore, people within a group can say, “My red is the same as your red.” But what was red for one group might effectively be green for another. Tsuchiya describes the intermediate group as a “bridge” between atypical and typical color experiences, capable of understanding both.

The project is now extending its methods to other experiences. To study emotion, it shows participants video pairs intended to evoke certain emotions and asks them to compare their feelings. So far, Tsuchiya reports, people with difficulty expressing emotions—known as alexithymia—make the same distinctions as others, indicating that an inability to articulate emotions does not necessarily mean a lack of feeling.

The periodic table of experience

In the 19th century, structural psychologists likened their analyses to chemistry, breaking complex mental states into basic units. Following this metaphor, Tsuchiya aims to categorize qualia into a periodic table, with sections for color, pitch, and more. He acknowledges that this metaphor has limitations, as the periodic table’s power lies in repeating patterns, which he does not see in qualia. Instead, each has different structures, which he believes explains why sight and sound feel different. But Tsuchiya theorizes that, if experience is holistic, the various qualia might share some features. “Maybe there’s some kind of underlying similarity between the different modalities,” he says. “It must be the case, I think.”

With experiences sorted into these categories, researchers have new tools to test consciousness theories. Beyond a consciousness detector, they can search for brain activity patterns that match the mapped structure of experience. “We need to understand what exactly physically makes something feel red, or feel blue, or be painful, or be joyful,” says João Pedro Parreira Rodrigues at the Einstein Center for Neurosciences in Berlin.

Neuroscientists have long used similarity judgments to pinpoint brain areas’ functions, including parts of the visual cortex involved in color differentiation. “The signals that come out of this part of the brain, when I measure those collections of stimuli, have the same similarity structure as the ones I got from behaviour,” explains neuroscientist Brian Wandell at Stanford University in California. Tsuchiya’s project is now conducting a larger study along these lines. The aim is to use qualia’s structure to identify relevant brain activity, measured with an fMRI, which can then be compared against the predictions of different consciousness theories.

Lucia Melloni, a neuroscientist at Ruhr University Bochum in Germany, appreciates the project overall but has reservations. Tsuchiya and his team collect data by presenting people with two colors or other stimuli, clearing the screen, and asking participants to rate their similarity on a scale of 1 to 8. Participants must form a judgment, remember it, and assign it a number. Melloni is concerned something might be lost in translation: “I wonder whether he’s not just testing memory.”

Collaborating with Parreira and Zefan Zheng at Ruhr, Melloni is conducting a basic reality check on the structural approach. While Tsuchiya asks participants to compare two conscious stimuli, her team prompts comparisons of stimuli that participants aren’t aware of.

That seems impossible by definition. How can the brain compare things it isn’t even aware of? So, I was eager to try the experiment myself. Parreira guided me through installing the software. It resembled a tedious 1980s arcade game. A colored circle would flash, followed by a colored ring; sometimes, there was just a ring. I pressed the left arrow if the ring was red and right for green. This continued, cycling through different colors, for 40 minutes.

Surprisingly, I thought a circle only flashed half the time, but Zheng and Parreira explained one always flashed. If I didn’t notice it, it was because the ring appeared so quickly afterward that it confused my visual cortex, preventing the earlier stimulus from entering conscious awareness. By adjusting the timing, the experiment controlled what I consciously registered to compare my responses.

Even when the circle went unnoticed, it subtly affected my brain: it primed me. When a green circle preceded a green ring, I consistently pressed “green” about 50 milliseconds faster than if the circle was red. Thus, my unconscious mind was making color-similarity judgments of its own. I couldn’t articulate these judgments, but my reaction time revealed them.

It turned out that my conscious and unconscious color structures were quite different. The conscious one organized colors into a neat sequence from green to blue to purple to red; my unconscious one resembled a toddler’s chaotic crayon spill. “We found that there is no unconscious color space at all,” says Zheng. That’s a win for structuralism. Although Zheng cautions that the experiment is basic and needs validation, it suggests that structure differentiates conscious from unconscious perception and can, therefore, be used to identify consciousness in brain activity. “The structuralist turn is a very exciting development,” he says.

The hard problem of consciousness

Ultimately, Tsuchiya is not only seeking evidence that could support a specific consciousness theory; his goal is to address the hard problem, believing that qualia structure holds the key. One way to phrase the hard problem is to state that experiences have an intrinsic or unanalysable quality, like redness or joyfulness. Red appears to us as something independent of any associations. “Think of red while just ignoring all its relations,” suggests philosopher Hedda Hassel Mørch at the University of Inland Norway. “There’s clearly something still left to think about.”

However, science does not deal in unanalysable qualities. Everything in science is ultimately about relations. That’s what equations do: express relations. “Most scientific descriptions are framed in structural terms,” says Kristjan Loorits, a philosopher at the University of Helsinki in Finland.

Tsuchiya believes that experiences might be entirely structural, lacking intrinsic qualities. This idea aligns with Buddhism and integrated information theory, which associates consciousness with the causal structure of information-processing networks. “Experience is all about the relationships,” he explains. What seems like an intrinsic property might be just a complex web of these relationships. If so, it can be captured in an equation or other mathematical object; Tsuchiya has turned to a branch of mathematics called category theory for a descriptive language. Thus, science might be able to explain experience, making the hard problem less daunting.

Most philosophers who generally support structuralism do not take it that far. However, they agree that experience is, to some extent, analysable, and the real question is why we perceive it as unanalysable. Lyre suggests our brains have some faculty that gives our experience the impression of having intrinsic qualities. Ron Chrisley, a cognitive scientist at the University of Sussex, UK, suggests a reason for such a mechanism. The brain constantly monitors itself and could easily spiral into self-doubt. “Yes, I believe there’s a plate on the table, because it looks to me that there’s this oval thing, and there’s this shading here,” he says. “But why do you believe that there’s that oval shape and that shading? If it always has to give an answer, it’ll be caught in this infinite regress of justification.” At some point, the brain must stop analysing and accept its experiences as unanalysable.

For Loorits, the unanalysability of qualia is an essential aspect of human psychology. Great art often defies our expectations and leaves us in awe. “It’s beautiful, and wow, I can’t say why,” he says. But this is temporary. We talk to friends, read critics, and find ways to analyse our feelings. “Beauty that seems unanalysable to me now maybe becomes analysable by me,” he says. Experiences are purely structural from a third-person view, but we can perceive them as unanalysable from a first-person view, until we adopt some critical distance and see them structurally.

Whether sublime beauty is just one of many qualia that Tsuchiya’s methods can address remains to be seen. Yet, these efforts to map the connections among various experiences will continue to change how we perceive ourselves. If it’s true that experiences are all relative, then viewing one thing differently will transform how you view everything else, Lyre notes. People with synaesthesia, who might taste words or smell colors, perceive these connections directly, but we all make them. “To a certain extent, we are all synaesthetes,” he says.