"Putamen" by Woutergroen is licensed under Public Domain

How the brain’s face code might unlock the mysteries of perception

Dec 13, 2018

By Alison Abbott

Doris Tsao launched her career deciphering faces — but for a few weeks in September, she struggled to control the expression on her own. Tsao had just won a MacArthur Foundation ‘genius’ award, an honour that comes with more than half a million dollars to use however the recipient wants. But she was sworn to secrecy — even when the foundation sent a film crew to her laboratory at the California Institute of Technology (Caltech) in Pasadena. Thrilled and embarrassed at the same time, she had to invent an explanation, all while keeping her face in check.

It was her work on faces that won Tsao awards and acclaim. Last year, she cracked the code that the brain uses to recognize faces from a multitude of minuscule differences in shapes, distances between features, tones and textures. The simplicity of the coding surprised and impressed the neuroscience community.

“Her work has been transformative,” says Tom Mrsic-Flogel, director of the Sainsbury Wellcome Centre for Neural Circuits and Behaviour at University College London.

But Tsao doesn’t want to be remembered just as the scientist who discovered the face code. It is a means to an end, she says, a good tool for approaching the question that really interests her: how does the brain build up a complete, coherent model of the world by filling in gaps in perception? “This idea has an elegant mathematical formulation,” she says, but it has been notoriously hard to put to the test. Tsao now has an idea of how to begin.

Continue reading by clicking the name of the source below.

29 comments on “How the brain’s face code might unlock the mysteries of perception

  • This is so, so exciting for me. The start of this work began with one of my three hero Richards, Richard Gregory, in the sixties and his work with cats. He could observe some of the early levels of visual processing (edge detection). He also proposed that what we “saw” was what we expected to see only later modified by observational errors. This is not only confirmed but being richly detailed. This also allows us to better understand the contingency and uniqueness of our emotional experience, as Lisa Feldman Barrett reveals in her neuro-constructivist book “How Emotions are Made”.

    This is also far deeper analysis of the elements perceived, how they are divided up into discernible properties that can be evaluated. Rather than just the shadowy outlines of an image seen on the surface of a visual cortex higher level informational attributes can be externally read, a face looking away, a spiky thing, and the seen object inferred.

    The simplicity of the mathematical transformations to get to some of this meta-data, shouldn’t be that surprising, but it sweetly confirms the ready evolvability of inferential visual processing, that eyes can rapidly be made smarter with a few of the right neurons both evolving to the broad task and wiring to the specific of the individual.

    Report abuse

  • Amusingly enough, a picture of the researcher (visible on the source Nature article) shows her wearing a pendant necklace which appears to be a neuron shaped like a christian cross… if that is the case, I hope she’ll go on to discover the networks and codes underpinning that specific product of her hallucination engine…

    Report abuse

  • I have worked, since the age of fourteen with noisy drills and not much in the way of health and safety and ear protectors. Before the introduction of pneumatic drills into mainstream working life, the hammer drills were extremely noisy and high pitched. The damage that did and my age now has me hearing sentences that were not uttered. Being someone who has to learn how things work, I pay attention when someone asks me something and I hear something completely different. Why doesn’t my brain just say ‘I didn’t understand’ instead of making up a sentence that has me giving an answer that makes no sense?

    Report abuse

  • Cantaz #5


    But what and where is the switch? Is it ego? Is it the brain takes all past experiences, in such situations, and makes its best guess? Regardless, it shows perception has moments of uniqueness.

    Report abuse

  • Ollie,


    What we perceive is first what our brain predicts we’ll perceive. Our brain burbles endlessly away with regions gossiping to other regions and below conscious perception mostly saying things like “I think she’s going to say no” or “Thursdaaaayyyy” or “he’ll snarl”, if not as formed sentences, then as their precursor concepts. Quite often we’re right and if wrong may may go “huh” mentally and rewrite our simulation of the world.

    Oliver Sacks wrote a book about these experiences of “Hallucination” that our notable experiences of perceptual error arise when some data (always anticipated) isn’t forthcoming and our simulation is left to stand uncorrected. The newly blind experience Charles Bonnet Syndrome and have wonderful detailed visions. The schizophrenic unable to recall how the world works make shit up, god, aliens, the commies did it. And the old with failing everything experience more hallucinations than any other healthy group.


    Those of us with declining hearing (I did mine in with high powered PA systems) suffer tinnitus and or mishearing, our brain “helpfully” adding back a high frequency whistle because our brain can see there is not enough HF in the data stream. I not only mis-complete others sentences, (people mumble so, nor say what they are supposed to) but I hear clearly the odd word or phrase from nowhere. At those times I become aware that I am anticipating speech from someone.

    Ooo, the jewelry looks good. The Oxytocin could stand in for mistleltoe.

    Report abuse

  • Phil #7


    Thanks Phil. Stuff for me to follow up on. Is deja vu the time that brain gets what’s going to be said right and surprises itself?


    I have not not been diagnosed with tinnitus but I hear the high pitched noise when I go to bed. The reason I mention not being diagnosed is because I think I can make it go away. It’s taken a while to perfect my technique but I concentrate on the low, quieter notes and gradually the high pitch seems to lesson and disappear. Like turning down manually. I feel nauseous almost as soon as the tinnitus starts but that disappears along with the noise. Not very scientific but it works for me.

    Report abuse

  • I think it a very scientific approach. I do something similar and when I can I add in some natural sounds (leaves in trees, water from a weir outside) which drown out the whistle more of less.

    I used to have golden ears and could hear up to 22kHz . Testing them now I can’t hear above 10kHz.

    You can get a pretty good test from a phone app. called “Hearing Test”. Plenty of others exist.

    We have multiple paths for some of our apprehensions. It could be double reporting, but it could be the reporting of an error that’s not an error.  The ACC reports perceptual errors and calls you to “look again” Looking again when you didn’t need to might well be the experience of deja vu. I think you could be on to something…..

    Report abuse

  • Phil #9


    An old school friend can’t go to sleep without earphones and listening to football chat mainly. I tried that but wasn’t comfortable with the earphones. As disruptive for me.


    I enjoy joy finding out about things that happen like deja vu. I find it happens when my senses are heightened when something new is happening. The record, for me, has got to be when we got married in Indonesia. They just kept coming.

    Report abuse

  • Cantaz   #11


    That sounds like a credible theory. Those two memory types seem indistinct one from another at times, yet possess possibly different access paths.


    Might the deja vu incidents be potentially archetypal in some element or other?

    Report abuse

  • Phil #12

    Might the deja vu incidents be potentially archetypal in some element or other?

    You mean, in a quasi-Jungian sense?

    Don’t know, but I guess it could lead to testable predictions – first off, one would expect deja-vu incidents to happen more likely (if not universally, at least within a given cultural group) in response to relatively specific stimuli patterns/contexts rather than being random…

    Ollie’s mention above of his deja-vu-palooza during his wedding might open another visual angle on deja-vu as, possibly, a byproduct of the tendency of the brain to lay down strong memory traces of new events that are experienced as highly salient and emotionally relevant. Perhaps this can trip the brain to interpret the new salient event as an episodic memory even if the memory trace itself is being laid down as such for the first time?

    Report abuse

  • Cantaz,


    Less specific than Jungian. Just meaning as you say particularly salient as a class of event. It is of course potentially episodic if it gets into short-term memory, but when does it get assessed as a new class or selected as an existing class of experience?

    Episodic memories can be stripped of details if they can be part substituted by an archetypal characteristic, and archetypal characteristics are the essential ingredient of semantic knowledge. Spiky things hurt, say. And like that spiky thing attribute in the article, these kinds of interpretation might cause the perceived entity to get data compressed into sets of pre-existing attributes, like spiky thing or Jennifer Anniston-ness. Parts of weddings or social gatherings might have generalised attributes also.


    I strongly suspect my poor facial recognition/memory is the result of having a limited set of facial attributes available. I think  I over compress faces and lose too much differentiating material. Or with the kids I have remembered a whole lot of uncompressed detail. One guy I used to work with 20 years had broadly the same type of features as another guy I worked with ten years later. The face of the first one has written over the face of the second seemingly irretrievably for me. They are two people who did different things, but they have the same face….


    Ah, there’s a paper I need to find….

    Report abuse

  • This is the gist of what I am trying to say…


    “The studies reviewed here have provided evidence for a range of interdependencies between semantic and episodic memory. Semantic memory facilitates the acquisition of new episodic memories, and episodic memory facilitates the addition of new information to the semantic store. Similarly, episodic memories facilitate the retrieval of information from semantic memory, and semantic memories are the basic material from which complex and detailed episodic memories are constructed.”


    In a sense if reading faces depends on collections of attributes, faces like this, gendered, non-gendered, wide eyes, eye contact, looking away/down, etc.  and the useful output of identity  corresponds to some combinations of these attributes, then these may be the basis for semantic memory. Wide eyes are read often as young/vulnerable as they appear wider on younger faces.


    We can see that episodic memory derives on those particular occasions from identity of another, an action say, a location say, etc. all further combining into a personal episodic narrative. We can see, too, the need for these semantic/analytic entities being derived from experience and being established in Bayesian type weightings. It is notable, though, that we clearly are developing semantic/analytic memories before we acquire episodic memories. I have one memory from age three but several from age four. Later we may well take bulky “uncompressed/under analysed” episodic memories and break them down into semantic groupings or even grow a new category. Hmmm…


    Link in a post on its own.

    Report abuse

  • Phil #16


    For my benefit, in layman terms;

    one is the things itself. The other is the common relationship with the thing and the other is my personal relationship with the thing?


    I am struggling to relate that with what I feel when deja vu happens to me. It feels more like you described in #9 or that a glitch plays the same episode twice acknowledging the first time as a memory albeit only a nano second before?

    Report abuse

  • To be honest, I haven’t got to the deja vu thing yet. Shortly.

    I’m still trying to put the semantic/episodic thing together with the Nature article here. I find when building mental models I have to take them for a couple of test drives until using them becomes more automatic. Then I can use them to take a proper trip somewhere.

    What bugs me at the moment is that I have been insufficiently detailed on memories in stuff I have written on what specifically appears in the conscious window.

    I have new problems. Hurrah!

    Thanks, chaps.

    Report abuse

  • We broke Phil

    No, Olgun, we didn’t… I hate to have to break it to you… Phils is not an actual person, but an algorithm which pretends very shrewdly to be a person…

    Report abuse

  • Phil, #16

    …It is notable, though, that we clearly are developing semantic/analytic memories before we acquire episodic memories. I have one memory from age three but several from age four. Later we may well take bulky “uncompressed/under analysed” episodic memories and break them down into semantic groupings or even grow a new category…

    That makes sense; on the other hand, to me it seems more likely that semantic memory may arise progressively from the decontextualization of episodic memories rather than the other way around – It’s true that we typically have very few episodic memories from very early childhood, but who knows how much semantic/analytic knowledge we actually had at the same age? The problem is, semantic memories are constantly refreshed/updated all throughout our lifetime, but individual episodic memories are, for the most part, not nearly as much…

    Back to thinking about episodic memory, the saliency/emotionally relevance of an event and their possible connection to déjà vu; the “I saw my life flash before my eyes” phenomenon just popped into my head. If something like that really happens in some near-death events, it could be described as the uncontrolled unloading of episodic memory happening in a highly salient/emotionally relevant state. Perhaps this could be consistent with the idea that some déjà vu experiences may be due to some sort of priming of episodic memory processes that somehow trips the brain up into interpreting a currently experienced salient event as an episodic memory trace… Hmm, need to think about this more.

    Report abuse

  • Cantaz


    semantic memory may arise progressively from the decontextualization of episodic memories rather than the other way around

    Thats what I used to think. The data is gathered then distilled and essences found. But after the depth of analysis that Tsao and others have done showing how specific the metadata processing can get in visual perception, so much more specific than Richard Gregory’s edge detection, then I think we can see how those attributes can evolve themselves.


    Neuro-constructivism may have an account where neural hardware for meta-data (like the spiky thing node) is sculpted by a Bayesian application of a stimulus like thus and so netting a maximally useful extracted attribute. We have to imagine how this kind of vision can emerge in monkeys or mice or fish….We know that edge data extracted from what we see, confers a value in alerting us to a possible real world feature, a rock edge or a branch. We know that a near closed loop of edges may constitute a discrete object. That a moving closed loop may confer an object with agency. The whole loop moving quickly vertically down in our field of view might well be followed by a bite on the shin. Or a spiky loop warns us of a finger prick if touched.

    We certainly can accumulate semantic knowledge without any need for that big elaborate thing of episodic memory. The meta data we extract depends on the flux of our experience and how we react to it, but not on the more demanding constructions of a narrative, no matter how fragmentary. Episodic memory is exclusively conscious, and while I am not excluding some degree of consciousness from any animal, I suspect it is maxxed in us  big cortexxed spindle celled mammals. I think consciousness arises from our inclusion of ourselves in our simulations of the world. I think we check our status frenetically and I think episodic memory and narrative are key to being able to do this.

    Phffffffphzzzzzzzt,                   splickkkekekertttt…

    Report abuse

  • I think consciuosness arises from our inclusion of ourselves in our simulation of the world… I think episodic memory and narrative are key to being able to do this.

    If this is true, then does it follow that deja-vu could be an instance of doubled consciousness?

    Report abuse

  • Phil,

    Is there, in your opinion, a relationship between the countless predictions consciousness makes in order to deal with fast-moving reality, and the various disciplines developed by humans to keep attention focused?  Personally I have little experience of meditation and the like. Other than playing with my dogs, who seem to have lowered my systolic ten points.


    Report abuse

  • Cantaz,


    Multiple personality is something you hear little of these days, but that might be an instance of double consciousness starting to form, though the personalities are traditionally serial, only overlapping for comedic effect. Need to think about that one too.



    Just measuring my systolic repeatedly, brings the number down. Exercise does it for me but I also seem to panic mightily with that first tight grip on my arm. I think I am much more anxious than the front I project. I’ll start 30 points up on the first (160) and fall rapidly in four measurements asymptotically to 130 where it’ll stay. The pulse pressure (diastolic subtracted from systolic) is terrifying at first, 65 or so but falls to 45 showing much more elasticity of the blood vessels. I think this is just a measure of the BP fall actually during the measurement and diastolic is always assessed second.


    I don’t seem to learn about the tight grip on my arm and always look for the asymptote. My subconscious semantic assessment is something like this is a precursor to a beating from some bully, perhaps. (I experienced more than a few at school.).


    I’m sure pets calm us, meditation too. I’m seriously contemplating getting into formal mindfulness exercises. (I know it can be good for kids getting stressed by school.) And yes, I think this is related to our frenetic self evaluations that feed our self, conscious experience.

    Report abuse

Leave a Reply

View our comment policy.