Perceptual reorganisation in deaf participants: Can high-level auditory cortex become selective for visual timing?

A paper recently published in PNAS reports a fascinating example of task-specific perceptual reorganisation in deaf participants that raises interesting questions regarding the involvement of high-level auditory cortex in temporal processing.


The study found that a rhythmic sequence task involving visual stimuli (a flashing disc) evoked activity in a high-level auditory region in deaf participants. The region – called area Te3 – showed stronger responses to temporally patterned sequences of visual flashes compared to visual sequences comprised of isochronous stimulation. In participants with intact hearing however, area Te3 showed rhythm selective responses only to auditory sequences, confirming that this region is typically involved in auditory processing. The authors concluded that auditory sensory depravation led to a reorganisation of the pathways servicing high level auditory cortex, a suggestion supported by connectivity analysis showing increased connectivity between area Te3 and visual area MT/V5 in deaf participants.


Although a striking example of perceptual reorganization, what is interesting about this conclusion is that the authors interpret the results as evidence of task-specific reorganization of high-level cortex. The implication here being that area Te3 is specialized for rhythmic processing in a modality independent manner. To support their argument, the authors note similar evidence of modality independent functional specialization in blind participants who show activation in visual cortex to auditory stimuli.


How could it be that an auditory selective region could come to be visually selective in deaf participants? One answer may lie in the residual hearing reported by the deaf participants. A table in the supplementary materials indicates that all the participants used hearing aids (outside the study) and that most reported their speech perception to be poor-moderate. This is interesting since listeners with low hearing rely more on visual temporal cues from the face to facilitate speech intelligibility. The increased utilization of visual timing cues to improve auditory processing may have led to a strengthening of the structural pathways between higher auditory cortex and visual cortex.


If so this raises questions regarding the degree to which area Te3 should be considered a task-specific region (i.e., modality independent, selective for timing tasks), or an auditory region typically involved in the temporal organisation of speech. Posterior STS is a multi-sensory region and shows many areas that are strongly selective for audio-visual speech perception. To identify the properties of area Te3, a more careful analysis of stimulus specific and task specific responses would need to be carried out within individual participants before any definitive claim can be made regarding the functional properties of this region.

Causal evidence for the right TPJ in temporal attention

Attention involves selecting a subset of the environment to undergo more elaborate processing in the brain. To respond appropriately to events in the world one must not only orient attention in space, but also in time. As it turns out, the brain regions most clearly implicated in spatial attention – the parietal lobes – are also thought to be involved in temporal attention, particularly ventral parietal regions such as the temporo-parietal junction (TPJ).


A recent study reported in the Journal of Cognitive Neuroscience provides further causal evidence in support of the view that the right TPJ in particular is dominant for temporal processing. The study utilized a novel simultaneity judgement task in two experiments that involved patients with lesions to the TPJ and healthy participants that had inhibitory TMS delivered to the TPJ. Participants were presented 4 flashing discs for 3 seconds (alternating uniform black and white) that were presented in the corners of an invisible square. On each trial, one disc was randomly selected to flash in counter phase to the other 3 discs (i.e., oddball disk was white when other disks were black). Prior to target onset, either the left or the right pair of disks was cued and participants were asked to judge whether cued pair flashed synchronously.


A staircase procedure showed that healthy controls and patients with damage to left TPJ could perform the simultaneity judgement at 80% accuracy when the flash rate for the array of items was approximately 9 Hz. In contrast, average flash thresholds for right TPJ patients were markedly worse, with 80% threshold observed when the flash rate was approximately 4 Hz.


The follow up experiment involving Transcranial Magnetic Stimulation (TMS) with healthy controls showed a similar pattern of results. In this experiment, inhibitory 1Hz TMS was applied for 20 minutes either to the left TPJ, the right TPJ or over early visual cortex. Simultaneity thresholds after TMS were worse (compared to pre-stimulation thresholds) only when the right TPJ was inhibited. Thresholds did not vary from baseline after inhibition of left TPJ, and inhibition of early visual cortex showed a slight improvement in flash thresholds.


By combining lesion and TMS methods, the results of the study provide convincing causal evidence that the TPJ is involved in temporal attention. Brain-imaging studies have previously reported TPJ activation during simultaneity tasks, however imaging studies are correlational and cannot say anything about the causal role of the TPJ in these processes. Indeed, the inclusion of TMS is important since many of the patients that participated in the study had very large lesions, whereas the effect of TMS is comparatively more focal.


However, the evidence that the right TPJ is dominant for temporal attention is somewhat ambiguous. It is difficult to tell from the data presented in the paper whether the extent of the brain damage observed in the left and right TPJ patients was the same. Moreover, the results of the TMS experiment did not provide strong evidence for the dominance of right TPJ for temporal processing. Although the right TPJ impaired temporal processing (compared to baseline), the magnitude of the impairment was not significantly different from the impairment observed in the left TPJ (however there was a trend toward significance). Strictly speaking then, the results of the TMS study do not provide firm support for the claim of selectivity. Nevertheless, the present paper adds to a growing number of studies examining the neural correlates of time perception using techniques that infer causality (TMS, TDCS etc) in a field that is largely dominated by brain imaging techniques.


Bronson Harry

The MARCS Institute, Western Sydney University

Twitter | ResearchGate | Web

Iterated reproduction task reveals rhythmic priors associated with exposure to music

According to Bayesian theories of cognition, perception involves integration of noisy sensory information with probabilistic internal models. These internal models reflect the net sum of all of our prior experiences and assist in structuring perception in the presence of unreliable sensory input. Known as priors, the influence of internal models can most clearly be observed in situations where sensory input is weak. In these cases, the prior makes a much larger contribution to perception, effectively biasing perception toward events that are more commonly encountered.


Musical practices are observed throughout all cultures and each musical system emphasises different rhythmic signatures. Is it possible that exposure to music forms rhythmic priors that help structure our perception of auditory sequences, and if so, are these rhythmic priors influenced by culture?


To assess rhythmic priors, Nori Jacoby and Josh McDermott from MIT devised an iterated reproduction task wherein participants tapped in time with auditory sequences comprised of repeating three-interval rhythms (e.g., 3:2:1, 1:2:1). On each trial the researchers surreptitiously replaced the auditory sequence with the rhythm produced by the participant from the previous trial. The idea of this procedure is that if temporal priors help structure the perception of musical sequences, then the rhythms produced by participants over successive trials should gradually become biased toward these priors. Indeed, the authors showed that reproductions tended to drift from the initial sequence and then stabalise after only five trials.


However, to ensure that the task itself was not biased toward cultural norms, the initial rhythm was randomly generated. In western music, interval ratios are usually comprised of integers. So to prevent the task from being influenced by western music conventions, the initial trial was randomly selected from all possible interval ratios, including non-integer values.


Despite the rhythms being randomly generated, reproductions tended to converge toward sequences with integer ratios. Importantly this effect was observed in a range of control experiments designed to rule out the role of motor demands. For example, the result was not specific to the effector since an integer bias was found when participants provided a verbal response. Likewise, a ratio bias was observed when sequences were reproduced from memory, indicating that the effect was not due to auditory-motor entrainment associated with synchronisation tasks.


Indeed, the effect of priors was also apparent in perceptual discrimination tasks. Participants were presented sequences that varied along a continuum between 3:2:3 and 1:1:1 and performed a same-different judgement task on pairs of sequences. Discrimination performance showed a pattern characteristic of categorical perception, with increased sensitivity found for non-integer rhythms and decreased sensitivity for rhythms near to integer ratios. The loss of perceptual sensitivity near integer patterns is indicative of a prior drawing the perception of patterns toward integer rhythms.


Crucially, the integer bias uncovered by the iterated reproduction task was influenced by exposure to music. In American participants, biases were observed only for ratios commonly found in western music. Likewise, a remote Amazonian population – the Tsimane – also showed a bias for integer ratios, however in this case, biases were only shown for intervals found in Tsimane music. However, the effect of the priors appeared to reflect passive exposure to common rhythmic structures, as American musicians also showed the same pattern of integer bias as Americans with no musical training.


Although Amercian and Tsimane cultures differed in the profile of intervals associated with priors, both cultures showed preferences for integer ratios. The Tsimane are a remote population with almost no exposure to western culture so it is unlikely that cultural transmission can explain a preference for integer ratios in the Tsimane. So this begs the question, how is it that both groups show priors for integer rhythms? Although iterated reproductions are often used in social science to explore the dynamics associated with the formation of shared practices, attitudes and beliefs, the authors’ stress that the task used here does not recapitulate the development of rhythmic preferences. Instead they argue the task only uncovers pre-existing internal preferences. How widespread such preferences are across different cultures and why preferences for integer rhythms emerge remains to be seen.


Bronson Harry

The MARCS Institute, Western Sydney University

Twitter | ResearchGate | Web

Frequency tagging indexes cortical entrainment related to temporal prediction

Listening to music typically induces a strong sense of an underlying beat. Although clearly related to periodicities contained in the stimulus, beat perception is internally generated since beat induction can occur in instances where no stimulation is present (i.e., in syncopated rhythms). Thus the capacity to extract a beat from periodic input provides an interesting phenomenon for examining the neural processes that temporally organise and ulitmately structure our perception of the world.

The neural mechanisms associated with beat-induction can be measured with a frequency tagging technique applied to EEG data recorded while participants listen to music. This approach involves examining the peaks in the frequency spectrum of EEG data that correspond to periodicities contained in the stimulus. By examining the activity evoked by syncopated rhythms (where the beat percept does not correspond to the sensory input), previous studies have shown that this technique indexes both periodic activity associated with the sensory input, and endogenous processes involved in extracting temporal structures from periodic input.

To further demonstrate the functional significance of this approach, Nozaradan, Peretz and Keller examined how neural entrainment measured with frequency tagging is correlated with individual differences in rhythmic motor control. The authors presented both unsyncopated and syncopated auditory rhythms whilst brain activity was recorded with EEG. Individual differences in ability to detect the beat in these rhythms were assessed offline with finger tapping tasks. In addition participants also completed another finger tapping task designed to assess temporal prediction. This stimulus comprised an aperiodic, predictable sequence of tones whereby the tempo continually varied between 400-600 ms (with a sinusoidal contour). This sequence was used to assess participants’ ability to anticipate the upcoming stimulus interval by quantifying the lag-1 and lag-0 correlation between the inter-stimulus interval and the inter-tap interval. Accurate prediction of the stimulus sequence would result in a larger lag-0 correlation than the lag-1 correlation, whereas tracking behaviour would be observed as a larger lag-1 correlation than lag-0 correlation.

The results showed that periodic stimulation produced peaks in the frequency spectrum of the recorded EEG data that corresponded to beat and non-beat frequencies. Importantly however, behavioural performance correlated selectively with the strength of entrainment in the beat frequencies. Tapping accuracy (mean asynchrony in the beat perception tasks) and the temporal prediction index (from the tempo change task) correlated positively with the height of the peaks for the beat induced frequencies, whereas the degree of entrainment in non-beat frequencies was negatively correlated with periodic tapping accuracy and was uncorrelated with prediction in tempo changing sequences. Together, these results highlight the functional significance of processes indexed by the frequency tagging approach, and show that beat perception is related to selective entrainment of neural activity to beat related frequencies.

The authors argue that the relationship between neural entrainment and temporal prediction is consistent with predictive coding models, whereby the brain optimises behaviour by forming internal models of the causes of sensory events. These internal models act as templates based on past experience that optimise sensory processing by providing predictions about the timing of upcoming sensory input. This argument is supported by another study published in 2016, which showed that temporal predictions associated with both periodic and aperiodic sequences lower sensory thresholds in a pitch discrimination task. Intriguingly, this study showed that isochronous stimulation also produced faster response times, whereas aperiodic stimulation did not. The authors of this paper argued that this dissociation may reflect the lowering of motor thresholds caused by simple sensory-motor coupling in the isochronous context, whereas changes in sensory processing may have been due to controlled processes that update internal models (i.e., linked to period correction). It remains to be seen whether the frequency tagging approach can be used to further dissociate the exogenous and endogenous processes involved in temporal prediction.


Bronson Harry

The MARCS Institute, Western Sydney University

Mental context biases retrospective temporal judgements

Our sense of time spans multiple scales; from seconds to minutes and days to years. Judgements for different temporal intervals however do not rely on a single unitary timing system, but instead rely on separate neural networks. For example, judgements within the sub- and supra-second range rely on networks involved in motor control (basal ganglia and cerebellum), whereas brain regions involved in long term memory and spatial navigation (prefrontal cortex and hippocampus) are involved in judgements spanning weeks and years.

A new functional magnetic resonance imaging study, led by researchers from Princeton, fills the gap in our understanding of how we judge time across intermediate timescales. The study examined time perception in the range of minutes by testing how accurately participants could estimate the amount of time that had elapsed between short excerpts taken from a 25-minute, science-fiction radio story.

The study investigated the idea that temporal estimation for intermediate intervals is related to the degree to which events are associated with similar contextual cues. This idea is based on theories of memory which posit that the recency of events can be ascertained by retrieving slowly varying contextual representations associated with global mental states. These representations include external environmental features (i.e., spatial location) and internal states such as goals and emotions. According to this theory, contextual cues bias temporal judgements such that the interval between events containing similar contextual features should be underestimated, whereas the the interval between events containing few contextual features should be overestimated.

To test this theory, participants listened to the radio story while brain activity was measured with functional magnetic resonance imaging. After the scanning session, participants completed a surprise temporal judgement test. Participants were presented two short clips that were either 2 or 6 minutes apart and were asked to estimate the time that elapsed between each excerpt. The degree to which these target clips were associated with similar mental context was estimated by examining brain activity recorded while participants initially listened to each clip. The authors used multi-voxel pattern analysis, a method that exploits distributed patterns of brain activity within a region of interest to measure the neural representations formed by different perceptual and cognitive states. MVPA was carried out by correlating the pattern of neural responses (across voxels) evoked by each clip with the idea that clips that share similar content, should also evoke highly correlated patterns of brain activity.

A region of interest analysis showed that pattern similarity in the right entorhinal cortex was correlated with temporal estimates. That is, clips that evoked similar patterns of brain activity within this region were associated with shorter duration estimates in the temporal judgement test. This result was also found when the correlations between judgements and pattern similarity were calculated for each clip pair across participants, indicating that variations in temporal judgements were not solely due to clips sharing perceptual features (i.e., if clips shared similar music).

Evidence that temporal judgements are based on representations formed in the entorhinal cortex is consistent with this region’s’ role in binding event content (i.e., objects, people, actions) within a broader spatial and temporal context. Indeed, a follow up analysis that examined the auto-correlation of evoked patterns within the entorhinal cortex showed that pattern similarity in this region fluctuated more slowly throughout the story than in neighbouring lateral temporal lobes. Together these results confirm the major predictions of the mental context theory of temporal estimation: temporal judgements are based on slowly varying representations that bind event content within broader contextual cues.

It remains to be seen whether the entorhinal cortex plays a general role in retrospective duration estimates for different tasks, contexts and timeframes. One possibility is that this region is particularly attuned to the temporal relations between events in spoken narratives. Depending on the nature of the story, the temporal relationships between events in a spoken narrative may be somewhat compressed compared to everyday experiences. In this case, it might be expected that the entorhinal cortex usually supports temporal judgements over hours and days in more naturalistic contexts.

Research published by the same group has shown that brain regions appear to be attuned to different temporal frequencies, a finding that most likely reflects the kind of representations formed within a region. It might be possible that temporal judgements for other timeframes (i.e., tens of seconds or hours) or different event content (daily activities, details of a conversation) may rely on other brain regions that are better suited for retrieving key information about different experiences.


Bronson Harry

The MARCS Institute, Western Sydney University

Structural coupling between auditory and motor networks is associated with sensorimotor synchronisation performance

Paced finger tapping tasks have been used extensively in brain imaging research to investigate the sensory and motor networks involved in the coordination of rhythmic movements. In comparison, much less is known about how these networks communicate to produce precisely timed actions. A paper published recently in Neuroimage provides new insight into the structural brain connections that underpin sensorimotor synchronisation (SMS) performance.

The study, conducted by Tal Blecher, Idan Tal and Michael Ben-Shachar at Bar Ilan University, explored the structural networks associated with two latent processes widely assumed to be associated with SMS performance: adaptation and anticipation. Adaptation and anticipation are two dissociable sensorimotor processes that are argued to help stabilise performance in SMS tasks. Adaptation refers to various reactive correction mechanisms that fine tune motor plans to minimise the asynchrony between actions and events. Anticipation on the other hand has been linked to the observation that actions (typically finger taps) tend to precede the pacing stimulus. Termed the negative mean asynchrony – the propensity for actions to occur before stimulus onset in SMS tasks suggests that participants do not merely react to stimulus onsets, but instead predict the timing of future events to ensure motor commands coincide with target stimuli.

To assess anticipation and adaptation, participants were instructed to tap in time with an auditory pacing stimulus that incorporated meter. Meter was marked by emphasising either every second tone (1 / 2 meter) or every third tone (3 / 4 meter). Participants were instructed to tap with their index finger for each emphasised tone, and to tap with their middle finger for all tones that were not emphasised. To assess adaptation, the meter presented to participants was changed at random intervals. The time taken to adjust the coordination of the index and middle fingers to the new meter – called time to resynchronise – was used as an index of adaptation. In contrast, to measure anticipation the mean asynchrony was calculated from performance data collected during auditory sequences that did not incorporate changes in meter (constant meter condition).

To examine the structural brain networks associated with adaptation and anticipation, mean asynchrony and time to resynchronise were correlated with brain imaging measures of white matter integrity. The authors used diffusion tensor imaging (DTI) – a technique that measures water diffusion – to identify the major white matter pathways in the brain. DTI exploits the propensity of water to diffuse freely only along the longitudinal axis of axons to delineate tissues that are composed of axons with uniform orientation, such as the major fibre tracts. In addition to tract identification, DTI can be used to estimate the microstructural integrity of the white matter pathways. One measure – called fractional anisotropy – quantifies the proportion of the total diffusion observed within a voxel that coincides with the primary direction of diffusion. High fractional anisotropy is related to microstructural tissue properties, such as the degree of axonal myelination, that are argued to facilitate communication between connected brain regions.

Using deterministic tractography, the authors focused their analysis on two white matter pathways involved in sensorimotor integration: the arcuate fasciculus and the corpus callosum. The arcuate fasciculus connects the superior temporal, inferior parietal and frontal lobes, and plays a prominent role in speech production, speech perception and action observation. In contrast, the corpus callosum connects homologous cortical regions in the left and right hemisphere. To limit the analysis of the corpus callosum to fibre tracts that link motor and auditory regions, the authors only examined the sections of the corpus callosum that corresponded to the pathways connecting bilateral pre-central gyrus (i.e., motor cortex) and bilateral temporal lobes (auditory cortex).

Analysis of the left arcuate fasciculus revealed a significant positive correlation between mean asynchrony and fractional anisotropy that was confined to an anterior portion of the tract. Given that observed mean asynchrony values were negative (i.e., distributed between -150ms and 0ms), this result indicates that participants with higher fractional anisotropy values were better able to synchronise with the auditory stimulus. The authors concluded that this finding adds evidence to the view that sensory motor integration relies on bidirectional coupling of brain regions involved in perception and action. Interpreting mean asynchrony as a measure of anticipation, these findings suggest that feedforward and feedback signals between frontal and temporal regions may be used to form predictions about the timing of upcoming auditory stimuli.

Fractional anisotropy in the pre-central segment of the corpus callosum was found to be negatively correlated with the time to resynchronise measure, indicating that increased integrity of the tract linking the left and right motor cortex was related to faster adaptation to changes in meter. To understand the behavioural significance of this finding, the authors decomposed the changing meter task into several underlying cognitive processes; meter change detection, new meter analysis, old meter inhibition, and execution of new motor plans. Based on evidence that callosal connections are predominantly inhibitory, the authors suggest that the pre-central callosal connections facilitate adaptation in the changing meter task via inhibition of the old meter.

Unexpectedly, fractional anisotropy in the temporal segment of the corpus callosum was found to be negatively correlated with mean asynchrony. Moreover, fractional anisotropy in this tract also correlated negatively with the standard deviation of asynchronies observed in the constant meter task. Taken together, these results indicate that participants with increased fractional anisotropy in this tract demonstrated less accurate and more variable performance in the tapping tasks. The authors provide two possible explanations to account for these apparently contradictory findings. Firstly the authors point out that the transmission of action potentials can be facilitated by either increased myelination and thicker axons. However, fibres comprising neurons with thicker axons should also demonstrate lower fractional anisotropy, as water would be free to diffuse more in directions perpendicular to the orientation of the axon. Alternatively, the authors also suggest that analysis of the auditory input might simply benefit from more lateralised analysis. In this case, sensorimotor synchronisation performance would benefit from decreased communication between the hemispheres.

In summary, these results seemingly point to the view that SMS performance is related to intra-hemispheric coupling between sensorimotor networks, with inter-hemispheric communication benefiting more complex tasks incorporating inhibitory processing. However, it is worth noting that the measures of SMS performance, particularly adaptation, depart considerably from those typically examined in sensorimotor synchronisation research. As noted by the authors, the changing meter task is likely associated with a range of cognitive processes. In contrast, models of SMS focus on much simpler forms of adaptation namely phase correction and period correction. These processes are thought to be carried about by functionally segregated timing networks not examined in this study. Future studies will need to examine these fundamental adaptive processes to determine whether they rely on different timing networks.