BERT 入門

浅川伸一 (東京女子大学) asakawa@ieee.org

14/Jun/2020

心理学に現れた注意のまとめ

Dicotomy

関連脳領域

認知心理学分野

計算モデル (Implementation)

総説論文

深層学習系

温故知新

分離脳 Split brain


From (Sperry 1968) Fig. 5

半側空間無視


From (Bloom and Lazerson 1988) Fig. 17-6

ポズナーとコーヘン


From (Posner 1980) Fig. 1, Fig.6: 右頭頂葉障害を呈した患者 (R.S.) の結果。 円:ターゲットが左視野提示， 三角:ターゲット右視野提示。 白点線:非有効手がかり，黒実線:有効手がかり。横軸は ISI。縦軸は反応時間中央値

特徴統合理論 (FIT)


From (Treisman and Souther 1985) Fig. 9

探索非対称性 search asymmetry}


From [Treisman (1988)} Fig. 3

上図右の結果は横軸に同時に提示された刺激の個数であり，縦軸は反応時間です。 線分特徴が存在する刺激 (Q) が目標となるか，存在しない (O) が目標となるか によって反応時間に差が認められます。結果は点線，すあんわち特徴が存在しな い目標を探索する条件，点線で描画，では同時に提示された刺激数が増加するに 従って反応時間が増大します。一方，特徴が存在する目標を探索する条件では， 同時提示された刺激の個数によらず反応時間は平坦になります。 以下に同様な実験結果を示しました。

スポットライトメタファー


From (Koch and Ullman 1985) Fig. 5

Inhibition of Return (IOR)

IOR
From http://www.scholarpedia.org/article/Inhibition_of_return

From The superior colliculus (SC) has been implicated as the neural substrate for IOR through four converging, but indirect, lines of evidence.

  1. IOR is abnormal in patients with midbrain degeneration due to progressive supranuclear palsy (PSP).
  2. It is preserved in patients with hemianopia, a condition in which only extrageniculate pathways are available to process visual information.
  3. It is present in newborn infants, in whom the geniculostriate pathways are not yet developed.
  4. It is generated asymmetrically in temporal and nasal visual fields, suggesting retinotecal mediation.

ガイド付き探索モデル Guided Search 2.0

最初にトップダウン注意を明示的に示した *ガイド付き探索モデル** [Wolfe (1994)}

From (Wolfe 1994) Fig. 2

(Itti and Borji 2015) の総説論文からそれまでのモデルの概説図

From (Itti and Borji 2015) Fig. 2

Friston’s attetion


From (Friston et al. 2014) Fig. 1

上丘 SC


From (Olshausen, Anderson, and Essen 1993) Fig. 10a

リズム現象


From (Fiebelkorn, Saalmann, and Kastner 2013) Fig. 1 and Fig. 2a

リズム現象 (2)


From (Buschman and Kastner 2015) Fig. 3b}

リズム現象 (3)


From (Buschman and Kastner 2015) Fig. 3a, Fig. 6}

DeepGaze II


From (Kümmerer et al. 2017) Fig. 2

DeepGaze II (2)


From (Kümmerer et al. 2017) Fig. 2

DeepGazeII より成績の良い最右の棒は人間の眼球運動データ

DeepGaze II (3)


From (Kümmerer et al. 2017) Fig. 3

IG: 情報ゲイン, IGE: 修正情報ゲイン, ACU: area under the ROC curve, sAUC: シャッフル精度, NSS: 正規化済キャンパス顕在性 normalized scanpath saliency

DeepGaze III


From (Kümmerer, Wallis, and Bethge 2019) Fig. 1

ヘルムホルツマシン


(Dayan et al. 1995);(Hinton et al. 1995)

ヘルムホルツマシン

\[ \log p(d\vert\theta) = -\sum Q_aE_a-\sum Q_a\log Q_a + \sum Q_a\log\left(\frac{Q_a}{P_a}\right)\\ =- F(d;\theta,Q)+\sum_a Q_a\log\left(\frac{Q_a}{P_a}\right) \]

\[ q^{(l)}\left(\phi,\mathbf{s}^{(l-1)}\right)=\sigma\left(\sum s^{l-1}\phi^{(l-1,l)}\right) \]

\[ Q_\alpha(\phi,d)=\prod\prod\left[q^{(l)}\left(\phi,\mathbf{s}^{(l-1)}\right)\right]^{s^{l}} \left[1-q^{(l)}\left(\phi,\mathbf{s}^{(l-1)}\right)\right]^{1-s} \]

\[ p_j^{(l)}\left(\theta,\mathbf{s}^{(l+1)}\right)=\sigma\left(\sum s^{(l+1)}\theta^{(l+1)}\right) \]

\[ p(\alpha\vert \theta)=\prod\prod\left[p_j^{(l)}\left(\theta,\mathbf{s}^{(l+1)}\right)\right] \]

モデル: ヘルムホルツマシン


From (Kawato, Hayakawa, and Inui 1993) Fig. 1 より

From (Hinton et al. 1995) Fig. 1 より

ボトムアップ処理による認識とトップダウン処理による(こう見えるはずだという思い込みの)生成を \(n\) (\(n=2,\ldots,4\)) 回繰り返す \(\rightarrow\)

定式化

思い込みの印象 \(\alpha\) と入力画像 \(d\) を用いて%%の記述長は，単なる前隠れ層ユニットの記述損失であり

\[ C(\alpha,d)=C(\alpha)+C(d\vert\alpha)\\ =\sum_{\ell\in L}\sum_{j\in\ell} C(s_j^\alpha)+\sum_i C(s_i^d\vert\alpha) \]

上式を用いて結合係数の更新を行う \[ \Delta w_{kj}=\epsilon s_k^\alpha \left(s_j^\alpha-p_j^\alpha\right), \]

\[ C(d) = \sum_\alpha Q(\alpha\vert d) C(\alpha, d) - \left[-\sum_\alpha Q(\alpha\vert d) \log Q(\alpha\vert d)\right]. \]

\[ p\left(\alpha\vert d\right)=\frac{e^{-C(\alpha,d)}}{\sum_\beta e^{-C(\beta,d)}} \]

\[ \Delta s_{j,t+1}=\epsilon s_{j,t}^\gamma(s_{j,t}^\gamma-q_{j,t}^\gamma) \]

全体の良い表象が得られるまで，すなわち下位層の活性を再構築するように複数回繰り返す

計算例


計算例


計算例 (2) 眼球運動のサンプリング}


文献

Bahdanau, Dzmitry, Kyunghyun Cho, and Yoshua Bengio. 2015. “Neural Machine Translation by Jointly Learning to Align and Translate.” In Proceedings in the International Conference on Learning Representations (ICLR), edited by Yoshua Bengio and Yann LeCun. San Diego, CA, USA. http://arxiv.org/abs/1409.0473.

Bichot, Narcisse P., Matthew T. Heard, Ellen M. DeGennaro, and Robert Desimone. 2015. “A Source for Feature-Based Attention in the Prefrontal Cortex.” Neuron 88 (November): 832–44.

Bloom, Floyd E., and Arlyne Lazerson. 1988. Brain, Mind, and Behavior. 2nd ed. New York, NY: Freeman.

Borji, Ali, and Laurent Itti. 2013. “State-of-the-Art in Visual Attention Modeling.” IEEE Transaction on Pattern Analysis and Machine Intelligence 35 (1): 185–207.

Broadbent, Donald E. 1958. Perception and Communication. Oxford,UK: Pergamon.

Buschman, Timothy J., and Sabine Kastner. 2015. “From Behavior to Neural Dynamics: An Integrated Theory of Attention.” Neuron 88 (October): 127–44.

Crick, Francis. 1984. “Function of the Thalamic Reticular Complex: The Search Light Hypothesis.” Proceedings of the National Academy of Sciences 81 (July): 4586–90.

Dayan, Peter, Geoffrey E. Hinton, Radford M. Neal, and Richard S. Zemel. 1995. “The Helmholtz Machine.” Neural Computation 7: 889–904.

Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. “BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding.” arXiv Preprint.

Duncan, John, and Glyn W. Humphreys. 1989. “Visual Search and Stimulus Similarity.” Psychological Review 96 (3): 433–58.

Eriksen, Charles W., and James D. St.James. 1986. “Visual Attention Within and Around the Field of Focal Attention: A Zoom Lens Model.” Perception and Psychophysics 40 (4): 225–40.

Fiebelkorn, Ian C., Yuri B. Saalmann, and Sabine Kastner. 2013. “Rhythmic Sampling Within and Between Objects Despite Sustained Attention at a Cued Location.” Current Biology 23 (December): 2553–8.

Friston, Karl J, Klaas Enno Stephan, Read Montague, and Raymond J Dolan. 2014. “Computational Psychiatry: The Brain as a Phantastic Organ.” The Lancet Psychiatry 1: 148–58.

Graves, Alex, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwi nska, Sergio Gómez Colmenarejo, et al. 2016. “Hybrid Computing Using a Neural Network with Dynamic External Memory.” Nature 538: 471–76. https://doi.org/10.1038/nature20101.

Heilman, Kennerh M., and Edward Valenstein. 1979. “Mechanisms Underlying Hemispatial Neglect.” The Annals of Neurology 5 (2): 166–70.

Hinton, Geoffrey E., Peter Dayan, Brendan J. Frey, and Radford M. Neal. 1995. “The "Wake-Sleep" Algorithm for Unsupervised Neural Networks.” Science 268 (5214): 1158–61.

Hopfield, John Joseph. 1982. “Neural Networks and Physical Systems with Emergent Collective Computational Abilities.” Proceedings of the National Academy of Sciences 79: 2554–8.

Itti, Laurent, and Ali Borji. 2014. “Computational Models: Bottom-up and Top-down Aspects.” In The Oxford Handbook of Attention, edited by Anna C. Nobre and Sabine Kastner, 1122–58. Oxford University Press.

———. 2015. “Computational Models of Attention.” ArXiv Preprint.

Itti, Laurent, and Christof Koch. 2001. “Computational Modelling of Visual Attention.” Nature Reviews Neuroscience 2: 1–11.

Itti, Laurent, Christof Koch, and Ernst Niebur. 1998. “A Model of Saliency-Based Visual Attention for Rapid Scene Analysis.” IEEE Transactions on Pattern Analysis and Machine Intelligence 20 (11): 1254–9.

Kawato, Mitsuo, Hideki Hayakawa, and Toshio Inui. 1993. “A Forward-Inverse Optics Model of Reciprocal Connections Between Visual Cortical Areas.” Network: Computation in Neural Systems 4 (4): 415–22.

Kimura, Akisato, Ryo Yonetani, and Takatsugu Hirayama. 2013. “Computational Models of Human Visual Attention and Their Implementations: A Survey.” IEICE Transactions of Information & Systems E96-D (3): 562–78.

Knudsen, Eric I. 2007. “Fundamental Components of Attention.” Annual Revivew of Neuroscience 30: 57–78.

Koch, Christoh, and Simon Ullman. 1985. “Shifts in Selective Visual Attention: Towards the Underlying Neural Circuitry.” Human Neurobiology 4: 219–27.

Krauzlis, Richard J., Lee P. Lovejoy, and Alexandre Zénon. 2013. “Superior Colliculus and Visual Spatial Attention.” Annual Review of Neuroscience 36 (165–182).

Kümmerer, Matthias, Thomas S. A. Wallis, and Matthias Bethge. 2019. “DeepGaze III: Using Deep Learning to Probe Interactions Between Scene Content and Scanpath History in Fixation Selection.” In Proceedings of Cognitive Computational Neuroscience, 542–45. Berlin, Germany. https://doi.org/https://doi.org/10.32470/CCN.2019.1235-0.

Kümmerer, Matthias, Thomas S. A. Wallis, Leon A. Gatys, and Matthias Bethge. 2017. “Understanding Low- and High-Level Contributions to Fixation Prediction.” In The IEEE International Conference on Computer Vision (ICCV), 4789–98. Venice, Italy.

Milanese, Ruggero, Harry Wechsler, Sylvia Gill, Jean-Marc Bost, and Thierry Pun. 1994. “Integration of Bottom-up Integration of Bottom-up and Top-down Cues for Visual Attention Using Non-Linear Relaxation.” In The Proceedings of CVPR, IEEE – Institute of Electrical and Electronics Engineers, 781–85. Dallas Texas, USA: Computer Vision; Pattern Recognition (CVPR).

Miller, Earl K., and Jonathan D. Cohen. 2001. “An Integrative Theory of Prefrontal Cortex Function.” Annual Revivew of Neuroscience 24 (167–202).

Monosov, Ilya E., and Kirk G. Thompson. 2009. “Frontal Eye Field Activity Enhances Object Identification During Covert Visual Search.” Journal of Neurophysiology 102 (October): 3656–72.

Moore, Tirin, and Marc Zirnsak. 2017. “Neural Mechanisms of Selective Visual Attention.” Annual Review of Psychology 68 (January): 47–72. https://doi.org/10.1146/annurev-psych-122414-033400.

Olshausen, Bruno A., Charles H. Anderson, and David C. Van Essen. 1993. “A Neurobiological Model of Visual Attention and Invariant Pattern Recognition Based on Dynamic Routing of Information.” The Journal of Neuroscience 13 (11): 4700–4719.

Petersen, Steven E., and Michael I. Posner. 2012. “The Attention System of the Human Brain: 20 Years After.” Annual Review of Neuroscience 35: 73–89.

Posner, Michael I., and Steven E. Petersen. 1990. “The Attention System of the Human Brain.” Annual Review of Neuroscience 13: 25–42.

Posner, Michel I. 1980. “Orienting of Attention.” Quarterly Journal of Experimental Psychology 32: 3–25.

Salinas, Emilio, and L. F. Abbott. 1997. “Invariant Visual Responses from Attentional Gain Fields.” Journal of Neurophsiology 77: 3267–72. https://doi.org/10.1152/jn.1997.77.6.3267.

Sperry, Roger W. 1961. “Cerebral Organization and Behavior.” Science 133: 1749–57.

Sperry, Roger W. 1968. “Hemisphere Deconnection and Unity in Conscious Awareness.” American Psychologist 28: 723–33.

Summerfield, Jennifer J., Jöran Lepsien, Darren R. Gitelman, M. Marsel Mesulam, and Anna C. Nobre. 2006. “Orienting Attention Based on Long-Term Memory Experience.” Neuron 49: 905–16. https://doi.org/10.1016/j.neuron.2006.01.021.

Treisman, Ann. 1964. “Selective Attention in Man.” British Medical Bulletin 20: 12–16.

———. 1988. “Feature and Objects: The Fourteenth Bartlett Memorial Lecture.” The Quarterly Journal of Experimental Psychology 40A: 201–37.

Treisman, Anne M. 1969. “Strategies and Models of Selective Attention.” Psychological Review 76 (3): 282–99.

Treisman, Ann, and George Gelade. 1980. “A Feature Integration Theory of Attention.” Cognitive Psychology 12: 97–136.

Treisman, Ann, and J. Souther. 1985. “Search Asymmetry: A Diagnostic for Preattentive Processing of Separable Features.” Journal of Experimental Psychology: General 114 (3): 285–310.

Vinyals, Oriol, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. “Show and Tell: A Neural Image Caption Generator.” In Computer Vision and Pattern Recognition (CVPR). Boston, MA, USA. http://arxiv.org/abs/1411.4555v1.

Wang, Wenguan, and Jianbin Shen. 2018. “Deep Visual Attention Prediction.” IEEE Transactions on Image Processing 27 (5): 2368–78.

Wardak, Claire, Etienne Olivier, and Jean-René Duhamel. 2004. “A Deficit in Covert Attention After Parietal Cortex Inactivation in the Monkey.” Neuron 42 (May): 501–8.

Wolfe, Jeremy M. 1994. “Guided Search 2.0 a Revised Model of Visual Search.” Psychonomic Bulletin and Review 1 (2): 202–38.