Architectures matérielles genériques pour les réseaux neuronaux dynamiques et évolutifs appliqués à l'apprentissage continu

Offre de thèse

Date limite de candidature

16-10-2023

Date de début de contrat

16-10-2023

Directeur de thèse

JOVANOVIC Slavisa

Encadrement

La thèse sera encadrée à hauteur de 50% par le directeur et co-directeur de thèse.

Type de contrat

ANR Financement d'Agences de financement de la recherche

Candidater à cette offre target="_blank"

école doctorale

IAEM - INFORMATIQUE - AUTOMATIQUE - ELECTRONIQUE - ELECTROTECHNIQUE - MATHEMATIQUES

équipe

DEPARTEMENT 4 - N2EV : 406 - Mesures et architectures électroniques

contexte

Ce sujet de thèse s'insère dans le cadre du projet ANR SORLAHNA (Self-organizing representation [for continual] learning on adaptive hardware neural architectures) financé par l'Agence Nationale de Recherche (ANR). Il s'agit d'un projet collaboratatif entre les laboratoires de recherche nancéens LORIA (Laboratoire lorrain de recherche en informatique et ses applications) et l'IJL (Institut Jean Lamour). Les travaux de recherche dans le cadre de cette thèse seront menés à l'Institut Jean Lamour (https://ijl.univ-lorraine.fr), au sein de l'équipe Mesures et Architectures Electroniques (MAE) (Département 4, équipe 406), dont les travaux de recherche portent sur les architectures matérielles et accélérateurs adaptés aux réseaux de neurones et approches neuromorphiques.

spécialité

Systèmes électroniques

laboratoire

IJL - INSTITUT JEAN LAMOUR

Mots clés

Architecture matérielle, Electronique numérique, Verilog/VHDL/SystemC, FPGA/ASIC, Réseaux de neurones, Intelligence artificielle

Détail de l'offre

Face à la quantité exponentiellement croissante des données numériques recueillies et stockées dans tous les domaines, le prétraitement, la catégorisation et la visualisation des données jouent un rôle de plus en plus essentiel. Si l'apprentissage profond (Deep Learning) actuellement en plein essor offre de multiples possibilités pour répondre à une partie de ces besoins, l'apprentissage non supervisé est de plus en plus mis en avant pour en dépasser certaines limites.En effet, l'apprentissage profond repose sur l'ajustement d'un modèle paramétrique complexe à un ensemble gigantesque de données, fournies lors de cette phase d'ajustement. Le modèle une fois ajusté est alors déployé dans les applications réelles, partant du principe que la statistique des données reste alors la même que celle qui a servi à la phase d'apprentissage. Toutefois, certains contextes fournissent des données non-stationnaires, dont la statistique dérive peu à peu au cours du temps. Disposer d'un modèle paramétrique de ces données suppose que ce modèle puisse dériver avec elles. Les modèles supportant l'apprentissage continu ou incrémental doivent ainsi être privilégiés pour traiter dynamiquement de telles données non stationnaires, notamment rencontrées par de nombreux systèmes embarqués (internet des objets - IoT, edge computing). Parmi les modèles envisageables, nous nous intéressons aux modèles basés sur la quantification vectorielle topographique (cartes auto-organisatrices, réseaux incrémentaux). La simplicité algorithmique et la nature distribuée des calculs de tels modèles permet d'envisager une implémentation matérielle de ces algorithmes, qui prend tout son sens dans le contexte de systèmes embarqués. Le projet que nous proposons vise donc à combiner des compétences complémentaires en informatique et électronique pour co-concevoir des algorithmes modernes de quantification vectorielle topographique de sorte à intégrer dès leur conception la double exigence d'une adéquation avec l'apprentissage en ligne de données non-stationnaires et d'une compatibilité avec une implémentation matérielle réalisable et efficace, notamment à l'aide de circuits reconfigurables qui autorisent une flexibilité impossible sur les circuits ASIC. Cette approche de co-conception conduira à proposer des architectures matérielles génériques basées sur des unités de traitement neuronales (NPU) innovantes, hautement configurables et évolutives, qui aideront à réduire la haute dimensionalité des flux de données incessants générés par les infrastructures IoT, ou encore à construire des couches optimisées pour des modèles neuronaux hybrides visant un apprentissage continu.

Keywords

HW architecture, Digital design, Verilog/VHDL/SystemC, FPGA/ASIC, Neural networks, AI

Subject details

The preprocessing, categorization and visualization of data play an increasingly essential role with the exponentially increasing amount of digital data collected and stored in all fields. If the currently booming field of deep learning (DL) offers multiple possibilities to meet some of these needs, unsupervised learning is increasingly put forward to overcome some of its limits. Indeed, DL is based on the training of a complex parametric model to a huge set of data, provided during this training phase. The model, once trained, is then deployed in real applications, assuming that the statistics of the data then remain the same as those used in the learning phase. However, some contexts provide non-stationary data, whose statistics gradually drift over time. Having a parametric model of these data supposes that this model can derive with them. Models supporting continual or incremental learning must therefore be favoured to dynamically process such non-stationary data, in particular encountered by many embedded systems (internet of things - IoT, edge computing). Among the possible models, we are interested in models based on topographic vector quantization (self-organizing maps, incremental networks). The algorithmic simplicity and the distributed nature of the calculations of such models makes it possible to consider a hardware implementation of these algorithms, which takes on its full meaning in the context of embedded systems. The project that we propose therefore aims to combine complementary skills in computer science and electronics to co-design modern topographic vector quantization algorithms so as to integrate from their design the double requirement of an adequacy with online learning of non-stationary data, and a compatibility with a feasible and efficient hardware implementation, in particular using reconfigurable circuits allowing a flexibility that is unreachable on ASIC circuits. This co-design approach will lead to proposing generic hardware architectures based on innovative, highly configurable and scalable neural processing units (NPUs), which will help reduce the high dimensionality of the permanent data streams generated by IoT infrastructures, or even help building optimized layers for hybrid neural models aimed at continual learning.

Profil du candidat

Le candidat à la thèse doit avoir les compétences suivantes:
• une bonne maitrise de l'électronique numérique et de la conception matérielle (HW) (VHDL/Verilog, SystemC, HLS) - FPGA et/ou ASIC,
• un bon niveau en programmation (C/C++, Python),
• un bon niveau d'anglais (oral et écrit) est obligatoire, un niveau élémentaire en français est souhaitable,
• un très grand intérêt et motivation pour la recherche et développement.

Candidate profile

The future PhD candidate should have the following skills:
• strong knowledge of hardware (HW) digital design (VHDL/Verilog, SystemC, HLS) - FPGA
and/or ASIC,
• good level of programming skills are expected (C/C++, Python),
• good knowledge of English (oral and written) is mandatory, basic knowledge of French would
be an advantage,
• high motivation for research and development.

Référence biblio

[1] Plamen Angelov and Eduardo Soares. Towards explainable deep neural networks (xDNN). Neural Networks, 130:185–194, October 2020.
[2] Pouya Bashivan, Martin Schrimpf, Robert Ajemian, Irina Rish, Matthew Riemer, and Yuhai Tu. Continual Learning with Self-Organizing Maps, April 2019.
[3] Eden Belouadah, Adrian Popescu, and Ioannis Kanellos. A comprehensive study of class incremental learning algorithms for visual tasks. Neural Networks, 135:38–54, March 2021.
[4] Yann Bernard. Calcul neuromorphique pour l'exploration et la catégorisation robuste d'environnement visuel et multimodal dans les systèmes embarqués.
[5] David Chen, Sijia Liu, Paul Kingsbury, Sunghwan Sohn, Curtis B. Storlie, Elizabeth B. Habermann, James M. Naessens, David W. Larson, and Hongfang Liu. Deep learning and alternative learning strategies for retrospective real-world clinical data. npj Digital Medicine, 2(1):43, May 2019.
[6] Kuilin Chen and Chi-Guhn Lee. INCREMENTAL FEW-SHOT LEARNING VIA VECTOR QUANTIZATION IN DEEP EMBEDDED SPACE. 2021.
[7] Giansalvo Cirrincione, Vincenzo Randazzo, and Eros Pasero. The Growing Curvilinear Component Analysis (GCCA) neural network. Neural Networks, 103:108–117, July 2018.
[8] Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–1, 2021.
[9] Songlin Dong, Xiaopeng Hong, Xiaoyu Tao, Xinyuan Chang, Xing Wei, and Yihong Gong.
Few-Shot Class-Incremental Learning via Relation Knowledge Distillation. Proceedings of the AAAI Conference on Artificial Intelligence, 35(2):1255–1263, May 2021.
[10] Tetsuo Furukawa. SOM of SOMs. Neural Networks, 22(4):463–478, May 2009.
[11] Bernard Girau and Cesar Torres-Huitzil. Fault tolerance of self-organizing maps. Neural Computing and Applications, 32(24):17977–17993, December 2020.
[12] Slavisa Jovanovic, Hassan Rabah, Serge Weber, Khaled Ben Khalifa, and Mohamed Hedi Bedoui. Scalable, dynamic and growing hardware self-organizing architecture for real-time vector quantization. In 2020 27th IEEE International Conference on Electronics, Circuits and Systems (ICECS), pages 1–4, Glasgow, UK, November 2020. IEEE.
[13] Lyes Khacef. Exploration of brain-inspired computing with self-organizing neuromorphic architectures.
[14] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521–3526, March 2017.
[15] Zhiyuan Li, Xiajun Jiang, Ryan Missel, Prashnna Kumar Gyawali, Nilesh Kumar, and Linwei Wang. CONTINUAL UNSUPERVISED DISENTANGLING OF SELF-ORGANIZING REPRESENTATIONS. 2023.
[16] Shutao Li, Weiwei Song, Leyuan Fang, Yushi Chen, Pedram Ghamisi, and Jón Atli Benediktsson. Deep Learning for Hyperspectral Image Classification: An Overview. IEEE Transactions on Geoscience and Remote Sensing, 57(9):6690–6709, September 2019.
[17] Shuo Li, Fang Liu, Licheng Jiao, Puhua Chen, and Lingling Li. Self-Supervised Self-Organizing Clustering Network: A Novel Unsupervised Representation Learning Method. IEEE Transactions on Neural Networks and Learning Systems, pages 1–15, 2022.
[18] P J G Lisboa, S Saralajew, A Vellido, and T Villmann. The Coming of Age of Interpretable and Explainable Machine Learning Models. Computational Intelligence, 2021.
[19] Yat Long Lo and Sina Ghiassian. Overcoming Catastrophic Interference in Online Reinforcement Learning with Dynamic Self-Organizing Maps, October 2019.
[20] Pierre-Emmanuel Novac. MicroIA : Intelligence Artificielle Embarquée pour la Reconnaissance d'Activités Physiques sur Lunettes Intelligentes.
[21] German I. Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54–71, May 2019.
[22] German I. Parisi. Human Action Recognition and Assessment via Deep Neural Network Self-Organization, February 2020.
[23] German I. Parisi, Jun Tani, Cornelius Weber, and Stefan Wermter. Lifelong learning of human actions with deep neural network self-organization. Neural Networks, 96:137–149, December 2017.
[24] Duvindu Piyasena, Miyuru Thathsara, Sathursan Kanagarajah, Siew Kei Lam, and Meiqing Wu. Dynamically Growing Neural Network Architecture for Lifelong Deep Learning on the Edge. In 2020 30th International Conference on Field-Programmable Logic and Applications (FPL), pages 262–268, Gothenburg, Sweden, August 2020. IEEE.
[25] Basheer Qolomany, Ala Al-Fuqaha, Ajay Gupta, Driss Benhaddou, Safaa Alwajidi, Junaid Qadir, and Alvis C. Fong. Leveraging Machine Learning and Big Data for Smart Buildings: A Comprehensive Survey. IEEE Access, 7:90316–90356, 2019.
[26] Bharathkumar Ramachandra, Michael J. Jones, and Ranga Raju Vatsavai. A Survey of Single-Scene Video Anomaly Detection, August 2020.
[27] Lukas Ruff, Jacob R. Kauffmann, Robert A. Vandermeulen, Gregoire Montavon, Wojciech Samek, Marius Kloft, Thomas G. Dietterich, and Klaus-Robert Muller. A Unifying Review of Deep and Shallow Anomaly Detection. Proceedings of the IEEE, 109(5):756–795, May 2021.
[28] Sascha Saralajew, Lars Holdijk, Maike Rees, and Thomas Villmann. Prototype-based Neural Network Layers: Incorporating Vector Quantization, January 2019.
[29] Lukas Schott, Jonas Rauber, Matthias Bethge, and Wieland Brendel. Towards the first adversarially robust neural network model on MNIST, September 2018.
[30] Ajay Shrestha and Ausif Mahmood. Review of Deep Learning Algorithms and Architectures. IEEE Access, 7:53040–53065, 2019.
[31] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical Networks for Few-shot Learning. 2017.
[32] Qianru Sun, Hong Liu, and Tatsuya Harada. Online growing neural gas for anomaly detection in changing surveillance scenes. Pattern Recognition, 64:187–201, April 2017.
[33] Xiaoyu Tao, Xiaopeng Hong, Xinyuan Chang, Songlin Dong, Xing Wei, and Yihong Gong. Few-Shot Class-Incremental Learning, April 2020.
[34] Simen Thys, Wiebe Van Ranst, and Toon Goedeme. Fooling Automated Surveillance Cameras: Adversarial Patches to Attack Person Detection. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 49–55, Long Beach, CA, USA, June 2019. IEEE.
[35] Rui-Qi Wang, Xu-Yao Zhang, and Cheng-Lin Liu. Meta-Prototypical Learning for Domain-Agnostic Few-Shot Recognition. IEEE Transactions on Neural Networks and Learning Systems, 33(11):6990–6996, November 2022.
[36] Chathurika S. Wickramasinghe, Kasun Amarasinghe, Daniel Marino, and Milos Manic. Deep Self-Organizing Maps for Visual Data Mining. In 2018 11th International Conference on Human System Interaction (HSI), pages 304–310, Gdansk, July 2018. IEEE.
[37] Chathurika S. Wickramasinghe, Kasun Amarasinghe, and Milos Manic. Deep Self-Organizing Maps for Unsupervised Image Classification. IEEE Transactions on Industrial Informatics, 15(11):5837–5845, November 2019.
[38] Chathurika S. Wickramasinghe, Kasun Amarasinghe, and Milos Manic. Deep Self-Organizing Maps for Unsupervised Image Classification. IEEE Transactions on Industrial Informatics, 15(11):5837–5845, November 2019.
[39] Chathurika S Wickramasinghe, Kasun Amarasinghe, Daniel L. Marino, Craig Rieger, and Milos Manic. Explainable Unsupervised Machine Learning for Cyber-Physical Systems. IEEE Access, 9:131824–131843, 2021.
[40] Chayut Wiwatcharakoses and Daniel Berrar. A self-organizing incremental neural network for continual supervised learning. Expert Systems with Applications, 185:115662, December 2021.
[41] Chayut Wiwatcharakoses and Daniel Berrar. SOINN+, a Self-Organizing Incremental Neural Network for Unsupervised Learning from Noisy Data Streams. Expert Systems with Applications, 143:113069, April 2020.
[42] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S. Yu. A Comprehensive Survey on Graph Neural Networks. IEEE Transactions on Neural Networks and Learning Systems, 32(1):4–24, January 2021.
[43] Wenlong Wu, James M. Keller, Jeffrey Dale, and James C. Bezdek. StreamSoNG: A Soft Streaming Classification Approach. IEEE Transactions on Emerging Topics in Computational Intelligence, 6(3):700–709, June 2022.
[44] Noémie Gonnier, Yann Boniface, and Hervé Frezza-Buet. Consensus Driven Self-Organization: Towards Non Hierarchical Multi-Map Architectures. In Haiqin Yang, Kitsuchart Pasupa, Andrew Chi-Sing Leung, James T. Kwok, Jonathan H. Chan, and Irwin King, editors, Neural Information Processing, volume 1333, pages 526–534. Springer International Publishing, Cham, 2020.
[45] Hong-Ming Yang, Xu-Yao Zhang, Fei Yin, and Cheng-Lin Liu. Robust Classification with Convolutional Prototype Learning. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3474–3482, Salt Lake City, UT, USA, June 2018. IEEE.
[46] Lorijn Zaadnoordijk, Tarek R. Besold, and Rhodri Cusack. Lessons from infant learning for unsupervised machine learning. Nature Machine Intelligence, 4(6):510–520, June 2022.
[47] Hai-Tian Zhang, Tae Joon Park, A. N. M. Nafiul Islam, Dat S. J. Tran, Sukriti Manna, Qi Wang, Sandip Mondal, Haoming Yu, Suvo Banik, Shaobo Cheng, Hua Zhou, Sampath Gamage, Sayantan Mahapatra, Yimei Zhu, Yohannes Abate, Nan Jiang, Subramanian K. R. S. Sankaranarayanan, Abhronil Sengupta, Christof Teuscher, and Shriram Ramanathan. Reconfigurable perovskite nickelate electronics for artificial intelligence. Science, 375(6580):533–539, February 2022.
[48] Kai Zhu, Yang Cao, Wei Zhai, Jie Cheng, and Zheng-Jun Zha. Self-Promoted Prototype Refinement for Few-Shot Class-Incremental Learning. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6797–6806, Nashville, TN, USA, June 2021. IEEE.
[49] Slaviša Jovanovi´c and Hiroomi Hikawa. A survey of hardware self-organizing maps. IEEE Transactions on Neural Networks and Learning Systems, pages 1–20, 2022.
[50] Mehdi Abadi, Slavisa Jovanovic, Khaled Ben Khalifa, Serge Weber, and Mohammed Hedi Bedoui.
A hardware configurable self-organizing map for real-time color quantization. In 2016 IEEE Intern. Conf. Elec., Cir. Syst. (ICECS), pages 336–339, Monte Carlo, Monaco, December 2016. IEEE.
[51] Mehdi Abadi, Slavisa Jovanovic, Khaled Ben Khalifa, Serge Weber, and Mohamed Hedi Bedoui. A Multi-Application, Scalable and Adaptable Hardware SOM Architecture. In 2019 Intern. Joint Conf. Neural Netw. (IJCNN), pages 1–8, Budapest, Hungary, July 2019. IEEE.
[52] Mehdi Abadi, Slavisa Jovanovic, Khaled Ben Khalifa, Serge Weber, and Mohamed Hédi Bedoui. A Scalable Flexible SOM NoC-Based Hardware Architecture. In Erzsébet Merényi, J. Michael Mendenhall, and Patrick O'Driscoll, editors, Advances in Self-Organizing Maps and Learning Vector Quantization: Proceedings of the 11th International Workshop WSOM 2016, Houston, Texas, USA, January 6-8, 2016, pages 165–175. Springer International Publishing, Cham, 2016.
[53] Slaviša Jovanovic, Hassan Rabah, Serge Weber, Institut Jean Lamour, and Universite de Lorraine. High performance scalable hardware SOM architecture for real-time vector quantization. In 2018 IEEE Intern. Conf. Image Proc., Appl. and Syst. (IPAS), page 6, Sophia Antipolis, France. IEEE.
[54] Slaviša Jovanovi´c, Hassan Rabah, Serge Weber, Khaled Ben Khalifa, and Mohamed Hédi Bedoui. Scalable, dynamic and growing hardware self-organizing architecture for real-time vector quantization. In 2020 27th IEEE Intern. Conf. Elec., Circ. Syst. (ICECS), pages 1–4, 2020.
[55] Mehdi Abadi, Slavisa Jovanovic, Khaled Ben Khalifa, Serge Weber, and Mohamed Hédi Bedoui. A Scalable Flexible SOM NoC-Based Hardware Architecture. In Advances in Self-Organizing Maps and Learning Vector Quantization.