Extended Seminar – Systems and Machine Learning

The seminar will be jointly held by Profs. Carsten Bin­nig, Kristian Kersting, Andreas Koch, and Mira Mezini.

This seminar serves the purpose of discussing new research papers in the intersection of hardware/software-systems and machine learning. The seminar aims to elicit new connections amongst these fields and discusses important topics regarding systems questions machine learning including topics such as hardware accelerators for ML, distributed scalable ML systems, novel programming paradigms for ML, Automated ML approaches, as well as using ML for systems.

The top­ics will be as­signed based on an on-line bid­ding pro­cess, which will be opened after the kick-off.​ The final as­sign­ment will be made a week later.

Quick Links

Course Material (Moodle) – Registration (TUCaN)

Organization

Course Infos

Prerequisites

It is not necessary to have prior knowledge in artificial intelligence, but prior knowledge in software/hardware systems and machine learning is helpful.​ Participation is limited to 20 students.​

For further questions feel free to send an email to dm@​cs.​tu-darmstadt.​de. No prior registration is needed, however, please still send us an email so that we are able to estimate beforehand the number of participants, and have your E-mail address for possible announcements.​ Also make sure that you are registered in TUCaN.

Ex­tend­ed Sem­i­nar

What is “Ex­tend­ed” about this sem­i­nar? Stu­dents are not only ex­pect­ed to give a short talk, but also to pre­pare a small write-up.​ The write-up will be pre­pared in groups, each group will cover one theme, con­sist­ing of four topics.​ The final write-up must be con­cise and short, and should give a short overview of the theme (not nec­es­sar­i­ly lim­it­ed to the stud­ied pa­pers).

In ad­di­tion, we will also do a peer re­view­ing pro­cess, as it is usu­al­ly done at sci­en­tif­ic conferences.​ This means that you also have to read (some) of the other write-ups and pro­vide feed­back by fill­ing out a re­view form.

Be­cause they are more work for stu­dents, stu­dents re­ceive 4 CPs for Ex­tend­ed Sem­i­nars (in­stead of 3 CPs for reg­u­lar sem­i­nars).

Write-up

The talks are or­ga­nized in top­i­cal groups.​ Each group must pre­pare one short write-up of their work.

Content: The pa­pers are re­lat­ed to each other.​ Your task is to use these pa­pers to cre­ate a mi­ni-sur­vey that com­bines the re­sults of all pa­pers, and pos­si­bly other papers.​ The con­tri­bu­tion of each in­di­vid­u­al paper can be lim­it­ed to the most im­por­tant points that are con­tribut­ed by this paper to the topic.​ There must be a clear “red thread” with­in each sur­vey, a con­cate­na­tion of in­di­vid­u­al paper sum­maries is not enough.​ A pos­si­ble out­line can con­sist of an in­tro­duc­tion to set the stage and out­line the cross-cut­ting themes of all pa­pers, mul­ti­ple sec­tions on in­di­vid­u­al con­tri­bu­tions w.​r.​t.​ cross-cut­ting themes and com­par­i­son of dif­fer­ent ap­proach­es, a joined re­lat­ed work sec­tion, and a sum­ma­ry and out­look.

For­mat: The for­mat for the write-up is pre­de­fined, and fol­lows con­ven­tions that are typ­i­cal­ly used for pub­li­ca­tions in com­put­er science.​ In par­tic­u­lar, we re­quire each paper to be for­mat­ted ac­cord­ing to the Tem­plate for Proceedings in ACM Conferences (2-column layout). Each paper should have no more than 6 pages in this for­mat (the bib­li­og­ra­phy is not count­ed, and can be as long as nec­es­sary). The for­mat must not be changed in order to gen­er­ate more space.​ Each paper also must, of course, have a title, au­thors, and an abstract.​ The tem­plates are avail­able in Word and LaTeX, but we strong­ly rec­om­mend that you try to use LaTeX.​ Environ­ments such as MiK­TeX and TeXs­tu­dio make local La­TeX-edit­ing quite easy, and web-sites like Over­leaf offer col­lab­o­ra­tive work­ing en­vi­ron­ments for LaTeX.

Dead­line: The write-ups are due 29 January, 2018.

Re­view­ing

Re­view­s are required for all three other writeups.​ A review­ing form will be pro­vid­ed by then.​ The dead­line of the stu­dents’ re­views will be 19 February, 2018.

Grad­ing

The slides, the pre­sen­ta­tion, the an­swers given to ques­tions in your talk will in­flu­ence the over­all grade, as will the write-up and the reviews.​ Further­more, it is ex­pect­ed that stu­dents ac­tive­ly par­tic­i­pate in the dis­cus­sions, and this will also be part of the final grade.

To achieve a grade in the 1.​x range, the talk and write-up needs to ex­ceed the recita­tion of the given ma­te­ri­al and in­clude own ideas, own ex­pe­ri­ence or even examples/demos.​ An exact recita­tion of the pa­pers will lead to a grade in the 2.​x range.​ A weak pre­sen­ta­tion and lack of en­gage­ment in the dis­cus­sions may lead to a grade in the 3.​x range, or worse.​ For the write-ups it is im­por­tant that they pro­vide a co­her­ent view (like a sur­vey paper), and do not sim­ply con­sist of a con­cate­na­tion of four paper sum­maries.

Sched­ule

See schedule on Moodle (link above)

Topics

All pa­pers should be avail­able on the in­ter­net or in the ULB.​ Note that Springer link often only works on cam­pus net­works (some­times not even via VPN). If you can­not find a paper, con­tact us.

Machine Learning and Data Management

Machine Learning to enhance Database Systems (Binnig)

• (A1) Tim Kraska, Alex Beutel, Ed H. Chi, Jeffrey Dean, Neoklis Polyzotis: The Case for Learned Index Structures. SIGMOD Conference 2018.
• (A2) Ma, Lin, et al. Query-based Workload Forecasting for Self-Driving Database Management Systems. SIGMOD Conference 2018.
• (A3) Krishnan, S., Yang, Z., Goldberg, K., Hellerstein, J., & Stoica, I. Learning to optimize join queries with deep reinforcement learning. arXiv 2018
• (A4) Kipf, A., Kipf, T., Radke, B., Leis, V., Boncz, P., & Kemper, A. Learned Cardinalities: Estimating Correlated Joins with Deep Learning. arXiv 2018.
• (A5) Li, T., Xu, Z., Tang, J., & Wang, Y. (2018). Model-free control for distributed stream data processing using deep reinforcement learning. PVLDB 2018

Machine Learning for Knowledge Base Construct (Kersting)

• (B1) Ismail Ilkan Ceylan, Adnan Darwiche, Guy Van den Broeck: Open-World Probabilistic Databases. KR 2016: 339-348.
• (B2) Benny Kimelfeld, Christopher Ré: A Relational Framework for Classifier Engineering. SIGMOD Record 47(1): 6-13 (2018).
• (B3) Ce Zhang, Christopher Ré, Michael J. Cafarella, Jaeho Shin, Feiran Wang, Sen Wu: DeepDive: declarative knowledge base construction. Commun. ACM 60(5): 93-102 (2017).
• (B4) Parisa Kordjamshidi, Dan Roth, Kristian Kersting: Systems AI: A Declarative Learning Based Programming Perspective. IJCAI 2018: 5464-5471.

Machine Learning Systems

Distributed Machine Learning (Binnig)

• (A6) Mu Li, David G. Andersen, Jun Woo Park, Alexander J. Smola, Amr Ahmed, Vanja Josifovski, James Long, Eugene J. Shekita, Bor-Yiing Su: Scaling Distributed Machine Learning with the Parameter Server. OSDI 2014
• (A7) Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, William Paul, Michael I. Jordan, Ion Stoica: Ray: A Distributed Framework for Emerging AI Applications. Arxiv 2017
• (A8) Jiawei Jiang, Fangcheng Fu, Tong Yang, Bin Cui: SketchML: Accelerating Distributed Machine Learning with Data Sketches. SIGMOD Conference 2018
• (A9) Hantian Zhang, Jerry Li, Kaan Kara, Dan Alistarh, Ji Liu, Ce Zhang:
• ZipML: Training Linear Models with End-to-End Low Precision, and a Little Bit of Deep Learning. ICML 2017
• (A10) Anthony Thomas, Arun Kumar: A Comparative Evaluation of Systems for Scalable Linear Algebra-based Analytics. PVLDB 2018
• (A11) Tian Li, Jie Zhong, Ji Liu, Wentao Wu, Ce Zhang: Ease.ml: Towards Multi-tenant Resource Sharing for Machine Learning Workloads. PVLDB 2018

Automating Machine Learning (Kersting)

• (B5) Alexander J. Ratner, Christopher De Sa, Sen Wu, Daniel Selsam, Christopher Ré: Data Programming: Creating Large Training Sets, Quickly. NIPS 2016: 3567-3575.
• (B6) Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Tobias Springenberg, Manuel Blum, Frank Hutter: Efficient and Robust Automated Machine Learning. NIPS 2015: 2962-2970.
• (B7) Yutian Chen, Matthew W. Hoffman, Sergio Gomez Colmenarejo, Misha Denil, Timothy P. Lillicrap, Matthew Botvinick, Nando de Freitas:
• Learning to Learn without Gradient Descent by Gradient Descent. ICML 2017: 748-756.
• (B8) Antonio Vergari, Alejandro Molina, Robert Peharz, Zoubin Ghahramani, Kristian Kersting, Isabel Valera: Automatic Bayesian Density Analysis. CoRR abs/1807.09306 (2018).
• (B9) James Robert Lloyd, David K. Duvenaud, Roger B. Grosse, Joshua B. Tenenbaum, Zoubin Ghahramani: Automatic Construction and Natural-Language Description of Nonparametric Regression Models. AAAI 2014: 1242-1250.

Machine Learning and Software Engineering

Programming Abstractions for Machine Learning (Mezini)

• (C1) Abadi et al. TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems.
• (C2) Tran, Hoffman, Saurous, Brevdo, Murphy, Blei. Deep probabilistic programming. arXiv preprint arXiv:1701.03757 (2017).
• (C3) PyTorch Machine Learning Library.
• (C4) Pyro Programming Language.
• (C5) Gelman, Lee, Guo. Stan: A probabilistic programming language for Bayesian inference and optimization. Journal of Educational and Behavioral Statistics 40.5 (2015): 530-543.
• (C6) Gordon, Henzinger, Nori, Rajamani. Probabilistic programming. In International Conference on Software Engineering (ICSE, FOSE track), 2014.
• (C7) Gordon et al. Probabilistic Programs as Spreadsheet Queries. ESOP 2015.
• (C8) Baudart, Hirzel, Mandel. Deep probabilistic programming languages: A Qualitative Study.
• (C9) Wang, Wu, Essertel, Decker, Rompf. Demystifying Differentiable Programming: Shift/Reset the Penultimate Backpropagator.
• (C10) Hur, Nori, Rajamani, Samuel. Slicing Probabilistic Programs. PLDI 2014.

Machine Learning for Software Engineering (Mezini)

• (C11) Raychev, Vechev, Krause. Predicting Program Properties from “Big Code”. POPL 2015.
• (C12) Proksch, Lerch, Mezini. Intelligent code completion with Bayesian networks. TSE 2015.
• (C13) Raychev, Vechev, Yahav. Code completion with statistical language models. PLDI 2014.
• (C14) Bichsel, Raychev, Tsankov, Vechev. Statistical deobfuscation of android applications. CCS 2016.
• (C15) Amann, Nguyen, Nadi, Nguyen, Mezini. A Systematic Evaluation of Static API-Misuse Detectors. TSE 2018.
• (C16) Amann, Nguyen, Nadi, Nguyen, Mezini. MuDetect: The Next Step in Static API-Misuse Detection (Available on request).

Hardware for Machine Learning (Koch)

• (D1) C. Farabet, B. Martini, B. Corda, P. Akselrod, E. Culurciello and Y. LeCun, “NeuFlow: A runtime reconfigurable dataflow processor for vision,” CVPR 2011 WORKSHOPS, Colorado Springs, CO, 2011, pp. 109-116.
• (D2) Jouppi, Norman P., et al. “In-datacenter performance analysis of a tensor processing unit.” Computer Architecture (ISCA), 2017 ACM/IEEE 44th Annual International Symposium on. IEEE, 2017.
• (D3) Chen, T., Du, Z., Sun, N., Wang, J., Wu, C., Chen, Y., & Temam, O. (2014). Diannao: A small-footprint high-throughput accelerator for ubiquitous machine-learning. ACM Sigplan Notices, 49(4), 269-284.
• (D4) Y. Chen, T. Krishna, J. S. Emer and V. Sze, “Eyeriss: An Energy-Efficient Reconfigurable Accelerator for Deep Convolutional Neural Networks,” in IEEE Journal of Solid-State Circuits, vol. 52, no. 1, pp. 127-138, Jan. 2017.
• (D5) S. Han et al., “EIE: Efficient Inference Engine on Compressed Deep Neural Network,” 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA), Seoul, 2016, pp. 243-254.
• (D6) A. Parashar et al., “SCNN: An accelerator for compressed-sparse convolutional neural networks,” 2017 ACM/IEEE 44th Annual International Symposium on Computer Architecture (ISCA), Toronto, ON, 2017, pp. 27-40.
• (D7) H. Sharma, J. Park, D. Mahajan, E. Amaro, J. K. Kim, C. Shao, A. Mishra, H. Esmaeilzadeh, “From High-Level Deep Neural Models to FPGAs”, in the Proceedings of the 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), 2016.
• (D8) Kwon, Hyoukjun, Ananda Samajdar, and Tushar Krishna. “MAERI: Enabling Flexible Dataflow Mapping over DNN Accelerators via Reconfigurable Interconnects.” Proceedings of the Twenty-Third International Conference on Architectural Support for Programming Languages and Operating Systems. ACM, 2018.