My name is Falk Pollok. I am a Senior Software Engineer at IBM Research Europe and a Senior Research Software Engineer at the MIT-IBM Watson AI Lab.

I specialize in Foundation Models and Retrieval Augmented Generation.



Optional background music:


Professional Profile

Short Bio

Falk Pollok is a Senior Software Engineer in IBM Research Europe and a Senior Research Software Engineer for the MIT-IBM Watson AI Lab specializing in Foundation Models and Retrieval Augmented Generation. Falk was a member of the MIT-Harvard-Stanford team on DARPA’s Machine Common Sense project, contributed to IBM watsonx, Watson Core, Watson Orchestrate & Watson Machine Learning, was the lead developer for IBM Sapphire and founded IBM’s Engineering Excellence program. He holds a master’s degree in CS from RWTH Aachen, leadership certificates from Cornell and IBM’s highest developer profession rank. Moreover, he published 8 papers in top conferences like NeurIPS, AAAI & Middleware and 2 patents, was named a Face of IBM Research and received multiple awards incl. IBM’s Outstanding Technical Achievement and InfoWorld’s BOSSIE award.

Falk Pollok is a Senior Software Engineer in IBM Research Europe specializing in foundation models. As a tech lead in the Global Advanced Prototyping Team he strengthens IBM as a preferred business innovation partner by building client-centered prototypes from cutting-edge research thus maximizing IBM’s lab-to-market impact.

Falk founded IBM’s Engineering Excellence program for which he was featured as one of five Faces of IBM Research, now serves on its steering committee and still leads its core speaker series which so far has reached almost 2,500 unique IBMers in 70 business units across 48 countries. He holds the highest Developer Profession Degree (Level 3 Expert) and was awarded with IBM’s highest technical achievement (Outstanding Technical Achievement) in 2022 as well as a Research Accomplishment in 2021 for his work on Neuro-Symbolic Models and World Simulators for Machine Common Sense which he conducted as Senior Research Engineer at the MIT-IBM Watson AI Lab (which he still collaborates with) as part of an interdisciplinary MIT-Harvard-Stanford team working on a multi-year DARPA project on Machine Common Sense. In this context of mimicking the cognitive abilities of small children he also implemented a neuro-symbolic visual question answering prototype on a TIAGo++ robot.

Before that, while working for IBM Research Cambridge and the T.J. Watson Research Center in Yorktown, he was the first engineer on IBM Verdi, the AI orchestration research system behind Watson Orchestrate, with many core contributions to services, integrations and its architecture. In 2018, he and his team were awarded InfoWorld’s Best of Open Source for Machine Learning (BOSSIE) Award for their work on IBM’s Fabric for Deep Learning, a cloud-based AI training platform. Before his career at IBM Falk was the lead developer of IBM Sapphire, a $6.5M project at the University of Michigan investigating dialog systems which was led by IBM’s Speech CTO David Nahamoo as well as University of Michigan Professor Satinder Singh.

Falk has published papers in top conferences like NeurIPS, AAAI and Middleware, earned multiple Cornell certificates (Change Management, Performance Leadership, Executive Leadership, Project Management 360) and gives back to the community (book and paper reviews, Blue Core Mentor, IBM Profession Champion, IBM Recognized Teacher/Educator, IBM Recognized Speaker/Presenter, IBM Intellectual Capital / Intellectual Property Creator, multiple Manager’s Choice awards). He earned his Master’s degree from RWTH Aachen.

Curriculum Vitae / Resume

Please find my CV here.

Selected Projects

  • watsonx: I was the first engineer within MIT-IBM working on foundation models and closely collaborated both with watsonx Platform Engineering and the Advanced Prototyping Team. I delivered prompt and prefix tuning capabilities, worked with clients on leveraging foundation models and have worked on Retrieval Augmented Generation. You can read more about the product on Forbes.
  • DARPA Machine Common Sense: I worked on the DARPA MCS project as part of the MIT-Harvard-Stanford team to mimic the development of commonsense in young children working in close collaboration with Josh Tenenbaum, Dan Gutfreund and Vikash Mansinghka. We were able to defeat Berkeley and OSU by total score two times in a row. Furthermore, I demonstrated neuro-symbolic visual question answering on a robot to bring our AI capabilities into the real-world.
  • IBM Verdi / Watson Orchestrate: Multi-Agent System for Business Process Optimization and Automation, foundation of Watson Orchestrate
  • IBM’s Fabric for Deep Learning (FfDL) / Watson Machine Learning: Open source heart of Watson Machine Learning (WML) for framework-agnostic distributed deep learning on top of Kubernetes that integrates with a wide ecosystem of related open source projects like Uber Horovod, the Adversarial Robustness Toolkit (ART), AI Fairness 360 (AIF360), H20.ai, Seldon and kube-batch. Won InfoWorld’s Best of Open Source Award 2018.
  • IBM Sapphire: I was the lead developer of IBM Sapphire at the University of Michigan, a dialog system for academic advising which consisted of over 30 neural and non-neural AI services.
  • Global Advanced Prototyping Team: While remaining part of the MIT-IBM team, I have devoted some of my time to start collaborating with our Global Advanced Prototyping Teams as a catalyst to accelerate the transfer of cutting edge research into product. For instance, I have delivered Multivariate Singular Spectrum Analysis (mSSA) into our Smarter Resources and Operations Management solution and built the multi-cloud capabilities of IBM’s plant optimization solutions such as the Cognitive Plant Advisor.
  • Engineering Excellence: I have founded IBM’s Engineering Excellence Platform and led it for the first year. Now that it has grown bigger and has executive support, I serve on its steering committee. I am most proud of the Engineering Excellence Speaker series which has reached about 2,500 unique IBMers across 70 business units in 48 countries. You can read more about it here.

Me with my colleagues Grady Booch and Lee Martie at the MIT-IBM Lab in Cambridge on the left as well as with Scott Boag at Stanford on the right.

Professional History

Senior Software Engineer & Tech Lead,
IBM Global Advanced Prototyping Team
IBM Research Europe (since 2022/11 – 2023/09)
           Foundation Models incl. Prompt Tuning and Retrieval Augmented Generation

Senior Research Software Engineer,
MIT-IBM Watson AI Lab (since 2020/02 – 2023/09)
           DARPA Machine Common Sense Project, Lab $250 million (10y),
           Member of MIT-Harvard-Stanford Team, Research Affiliate MIT,
           Neuro-Symbolic Visual Question Answering on TIAGo++ Robot

Software Engineer, Core AI, IBM Research AI, Cambridge (2018/03 – 2020/02)
           Grouped with Thomas J. Watson Research Center and MIT-IBM Watson AI Lab
           Fabric for Deep Learning    Business Process Automation via Multi-Agent System
           InfoWorld’s Best of Open Source (BOSSIE) Award 2018

CSE Software Engineer & Lead Developer (2016/05-2018/02),
IBM Sapphire Project at University of Michigan ($6.5 million)
 Dialog systems   Microservice architecture   Recording pipeline   Demos            
          Academic Affiliate (May-Nov. 2016)

Education

I have a Bachelor’s and Master’s degree of Computer Science from RWTH Aachen.

Master of Science, RWTH Aachen

           Machine Learning   Computer Vision   Artificial Intelligence   Seminar IT Sec.
           Current Topics in Media Computing & HCI   Designing Interactive Systems II
           Advanced Learning Technologies   e-Learning   Graph Theory   SAT Checking

Thesis: : Leveraging UIMA-Based Question and Answering Systems for eLearning and Research

              Voluntary Courses (with exercises)
              Knowledge Representation   Logic Programming   Functional Programming
              IT Security I & II   Computer Graphics   Compiler Construction   Model Checking
              Virtual Reality   Software Quality Assurance

Bachelor of Science, RWTH Aachen

           Linear Algebra   Calculus   Logic   Applied Stochastics   Discrete Structures
           Operating Systems & System Software   Computability & Complexity
           Formal Systems, Automata, Processes   Efficient Algorithms
           Secure Distributed Systems   Computer Engineering   Programming
           Software Engineering   Numerical Computations   Embedded Systems
           Designing Interactive Systems I   Databases and Information Systems
           Data Structures & Algorithms   Lab Hardware Programming
           Lab Electrical Basics of Computer Science   Seminar Cloud Computing
           Proseminar Artificial Intelligence

Thesis: Real-Time Tracking on Mobile Devices for Augmented Reality

Furthermore, I have for multiple years worked for the University of Michigan and with MIT (as part of an MIT-Harvard-Stanford team).

Expertise

I am a backend developer focusing on AI and cloud. While I have written both native and web UIs (e.g., Qt, GWT, Angular, React) in the past as well as taken three HCI lectures, I prefer focusing on the backend rather than being a jack of all trades and master of none.

Major Fields of Expertise

  • Enterprise AI and Architecture of Complex AI Systems
  • AI Orchestration and Software Agents as Digital Workers
  • ML Systems in Hybrid and Multi Cloud Settings

Languages: English (fluent), German (fluent), Latin

Programming Languages: Python, Java, Julia [C/C++, JavaScript, Haskell, Prolog, Go, Visual Basic, Bash, Wolfram Language, MATLAB, R, SAS]

AI: Foundation Models, NLP, Vision, Machine Learning, Deep Learning

Cloud: Linux, Cloud, Kubernetes. I have primarily worked against IBM Cloud and Watson, but also briefly worked with Azure, GCP and AWS.

Frameworks and Tools I have worked with in the past include PyTorch, Tensorflow, Huggingface (transformers, evaluate, peft), langchain, scikit-learn, spacy, nltk, Docker, Kubernetes and various Linux distributions.

Interests: Foundation Models, Retrieval Augmented Generation, NLP, CV, Multimodal Q&A, Augmented Reality, Hybrid and Multi Cloud, Embodied AI.

Publications, Patents, Certificates and Microcredentials

My publications and patents can be found here.

I hold several leadership certificates from Cornell University: Executive Leadership, Performance Leadership, Change Management and Project Management 360 (comprises Project Management and Project Leadership). I also have an Advanced Data Science certificate from Coursera.

My microcredentials can be found here which include:


Awards

Outstanding Technical Achievement 2022: Neuro-Symbolic Models and World Simulators for Machine Common Sense

Congratulations Falk for being selected to receive an Outstanding Technical Achievement Award for your research on Neuro-symbolic Models and World Simulators for Machine Common Sense. Thank you for the work that you and your team do to solve difficult AI problems by collaborating with top research institutions on innovative ideas. Together with your collaborators you have demonstrated scientific leadership in the important AI field of common-sense reasoning. Your contributions include both algorithmic development and dataset creation, and through your work you have shown how common sense can be incorporated into neuro-symbolic AI models. The importance of your work on common-sense reasoning, and the strength of your team which includes collaborators from MIT, Harvard and Stanford, has also been recognized by the contract awarded to you by DARPA. In addition, you have impacted the broader AI research community by publishing numerous papers, giving many invited talks and tutorials at top conferences, and sharing datasets, challenges and your photo-realistic simulation platform for training and testing AI agents. Altogether this is an impressive example of cross institution teaming. Your research leadership on important and fundamental AI problems is what helps to make IBM Research known for its innovation, and your collaboration with the broader research community enriches IBM as a research institution and helps make it a great place to work. I and the Exploratory Science team appreciate and applaud your excellent work and look forward to future developments.

Jeff Welser, COO IBM Research / VP Exploratory Science & University Collaborations / VP Almaden, Brazil & Japan Research

You can find a scan of the certificate here.

Research Accomplishment 2021: Neuro-Symbolic Models and World Simulators for Machine Common Sense

From a very young age, humans are able to grasp basic physical and social principles that govern their environment. For example, at around 4 months of age, babies understand that an object does not vanish out of existence even when it is out of sight. Often such principles are not labeled or taught in a direct supervised manner and we refer to them as common sense. As humans grow and develop the ability to communicate in natural language, their common sense understanding develops as well. For example, adults possess an intuitive understanding of Newtonian physics and are able to make accurate predictions and answer questions about dynamic scenes involving interactions between several objects. Common sense is necessary for humans, and other animals, to successfully interact with their environment and achieve their goals, and it is often a prerequisite for learning and solving higher-level tasks, e.g. changing a flat tire. It is likely that the road to general AI and embodied agents will require developing models and systems that have common sense. This accomplishment is for a body of work that led to a significant scientific impact, including more than 20 publications in top tier conferences, invited talks and organized workshops and patents, as well as [budget redacted since confidential, but a multi-million] 4-year DARPA contract, several open source projects including a highly realistic simulation environment for training and testing AI agents, numerous released datasets and benchmarks, and media coverage. This impact is a result of a long-term, cross-organizational and multi-disciplinary teaming effort between researchers and engineers from IBM and our academic partners at MIT, Harvard and Stanford. [If you are within the IBM network, you can read more here.]

One of Five Faces of IBM Research 2021

I was featured as one of the five Faces of IBM Research in 2021 for leading IBM’s Engineering Excellence platform. If you are within the IBM network, you can read more here (“Spotlighting our Engagement Catalysts – 5 researchers driving positive change and culture”). [I also received two Manager’s Choice awards in the years before.]

InfoWorld’s Best of Open Source (BOSSIE) Award 2018

My team received InfoWorld’s Best of Open Source (BOSSIE) award in 2018 for our work on the Fabric for Deep Learning.

Social Media






You can connect with me on LinkedIn and Twitter.





Press Coverage

Here is a selection of press coverage about projects I worked on.

Fabric for Deep Learning

IBM Sapphire

Watson Orchestrate

There were multiple articles about the DARPA Machine Common Sense project that I am currently working on, but since they tend to be about the whole project and not my team in particular, I will skip them for now.

Testimonials

“Falk’s leadership role as an experienced engineer at the MIT-IBM Watson AI lab has been crucial in achieving the goals and deliverables on some of the most central projects in the lab.”

— Dan Gutfreund, Principal RSM and Manager at MIT-IBM Watson AI Lab

“Falk is a research engineer at the MIT-IBM Watson AI Lab. Since he joined IBM, he made important contributions to several projects touching on different aspects of AI such as NLP, vision and reasoning. His expertise in developing integrated AI architectures is invaluable. He is also known to work relentlessly, making sure no problem remains unsolved before he goes to sleep.“

— Jessie Rosenberg and Dan Gutfreund (managers at MIT-IBM Watson AI Lab) in my nomination for IBM’s Stage for the Research Engineer

“I don’t think I’ve had a collaborator who writes such nice code (and cleans up my mistakes nearly as well)!“

— Dan Bear, Postdoctoral Research Fellow at Stanford

“I’ve worked with Falk for several years, who was also one of the original instigators of the Verdi project (which WO wrapped). Falk is AWESOME!“

— Scott Boag, Senior Technical Staff Member, IBM Research Cambridge

“Falk, I want to thank you for your incredible contribution to the Engagement Catalysts efforts in 2021. Yours is the most significant contribution – you laid the ground for our Engineering Excellence platform (announced by Dario!). Your Engineering Excellence talk series is an admirable achievement, that professional event organizers would envy”

— Michal Jacovi, Global Engagement Leader for IBM Research

“I am convinced that with his abilities and dedication he will be a great addition to any team aiming at pushing the boundaries of machine learning, reasoning and natural language processing for smarter and more natural interaction between humans and computer systems.”

— Prof. Ulrik Schroeder, Professor at RWTH Aachen

“Falk is an exceptionally hard-working and dedicated member of the UMich team making this project happen. He is the lead developer for the project, and as such, manages the integration of research ideas into stable, well-engineered software. He has proven himself to be a critical piece of this multi-million-dollar collaboration between industry and academia.”

— Prof. Walter Lasecki, Professor and PI at University of Michigan

“Your insights and suggestions have helped immensely in shaping the final drafts of the book. Your expertise is truly impressive, and your contribution has been valuable as it has helped us to refine the content and even the author to present a more thorough analysis of the book.“

— Hemangi Lotlikar, Packt Publishing