I'm currently a final-year Ph.D. candidate in the Computer Science Department at UCLA, supervised by Kai-Wei Chang and Song-Chun Zhu. I am a member of UCLA Natural Language Processing Group (UCLA NLP) and the Center for Vision, Cognition, Learning, and Autonomy (VCLA). Previously, I received a Research Master's degree at Tsinghua University, advised by Jianyong Wang. My research has been funded by Amazon PhD Fellowship, Bloomberg PhD Fellowship, Qualcomm Innovation Fellowship, UCLA Dissertation Year Fellowship, DARPA, Naval Research Grant, and NeurIPS Scholar Award.

My research goal is to build machines that can reason and collaborate with humans for the common good. My primary research focuses on machine reasoning and trustworthy NLP, particularly in the domains of mathematics, science, and medicine:

In addition, I am interested in:


News


  • [01/2023]  New! One paper on in-context learning for math reasoning (PromptPG) is accepted to ICLR 2023.
  • [12/2022]  New! A survey paper on deep learning for mathematical reasoning is available at Preprint.
  • [12/2022]  New! One paper is accepted to AAAI’23 KnowledgeNLP Workshop as an Oral Presentation.
  • [12/2022]  New! I am excited to join Microsoft Research as a research intern!
  • [10/2022]  New! Happy to receive the NeurIPS 2022 Scholar Award.
  • [10/2022]  New! Two papers on mathematical reasoning are accepted to EMNLP 2022.
  • [09/2022]  New! One paper on prompt learning for math reasoning (PromptPG) is submitted to Preprint.
  • [09/2022]  New! One paper on chain-of-thought reasoning for ScienceQA is accepted to NeurIPS 2022.
  • [07/2022]  New! I am co-organizing the 2nd MATH-AI Workshop at NeurIPS 2022. See you in New Orleans!
  • [07/2022]  New! One paper on socially intelligent agents is accepted to SIGDIAL 2022.
  • [04/2022]  Excited to be listed as a Highlighted Reviewer for ICLR 2022.
  • [03/2022]  I am excited to join Allen Institute for AI (AI2) as a research intern!
  • [03/2022]  One paper on character animation sampling is submitted to Preprint.
  • [12/2021]  Two papers are accepted to AAAI 2022.
  • [10/2021]  One paper on visual question answering for icon images (IconQA) is accepted to NeurIPS 2021.
  • [07/2021]  I am co-organizing the MATHAI4ED Workshop at NeurIPS 2021. Welcome to participate!
  • [07/2021]  Our workshop proposal for Math AI for Education (MATHAI4ED) is accepted to NeurIPS 2021.
  • [05/2021]  One paper on interpretable geometry problem solving is accepted to ACL 2021 as an Oral Presentation.
  • [05/2021]  One paper on social relation inference in dialogues is accepted to ACL 2021 as an Oral Presentation.
  • [03/2021] One paper on socially intelligent agents is submitted to Preprint.

Selected Publications


MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, Jianfeng Gao
ICLR 2024  [Project]  [Paper]  [PDF]  [Code]  [Dataset]  [Leaderboard]  [Visualize]  [Twitter]  [Coverage]  [BibTex]
Oral Presentation (85 in 7304 submissions, 1.2%)
CryptoRank News Feature (29 October 2023)

MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?
Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, Hongsheng Li
arXiv:2403.14624  [Project]  [Paper]  [PDF]  [Code]  [Data]  [Visualization]  [Coverage]  [Daily Papers]  [BibTex]

Are LLMs Capable of Data-based Statistical and Causal Reasoning? Benchmarking Advanced Quantitative Reasoning with Data
Xiao Liu, Zirui Wu, Xueqing Wu, Pan Lu, Kai-Wei Chang, Yansong Feng
arXiv:2402.17644  [Project]  [Paper]  [PDF]  [Code]  [Data]  [Twitter]  [BibTex]

SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models
Peng Gao, Renrui Zhang, Chris Liu, Longtian Qiu, Siyuan Huang, Weifeng Lin, Shitian Zhao, Shijie Geng, Ziyi Lin, Peng Jin, Kaipeng Zhang, Wenqi Shao, Chao Xu, Conghui He, Junjun He, Hao Shao, Pan Lu, Hongsheng Li, Yu Qiao
arXiv:2402.05935  [Paper]  [PDF]  [Code]  [Doc]  [Hugging Face]  [Twitter]  [Coverage]  [BibTex]

Model Editing Can Hurt General Abilities of Large Language Models
Jia-Chen Gu, Hao-Xiang Xu, Jun-Yu Ma, Pan Lu, Zhen-Hua Ling, Kai-Wei Chang, Nanyun Peng
arXiv:2401.04700  [Paper]  [PDF]  [Code]  [Twitter]  [BibTex]

LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention
Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, Yu Qiao
ICLR 2024  [Paper]  [PDF]  [Code]  [Twitter]  [Coverage]  [BibTex]
LightningAI Blog Feature (14 April 2023)

Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models
Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Jianfeng Gao
NeurIPS 2023  [Project]  [Paper]  [PDF]  [Code]  [Twitter]  [Coverage]  [BibTex]
Best Weekly AI Paper (by AlphaSignal, 1st in 1682, 0.06%)
Awesome NeurIPS 2023 Papers (40 in 3584, 0.01%)
NeurIPS 2023 Top 10 Multimodal ML Papers

SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models
Xiaoxuan Wang*, Ziniu Hu*, Pan Lu*, Yanqiao Zhu*, Jieyu Zhang, Satyen Subramaniam, Arjun R. Loomba, Shichang Zhang, Yizhou Sun, Wei Wang
arXiv:2305.00970  [Paper]  [PDF]  [Code]  [Twitter]  [BibTex]
(*Equal Contribution)
Nature News Feature (15 November 2023)

KokoMind: Can LLMs Understand Social Interactions?
Weiyan Shi*, Liang Qiu*, Dehong Xu, Pengwei Sui, Pan Lu, Zhou Yu
[Project]  [Code]  [Twitter]  [Twitter]  [BibTex]
(*Equal Contribution)

TheoremQA: A Theorem-driven Question Answering Dataset
Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang, Tony Xia
EMNLP 2023  [Paper]  [PDF]  [Code]  [Twitter]  [BibTex]

ArK: Augmented Reality with Knowledge Emergent Infrastructure
Abhinav Gupta*, Qiuyuan Huang*, Jae Sung Park*, Pan Lu*, Paul N. Bennett, Ran Gong, Subhojit Som, Baolin Peng, Owais Khan Mohammed, Christopher Pal, Yejin Choi, Jianfeng Gao
arXiv:2305.00970  [Paper]  [PDF]  [BibTex]
(*Equal Contribution)

Multimodal Procedural Planning via Dual Text-Image Prompting
Yujie Lu, Pan Lu, Zhiyu Chen, Wanrong Zhu, Xin Eric Wang, William Yang Wang
arXiv:2305.01795  [Paper]  [PDF]  [Code]  [Twitter]  [Coverage]  [BibTex]

LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model
Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, Yu Qiao
arXiv:2304.15010  [Paper]  [PDF]  [Code]  [Gradio]  [Gradio-Multimodal]  [Twitter]  [YouTube]  [BibTex]

A Survey of Deep Learning for Mathematical Reasoning
Pan Lu, Liang Qiu, Wenhao Yu, Sean Welleck, Kai-Wei Chang
ACL 2023  [Paper]  [PDF]  [Code]  [Poster]  [Twitter]  [Coverage]  [BibTex]

Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning
Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, Ashwin Kalyan
ICLR 2023  [Paper]  [PDF]  [Project]  [Data]  [Code]  [Explore]  [Leaderboard]  [Twitter]  [BibTex]

Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, Ashwin Kalyan
NeurIPS 2022  [Paper]  [PDF]  [Project]  [Data]  [Huggingface]  [Code]  [Explore]  [Leaderboard]  [Twitter]  [BibTex]

LILA: A Unified Benchmark for Mathematical Reasoning
Swaroop Mishra*, Matthew Finlayson*, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, Ashwin K. Kalyan
EMNLP 2022  [Paper]  [PDF]  [Project]  [Data]  [Code]  [Huggingface]  [BibTex]
(*Equal Contribution)

UniGeo: Unifying Geometry Logical Reasoning via Reformulating Mathematical Expression
Jiaqi Chen, Tong Li, Jinghui Qin, Pan Lu, Liang Lin, Chongyu Chen and Xiaodan Liang
EMNLP 2022  [Paper]  [PDF]  [Code]  [BibTex]

Towards Socially Intelligent Agents with Mental State Transition and Human Utility
Liang Qiu*, Yizhou Zhao*, Yuan Liang, Pan Lu, Weiyan Shi, Zhou Yu, Song-Chun Zhu
SIGDIAL 2022  [Paper]  [PDF]  [BibTex]
(*Equal Contribution)

Triangular Character Animation Sampling with Motion, Emotion, and Relation
Yizhou Zhao, Liang Qiu, Wensi Ai, Pan Lu, Song-Chun Zhu
arXiv:2203.04930  [Paper]  [BibTex]



GenMotion: Data-driven Motion Generators for Real-time Animation Synthesis
Yizhou Zhao, Wensi Ai, Liang Qiu, Pan Lu, Feng Shi, Tian Han, Song-Chun Zhu
arXiv:2112.06060  [Paper]  [BibTex]

IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning
Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, Song-Chun Zhu
NeurIPS 2021  [Paper]  [PDF]  [Project]  [Code]  [BibTex]
Datasets and Benchmarks Track

Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning
Pan Lu*, Ran Gong*, Shibiao Jiang*, Liang Qiu, Siyuan Huang, Xiaodan Liang, Song-Chun Zhu
ACL 2021  [Paper]  [PDF]  [Project]  [Code]  [BibTex]
Oral Presentation (*Equal Contribution)

SocAoG: Incremental Graph Parsing for Social Relation Inference in Dialogues
Liang Qiu, Yuan Liang, Yizhou Zhao, Pan Lu, Baolin Peng, Zhou Yu, Ying Nian Wu, Song-Chun Zhu
ACL 2021  [Paper]  [PDF]  [BibTex]
Oral Presentation

Learning Long- and Short-Term User Literal-Preference with Multimodal Hierarchical Transformer Network for Personalized Image Caption
Wei Zhang, Yue Ying, Pan Lu, Hongyuan Zha
AAAI 2020  [Paper]  [PDF]  [BibTex]

Knowledge Aware Semantic Concept Expansion for Image-Text Matching
Botian Shi, Lei Ji, Pan Lu, Nan Duan
IJCAI 2019  [Paper]  [BibTex]
Oral Presentation

Dynamic Fusion with Intra- and Inter-modality Attention Flow for Visual Question Answering
Peng Gao, Zhengkai Jiang, Haoxuan You, Pan Lu, Steven CH Hoi, Xiaogang Wang, Hongsheng Li
CVPR 2019  [Paper]  [Code]  [BibTex]
Oral Presentation

Knowledge-Aware Deep Dual Networks for Text-Based Mortality Prediction
Ning Liu, Pan Lu, Wei Zhang, Jianyong Wang
ICDE 2019  [Paper]  [BibTex]


R-VQA: Learning Visual Relation Facts with Semantic Attention for Visual Question Answering
Pan Lu, Lei Ji, Wei Zhang, Nan Duan, Ming Zhou, Jianyong Wang
SIGKDD 2018  [Paper]  [Project]  [Video]  [BibTex]
Oral Presentation

Co-attending Free-form Regions and Detections with Multi-modal Multiplicative Feature Embedding for Visual Question Answering
Pan Lu, Hongsheng Li, Wei Zhang, Jianyong Wang, Xiaogang Wang
AAAI 2018  [Paper]  [Code]  [BibTex]
Oral Presentation


Education



Selected Experience



Teaching



Professional Service


Conferences

Workshops and Tutorials

Program Committee Member

Journal Reviewer

Organizations

  • Chair, IEEE Student Branch at Tsinghua University, Beijing, 2015.10 - 2016.10

  • Selected Awards



    Contact


    UCLA Computer Science Department
    404 Westwood Plaza
    Los Angeles, CA 90095
    lupantech [at] gmail [dot] com
    [Google Scholar]  |   [Semantic Scholar]  |  [GitHub]  |  [LinkedIn]

    © Pan Lu 2024