This study examines question-answering (QA) abilities across human and AI agents. Our framework CAIMIRA addresses limitations in traditional item response theory, by incorporating multidimensional analysis, identifiability, and content awareness,
enabling nuanced comparison of QA agents.
Analyzing responses from ~ 30 AI systems and 155 humans over
thousands of questions, we identify distinct knowledge domains and
reasoning skills where these agents demonstrate differential proficiencies.
Humans outperform AI systems in scientific reasoning and understanding nuanced
language, while large-scale LLMs like GPT-4 and LLAMA-2 excel in
retrieving specific factual information.
The study identifies key areas for future QA tasks and model development,
emphasizing the importance of semantic understanding and scientific reasoning in
creating more effective and discriminating benchmarks.
@article{gor2024caimira,title={Do great minds think alike? Investigating Human-AI Complementarity for Question Answering},author={Gor, Maharshi and Zhou, Tianyi and Daum\'e, Hal, III and Boyd-Graber, Jordan},year={2024},}
Deep image prior (DIP) is a recently proposed technique for solving imaging inverse problems by fitting the reconstructed images to the output of an untrained convolutional neural network.
Unlike pretrained feedforward neural networks, the same DIP can generalize to arbitrary inverse problems, from denoising to phase retrieval, while offering competitive performance at each task.
The central disadvantage of DIP is that, while feedforward neural networks can reconstruct an image in a single pass, DIP must gradually update its weights over hundreds to thousands of iterations, at a significant computational cost.
In this work we use meta-learning to massively accelerate DIP-based reconstructions.
By learning a proper initialization for the DIP weights, we demonstrate a 10x improvement in runtimes across a range of inverse imaging tasks.
Moreover, we demonstrate that a network trained to quickly reconstruct faces also generalizes to reconstructing natural image patches.
@article{zhang2022metadip,title={MetaDIP: Accelerating Deep Image Prior with Meta Learning},author={Zhang, Kevin and Xie, Mingyang and Gor, Maharshi and Chen, Yi-Ting and Zhou, Yvonne and Metzler, Christopher A.},journal={arXiv preprint arXiv: 2209.08452},year={2022},}
The overwhelming vulnerability of deep neural networks to carefully crafted perturbations known as adversarial attacks has led to the development of various training techniques to produce robust models.
While the primary focus of existing approaches has been directed toward addressing the worst-case performance achieved under a single-threat model, it is imperative that safety-critical systems are robust with respect to multiple threat models simultaneously.
Existing approaches that address worst-case performance under the union of such threat models (e.g., \(\ell_∞), \ell_2, \ell_1) either utilize adversarial training methods that require multi-step attacks which are computationally expensive in practice, or rely upon fine-tuning of pre-trained models that are robust with respect to a single-threat model.
In this work, we show that by carefully choosing the objective function used for robust training, it is possible to achieve similar, or even improved worst-case performance over a union of threat models while utilizing only single-step attacks during the training, thereby achieving a significant reduction in computational resources necessary for training.
Furthermore, prior work showed that adversarial training against the \ell_1 threat model is relatively difficult, to the extent that even multi-step adversarially trained models were shown to be prone to gradient-masking and catastrophic over-fitting.
However, our proposed method—when applied on the \ell_1 threat model specifically—enables us to obtain the first \ell_1$ robust model trained solely with single-step adversarial attacks.
Finally, to demonstrate the merits of our approach, we utilize a modern set of attack evaluations to better estimate the worst-case performance under the considered union of threat models.
@inproceedings{sriramanan:gor:feizi-neurips2022,title={Toward Efficient Robust Training against Union of L<sub>p</sub> Threat Models},author={Sriramanan, Gaurang and Gor, Maharshi and Feizi, Soheil},booktitle={Advances in Neural Information Processing Systems},year={2022},location={New Orleans, LA},}
This work presents a sparse-attention Transformer architecture for modeling documents that contain large tables.
Tables are ubiquitous on the web, and are rich in information.
However, more than 20% of relational tables on the web have 20 or more rows (Cafarella et al., 2008), and these large tables present a challenge for current Transformer models, which are typically limited to 512 tokens.
Here we propose MATE, a novel Transformer architecture designed to model the structure of web tables. MATE uses sparse attention in a way that allows heads to efficiently attend to either rows or columns in a table.
This architecture scales linearly with respect to speed and memory, and can handle documents containing more than 8000 tokens with current accelerators.
MATE also has a more appropriate inductive bias for tabular data, and sets a new state-of-the-art for three table reasoning datasets.
For HybridQA (Chen et al., 2020b), a dataset that involves large documents containing tables, we improve the best prior result by 19 points.
@inproceedings{eisenschlos2021mate,title={MATE: Multi-view Attention for Table Transformer Efficiency},author={Eisenschlos, Julian Martin and Gor, Maharshi and M{\"u}ller, Thomas and Cohen, William Weston},booktitle={Empirical Methods in Natural Language Processing},publisher={Association for Computational Linguistics,},year={2021},location={Punta Cana},}
The goal of question answering (QA) is to answer any question. However, major QA datasets have skewed distributions over gender, profession, and nationality. Despite that skew, model accuracy analysis reveals little evidence that accuracy is lower for people based on gender or nationality; instead, there is more variation on professions (question topic). But QA’s lack of representation could itself hide evidence of bias, necessitating QA datasets that better represent global diversity.
@inproceedings{Gor:Webster:Boyd-Graber-2021,title={Toward Deconfounding the Influence of Entity Demographics for Question Answering Accuracy},author={Gor, Maharshi and Webster, Kellie and Boyd-Graber, Jordan},booktitle={Empirical Methods in Natural Language Processing},publisher={Association for Computational Linguistics,},month=nov,year={2021},location={Punta Cana},}
Despite the remarkable success of generative adversarial networks, their performance seems less impressive for diverse training sets, requiring learning of discontinuous mapping functions. Though multi-mode prior or multi-generator models have been proposed to alleviate this problem, such approaches may fail depending on the empirically chosen initial mode components. In contrast to such bottom-up approaches, we present GAN-Tree, which follows a hierarchical divisive strategy to address such discontinuous multi-modal data. Devoid of any assumption on the number of modes, GAN-Tree utilizes a novel mode-splitting algorithm to effectively split the parent mode to semantically cohesive children modes, facilitating unsupervised clustering. Further, it also enables incremental addition of new data modes to an already trained GAN-Tree, by updating only a single branch of the tree structure. As compared to prior approaches, the proposed framework offers a higher degree of flexibility in choosing a large variety of mutually exclusive and exhaustive tree nodes called GAN-Set. Extensive experiments on synthetic and natural image datasets including ImageNet demonstrate the superiority of GAN-Tree against the prior state-of-the-art.
@inproceedings{Kundu:Gor:Agrawal:Babu-ICCV2019,author={Kundu, Jogendra Nath and Gor, Maharshi and Agrawal, Dakshit and Babu, R. Venkatesh},title={GAN-Tree: An Incrementally Learned Hierarchical Generative Framework for Multi-Modal Data Distributions},booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},month=oct,year={2019},location={Seoul, South Korea},}
Human motion prediction model has applications in various fields of computer vision. Without taking into account the inherent stochasticity in the prediction of future pose dynamics, such methods often converges to a deterministic undesired mean of multiple probable outcomes. Devoid of this, we propose a novel probabilistic generative approach called Bidirectional Human motion prediction GAN, or <em>BiHMP-GAN</em>. To be able to generate multiple probable human-pose sequences, conditioned on a given starting sequence, we introduce a random extrinsic factor <em>r</em>, drawn from a predefined prior distribution. Furthermore, to enforce a direct content loss on the predicted motion sequence and also to avoid mode-collapse, a novel bidirectional framework is incorporated by modifying the usual discriminator architecture. The discriminator is trained also to regress this extrinsic factor <em>r</em>, which is used alongside with the intrinsic factor (encoded starting pose sequence) to generate a particular pose sequence. To further regularize the training, we introduce a novel recursive prediction strategy. In spite of being in a probabilistic framework, the enhanced discriminator architecture allows predictions of an intermediate part of pose sequence to be used as a conditioning for prediction of the latter part of the sequence. The bidirectional setup also provides a new direction to evaluate the prediction quality against a given test sequence. For a fair assessment of <em>BiHMP-GAN</em>, we report performance of the generated motion sequence using (i) a critic model trained to discriminate between real and fake motion sequence, and (ii) an action classifier trained on real human motion dynamics. Outcomes of both qualitative and quantitative evaluations, on the probabilistic generations of the model, demonstrate the superiority of <em>BiHMP-GAN</em> over previously available methods.
@inproceedings{Kundu_Gor_Babu_2019,title={BiHMP-GAN: Bidirectional 3D Human Motion Prediction GAN},author={Kundu, Jogendra Nath and Gor, Maharshi and Babu, R. Venkatesh},booktitle={Proceedings of the AAAI conference on Artificial Intelligence},year={2019},month=jul,volume={33},number={01},pages={8553--8560},url={https://ojs.aaai.org/index.php/AAAI/article/view/4874},doi={10.1609/aaai.v33i01.33018553},}
This work introduces a novel unsupervised framework for human action modeling, which separates the learning of individual pose representations from sequences of poses.
It employs a novel Encoder GAN (EnGAN) for continuous pose embedding and an RNN auto-encoder architecture, PoseRNN, for action modeling.
It demonstrates superior transferability in action recognition tasks and provides qualitative insights through skeleton pose reconstructions and visualizations in the pose-embedding space.
@inproceedings{Kundu:Gor:Uppala:Babu-WACV2019,author={Kundu, Jogendra Nath and Gor, Maharshi and Uppala, Phani Krishna and Babu, R. Venkatesh},booktitle={IEEE Winter Conference on Applications of Computer Vision (WACV)},title={Unsupervised Feature Learning of Human Actions As Trajectories in Pose Embedding Manifold},year={2019},volume={},number={},pages={1459-1467},doi={10.1109/WACV.2019.00160},tldr={This work introduces a novel unsupervised framework for human action modeling, which separates the learning of individual pose representations from sequences of poses.
It employs a novel Encoder GAN (EnGAN) for continuous pose embedding and an RNN auto-encoder architecture, PoseRNN, for action modeling.
It demonstrates superior transferability in action recognition tasks and provides qualitative insights through skeleton pose reconstructions and visualizations in the pose-embedding space.}}