Research

This creates interactive web visualizations of factor graphs (e.g., output from py-factorgraph) using the d3-force component of D3.js. I use it in the Verb Physics demo.

This is the NLP backend for our ICRA 2015 paper, Robot Programming by Demonstration with Situated Spatial Language Understanding.

I think it’s a great experience to write a rule-based parser. You quickly learn how exhausting it is to support a wide variety of language, and how brittle the resulting system is. But it’s also fun to use, because you kind of make your own formal, constrained language (i.e., stuff that works for your parser) that you know how to use.

The cool part of this is that, unlike all of my “actual” NLP research projects after, this one actually worked in the real world. I could type in commands, or talk to it with voice recognition, then it would parse my commands, send the commands to the robot, and the robot would plan and execute them. You could walk up to the robot and write a simple program for it using your voice, and the robot would actually manipulate objects as a result.

This is the model code, data, and documentation for the CogSci 2019 paper, Do Neural Language Representations Learn Physical Commonsense?.

The readme has an index of the branches containing the directions I took this code base. One research project lives in the hands-free branch, which is the robotics portion of the code for the ICRA 2015 paper, Robot Programming by Demonstration with Situated Spatial Language Understanding. A large collection of branches (da, generate-objects, interface-video, mock-objects, mock-objects-http, mock-objects-pr2, success-testing, success-testing-pr2) were involved in the HCOMP 2014 paper, Robot Programming by Demonstration with Crowdsourced Action Fixes.

Looking back, it’s wild to remember how challenging it is to build high level robotics software. Of course there’s the robot itself, which can fail in myriad hilarious ways. But there’s also the massive, ever-shifting, ever-slightly-failing robotics software stack you must build on top of, including partially working components, quirks, edge cases, and general sluggishness.

Our broader NLP group had struggled with collectively sharing corpora: who has what, where do we store it, how do we make sure it’s unmodified, and how do we share any processed versions? I built this tool to help us solve these problems. It scans over a directory on a shared filesystem, performing checks on the structure and permissions of corpora, and uploads the results to a browsable index. I have it run daily as a cron job. (This is also why it looks like I commit every day on GitHub.)

This is a tiny Python library for building factor graphs and running the (loopy) belief propagation algorithm. At the time, I could not find an easy-to-use library for building factor graphs, only huge packages specializing in other graphical models. Those are probably still better choices, but I learned a lot implementing this one. Made for the Verb Physics paper.

A tiny Python library for loading a POMDP setup (environment) and the output (policy) of a POMDP solver run on it.

This is the model code, data schema, and documentation for the EMNLP 2020 paper, Social Chemistry 101: Learning to Reason about Social and Moral Norms.

If you try to run any of the standard machine text evaluation metrics, you discover a surprising world of pain. BLEU is a Perl script, METEOR is a Java program, ROUGE is a problematic Perl script, and they all have inconsistent interfaces and expectations. If you instead went with a Python reimplementation, you’d learn there were inconsistencies in preprocessing that change the resulting scores.

textmetrics was my stab at providing a consistent Python interface still that ran the original programs (e.g., BLUE in Perl, METEOR in Java) under the hood. I also included some additional metrics by computing various ngram statistics. When we were studying neural text (de)generation, we realized that the standard metrics only told us so much, and we drew from auxiliary sources to get a more holistic picture of what models were doing.

Today, I no longer think it’s important to use the exact reference implementations (unless you’re doing a shared task with standard script, of course). There is enough statistical noise from other sources that I don’t think it’s vital to exactly replicate the reference implementations’ decisions. Instead, I would simply use the most widespread reimplementation in Python, perhaps the ones in huggingface/datasets.

This is the model code, data, and documentation for the ACL 2017 paper, Verb Physics: Relative Physical Knowledge of Actions and Objects.