Aug 18, 2020

Use Examples

They are densely impactful

Examples vastly improve communication.

Teaser: Examples take significantly more effort to make, but they are densely impactful to a work. Examples should illustrate interesting ideas, but counterintuitively, the impact of the examples is greater than the impact of the ideas themselves. Whether in writing, talking, or teaching, examples serve both the ideas (by making the ideas specific) and the communication (as the examples become idea vessels).

Let’s jump right to some examples.

Talks

One talk really sticks out to me in its great use of examples.

Making Fluid and Powerful Animations For Skullgirls
(Mariel Cartwright, 2014)

This presenter is feeling the pressure on stage (“I’m so nervous!”), she hedges on her qualifications (“I’m still kind of a newcomer”) , and it’s her first talk.

None of that matters. She absolutely kicks ass. This talk is fantastic.

Why? Cartwright has one or even three examples to illustrate every single one of her insights. The presentation of the examples is top-notch: she plays animations in the slides to accompany each idea. The structure of the examples is exemplary: she often shows both the before and the after, and offers practical reasons why it improves. And as a bonus, she selects the examples from both her own work as well as famous sources. This makes them more convincing.

Books

Many of us are familiar with academic writing that seems to be saying a lot, but might equally be saying many different things. This is because the writing is deprived of examples.

For example, check out this excerpt:

Once the original/copy distinction becomes the topic of “transjunctional” operations and can be accepted or rejected as a distinction, a new description is needed to resist the hegemony of the law of novelty. “Postmodernism” rebels against this law; but in so doing, it only returns to an older law, which states that the work of art, in one way or another, must mediate between variety and redundancy in order for the appeal of novelty to become intelligible.

— Art as a Social System by Niklas Luhmann (trans. Eva M. Knodt, Stanford University Press, 2000, orig. 1995)

First, I want to defend Luhmann (the author) a bit. The book isn’t as bad as I remembered. I had to read a bunch of Art as a Social System in college. I remember it being awful and abstract and saying a million things. Now that I opened it up again, I actually found most of the passages sufficiently illustrated with conceptual examples for me to follow.01

However. Most passages—like the above—rely on the reader to have a pretty monstrous set of examples in their head already in order to follow the argument. Then, as the argument develops over time, the reader has to continuously search for new examples on their own to make sense of it.

It can get much worse.

The infinite player in us does not consume time but generates it. Because infinite play is dramatic and has no scripted conclusion, its time is time lived and not time viewed. As an infinite player one is neither young nor old, for one does not live in the time of another. There is therefore no external measure of an infinite player’s temporarily. Time does not pass for an infinite player. Each moment of time is a beginning.

— Finite and Infinite Games by James P. Carse (Simon and Schuster, 1986)

This was just flipping to a random page. It’s easy to find other examples in academic writing. What on earth is going on?

Stopping to ground your arguments with actual examples limits how many ideas you can express because an example takes up more real estate. You can imagine this reading Luhmann—he’s going at the rate of about one big idea per sentence. So it may seem like an awful tradeoff: if you add examples, you may publish 10x fewer ideas per book.

But I think it’s worth it. Maybe you publish 10x or even 100x fewer ideas, but they are actually understood. And if you publish ideas that are easier to understand, your audience may grow dramatically. (How many people do you know that read Luhmann for fun in the evening?). Having more humans really understand what you’re saying seems like a good recipe for challenging and strengthening (or replacing) your ideas—which you would want if you’re trying to really figure out the nature of things, but something you might not like if you’re aiming to become a “dominant thinker” or capture “mindshare.”02

Teaching

Bad Slides

Let’s look at a slide (source withheld):

Hey, just apply proper engineering! Just do it! Oh and proper project management, do that too.03

Sometimes, when there’s just a comical lack of examples, I think: why even bother making the points at all?

Teaching Math

Math is especially bad without examples.

I think that most people teaching math (or something that involves math) don’t realize they aren’t giving examples. This is because in math, the core idea might be something pretty abstract, like a theorem:

Addition is commutative

A decent teacher will probably realize this needs an example. (A bad one will instead think it just needs a proof.04) So they’ll do something like:

1 + 2 = 2 + 1

In math, this is called an example. But it doesn’t go far enough. It’s still written in a symbolic language. It doesn’t have the groundedness that we require from examples everywhere else. Instead, we need what we’d call an “application:”

Take one orange, and add it to a pile of two oranges. Count them. You have three oranges. Now, split them up again. Put a single orange over there. Now, add two oranges to it. You get three oranges this way as well.

(Pardon the simple example here—you probably don’t need to talk about oranges to convince people of this. I just wanted to pick something accessible.)

If we arrange them into quality of instruction, from worst to best, we could maybe say:

  1. Idea (theorem)
  2. Proof
  3. Example (masquerading; just an instance of computation)
  4. Application (true example)

Up through finishing my undergrad, I never cared about applications / true examples in math. I was always good enough at the math part of it that things came pretty easily.

It wasn’t until I took a theory-heavy machine learning class (basically a statistics course) in the first year of my PhD that I experienced what many people probably experience much earlier in math. Here’s how the teaching went, along with my annotated monologue:

  1. Theorem (“why are we doing this? They just said ‘Let’s consider …’ and started writing equations and now we’re supposed to all be on board with this theorem?”)

  2. Proof (“OK, I’ll try to follow this, but … why?”)

  3. No example (“I guess I could look up some exercises in a book I can find on this topic. But they all seem so artificial. When am I just going to have distributions and random variables defined out of the blue like this?“)

  4. No application (“Are there even uses for this?”)

To show you how deep this goes, for those of you that have taken linear algebra, where did your teachers or professors stop in this chain?

  1. Definition of vector algebra
  2. Equation “examples” of vector algebra
  3. Drawn “examples” of vector algebra
  4. Situation where you can use vector algebra

It’s not only that examples clarify the intuition. It’s that the process of mapping a problem in the wild to the technique you’re learning is monstrously informative. Have you ever taken a class where the only problems you solved were those provided to you? (Maybe all of your classes were like this?) Conversely, have you ever found a problem in real life, made the connection to something you learned in a class but haven’t mastered, and in the act of struggling through and succeeding with the application, felt your brain twist and scream and then wonderfully burst through multiple levels of enlightenment and mastery?

To backpedal and clarify: I believe math as a research area does not need to be justified by applications. In fact, I think computer science research also doesn’t have to be justified by an application. There’s so much to explore that may be useful in 1 to 100 years. It’s too hard to predict.

But with that said, when you’re teaching math to people—especially people who aren’t going to become mathematicians—it seems like an absolutely wild disservice to not show true examples (i.e., applications) of how the math solves problems that come up in real life.05

Editing

I recently read a wonderful passage on how to write better. The reason it’s so great is that the author—Umberto Eco—actually gives you examples of badly written passages, and then rewrites them to be better.

You are not Proust. Do not write long sentences. If they come into your head, write them, but then break them down. Do not be afraid to repeat the subject twice, and stay away from too many pronouns and subordinate clauses. Do not write,

The pianist Wittgenstein, brother of the well-known philosopher who wrote the Tractatus Logico-Philosophicus that today many consider the masterpiece of contemporary philosophy, happened to have Ravel write for him a concerto for the left hand, since he had lost the right one in the war.

Write instead,

The pianist Paul Wittgenstein was the brother of the philosopher Ludwig Wittgenstein. Since Paul was maimed of his right hand, the composer Maurice Ravel wrote a concerto for him that required only the left hand.

– How to Write a Thesis by Umberto Eco (The MIT Press, 2015, orig. 1977). Here’s a longer passage from The MIT Press.

It is easy to find writing advice in the form of rules like “use short sentences.” But it is so much more valuable when writing advice actually shows you rewriting in practice.

Research Papers

I have focused so far on the use of examples that cause work to be excellent or damning. My examples with research papers are fuzzier.

As part of a research community—a relatively small number of peers who all know each other—we are somewhat honor-bound not to roast each other too badly. This is true for me as well, which means I cannot write this section as well as it deserves.

Pulling Punches

My first example is this paper: Troubling Trends in Machine Learning Scholarship (Lipton and Steinhardt, 2018). This is a survey paper06 that I was very excited to see come out, because I’d experienced all of the trends that they find troubling first-hand when learning how the research world works. It’s a great paper overall, but they purposely toned down examples they used. From the paper:

While we provide concrete examples, our guiding principles are to (i) implicate ourselves, and (ii) to preferentially select from the work of better-established researchers and institutions that we admire, to avoid singling out junior students for whom inclusion in this discussion might have consequences and who lack the opportunity to reply symmetrically.

Realistically, there’s no other way they—as successful members of the community—could have chosen examples without writing an absolute smack down. But as a result, most of the examples they use are watered down. For example:

We (JS) are guilty of [mathiness] in [70], where a discussion of “staged strong Doeblin chains” has limited relevance to the proposed learning algorithm, but might confer a sense of theoretical depth to readers.

To me, this is like pointing at a lightly smoking toaster when next door there’s an eight story building up in flames. So, while I get why they chose what they did, it’s a shame that they had to ignore hundreds of papers that vividly, clearly illustrate the problems they’re describing. Ironically, I think these omissions actually make it harder for someone new to the field to get an intuitive sense for the problems they’re describing.

Example Outputs

One problem research papers fall prey to is not giving examples to support their arguments or to illustrate a method. But I’ve already covered this kind of problem above with books.

Instead, I’m going to focus on example outputs. This likely isn’t applicable to all research. But for natural language processing (NLP), it is almost always applicable.

Take, for example, SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient (Yu et al., 2016). This was the first paper (that I know of) to apply generative adversarial nets (GANs) to NLP, which seemed like a big deal at the time.

Put plainly: the result of this paper is a computer program that generates text. Well, how good is the text that it generates? What does it actually look like? Who knows! The paper has zero example outputs.

To me, this is like writing an article about an extensive process for training monkeys to paint pictures, but never once showing the artwork they produce. It’s not just unsatisfying, it’s unconvincing.

It is tempting to get on a soapbox here and demand that we make space, explicitly, in all NLP papers for example outputs, and require all authors to produce them. But I will save this rant for another day. I will merely humbly claim that looking at the output of a system is hugely informative. For example, if your computer program just replies “I don’t know, I don’t know, I don’t know,” every time you ask it something, this is very easy to see and tells you kind of a lot about it.07

The field of computer vision seems to have figured this out a while ago. Their papers are filled with wonderful pictures. We may not have caught on as quickly because text isn’t as exciting to look at. I also suspect there’s something trivial but suffocating going on, like the facts that (a) all NLP papers must be (roughly) same number of pages, and (b) examples of text are annoying to format nicely in a research paper. But I truly hope we get better at this.08

Excuses

1. Negative Examples Roast Others

I mentioned this above in the “Research papers” section. This is a legit worry.

But let me risk an analogy. You must vote if you believe everyone should vote, even if you know your individual vote doesn’t matter. Same with recycling, or any greater-common-good activity.

I think there’s so much to be gained by employing examples that it’s worth roasting others a little bit to use them as bad examples sometimes. We can be gentle. And it’ll be better if everyone does it.

It’s a fine line to walk, especially because a good general tenant seems to be “don’t be negative.” I notice this in successful researchers and people doing a good job in leadership roles: they focus on positive aspects, and work on developing and strengthening ideas rather than blasting them apart. (I would guess there’s some deep psychological stuff going on with this.)

But when you’re actually doing the ground work, producing some kind of artifact, the fact of the matter is that it’s extremely helpful to study bad examples. Even better for everyone if, like Cartwright or Eco above, you show how it can be improved.

2. Examples Take Much More Effort

Yes, they do.

But here’s why you should expend this effort: using examples is work that scales. By illustrating your ideas with examples, you have done cognitive work for your audience. The larger your audience—which, for a book or talk online, could be in the thousands or millions—the more work you have saved everyone.

There’s another side to this, too. Sometimes, your audience cannot do the work of inventing examples behind your ideas. Then they will miss the ideas entirely. This is bad for you: people won’t recommend your book, remember your talk, or give you a nice teaching evaluation.

What if you believe that understanding your points should be difficult? You believe an audience should have to work at understanding your text? In other words…

3. Examples Prevent Art

If you’re a pro, and you’re capable of tastefully withholding understanding from your audience, I think this is legit.

Obfuscation, or even just doling out comprehension at a measured pace, can be a wonderful tactic. If you squint, you could say that these techniques can create big reveals, character development, and twists in stuff like movies and novels.

But apart from storytelling, I have never seen a lack of examples that caused confusion benefiting the work. In academic work, I believe that abstract example-less prose is acceptable may be one of the all time worst decisions of the academy.

Well, this has certainly got me all fired up. Let’s talk again about Luhmann, the German philosopher from the Books section above.

Wikipedia says this about why Luhmann’s ideas haven’t caught on much:

His relatively low profile [outside of Germany, Japan, and Russia] is partly due to the fact that translating his work is a difficult task, since his writing presents a challenge even to readers of German, including many sociologists.

His defense?

Luhmann … claimed he was deliberately keeping his prose enigmatic to prevent it from being understood “too quickly”, which would only produce simplistic misunderstandings.

I am going to take a stand. This is an utter fallacy. It is rubbish. It is a childish defense against being wrong by dancing away from having your ideas nailed down. “Oh that’s not what I meant. Oh, that’s not right either.”

Luhmann, and everyone else who is guilty of this, is writing about people and events and society and culture happening in our known universe, in our time, from our perception as human beings. When they make statements, they have examples in their mind. They could not abstract from nothing.

And it’s not just obfuscation. Writing down the examples behind your points would allow others to

I think that omitting examples purposely—for what I’m calling artistic reasons—is an excuse. The real reason is that they are more work. If you can get away without them, theories that you’ve built your career on can last longer because no one can understand them. Even those that do understand them can’t disprove them because you can always say they’ve gotten the mapping to reality wrong.

So, if someone like Luhmann made his prose concrete and understandable by providing concrete examples, how long would his books be?

4. Examples Take Up Real Estate

You can only write so many words. Won’t examples take up too much space?

This is a tempting concern. Say you have a very dense text that introduces a new idea every couple sentences. If each idea takes even one sentence to explain with an example, you’ve doubled the length of your work. And this is probably conservative. Some ideas might take a whole paragraph to illustrate.

I have three reasons why I don’t think we should worry about examples taking too many words (or too much time in a talk).

The first reason is that examples can be introduced in a variety of formats:

The second reason—hinted at by the last bullet—is that ideas aren’t introduced randomly in any sane text or talk. And since the ideas are connected, examples can flow with them. So it’s unlikely that you’ll need a fresh example with plentiful exposition for every new point. A reader or listener can draw connections for a while with some gentle help.

The third reason I think examples are worth real estate is that examples are densely impactful. This is in bold because it’s actually the whole thing I’m trying to communicate with this essay. An example pulls its weight much more than words communicating new ideas do. An ounce of examples may be worth two pounds of ideas. The best word I could come up with for “weight” here is “impact:” what your audience will remember the most, be moved by, have your points clarified with, or emotionally and deeply connect to.

I would venture a guess that people who omit examples have usually misestimated this last point. To them, ideas are the impactful thing, because they contain the most new information. But just stating raw ideas is gambling that they will invoke meaning in your audience. Examples are a superior communication device.09

How Many Examples?

At this point, you might notice that I am dancing around committing the folly I’m ranting against. None of the last several paragraphs have any examples at all.

I’m taking a risk that all of the examples above have sufficiently primed you for the abstract discussion about examples. I’m sort of cashing them all in.

This reveals that things are more complicated than I’ve been trying to make them seem. There is an art to knowing when your audience is on board, and when they’ll need assistance. If you have a small audience with a guaranteed shared understanding, you can omit a lot. This is probably where academic work falls: they write for their colleagues. But when you’re an expert in something, it’s easy to swing too far into the abstract, leaving even some of your field behind. And if that work ever escapes to the public? It’s hopeless.

In my opinion, we have the most to be gained just by remembering to use any examples at all. I think for a lot of folks, examples just aren’t on their radar.

Not everything we make has to be as good as Cartwright’s Skull Girls talk. But we ought to know what we can aim for.

Thanks to Alex Miller and Shelby Wilson for reading drafts of this. Alex suggested ISO Standards as nearly pathologically impenetrable writing—so bad you have to hire a consultant to help you make sense of it. I send him my condolences.

Footnotes

  1. A few things to say here. First, I called these “conceptual examples” because Luhmann is using commonly understood concepts in place of actual examples. Given how abstract the topics he’s trying to tackle are, this choice is understandable. But they’re still not as good as examples that point to specific things or events. Second, there’s something going on with quoting books that I don’t address in this essay: the concepts build upon previous concepts, sort of turning past digested concepts into examples. This is tenuous and fraught: fall off the conceptual bandwagon, and it’s very difficult to climb back on. But it does mean that opening to a random page and quoting a paragraph isn’t quite fair to the authors. It would kind of be like opening a novel to a random page and complaining that you don’t know the characters. Third, just on the book itself: right off the bat, Luhmann seems to be working through some pretty serious “what is the nature of reality and society” shit, which nearly sucked me in. I’m almost tempted to give this book another go. ↩︎

  2. Here I wrote “really figure out the nature of things” rather than our shorter word for this, which is “science.” That’s because staking out academic territory is serious business in our society’s implementation of science right now. (At least it is in computer science.) ↩︎

  3. This was a real slide from a real course I really took. I have a whole rant saved up about this class that I’ll save for another day. ↩︎

  4. I’m simplifying here. It depends on where the students are at. If you’re talking about addition, and writing a proof is even on the table, you’re probably in an advanced enough class that everyone intuitively gets it already, so a proof is fine. ↩︎

  5. You would think that physics class does this well, because all of the physics problems you solve are based on real situations. But I think to really get the benefit of examples, you have to give people physical problems to solve and have them fail first. You can’t just jump out of nowhere with a spring or electric current and expect it to turn on a light bulb. ↩︎

  6. By “survey” I mean that it presents a review of the state of the field, rather than a novel technical or theoretical contribution. ↩︎

  7. Epilogue: Language GANs sort of existed in this mysterious “does anybody really use them?” period for a couple years until Caccia et al. wrote the wonderful Language GANs Falling Short (2018) and put the whole thing to bed. (At least this is the timeline in my brain, I didn’t / don’t use Twitter so who knows if this is where everybody else fell on this.) ↩︎

  8. Problems do arise with examples in research papers. They’re either cherry-picked, or they say they’re not but you only half believe them. But anything at all tells you a lot. I think it’s reasonable to have a mix of the best ones along with some randos. ↩︎

  9. I’ve seen this abused. I watched an academic talk last year where the speaker spent over an hour giving examples of things we all already knew. I was bored out of my mind, but lots of the faculty seemed to love it. I don’t think this is the right balance to strike—ideally you want to propose some interesting ideas. But the fact that people loved it goes to show how depraved we might be for effective communication in academic talks: when someone stuffed their talk full of 100% examples and 0% new ideas, it was still satisfying. ↩︎

post info

Published Aug 18, 2020
Inbound
Outbound