How bad is what you read online?
The neglect of silent evidence is endemic to the way we study comparative talent, particularly in activities that are plagued with winner-take-all attributes. We may enjoy what we see, but there is no point reading too much into success stories because we do not see the full picture. The Black Swan, Nicholas Nassim Taleb, p 103
We read, watch TV, listen to music. Is what we read and watch and listen to good?
Sure, you might think. After all, you wouldn’t, for the most part, read or watch or listen them if they weren’t (in your view) good. But that neglects what the irascible statistician/philosopher/trader Nicholas Nasim Taleb, in the above quote, calls silent evidence: that to evaluate the performance of something — in this case culture as a whole — you need to know not only what it produces, but also what it fails to produce. Not only what gets accepted, but what gets rejected.
To see this, an example helps. Take your favourite magazine — Jacobin, or National Review, as it may be. You skim the most recent issue and find three good articles, two excellent ones, and one mediocre one. Overall, you’re pretty happy. They’re doing a good job, you think. Are you right to think so?
Not really. In order to be able properly to assess their performance, you need to know not only what they publish, but also what they fail to publish. Imagine they received ten submissions. And consider these two possibilities. In the first, all ten submissions are excellent. In the second, three are good, three are excellent, and four are mediocre.
In the first scenario, the magazine’s performance is very poor. It could, with better picking, have produced six excellent articles. In the latter, its performance is very good — they only missed one trick, publishing a mediocre one when an excellent one was available.
The point generalizes, to literature, TV, music, and so on. In order to know how good things are, we need to know what is rejected.
But that info is not easy to come by. Sticking with magazines, scroll to your magazine of choice, and look at the submissions page. They’ll probably say something like ‘Because we receive so many submissions, it will take ages to assess your submission, and we don’t have time to respond to ones we don’t pick’. But they don’t give us numbers, and they don’t — naturally enough — let us see the articles they don’t pick. So, I think, we have no reason to trust them.
This is an important question, both for consumers and producers of content. As consumers, we should care a lot if we are presented with sub-par content. For many of us, culture is a very important feature of our lives, and we should hope it’s as good as possible. As producers (writers, musicians, etc.), we need to know if the reason we aren’t successful is because others are simply better than us, or because the mechanisms that pick who gets attention are faulty.
Clever physics people come up with clever theories and clever experiments to try and estimate what they can’t directly see, like the amount of dark matter in the universe. Although I’m not a clever physics person, and I lack clever theories, I am going to try, here, to find a way of estimating, if not the quantity, then something of the nature of rejected submissions.
I will focus on two online culture magazines, and, presenting some data about what they accept, and how popular it is, I’ll try to work back and make a guess at the sort of things they reject. Before doing so, I want to introduce some ideas, and motivate the project a bit.
How Is Talent Distributed?
How is talent distributed among a community, such as the community of writers or the community of musicians?
This is a truly fascinating question, I think, and no answer jumps out as being obviously correct. It could be, again relativising to a particular community (say, literary talent in the case of a bunch MFAs), that talent is evenly distributed. People are pretty much as talented as each other, and there aren’t major outliers.
This wouldn’t be too surprising because other qualities are distributed like that. If you take a bunch of people, weigh them, and plot their weight, you’ll get a graph like this, which represents what is called a normal distribution (ignore all the terminology and Greek letters):
I don’t want to get into any IQ-quarrels (I know that for many the nature/existence/distribution of IQ is a burningly pressing intellectual question, but I don’t get it), but some people think that intelligence, too, is normally distributed: on this story, most people’s intelligence tends to clump around a middle value. Most of us are averagely smart.
With this in mind, we could then imagine our y axis above was recording — somehow! (this is a big somehow, I admit) — talent, conceived of as an intrinsic feature of a person that is independent of their success. Most people would be near the middle, and there would be few extreme outliers — few amazing and few very poor creators. This is prima facie plausible, I think.
But there are other possibilities. Some attributes have what is called a Paretian distribution (note, I might be using this term slightly incorrectly, but I don’t think it really matters to any point I make). You might have heard of this as the 80–20 rule — that 20% of the people, for example, own 80% of the land in a given country. Paretian distributions look like the following, where many people have very little and a small number have a lot. Paretian distributions have this sort of look (again, ignore the numbers of colours):
Many things follow a Paretian distribution: performance in sports, entertainment, and politics, for example (for this claim, see this paper.)
Fame, almost certainly, is Paretian: some are very famous and most not famous at all. What about talent? It certainly could be Paretian, and that would provide a nice explanation of why fame is Paretian too — some people are just better than others, and it’s they who get the fame.
We can consider mixed hypotheses, too. It could be, for example, that an author’s getting accepted for publication is Paretian (some get accepted a lot, most get accepted little), but the quality of work is normal (most people are pretty much as talented as each other).
Below, I want to present some tentative evidence to suggest that this melancholy hypothesis is realized. And then I’ll use it to present a way of trying to think about the general accurateness of rejection, and thus of the overall quantity of our venues.
Acceptance and Appreciation in Online Culture Magazines
The Outline is an online magazine of culture and technology. Although it isn’t overtly political, it certainly leans left if anywhere, if only because its target audience — hip young people — lean left. It publishes frequently, several articles a week, and online only, and uses a wide range of freelance writers. It is VC-funded and helmed by a new media stalwart, and has been running since 2016. It has about 44,000 Twitter followers.
Quillette is also an online magazine, from a roughly classical liberal/right perspective, about culture, philosophy, and academia. It also publishes frequently and online only, and also has many freelance writers. It is crowd-funded, has been running since 2015, and has about 153,000 followers.
I decided to take a look at the profile of the articles they accept, concentrating on the question of whether the distribution of authorship was Paretian, that is if a small number wrote a lot of the articles, and if appreciation was also Paretian — if a small number of articles or authors were most popular. For example, if both were Paretian, it would suggest that, when it comes to online literary journalism, there’s a small number of uber-talented writers who get most of the attention. If our magazine of choice then publishes such people, we could have faith in it.
For each magazine, I took a sample of thirty writers (actually, I only took twenty six for The Outline, a mistake I only noticed once I’d more or less finished. But I looked at what would have been the other four and nothing stood out), none of whom, as far as I could tell, were employed (as editors, columnists, or interns) by the magazine. I found the writers by searching in the domain (site:https://theoutline.com/contributor, and site:https://quillette.com/author/ respectively), and took the first thirty (twenty six) I found.
Before going on, let me note something — I’m an amateur at this, and it’s best thought of as kind of rigorous anecdata rather than proper serious-person data science. Please feel free to point out mistakes in my methods or analysis.
My first question was what the distribution of articles per author was like. Normal, Paretian, something else? Here is what I found — each bar represents a person, and the number is the number of articles they published. For The Outline:
For Quillette:
Pretty solidly Paretian. This is already interesting. If this were our only information, we might be inclined to side with our Paretian-talent hypothesis. On this story, there are a small number of exceptionally talented writers, and the magazines successfully track them.
So far, so good. But even if you are tempted by this hypothesis, you should be worried that the people the magazine pick again and again are not, in fact, the most talented. The Paretian distribution of authorship is compatible with a theory according to which, say, the editors have a handful of friends whom they disproportionately publish (I am not accusing either magazine of this; as will become clear, I don’t think it’s right).
So then I wondered — is there any way to assess the quality of an author’s work independently of how frequently it appears?
Here was my thought: both these magazines, as one would expect, tweet out the articles they publish. The following seems reasonable, at least prima facie: the better articles will be engaged with more, and in particular will be retweeted and favourited more. We can use engagement as a proxy for the hard-to-measure notion of intrinsic quality (it’s flawed, I admit, because engagement has a snowballing effect — retweeting something shows it to more people, increasing the likelihood of it getting retweeted).
Accordingly, we have a hypothesis: if the prolific writers are indeed uber-talented, then their work should get more engagement. If this were so, that would provide some solid evidence for the Paretian-talent hypothesis.
So what I did, first, was pick a bunch of authors who were prolific, moderately prolific, and not prolific, to look at (I didn’t look at them all because my taste for recreational data collection only goes so far). Then I picked either all or a majority (for the prolific ones) of their articles and worked out an engagement score for them. The engagement score is the number of favourites on Twitter multiplied by double the number of retweets (I doubled retweets assuming that retweeting something implies one likes it more than favouriting it). The first thing to note is that the distribution of engagement is Paretian. There are a small number of outliers for both venue. First The Outline:
Then Quillette:
If the prolific authors were uber-talented, then their articles should be in the well-engaged right of these distributions. But they’re not. In the Quillette case, for example, only one article by a prolific author has an over 500 engagement score. And when it comes to the two most prolific authors for The Outline, only one has an over 50 score. In the below, each dot represents an author, the x axis records number of times published, and the y average engagement number (i.e. the sum total of engagement divided by the number of articles published). For The Outline:
And for Quillette:
In neither case is there a correlation between being published a lot and receiving a lot of engagement. This suggests that the handful-of-uber-talented-authors hypothesis is no good.
We accordingly have a puzzle. Some writers get accepted a lot, and some papers get a lot of engagement, but it’s not the writers who get accepted a lot who get a lot of engagement. What should we make of this? And what, if anything, can we learn about rejections, and the quality of culture as a whole, based on this fact?
Added 23 June 2020: From summer 2020 I’m going to move my occasional writing from medium to tinyletter. If you want to read more from me in your inbox, please consider signing up: https://tinyletter.com/mittmattmutt. I’ll post relatively infrequently, and hopefully interestingly, on the same sort of themes as the blog, so: popular philosophy/explainers, culture, literature, politics/economics, etc. I might also do things like brief reviews of books I read and so on.
There are a couple of possibilities:
- Most submissions are roughly the same quality, and engagement is unpredictable
- There are many foreseeably high-engagement submissions that are rejected owning to incompetent editing
If the former is true, we can have faith in our institutions. They’re doing the best they can — there’s no way to predict in advance which article will take off, so they just pick among the many solid submissions they have. This is supported by the fact that what one might have thought would have been a predictor, namely prolificness, isn’t one.
If the latter is true, we shouldn’t have faith in our institutions. You might think that the presence of moderate engagement prolific writers supports this —the fact that editors keep on publishing the same people even though they don’t perform better than others, you might think, indicates editors aren’t doing their job well. And that leaves it open that there are many high-quality submissions they are overlooking
But, in fact, I think there’s an alternative explanation for this that in fact supports the claim that most submissions are roughly the same quality. Even if most articles are roughly equally talented, it could still be that there are prolific writers, for this reason. The prolific writers write a lot of relatively good articles; some get accepted. The authors work with editors and get a name as being reliable and easy to work with. And this could make it easier for them to get accepted when in competition with equally talented new writers: because the editor knows they are easy to work with, they will be more likely to favour them, provided the difference between the prolific people’s work and the new writer isn’t too great. By contrast, if there were a great variance in talent, and so many articles better than the typical article by a prolific author, the editors would surely pick them, and there would be no prolific writers. The equal talent hypothesis, I claim, explains the notable finding about prolificness.
In all, although this is tentative, I think the hypothesis that talent is relatively equally distributed is well supported, and that in turn, in a sort of knockon effect, supports the idea that what will be highly engaged with is unpredictable. And so, to return to our question about rejection, we can suggest this: there are probably many articles that get rejected roughly as good as the ones that get accepted, but which of those will be a success is unclear, and so our cultural institutions are trustworthy — they are not doing anything wrong which they could be doing better.
This is all of course wildly tentative, and one shouldn’t approach it with too much credulousness. But at the very least I hope I have asked an interesting question and shown how data can bear on its answer in ways that might seem surprising.
