← Return to search results
Back to Prindle Institute

lottery

Posted on

Rolling the Dice: The Ethics of Randomized Research Funding

photograph of bingo balls in a lottery machine

There is only so much money to go around.

We hear this reasoning all the time in our personal and political lives. Want to buy a new car and a next-gen console? Tough. You can only afford one (or quite possibly, none). So, decisions must be made about where you spend your money. Do you forgo the essentials to get the luxuries? Probably not. It’s usually best to buy what you need before what you want, and for many, paying for the former leaves little for the latter.

The same is true for governments. They can’t simply print money without consequences, as they don’t have an unlimited balance from which to draw. Indeed, discussions about fiscal responsibility – being economically sensible by balancing the books – permeate political debates worldwide. As now infamously encapsulated by former U.K. Prime Minister Theresa May’s “magic money tree” speech, when it comes to deciding where money is spent, just like us individuals managing our household budgets, those in charge have to make decisions that mean some things we’d want to dedicate money to get shafted.

But it is not only in the personal and political spheres that this is a reality. It also occurs in philosophy and, more broadly, in academia. It costs money to employ people to ponder life’s great mysteries. It also costs money to train them so they have the required skills. It costs money to build and maintain the administrative systems required to employ them and those they work with, and it costs money to send them to conferences to share their research with others. And while philosophers don’t typically need much resources (there’s no large hadron collider for ethics or metaphysics), we need our basic needs met; we need to get paid to afford to live. So, those holding the purse strings make similar decisions about what projects to fund and who to employ as you and I do about what to spend our money on. They have a limited fund to divvy up. For every researcher – senior, established, early career, or even pre-PhD – who secures funding to run their project or fund their post, countless more aren’t so lucky.

This places funding bodies in a somewhat unenviable situation as they must decide, from amongst the numerous applications they receive, which ones they should award funding to and which they have to reject; and there are always more rejections than acceptances. For instance, the British Academy – one of the U.K.’s largest research funders – runs an early career fellowship scheme with a typical success rate of less than 10%. Similar stats can be attributed to comparable schemes run by other U.K. funding bodies like the Wellcome Trust and the Leverhulme Trust. I suspect the same is true for funders in other jurisdictions.

So, how do funders decide which projects to support? Typically (hopefully), these decisions are made based on merit. Applicants identify a scheme they want to apply for and submit a research proposal, a CV, and referee statements (and maybe some other documentation). The funding body then considers these applications, ranks them according to a set list of criteria, and rewards the lucky few with funding. Those falling short receive a nicely worded email and a “better luck next time” metaphorical pat on the head.

At least, this is how it is supposed to work. Recently, however, funding bodies have been increasingly vocal about how hard it is to distinguish worthy from unworthy proposals. Or, to be more accurate, they’re receiving so many proposals of top quality that they can’t rank them. When it comes to selecting worthy projects, according to those funders, even after a rigorous selection process, they still have more in the “yes” pile than the available funding permits, and they simply can’t choose which projects deserve to be greenlit.

The question, then, which touches upon themes of fairness and responsibility, is what to do about this. How should funding bodies respond when faced with more worthy projects than they can fund and seemingly no way to choose between them? Some have decided that the best way forward is to leave it up to chance.

This method, typically called randomization, is seen as a way for funders to offload the work of selecting between seemingly equally deserving projects onto Lady Luck. In essence, projects are put into a hat, and those pulled out receive funding. This sidesteps the messy work of picking favorites and the need for splitting hairs. Of course, an entirely random selection process would be unfair as it would entail all projects, regardless of merit, being given an equal chance at receiving funding. So, when employed, the randomization is only partial. Prospective projects still go through the same evaluation process as before, thus maintaining the quality of work; it is only at the final step when only worthy projects remain, and if there is a need for it, that randomization is employed.

The aforementioned British Academy was the first major funder to trial partial randomization, trying it out in 2022 for a three-year period as part of their Small Research Grants scheme. Since then, other funders have followed its lead, including the Natural Environment Research Council, the Novo Nordisk Foundation, the Wellcome Trust, and the Carnegie Trust. It is not unreasonable to expect that other funders, upon seeing the increasing use of partial randomization, might also follow suit.

However, the justification for its use goes beyond simply making the funder’s life easier. According to those same funders, it also promotes diversity and fairness. The envisioned mechanisms powering these proposed benefits are relatively intuitive. If all the proposals selected for random selection meet the necessary standards, other factors that might inadvertently influence funding decisions – such as perceived socio-economic or cultural backgrounds – would not be a factor. In other words, partial randomization removes a layer of human bias from the selection process. Indeed, there’s evidence to support such an idea, as the British Academy has already announced that since their trial started, there has been a notable increase in successful projects originating from scholars from previously underrepresented backgrounds. As noted by Professor Simon Swain, the British Academy’s Vice-President for Research and Higher Education Policy:

The increase in successful applications from historically underrepresented ethnic backgrounds and those based in Scotland and Northern Ireland, along with broader institutional representation, suggests that awarding grants in this way [via partial randomization] could lead to more diverse cohorts of Small Research Grant-holders.

So, not only does partial randomization relieve decision pressures on the funders, but it also benefits those who have historically faced exclusion from such opportunities, which, in turn, enhances the quality of academic research overall. This is undoubtedly a good thing.

Provided that partial randomization is genuinely random, I believe it can also provide solace to researchers whose projects do not get selected. This is because it makes the luck aspect of grant chasing explicit. Like much in life, luck plays a massive role in whether a project gets funding. Even if your work is as good as possible, success depends on multiple factors outside an applicant’s control: is the reviewer familiar with the project’s field? Has another applicant got better-written references? Is the reviewer hungry? Or ill? Or tired? All these things, which shouldn’t influence funding decisions, inevitably do. By building into the system a degree of randomization – a quantifiable stage in which luck is explicit – prospective applicants can (or I think should) be able to take solace from the fact that their project may not get selected not because of something they did or didn’t do, but because it just wasn’t their day.

However, while partial randomization might have some genuinely desirable benefits, it leaves me slightly uneasy because it has an air of abandonment (maybe even a dereliction) of duty on the funder’s behalf.

It is the funder’s job, or at least the job of those on the relevant selection committees, to rank the projects according to the relevant criteria and decide which should be awarded funding. By outsourcing the final part of this process to a randomized system – be that as complex as a dynamic, multifactored algorithm or as simple as a hat full of names – the funders avoid discharging this duty. They avoid deciding which projects should get funding and avoid responsibility for the outcome. They can wipe their hands of the final selection stage and so wipe their hands of the joy and, crucially, disappointment they bring to applicants. While I think prospective applicants can take solace from knowing their project might fail based on nothing but luck, this robs those applicants of a figure at which to be mad; you can be angry at an envisioned funder or selection committee if you know that, somewhere, a person said your project shouldn’t receive funding. But, when a project is rejected based on luck, you have no one at which to direct any anger or sadness. An algorithm isn’t as good a target for frustration as a person or group.

Ultimately, while the anticipated benefits of partial randomization (increased diversity and fairness) are desirable, the selection method’s usage has an air of avoidance and ease. It’s the funder’s job to pick the appropriate projects. If they can’t do this fairly, do we want them to take the easy way out, or would we prefer they worked harder to make a justifiable decision?

Posted on

Ethics and Job Apps: Why Use Lotteries?

photograph of lottery balls coming out of machine

This semester I’ve been 1) applying for jobs, and 2) running a job search to select a team of undergraduate researchers. This has resulted in a curious experience. As an employer, I’ve been tempted to use various techniques in running my job search that, as an applicant, I’ve found myself lamenting. Similarly, as an applicant, I’ve made changes to my application materials designed to frustrate those very purposes I have as an employer.

The source of the experience is that the incentives of search committees and the incentives job applicants don’t align. As an employer, my goal is to select the best candidate for the job. While as an applicant, my goal is that I get a job, whether I’m the best candidate or not.

As an employer, I want to minimize the amount of work it takes for me to find a dedicated employee. Thus, as an employer, I’m inclined to add ‘hoops’ to the application process, by requiring applicants to jump through those hoops, I make sure I only look through applications of those who are really interested in the job. But as an applicant, my goal is to minimize the amount of time I spend on each application. Thus, I am frustrated with job applications that require me to develop customized materials.

In this post, I want to do three things. First, I want to describe one central problem I see with application systems — what I will refer to as the ‘treadmill problem.’ Second, I want to propose a solution to this problem — namely the use of lotteries to select candidates. Third, I want to address an objection employers might have to lotteries — namely that it lowers the average quality of an employer’s hires.

Part I—The Treadmill Problem

As a job applicant, I care about the quality of my application materials. But I don’t care about the quality intrinsically. Rather, I care about the quality in relation to the quality of other applications. Application quality is a good, but it is a positional good. What matters is how strong my applications are in comparison to everyone else.

Take as an analogy the value of height while watching a sports game. If I want to see what is going on, it’s not important just to be tall, rather it’s important to be taller than others. If everyone is sitting down, I can see better if I stand up. But if everyone stands up, I can’t see any better than when I started. Now I’ll need to stand on my tiptoes. And if everyone else does the same, then I’m again right back where I started.

Except, I’m not quite back where I started. Originally everyone was sitting comfortably. Now everyone is craning uncomfortably on their tiptoes, but no one can see any better than when we began.

Job applications work in a similar way. Employers, ideally, hire whosoever application is best. Suppose every applicant just spends a single hour pulling together application materials. The result is that no application is very good, but some are better than others. In general, the better candidates will have somewhat better applications, but the correlation will be imperfect (since the skills of being good at philosophy only imperfectly correlate with the skills of being good at writing application materials).

Now, as an applicant, I realize that I could put in a few hours polishing my application materials — nudging out ahead of other candidates. Thus, I have a reason to spend time polishing.

But everyone else realizes the same thing. So, everyone spends a few hours polishing their materials. And so now the result is that every application is a bit better, but still with some clearly better than others. Once again, in general, the better candidates will have somewhat better applications, but the correlation will remain imperfect.

Of course, everyone spending a few extra hours on applications is not so bad. Except that the same incentive structure iterates. Everyone has reason to spend ten hours polishing, now fifteen hours polishing. Everyone has reason to ask friends to look over their materials, now everyone has reason to hire a job application consultant. Every applicant is stuck in an arms race with every other, but this arms race does not create any new jobs. So, in the end, no one is better off than if everyone could have just agreed to an armistice at the beginning.

Job applicants are left on a treadmill, everyone must keep running faster and faster just to stay in place. If you ever stop running, you will slide off the back of the machine. So, you must keep running faster and faster, but like the Red Queen in Lewis Carrol’s Through the Looking Glass, you never actually get anywhere.

Of course, not all arms races are bad. A similar arms race exists for academic journal publications. Some top journals have a limited number of article slots. If one article gets published, another article does not. Thus, every author is in an arms race with every other. Each person is trying to make sure their work is better than everyone else’s.

But in the case of research, there is a positive benefit to the arms race. The quality of philosophical research goes up. That is because while the quality of my research is a positional good as far as my ability to get published, it is a non-positional good in its contribution to philosophy. If every philosophy article is better, then the philosophical community is, as a whole, better off. But the same is not true of job application materials. No large positive externality is created by everyone competing to polish their cover letters.

There may be some positive externalities to the arms race. Graduate students might do better research in order to get better publications. Graduate students might volunteer more of their time in professional service in order to bolster their CV.

But even if parts of the arms race have positive externalities, many other parts do not. And there is a high opportunity cost to the time wasted in the arms race. This is a cost paid by applicants, who have less time with friends and family. And a cost paid by the profession, as people spend less time teaching, writing, and helping the community in ways that don’t contribute to one’s CV.

This problem is not unique to philosophy. Similar problems have been identified in other sorts of applications. One example is grant writing in the sciences. Right now, top scientists must spend a huge amount of their time optimizing grant proposals. One study found that researchers collectively spent a total of 550 working years on grant proposals for Australia’s National Health and Medical Research Council’s 2012 funding round.

This might have a small benefit in leading research to come up with better projects. But most of the time spent in the arms race is expended just so everyone can stay in place. Indeed, there are some reasons to think the arms race actually leads people to develop worse projects, because scientists optimize for grant approval and not scientific output.

Another example is college admissions. Right now, high school students spend huge amounts of time and money preparing for standardized tests like the SAT. But everyone ends up putting in the time just to stay in place. (Except, of course, for those who lack the resources required to put in the time; they just get left behind entirely.)

Part II—The Lottery Solution

Because I was on this treadmill as a job applicant, I didn’t want to force other people onto a treadmill of their own. So, when running my own job search, I decided to modify a solution to the treadmill problem that has been suggested for both grant funding and college admissions. I ran a lottery. I had each applying student complete a short assignment, and then ‘graded’ the assignments on a pass/fail system. I then choose my assistants at random from all those who had demonstrated they would be a good fit. I judged who was a good fit. I didn’t try to judge, of those who were good fits, who fit best.

This allowed students to step off the treadmill. Students didn’t need to write the ‘best’ application. They just needed an application that showed they would be a good fit for the project.

It seems to me that it would be best if philosophy departments similarly made hiring decisions based on a lottery. Hiring committees would go through and assess which candidates they think are a good fit. Then, they would use a lottery system to decide who is selected for the job.

The details would need to be worked out carefully and identifying the best system would probably require a fair amount of experimentation. For example, it is not clear to me the best way to incorporate interviews into the lottery process.

One possibility would be to interview everyone you think is likely a good fit. This, I expect, would prove logistically overwhelming. A second possibility, and I think the one I favor, would be to use a lottery to select the shortlist of candidates, rather than to select the final candidate. The search committee would go through the application and identify everyone who looks like a good fit. They would then use a lottery to narrow down to a shortlist of three to five candidates who come out for an interview. While the shortlisted candidates would be placed on the treadmill, a far smaller number of people are subject to the wasted effort. A third possibility would use the lottery to select a single final candidate, and then use an in-person interview merely to confirm the selected candidate really is a good fit. There is a lot of evidence that hiring committees systematically overweight the evidential weight of interviews, and that this creates tons of statistical noise in hiring decisions (see chapters 11 and 24 in Daniel Kahneman’s book Noise).

Assuming the obstacles could be overcome, however, lotteries would have an important benefit in going some way towards breaking the treadmill.

There are a range of other benefits as well.

  • Lotteries would decrease the influence of bias on hiring decisions. Implicit bias tends to make a difference in close decisions. Thus, bias is more likely to flip a first and second choice, than it is to flip someone from making it onto the shortlist in the first place.
  • Lotteries would decrease the influence of networking, and so go some way towards democratizing hiring. At most, an in-network connection will get someone into the lottery but it won’t increase you chance of winning the lottery.
  • It would create a more transparent way to integrate hiring preferences. A department might prefer to hire someone who can teach bioethics, or might prefer to hire a female philosopher, but not want to restrict the search to people who meet such criteria. One way to integrate such preferences more rigorously would be to explicitly weight candidates in the lottery by such criteria.
  • Lotteries could decrease interdepartmental hiring drama. It is often difficult to get everyone to agree on a best candidate. It is generally not too difficult to get everyone to agree on a set of candidates all who are considered a good fit.

Part III—The Accuracy Drawback

While there are advantages accrue to applicants and the philosophical community, employers might not like a lottery system. The problem for employers is that a lottery will decrease the average quality of hires.

A lottery system means you should expect to hire the average candidate who meets the ‘good fit’ criteria. Thus, as long as trying to pick the best candidate results in a candidate at least above average, then the average quality of the hire goes down with a lottery.

However, while there is something to this point, the point is weaker than most people think. That is because humans tend to systematically overestimate the reliability of judgment. When you look at the empirical literature a pattern emerges. Human judgment has a fair degree of reliability, but most of that reliability comes from identifying the ‘bad fits.’

Consider science grants. Multiple studies have compared the scores that grant proposals receive to the eventual impact of research (as measured by future citations). What is found is that scores do correlate with research impact, but almost all of that effect is explained by the worst performing grants getting low scores. If you restrict your assessment to the good proposals, researchers are terrible at judging which of the good proposals are actually best. Similarly, while there is general agreement about which proposals are good and which bad, evaluators rarely agree about which proposals are best.

A similar sort of pattern emerges for college admission counselors. Admissions officers can predict who is likely to do poorly in school, but can’t reliably predict which of the good students will do best.

Humans are fairly good at judging which candidates would make a good fit. We are bad at judging which good fit candidates would actually be best. Thus, most of the benefit of human judgment comes at the level of identifying the set of candidates who would make a good fit, not at the level of deciding between those candidates. This, in turn, suggests that the cost to employers of instituting a lottery system is much smaller than we generally appreciate.

Of course, I doubt I’ll convince people to immediately use lotteries on major important decisions. Thus, for now I’ll suggest that for smaller less consequential decisions, try a lottery system. If you are a graduate department, select half your graduating class the traditional way, and half by a lottery of those who seem like a good fit. Don’t tell faculty which are which, and I expect several years later it will be clear that the lottery system works just as well. Or, like me, if you are hiring some undergraduate researchers, try the lottery system. Make small experiments and let’s see if we can’t buck the current status quo.