I’ve been in Amsterdam for fourteen long days working with many others on the intense process that judges the winners for the World Press Photo 2014 contest (which includes the award of the World Press Photo of the Year 2013).
This was my first year working as Secretary to the contest jury, though it was the second year I have witnessed the judging. Last year I was present for about two-thirds of the time, learning from my predecessor about the Secretary’s responsibilities. Unlike other jury members, the Secretary does not participate in debates or vote on photographs in any of the rounds. The Secretary is tasked with ensuring fairness in the process and is responsible for all procedural matters relating to the conduct of the judging, which means applying the written rules to ensure the jury can reach the best possible outcome. Looking back on the last two weeks I’m in no doubt we ran a judging process that treated all contestants and jurors equally and fairly.
I’ve had a long association on and off with World Press Photo, having presented the 2005 Sem Presser Lecture and directed the Multimedia Research Project in 2012-13. Up until the time I was employed by World Press Photo for the research project I occasionally wrote here about debates involving their procedures and the prizes awarded. When I was directing the research project I decided I could no longer do that. While the Secretary is not an employee of World Press Photo, I still maintain the position that it is not appropriate for me to comment one or another on winning photographs. I leave that for the time being to others, and you can see Martijn Kleppe’s independently curated list of commentary for most of those debates.
What I want to do here, therefore, is detail how the judging process operates so that the jurors can reach their decision. I am presenting some detailed information for others to debate if they so wish, but will not be debating these points directly myself. World Press Photo keeps its procedures under review and I will be contributing to those internal discussions. I am writing this because after two years of having an insider’s view, I feel that those on the outside will benefit from knowing more about the stages of what is a long and complex process. Sometimes I get the sense that without knowing what happens at various stages, people on the outside view the jury process as something akin to the cardinals electing a pope – an opaque process, in which deals are done and favours called in, that culminates mysteriously in the release of white smoke to signal a conclusion.
I’m sure there will be many who continue to hold that view even if they read this, but I feel it is my responsibility to the nineteen jurors who gave their time and energy to the process this year – without payment – to lay out how they worked. All the jurors and the Secretary are bound by a signed commitment to confidentiality, meaning we are unable to discuss what individual jurors do or do not say, and we are unable to discuss which specific images went out and why. This cardinal rule is designed to create an environment of openness and trust within the jury room amongst the jurors, so they can speak critically and freely in the knowledge their contributions will not be misrepresented later. This of course complicates the desire to be transparent about all aspects of the debates. Other competitions operate in different ways, with, for example, Picture of the Year International running live webcasts of its judges at work. While that seems to be the model of openness, I have heard from former PoYI judges it has the potential to constrain judges in speaking freely because of the concern that those up for awards might be watching. Which approach is superior is for others to debate, but there are good arguments on both sides, and as a result I can only discuss how the procedures of the World Press Photo contest work.
I know also there are a number of people who doubt the value of contests in principle, and that can be a valid view point. World Press Photo has been judging professionals annually for nearly sixty years and retains considerable respect and status within the international photographic community. While I am Secretary I want people who question the contest to do so from as informed a position as possible. As a result, this will necessarily be a long post, quite dry in large part. I will be covering:
- data on entries
- jury structure and principles
- the first and second week of judging
- conflict of interest procedures
- manipulation and processing issues
I hope those wanting to continue the debate will take the time to read it closely, because a good critique depends on some knowledge of what you are talking about.
Data on entries
This year the contest received 98,671 images from 5,754 photographers with 132 nationalities. Photographers enter their images in one of nine categories
Data on the national breakdown of entries is interesting. If we consider all countries from which 100 or more photographers entered we get this list of the proportion of entry nationalities:
That means entrants from thirteen countries constitute 61% of the total number of entries, and those from 119 countries make up the remaining 39% (with Australia, Bangladesh, Canada, Indonesia and Mexico prominent amongst those). We would have to look at other years to know whether this national breakdown was either typical or anomalous. It is worth noting that 61% of this year entries came from freelancers who work professionally, with 39% from directly employed professionals. Male photographers made up 86% of entrants, and 14% were female.
The jury structure and principles
In the first round their are specialised juries – News and Documentary, Nature, Sports and Portraits. – that consider their respective categories. You can see who was on each specialised jury here. News and Documentary is the largest specialised jury, with five voting members, because it considers the bulk of the entries – 68,023 photographs this year. The other juries have three voting members. A great deal of effort goes into having a diverse group of jurors. There were nineteen judges from more than a dozen nationalities, though judging always takes place in English.
The juries operate with three general principles:
- Over the two weeks juries are, in two stages, editing a large body of work down to smaller numbers in order to award prizes to singles and stories in each category.
- The jury process is a form of peer review, where people from the industry and wider photographic community make the judgements.
- The images are presented anonymously, without any credit identifying the names of photographer, agency or publication. Jurors are prohibited from mentioning any names if they recognised images and they are prohibited from speculating on names if they are unsure, and this prohibition applies both inside and outside the jury room. As it is not uncommon for different photographers to independently submit images from the same event or place, event those who think they recognise work are often surprised when the names of winners are revealed after the conclusion of judging. To enable discussion of anonymous entries, each image and story has a code number jurors can refer to if they wish to review something.
The contest has five voting rounds: the first, second, third and fourth round, and the finals. Each jury has a specific role, and each round had different voting requirements and procedures.
The first week of judging
The first stage of the judging process runs for the first week. In each specialised jury, singles or stories are passed to the second round if one member of the jury says ‘yes’ on the basis of seeing the screened image. All images are on the screen for roughly the same brief period of time, and there is no caption information available at this stage.
Because they are dealing with smaller bodies of work, the Nature, Portraits and Sports juries proceed to the second round in the first week. Work passed with a single ‘yes’ is then seen again, and requires at least two votes to proceed. If stories do not proceed, then each jury member can if so desired select a single image to be entered into the same singles category. These specialised juries then bring a minimum of six entries in each category to the second week of judging.
The second week of judging
The General Jury conducts the second week of judging. The Chair and Secretary of News and Documentary act as Chair and Secretary of the General Jury. They are joined by the Chairs of Nature, Portraits and Sports, and five new members who did not participate in the first round.
The General Jury begins with the third round. Voting is anonymous, with an electronic voting machine recording each jurors ‘yes’ or ‘no’. Only the Secretary sees and calls the results of each vote. It requires at least 6 votes of the 9 voting members for a picture or story to pass to the fourth round. Caption information (provided by the photographers when they enter) is available. It was read as a matter of course for stories and on request for singles. Work voted out can be recalled if proposed and the vote is supported by at least 6 jurors, and work can be moved to another category if proposed and supported by the same number of jurors.
In the fourth round, voting is also anonymous and conducted electronically. The aim of the fourth round is to edit each category of singles and stories down to between four and six entries for the final round. That is achieved by alternating negative and positive voting. Jurors first vote to reject work for the finals by voting ‘no’ with the machine, and it takes at least 6 ‘no’ votes for something to go out. If there are less than four or more than six entries left, the jury will then revert to positive voting (voting ‘yes’ with the machine) to reach the four to six entries required for finals. If this takes longer, then the process alternates between positive and negative voting.
In a new procedure for this year, all singles and stories voted through from the fourth round to the finals were reviewed by an independent digital expert to see they complied with the contest rules on manipulation and processing. I’ll detail that process in the section below. Those entires deemed eligible entered the finals for consideration for prizes, with a maximum of four entries in each category.
Extensive discussion is a feature of the process. From the third round onwards the jury engages in extensive debate on the singles or stories in front of them. This was like being in an advanced visual studies seminar – jurors explored aesthetics, ethics, editing, the depiction of violence, informed consent, stereotypes, narrative, story and more.
In the final round, a different voting process, involving the distribution of preferences, operates. Entries are still presented only with their code. Voting is anonymous, using paper ballots, and jurors are prevented from discussing their votes. Jurors have 10 points to distribute to singles or stories they wish to award first, second and third prizes too. They can award a maximum number of 7 points to any entry. This means they chose any combination of 10 points from 7/3/0 to 4/3/3 is given to the three winning entries they favour. Once they have voted these points are counted and the prizes determined.
After all first/second/third prizes are awarded in the nine categories for both singles and stories then the process of selecting the Photo of the Year begins. Only those singles which were first place in their respective categories, plus any frame from any of the first/second/third prize winning stories that was taken is 2013, are eligible to be considered for Photo of the Year. Jurors then nominate photographs – which are still anonymous, without the names of the photographer etc – to be considered and a long list is produced. After debate, successive rounds of positive and negative voting along the lines described above (each time requiring at least 6 votes to keep a photo in or remove it) are conducted until two photographs remain. After more debate, a secret paper ballot is conducted with jurors voting for photo A or photo B. The photo with at least an overall majority amongst the nine members is selected as the winner.
Conflict of interest procedures
I have laid out the jury structure, principles and process in detail because I don’t think these procedures are widely known. The complexity of the voting structure over five rounds is designed to ensure that to progress singles and stories have to build strong majority support and resist majority opposition. The anonymised images and confidential voting process is specifically designed to prevent one individual pushing a favoured image, for whatever reason, against the majority view. Having watched this for one year and operated it this year I have no doubt it works exceedingly well.
The judging process, as I observed at the outset, is a form of peer review. It is inevitable that when you have entries from thousands of professional photographers around the world, someone on a jury of peers could have some relationship with submitted work. This is dealt with in two ways. Firstly, if they know they have an interest jurors must publicly declare it to the jury so it can be factored into debate. Secondly, to check that jurors are declaring an interest, from the third round onwards in the General Jury, the Secretary is given data from the support staff that cross checks the entries remaining to highlight potential interests. I can make two general observations on this. The first is that to date the only reason jurors have not declared an interest before being told of the interest is because they do not know or recognise the work under consideration as being somehow related to them. People assume jurors automatically recognise all work from their colleagues or publications, but this is far from being the case. The second is that people should not assume jurors declaring an interest automatically go on to advocate for the work in which they have an interest – they are at least as likely to oppose it.
Of course, there are many ways organisations can handle the appearance of, or potential for, conflicts of interest. Pictures of the Year International represents its process in these terms:
Pictures of the Year International selects judges who maintain the highest journalistic and ethical standards. We have confidence that these same values will apply as jurors for POYi. We recognize that our profession is a close network and that the judges are also working journalists. So, we carefully research and consider any potential conflicts and then counsel all the members about their obligations to be fair and impartial. Any judge with entries in a category are asked to recuse themselves. The entire three weeks of judging is an open forum for anyone to quietly observe the process. POYi conducts the annual competition with complete transparency and integrity.
Without knowing the full detail of their research and counsel, this sounds similar to the World Press Photo approach. One big difference is that World Press Photo jurors cannot submit entries in the year they are judging, so there are no ground for recusal as there are for POYi judges.
The appearance of a conflict of interest has been a topic in the aftermath of the Photo of the Year award because the winning photographer, John Stanmeyer, is a member of VII, founded by Gary Knight and others. As Lens blog reported:
Mr Knight said that although he had asked to be removed from the final judging because of his friendship and professional relationship with Mr Stanmeyer, the World Press rules did not allow for it. He emphasised that at every level there was complete transparency. “If anything,” he said, “I was a hindrance for John getting the award, not a help.”
This is true. According to the rules, all jurors have one vote and must vote, and no one can abstain in the vote. There was complete transparency at every level and all the rules were followed strictly. The winning image had to progress through the various rounds with majority support as detailed above. Like all the votes from the third round onwards the final vote was anonymous and confidential so we do not know how any individual juror voted. To be blunt, it is an insult to the intelligence and integrity of the eight other voting members of the jury to suggest they made an award on the basis of a declared interest. As Secretary and the person responsible for the integrity of the process I have no doubt whatsoever that all the winners were decided on in accordance with both the rules and the spirt of fairness and equality. There is simply no basis in evidence for questioning the conduct and integrity of Gary Knight, the general jury chair, who at all times created an open environment for free debate on all entries. That was one of the things remarked on and appreciated by all jurors.
Peer review is the backbone of a good system of governance. Anyone with experience in research knows that peer review helps ensure the best quality outcome. In academia peer review determines the award of grants and publications that involve millions of dollars and secure employment. Yet procedures in the university sector often lack the levels of assurance and robustness found in the World Press Photo voting scheme.
The problem is that peer review in a relatively small professional community is going to create the appearance of a conflict of interest where none exists. That is a difficult issue we will struggle with so long as judging by peers is the principle. I’m sure those respected film, literary, musical and theatre awards which use peers from the highest levels of their respective industry struggle with similar issues. Given that 5,754 professional photographers from 132 nationalities entered, making the avoidance of the appearance of a conflict of interest the primary basis on which a jury was structured would eliminate anybody – photographers, editors, publishers, broadcasters, journalists, gallery owners, writers, curators, foundation funders, not to mention any personal friends or current and former partners – who in some way were associated with or worked with that global network. If you made recusal mandatory for any declared interest, then contests could be decided by a jury of peers reduced to a much smaller number of people that could alter with each vote on a single or story, which would distort the process and outcome in other ways. Would it be fair if there were nine jurors voting on one entry, six on the next and only two on the last? That would be a nightmare to administer in terms of equitable procedures.
Of course, you could advocate having a photography contest judged by people who had absolutely nothing to do with any aspect of the global network of photographers who entered. But then you would have a totally different contest and potentially a whole new set of problems. It might be a good idea for someone to set up such a contest and see how it works out, but it’s obviously unlikely to be under the auspices of World Press Photo.
In the end, I don’t know an easy way around the problem of communicating the integrity of a process based on peer review in a small professional community beyond what is covered here. I would be interested in any considered responses on that topic. My hope is that fair minded readers will appreciate that the detail of these reflections is necessary in order to understand how the structure of voting deals with the conflict of interest issues in advance of the specific processes for checking and declaring an interest. With regard to this year’s award, too many critics have, in ignorance of the procedures, remained at the level of appearance, confusing correlation with causation. We know we have a communications problem, and we know that appearances matter. What we do not have is a fairness or integrity problem.
Manipulation and processing issues
In a new development, this year World Press Photo had all entries being considered for prizes in the final round examined by an independent digital photography expert from the Netherlands before the jury proceeded to deliberate. This was to determine that all singles and stories going into the final round were eligible. To be eligible for prizes, entries must be valid according to the contest rules. The relevant rules states:
The content of an image must not have been altered. Only retouching which conforms to currently accepted standards in the industry are allowed. The jury is the ultimate arbiter of these standards…
The expert carried out a case-by-case analysis of the level of post-processing in the files that were entered in the contest by comparing them with the unprocessed files. 120 photographers were contacted with the request to submit the unprocessed files for analysis. The jury received a full briefing and this was followed by a thorough discussion.
In applying the contest rules, the jury affirmed the content of an image must not be altered. This means no significant material may be added or removed by either cloning or substantial toning. The jury based their decision on the outcome (whether significant material had been added or removed) irrespective of the technique (cloning or toning) used. The jury applied accepted standards in the industry, which, for example, allow for the cleaning of dust and scratches, and this judgement was applied in the same way to each entry in each category.
The jury decided that 10 entries were not eligible for the finals. That is 8% of the entries that were still in competition after the fourth round. There were 8 stories and 2 singles rules to be ineligible, entered in the Nature, Sports, People, Spot News, and Contemporary Issues categories. World Press Photo will be writing in confidence to those photographers whose work was ruled ineligible to make them aware of the decision.
Was this a large number of problem entries at that stage of the competition? Given that it is the first year this process has been in place, there is no point of comparison. We will have a better idea in the future, but then we would expect that percentage to drop as future entrants will be aware of what being caught means, so we won’t be able to say if that proportion is reflective of the industry as a whole. The Jury Chair reflected on these numbers:
I was really distressed, especially because so much of the post-processing that had made these images ineligible was absolutely unnecessary…It was materially minute but ethically significant. Or it was just laziness – it was photographers trying to turn a pig’s ears into a silk purse. One image in one story disqualified the whole story.
The review of eligible entries is an on-going process given the need to closely analyse unprocessed files as they come in from photographers. Work found to be ineligible can be disqualified after judging, as was the case in a previous year.
I am presenting this detailed information on the process for others to debate if they so wish, but will not be debating any points relating directly to this year’s competition myself. I am happy to accept comments below, but will only respond myself if there is a factual point about this years procedures to clarify, or a general observation to engage.
I have to conclude by saying I was honoured to accept the post of Secretary to the World Press Photo contest jury and I am already looking forward to next year. I am proud of the way this year’s judging was conducted by all concerned. There’s lots of analysis to do on what the contest can tell us about the state of the global visual economy and the representation of the world it gives us, and this was something this year’s jury debated too. But since returning form Amsterdam I have been sleeping well in the knowledge the results of this years contest were achieved fairly, equally and with unquestionable integrity on the part of all who deliberated.
Update: 18 February 2014
The World Press Photo Managing Director issued a statement which concludes: “World Press Photo has total confidence in its judging process and how it was applied this year. We trust absolutely in the integrity of our chairs and jurors and we honor the selections they have made.”
Photos credit: © Michael Kooren photography / Hollandse Hoogte — the general jury working, at World Press Photo HQ, 11 February 2014.