337. Castillo and Caturra, by the numbers
I recently concluded a series that examined the current campaign by Colombia’s coffee authorities to replace the country’s traditional coffee cultivars with the disease-resistant Castillo hybrid as part of their response to the coffee leaf rust epidemic.
As part of that series, I wrote:
At the risk of oversimplification, the debate has been framed by two positions: that of representatives of Colombia’s Federación Nacional de Cafeteros, who insist that Castillo will thrive in the specialty market because its cup quality equal to that of the country’s traditional varieties, and that of quality-obsessed specialty roasters who insist it won’t because it isn’t.
Well, we collected hundreds of coffee samples from participants in our Borderlands Coffee Project in Nariño last harvest as part of the project’s baseline survey, and asked allied importers and roasters to cup them for us. Caturra was by far the most common variety, with Castillo a distant second in terms of the numbers of samples collected. As it turns out, the data we collected did suggest that Caturra consistently outperformed Castillo, but only by the thinnest of margins.
I want to emphasize from the beginning (before I the critiques come in from the Ph.D.’s who have been carefully following the discussion here on impact assessment, or the quality-focused roasters who are wringing their hands over the rise of Castillo and the dwindling genetic diversity of coffee) that these data are far from conclusive. They may be more notable for what they don’t tell us than what they do. In fact, I will close this post with a partial list of all the ways we need to improve our data analysis to be more assertive in attributing our findings to the genetics of a particular variety.
And I should also mention that we do not see ourselves in any way as an arbiter in the debate over the relative qualities of different varieties currently cultivated in Colombia. That task belongs to others. In the meantime, we find ourselves working with smallholder farmers choosing from among those varieties as they try to minimize production risk on the farm and maximize their opportunities in the marketplace. As we try to support them in their decision-making, we are gathering and analyzing data on this and many other issues affecting their livelihoods. In service of the broader discussion over coffee disease, genetics and quality, we are publishing those data here.
CONTROLLING FOR TERROIR.
There may be a temptation to misread the graph above as a scoreboard: Caturra 83.5, Castillo 83.1.
But those values are average scores from samples collected from dozens of farms over a vast geographic area. And not all coffee farms created equal. Some are naturally endowed with more of what makes coffee extraordinary. Like elevation. Or volcanic soils. Or solar radiation. These factors, largely beyond the control of farmers, go a long way in influencing cup quality. If we don’t account for those differences in comparing Castillo and Caturra scores, we will end up erroneously attributing to coffee genetics quality differences that are based on terroir.
Thanks to our action-research partnership with the International Center for Tropical Agriculture in Colombia (CIAT), we were able to identify five distinct agroecological production zone in the areas of Nariño where we are implementing our Borderlands Coffee Project, and assign each sample to one of those zones. Here are the results by agroecological zone.
The identification of these distinctive production zones, based on years of careful CIAT research into the drivers of coffee quality, allows us to control for differences in agroecological conditions when comparing the scores of Castillo and Caturra. By comparing the results of different cultivars within specific agroecological zones, we can reduce the “noise” in our analysis. Unfortunately, the baseline included very few Castillo samples, just eight. There was only one zone with more than one Castillo sample. In that agroecological zone, despite a sample size too small to be statistically meaningful, the relationship between Castillo and Caturra was similar to what we saw overall — Caturra was slightly preferred by cuppers who participated in our baseline.
This relationship — Caturra scoring slightly higher than Castillo — also held up in the other two agroecological zones in which the baseline had scores for samples of each variety. While the number of samples of each variety is too low to put too much stock in the results, the fact that the two varieties move up and down together across the zones may suggest that both the agroecological zones and the narrow but consistent advantage for Caturra are meaningful.
WHAT’S WRONG WITH THESE DATA?
- Sample size.
We didn’t collect enough samples to be able to make generalizations about either Castillo or Caturra on the basis of our baseline cupping results.
Even if we had more samples, the accuracy of the varietal identification is less than scientifically certain. The samples in this data set were all self-identified by the farmers who submitted them to us for evaluation. We have taken them at their word and have a high degree of confidence in the identification on which we based this analysis, but there is no guarantee that the label on the outside of the bag corresponds to the genetic content of the coffee inside it.
- Omitted variable bias.
In my mind the biggest weakness of these data is what they leave out. In order to isolate the impact of coffee genetics on cup quality, we must control for all other variables that can also affect cup quality. The division of the project area into agroecological production zones is vitally important in controlling to some extent for terroir — the sources of cup quality that farmers can’t control. But plenty of the sources of coffee quality are the result of what a farmer does with it. Everything the farmer does throughout the coffee life cycle impacts the quality of her coffee in the cup — how she manages plant densities, shade, fertilization, weeding and pruning, pests and diseases, harvesting, post-harvest selection, pulping and washing, drying, storage, transport, etc. And these data do not control for anything that happens to the coffee between the time the farmer selects the seeds she will plant in her nursery and the time she prepares samples to send to the roaster. So much of the variability in the results observed in the baseline are due to differences in farming practices, harvesting and post-harvest processing. Improving performance from seed to cup will be a focus of our project’s work on the ground. Meantime, in order to get a more accurate sense of how much of cup quality can be attributed to coffee genetics, we need to control effectively for these variables.
- Blind sampling.
This time around, we included varietal information in the sample tags. Cuppers had access to information about what they were tasting before they cupped. This harvest, we will prepare blind samples and unveil the genetic identity of the samples only after they are cupped.
- Better analytical modeling.
We need to spend more time mining the rich trove of baseline data we have collected from project participants and allied importers and roasters, conducting multivariate regression analyses to more effectively attribute the results we observe in the field to specific causes. And not just for better understanding the quality frontiers of different coffee varieties. We need to do this in order to understand the effectiveness of everything we are doing in the field with farmers, from agronomic extension to building new post-harvest infrastructure. Fortunately, the full baseline report for our Borderlands project is due soon, and a Ph.D. widely recognized for his work in coffee will be joining the Borderlands team in 2013 to help us on this front.