Pope Francis makes his first visit to the United States this week, prompting reflections on the state of his papacy so far and speculation about what his legacy will be. Leah Libresco, a FiveThirtyEight staff writer and a practicing Roman Catholic, has a $10 bet with a friend riding on the success of Francis’s papacy. She realized when setting the wager that it’s difficult to quantify the success of a pope. To discuss the question of a pope’s impact, she recruited several colleagues and experts for a chat: Pascal-Emmanuel Gobry (known here as PEG), editor in chief of Aleteia and columnist at The Week; the Rev. Sam Sawyer, a Jesuit priest and an associate editor at America, a weekly magazine published by the Jesuit order; Mark Oppenheimer, who writes the Beliefs column for The New York Times; and Carl Bialik, the lead writer for news at FiveThirtyEight. Below is a lightly edited transcript of their conversation.
Leah: So, what would you guys expect to see from a pope who’s done well?
Mark: I think these things are well nigh impossible to quantify, but then again I was a history major and last took math junior year of high school. The AP score was not, shall we say, winning. But I suppose some measurable uptick in Mass attendance would be a kind of success. I mean, if it stayed completely flat, that wouldn’t be good, right?
PEG: I think before you even think of metrics, you have to think about what criteria constitute success. In the church’s self-understanding, it’s a machine for producing saints. Well, the results are only going to be in a few centuries from now.
Leah: I thought of doing a generational bet, which my friend and I would pass down to our children, to check on canonization rates of people alive during Francis’s papacy, but (a) it seemed to be vulnerable to changing attitudes about canonizations (witness Pope John Paul II canonizing a record-setting 480-plus saints) and (b) no fun for us.
Carl: The very social-media-savvy pope tweeted that “The world tells us to seek success, power and money; God tells us to seek humility, service and love.” What if we take him on his terms? Are people humbler, are more people serving each other and the world, and is love on the rise? I as a non-Catholic would love to find metrics for those.
Mark: I think we can be certain he’ll fail on those terms. If there’s one thing the statisticians can tell us, it’s that religious observance, or even proximity to observant people, has never produced measurable changes in human behavior, or at least not the kind religious people would like to see. If you start a megachurch or a new parish because you think it will reduce extramarital sex, contraception, abortion, lying, gossip, divorce or tax-cheating in its area, or among its congregants, you’re probably screwed.
Leah: I am interested in ways to try to quantify flourishing/kindness/caritas,1 both for the purposes of this bet and because I’m interested in being able to do effective altruism that focuses on more than the health of the body, but I’d agree we don’t have great outcome variables to track. I did wind up focusing more on people’s links to the church in my own bet.
PEG: I would certainly dispute the notion that religious observance isn’t associated with measurable behavior differences, although of course as always cause-effect is impossible to establish with any certainty because of omitted variable bias. But that takes us far from our topic of discussion.
Other obvious potential criteria for success for a pope: evangelism (both growth of the church and, more importantly for Catholicism today, growth in piety of existing members); increase in piety; increase in social justice efforts (however defined).
Even for more down-to-earth stuff it’s hard to measure. For example, Francis has been a very tough reformer when it comes to priestly sex abuse, but I assume the real criterion for success is future abuse prevented, which, of course, is impossible to measure.
Carl: I think we’d want to know what the trends were before he took over. Even if the measurable stuff has declined under his watch, we’d want to know if the decline accelerated or decelerated.
Leah: Yep, which forces us to stick to measures that have historical data, not something we began measuring after Francis became pope.
PEG: And even then you’re still dealing with omitted variable bias.
Mark: I think we could measure how eager people are to live in community, which I suspect correlates with other good things. I take it as axiomatic (you can quarrel with this) that suburban anomie, distance from others, loss of face-to-face time, etc., are bad for human flourishing. I think that people’s eagerness to re-urbanize (loosely defined: maybe it’s ingathering with other, say, Catholics, to make a thriving parish of like-minded people) would be a kind of success. Those parishes in Philly and D.C. where a lot of more traditional Catholics have chosen to live; the eight or 10 urban cores where Haredi/highly Orthodox Jews all gather in the U.S., to be near like-minded folks and have schools for their children; Colorado Springs for evangelicals — these seem to me clear markers of success, highly auspicious in an obvious way. What about if we look for increased numbers of Catholics who seek to live out their lives around other Catholics, be they in Catholic Worker communities, religious orders, or very committed parishes?
Sam: That’s a very interesting perspective: trying to measure something other than individual belief or changes in it, but to see how it’s expressed. Though I think we might need to expand it to also take into account missionary initiative — going out to carry the Gospel to those who haven’t yet encountered it (whether geographic frontiers or existential ones) or to those in material need, and that might cut against Catholics with other Catholics.
I think another angle to take, that might be a little easier to measure than trying to get at the impact in terms of changes in individual belief, is to look at the way the church is talked about in public, and how closely it coheres with how the church talks about itself. And by that standard, I’d argue, there’s already some success: We’re talking more about mercy in public than (as far as I know) we ever have before.
Leah: So you might commission surveys about what words people associate with the church and see how they fit with how it describes itself?
Sam: Or correlational studies on a published corpus of texts, from official media or social media.
PEG: I like that.
Mark: But as Scott Walker can tell you, polls are fickle. And easily swayed by a good PR campaign, which is why Mormons (among others) pay a lot for advertising.
Leah: I have Catholic friends with a lot of holy envy for the Mormons’ Web presence. Meanwhile, I’m still trying to track down a Vatican statistical yearbook that isn’t in Italian.
Sam: Rather than trying to learn Italian? ;)
PEG: Another enormous indicator of success would be an increase in people going to confession, and that’s also unknowable.
Mark: It also privileges the confessional booth, rather than the general confession, which is perfectly acceptable, no?2 So it privileges a more traditional/conservative/nostalgic take on success.
Sam: I wonder if any dioceses attempt to keep statistics on their “Keep the Light On” campaigns to get people to come to confession.
Leah: There are always self-reports, with all attendant caveats.
Mark: New idea. What if we see an uptick in Catholic students who want to do vacation/summer/gap-year Catholic service programs? When I taught at Boston College last year, that was a strong identifier of who took Catholicism seriously.
PEG: I like that.
Sam: I like that too, and I think that’s a little more robust to the problem of figuring out who’s deceiving themselves or others about the content of their belief. Also, if people are understanding the church better (i.e., closer to the church’s self-understanding and teaching), I’d consider that at least a partial win even if it doesn’t immediately result in increased affiliation or identification.
PEG: Another huge thing would be a great overlap between people who say they’re Catholic and people who actually believe what the church teaches (and not because people stop saying they’re Catholic!). Which is sort of what Father Sam is getting at, I think.
Mark: That assumes Catholicism is about belief, which I think is debatable. I don’t think religion generally is about belief, and Catholicism is no exception.
Carl: How about metrics that relate to the non-Catholic world? Leah ran some interesting numbers showing that Catholicism is losing to other faiths when Americans change faiths. Does the attitude of the rest of the world to Catholicism, and other religions’ relationship with the Vatican, matter, and is it worth measuring?
Leah: Though even there, it’s a little hard to decide what to measure! Catholics do about as well as “nones” (people with no religious affiliation) at retaining the people raised in the faith, but that still means that “nones” are on the rise, since there are a lot more Catholics than “nones” to start with, so losing 40 percent of the people raised as either means you have way more ex-Catholics.
Mark: I don’t see how you can ever improve on self-identification. When you start applying criteria — rooting out the liars or dissemblers — you will fail, in some profound ways.
Leah: And, in fact, self-identification numbers were what my friend and I settled on. But we had the problem of how to control for general changes in religiosity over time, so we went looking for a control group. Any guesses as to what we picked?
Mark: I give up. Remember, math is not what I do.
Sam: Did you measure against another religious self-identification?
Carl: Catholics under the prior pope?
Mark: Boston Celtics fans?
Leah: Sam and Carl (combined) are right! We decided to look at the rate of change in Catholic self-identification from 2010 to 2013 (as measured by the General Social Survey, or GSS) and 2013 to 2016 to see if there was a discontinuity. But we needed a control as a comparison, and we picked Episcopalians!
So the question for us is: Did rates of change under Francis vs. Benedict in those years beat whatever trend was going on for Episcopalians in that period?
Carl: I hope $10 isn’t enough for either of you to try to find GSS respondents and sway their answers!
Leah: We’re more betting for pride (which I guess I’ll talk to Father Sam about later) than anything else, but I’ve got a calendar reminder set for a year from now so that one of us pays up.
Carl: Also — so this is U.S. success, only?
Leah: Yep. Charitably, we’re betting where we’re informed enough to make predictions. Uncharitably, we’re pretty parochial, like most Americans.
Mark: With Episcopalians involved, may I suggest including some spirits in the bet?
Sam: Were you concerned about external events impacting the Episcopalians over the same time frame? They’re having plenty of their own news.
PEG: Why Episcopalians?
Leah: Yeah, that’s turned out to be more of a concern, but the main thing was that, although a lot of weird things could happen for them, we saw that more as noise than bias. And we picked the Episcopalians because they’re the closest we could get to Catholics on a number of fronts (the way the liturgy works, structure of churches, etc.), so it was the best we could do.
Sam: Am I allowed to ask whether or not you’re summing back in the groups that fractured off from the mainline Episcopalians under the same time frame?
Leah: Whatever the GSS does, we do. But if they don’t lump them, I won’t. (Pope Francis doesn’t get credit for Old Catholics.)
Leah: We’re wrapping up, so I’m curious: Which side would you all take on the bet my friend and I have running?
Sam: I’ll bet on the Catholic side (positive effect from Pope Francis), but I feel sort of like I’m gaming it based on the external factors.
PEG: Given the control group, I wouldn’t take the bet. Sorry. I suspect the outcome will just be noise. (But on a differently structured bet, my bet would be Francis has a positive impact on the church, however we choose to measure it.)
Carl: I’m backing the pope. Even though we know executives shouldn’t get full credit for results under their watch.
Mark: I don’t see how his visit could have a negative impact, but I’d be curious how positive it is.
PEG: Uh-oh. If there’s any reason to believe he’ll have a negative impact, it’s that we all believe he’s going to have a positive impact.
Leah: Well, thanks for your suggestions and predictions, everyone. I’m still feeling pretty good about my bet, and even better about the pope, but I guess I’ll know about the former sooner than the latter.
Read more:
Pope Francis Called For More Work From Priests, But 20 Percent Of Parishes Don’t Even Have One
The Pope’s Not Going To Like How Far We Are From Our Climate Change Goals