After the Webinar: Learning from Sentinel Events. Q&A with the Presenters

Webinar presenters Maureen McGough and Jim Doyle answered a number of your questions after their presentation, "Learning from Error in Criminal Justice: A Sentinel Events Approach." Here are a few of their responses.


Audience Question: Should agencies conduct more studies to examine the smaller individual variables that lead to these sentinel events? 

Jim Doyle: I'm not completely sure I understand the question but I think one of the things that these sentinel event reviews will help with is beginning to answer that question. To put it in another way, by beginning to surface good questions to ask. There's a lot of data out there, what we don't know is we have the data we need and we don't just know what to ask the data we have. When you look at an event like this, it gives you a way to query the data. So, you have, on the one hand, a statistical compilation and on the other hand, a narrative. The narrative can be very important in explaining why this case turned out to be one of the 10%, not one of the 90%, and what more you might need to know about this. One of the things to stress about is it's not immediately clear when you just start saying well let's look back at something that happened is these things are very focused on not in the performance review of individuals, the kind of thing we're used to looking at. Should somebody be fired or prosecuted or disciplined? These are event reviews where you try to understand the whole review, the whole event. Why things happened not just what happened and who did them. They can be very illuminating for agencies who might not quite know what questions they want to ask themselves about it. Certainly, because most things in criminal justice involve more than one agency. It's hard for me to think of an event where I couldn't name at least five that are involved. It helps if you ask each other questions too.

Maureen McGough: ​ In terms of the long-term implementation goal of this type of sentinel event reviews is, James has actually written a bit about this as well, is that the ideal venue might be something like a state center that was in a position to collect a multitude of these sentinel event reviews and conduct a meta-analysis to determine what are the common contributing factors that we are seeing consistently over time across jurisdictions where folks should really consider putting a lot of their attention in order to affect long-term system changes.


Audience Question: At what stage would you initiate a review? Would you wait until all of the civil and criminal criteria have been submitted or begin immediately? 

Jim Doyle: Let me start. This is one of the things that we hope to learn by this exploration because the process of choosing the events to review will also implicate when you want to begin a review. For example, if you have a near miss or a good catch that has nothing in terms of financial liability implications, well you might start out earlier. If you have something where there are financial implications but you know that the case has been settled or closed, well you might wait and do that later. Again, this falls in that area where we're not pushing down a federal set of guidelines about this. I will say that having, there will be a different answer to that question for different events and how you can create a kind safe harbor to have a discussion of these events will differ from place to place and from event to event. That is one of the things we want to learn about.

Maureen McGough: ​ That is a great question. Definitely, something we have wrestled with a bit on the pilot engagement. One of the things we learned in our pilot, the Philadelphia case was very high profile, the (Schurif?) murder case. There was a lot of associated activity around civil and criminal liability after that case happened. When we actually do the review of that case, it was decades old at that point. There were still lessons to be learned with practical implications for how the system can be improved in the present day. We learned that policies may change in the intervening years but practices on the ground are much slower to change. Even if you have a case that's a bit on the older side, there might be some value in your jurisdiction in examining it.


Audience Question: Judy says, "I'm a behavioral scientist and one of the areas I focus on is evaluating high risk, investigative and operational decisions. One major factor that is often overlooked is the negative effects of decision making bias on investigative and operational decision making. For example, how information is presented, the order, structure, language, complexity, significant — influences, what you think, what you feel and the decisions we make. Is this something that you're looking at as one of your pilot sets?" 

Jim Doyle: That's a great example of the kind of thing that we think can be mobilized in an event review where things that often look like just simple mistakes have deep, prevalent, cognitive roots that need to be examined where someone like Judy with an expertise in this area is called upon by the group viewing the event to shed light tremendous light on what happened. This applies across a lot of very superficially simple situations. For example, we talked about the example of a prosecutor who violates the ethical code and hides exculpatory evidence leading to a wrongful conviction. There's a whole body of social science, behavioral science, and behavioral ethics. It's not really just a question to say you violated this rule and stop at that because we can learn a lot about why do this person violate this rule. What was it at this situation that led to this ethical behavioral problem? Of course, the whole business of investigation and adjudication is frequently saturated with ongoing cognitive biases that we all have. So, having people at the table who can bring that out and help show that these things were not one-off events involving the performance of an individual but a natural consequence of basic cognitive human behavioral issues on repetitive situations can be, we hope, are very important part of these event reviews.


Audience Question: What were some of the obstacles your pilot sites encounter? In other words, what kind of likely challenges would my team be facing if we got involved in a sentinel events program? 

Maureen McGough: ​The ones that sort of comes to my head initially were in the Philadelphia case, for example, access to record in an older case was a bit challenging. Capacity for reviewing records was a need that was identified that crossed all three states so in order to do some of these reviews, depending on the issue of course, but a lot of them are very file intensive and they were having to come up with creative ways to have folks help out reviewing case files for relevant information to contribute to the review. Managing the information was a challenge. We ended up using a specific timeline software to help sort of get it back. Not necessarily in our pilot cases, but we imagine in some jurisdictions, establishing parameters for confidentiality and information sharing, at least initially, will be a bit challenging. One of the principals of these reviews that we feel are very important is that it's all ranked and all stakeholders. You have a situation where you might have a line officer in the same room as the law enforcement chief or executive. That makes disclosure a little bit difficult so we came up with a few creative ways to conduct interviews that help people feel comfortable so they can disclose relevant information without worrying about any blowback. I think it's one thing to say that these sentinel reviews are not blaming but until that's proved out over time, convincing people of that and encouraging them to be portrayed in a discussion is going to be challenging as well.

Jim Doyle: One of the things that the technical assistants provide or would bring to the demonstration site would be expertise and working around these problems. One thing, for example, to think through is whether you can have a successful review without actually having the enrolled individuals at the table but having the people who know very well the perspective from which the individuals would be operating. Again, allowing for the fact that there are these core ideas that people need to pursue, we do want to see people use their own creativity to find ways around these problems. We do think we have a very capable technical assistance provider who was involved, for example, in the Philadelphia review and can help bring the lessons from other people and coordinate the lessons learned by the demonstration sites as this goes forward in multiple places.


Audience Question: What can you say about the political and legal threats involved in the conducting sentinel event reviews? 

Jim Doyle: Let me start on that. Again, this is one of the events we are going to learn about. I'd gone out and listened to people about this problem. This question tends to arise not because the person I'm talking with is themselves worried about it. It's because they're worried about explaining to someone else why they should go ahead and do this. Most people who are involved in the front lines and have been or veterans of the front lines understand that there's only so much you can do to prevent liability. If you're going to be sued, one police chief told me, you're going to be sued. That doesn't change the fact that you don't want to repeat this so that you'd get sued again someplace down the line. Many of these situations are situations where having this review is not a binding cause of litigation. Litigation, if it happens, will happen, civil litigators will get the information that they're going to get. The actual downside of doing these reviews in terms of concrete outcomes, in terms of money, when people who think it through are probably not that bad. It doesn't change the fact that the first lawyer you talked to will probably say well the safe thing to do is to say nothing to anybody at any time. But because most of these things involve public officials in public agencies who had to satisfy the public, they understand that the best way to avoid paying out a lot of money is not to repeat the harm another time. Also, to not put yourself in a situation in terms of public trust in the law where something happens, you don't have someone to hang or even if you do have someone to hang, you don't learn anything and you make it appear that you don't much care whether this happens again. You want to be able to show in the same way that for example, the National Transportation Safety Board does after an air crash that you would really try to get to the bottom of this situation because you don't want to see this happen again. If you think through the kind of trust, we all have for the transportation system, a lot of that comes out of being open and proving that you do want to learn about this stuff. The liability issues are, they tend to be in the front of people's minds but they're not necessarily deep issues. There are workarounds for most of them. You're not wrong to raise the question because they will inhibit people, they will be a fact of life. When you want all stakeholders and your format depends on all stakeholders, one recalcitrant stakeholder can damage your effort. You need to satisfy everybody and get everybody to collaborate by consensus. Politically, there are actually advantages to being able to go to the microphones the day after the event and say we really consider whether someone's going to be prosecuted but whether someone is going to be prosecuted or not, I promise you we're going to do a feral examination of this event with everybody at the table and understand why it happens so we can make sure it does not happen again.


Audience Question: How does this process compare to a root cause analysis? Is there any benefit to join both simultaneously or could they possibly contradict and negate each other? 

Jim Doyle: A root cause analysis is a particular technique of event review. I don't see any reason to not mobilize a root cause analysis, but I also think there is something to be learned from an all stakeholders approach that involves many voices and doesn't turn into a purely technocratic kind of root cause analysis where experts are talking to other experts about what happened and trying to analyze things that way. There are people in the safety world who argue very strongly that there is a danger in just doing a root cause analysis because it ends up giving you a very schematic, oversimplified, set of explanations for what happened when in fact, things could be a lot messier than the root cause analysis makes them seem. There are other people involved in safety work in medicine, aviation, and industry who would say there's a danger in root cause analysis that what you get from a root cause analysis is something that ends up with kind of easy compromises being made by the players who happened to be at the table rather than a complete resolution of the problem with the underlined culture or the underlined environment. Again, here's something that I don’t think the BJA and NIJ planned to dictate in advance. We have a technical assistance provider available to the demonstration sites who is very very adept in doing the root cause analysis, can help people do that, and I think can help people discuss among themselves whether that needs to be complemented by other techniques for looking at the event and understanding the event. I think it's a great question. I think it's a question you look forward to answering during the course of working on these demonstration projects.


Audience Question: Could this work in and are you interested in piloting the program with a federal law enforcement agency such as Wildlife Law Enforcement with respect to wildlife crimes? 

Maureen McGough: ​ We are certainly open to partnering with whoever's interested in learning to better apply these approaches. One thing I will say is that while it is not hardened fast rule, one thing we do prioritize is having multiple stakeholders partners at the table. While we would certainly welcome a federal law enforcement partner, I think the question would be are there other entities whose actions or inactions contribute to the types of errors you're looking to review that we also could bring to the table.


Audience Question: When you assemble a sentinel review team, is it then to review multiple events or is a new review team assembled for each event? Then they have a follow-up question that I'll get into depending on how you answer that first part. 

Maureen McGough: ​I think James and I are saying this answer over and over again but I think that is largely dependent on the locality. Who they have available, capacity and the types of events that they are reviewing. One of the things that we have seen that has been effective is where you have a sort of standing body, a panel of stakeholders who can represent different stakeholder interests and then depending on specific events that are brought before the panel, they might bring different partners to the table, they might interview different people as part of the fact-finding mission. There's nothing prescriptive about whether it needs to be a sort of a standing body or whether the team is assembled ad hoc depending on the events that come forward.

Jim Doyle: One thing I could add to that is that the experience we've had with the places that have done this is that they want to do more. The people who take part in this find them to be among the most valuable experiences they've had in many cases in their very long professional careers.


Audience Question: If it’s for multiple events, what organizations at the minimum should be at the table? It sounds like it really depends on what kinds of events they're going to be looking at, is that right? 

Maureen McGough: ​ That is absolutely correct. So, in the current sort of very novel sites that we have going on now for the demonstrations projects, there are some sites that are contemplating at a minimum law enforcement prosecution court and defense, right? That'll be your sort of standard criminal case. In other areas they're looking at opioid fatalities of justice developed individuals. There you would have of course law enforcement, you might have public health at the table, emergency services at the table. It really is dependent on the type of data that you are looking for.

Jim Doyle: I just think if you're using this safety lens, you will begin to see more and more stakeholders that you want at the table. And more and more stakeholders who want to be at the table.  Public health people who are constantly inheriting problems from the criminal justice system and vice versa would both want to be at that table. Family members of those who are involved in that cycle would want to be at that table. Again, it will vary from place to place and from event to event but the logic of this safety approach is that more people would be involved that two or three agencies try to broker a solution between themselves.


Audience Question: Have you ever worked at a child fatality review panel? Do you see your ideas working in a relationship to various conviction review units with the prosecution's office or The Innocence Project and it sounds like there's nothing off the table? They can potentially be your partners; did I hear you out correctly? 

Jim Doyle: Yes, that's true. All of the people you mentioned are people we talked to very thoroughly about this kind of issue. There are lots of precursored kinds of things going on and if these things were all added up together and given a name, well it would look a lot like bigger than the sum of its parts right now. There are child death fatality reviews in places, elder death fatality reviews in some places. Opioid death fatality review in places. In all of these things, the criminal justice system is implicated and placed a role either by doing something or needing to do something. We talked very thoroughly with the Innocence Project people about their efforts and too many prosecutors’ offices about the conviction integrity units. Again, the focus here is on the forward-looking aspect of things. This would be an important compliment to the work of prosecutors office that is doing the work of looking back to find specific individuals and doing the emergency work of getting those people released. There's still work to be done about how to disperse and get wrongfully convicted in the first place. We are in touch with a very broad array of prosecutor's offices who can see that they want to get involved in that kind of event review once they have gotten past the immediate performance review of what happened in this case.



Audience Question: Is the eventual goal of this research is to create more guidelines for individual departments and agencies? Or will the guiding principle remain to be empowering jurisdictions to figure out how to best solve their own problems? 

Maureen McGough: I don't really see it as an either-or. In order for sentinel events reviews to work, we believe very strongly that each jurisdiction needs to come up with their own plan for implementation and how to track recommendations out of that implementation and make sure that it actually reflects the reality of their jurisdiction. That being said, we also believe that it is very helpful to jurisdictions to think through how other people have done it and lessons that have been learned. Maybe neighboring jurisdictions with similar characteristics. So, there's just another data point for them so that they don't have to start from scratch necessarily. I don't think we'll ever get to a point in this process where we come up with a sort of a handbook where there are boxes you must check in order to do a sentinel review. I don't think it'll ever work quite that way.

Jim Doyle: One of the things that the central maxim of people who do safety work is that they would say that essentially nothing is ever completely fixed. It's good to work on new guidelines, it's good to work on best practices but the minute you institute a new guideline or a new best practice, it is immediately under attack by its environment. The caseloads go up, the money goes down, people change, the law changes. All of these things are dynamic. They're never fixed in the way you wash your hands and say we handled that problem. What we're hoping is that there will be a way here for people to come up with a capacity to answer the challenge when some new event challenges the existing set of best practices or challenges the existing checklist. So that you would at least have the capacity to say why did that happen? Why didn't our checklist prevent that from happening? There are a lot of situations that arrive in criminal justice that aren't covered in advanced by the checklist. There's the famous US Air landing of Sully Sullenberger in the Hudson River. Well, you had a checklist but the checklist didn't cover that event. So, to understand what happened there, you have to understand the kind workmanship he displayed and why you would want to train the next pilot who comes along to be able to do that and to improve your checklist or amend your checklist or modernize your checklist. It's more of the ability to as an ongoing practice be able to keep looking into these events and keep reviewing them rather than something that has an endpoint where you produce a product and then you all wash your hands and walk away from it. No product that you come up with is going to be permanent. All of them are going to require that ongoing work of the people. From my point of view, maybe this is to feature my biography but I think the people at the front lines who are going to tell you that and provide an early warning system for when your checklist is going out of date or has been made unfeasible by some new checklist that somebody else from some other agency set up for their people to operate under.


Click here to watch a recording of "Learning from Error in Criminal Justice: A Sentinel Events Approach."

Additional Resources
2 years ago
Thoughts on Maximizing Human Resources from Gary Cordner
Love Gary Cordner's reminder about the importance of leveraging every employee to their fullest pote […]
3 years ago
LEADS Scholars and Agencies: an Interview with Gary Cordner
Evidence-Based Practices have become a significant part of conversation throughout the justice commu […]
3 years ago
Scottsdale’s Restorative Justice Intervention Program Case Study: An Interview with Cassie Johnson
Diversionary programs to keep kids out of the court system are important - but they're only as succe […]
3 years ago
Strategic Planning as a Management Philosophy: An Interview with Will Davis
While strategic planning might seem dull in comparison to what law enforcement officers do to protec […]