After the Webinar: Police-Led Field Experiments. Q&A with the Presenters

Webinar presenters Jeremiah Johnson and Jason Potts answered a number of your questions after their presentation, Police Led Field Experiments: Lessons from the NIJ LEADS scholars Program.  Here are just a few of their responses.


Audience Question:  You have the terms prime and not prime on your slide. Can you kind of explain that notes again?

Jason Potts: So there’s this thing called dispatch prime, a phenomenon where if you’re primed or brought in a dispatch broadcast presence with a gun, our officers are more inclined to shoot when they are primed with the presence of a gun and their cellphones, for example, is aggressively pulled out. So that’s that. Not priming is there’s no gun mentioned in the call. It essentially means the gun with mentioned in the call and that the person that they’re gonna go to contact had a gun or been told they have a gun. There’s a lot of that going on in policing where they call swatting in. They’re trying to get a police response by saying that they saw a gun or somebody has a gun. We see that in our jurisdiction.



Audience Question:  Gary, you mentioned that campus police can participate in the LEADS. What about tribal police? 

Dr. Gary Cordner: Actually Tribal Police are also eligible. I don’t think in the first five years we’ve had any representative for the Tribal Police but I expect this year we will.



Audience Question:  What’s the best way for us to learn about other LEADS research programs and their outcomes?

Dr. Gary Cordner: If you google NIJ LEADS Program, there are several webpages with a bunch of information on them and I would mention that part of also what’s available there is the information about applying to be a LEADS scholar. The application process usually opens up one time a year. Usually in May, and there’s about a developed month-long period with the windows open. It closes sometimes its June and then usually, the questions are out in July.



Audience Question:  Jeremiah, how did you collect the data from the officers? Did you give them a sheet of paper for tracking or how did that work?

Jeremiah Johnson: In the research that involved several different agencies, essentially there was a person designated in each organization to go through and collect the data for each shift. Our unit of analysis was the shift. Individual officers did not have to go anything out with themselves basically an administrator in the department. In one agency, they had a civilian crime analyst, collecting the shift data. In my agency, it was me and I think it was an administrative lieutenant in another organization that’s handling it. But yeah, there was one person that goes through the shifts and collects and collate all the data.



Audience Question:  How did you determine the link for the duration of the study the 122 days, the 61 days testing versus the 61 days of control, how did you determine that duration time period? 

Jeremiah Johnson: Certainly that’s a really good question because of one of the things that Jason mentioned the limitation of his study with what they call the Statistical Power. If you don’t have enough observations that can still get something called Statistical Significance, but you may not have a strong power. You’re just basically a Power Analysis based off of historical data that we look that. I forget how many years we look back but we look at the volume of the motor vehicle burglaries and auto-thefts that these different jurisdictions had. Then we basically made a prediction based off on past data to basically surmise how long we would need to run the trial to get a sufficient number of observations.



Audience Question:  This is really more for all of you but the question has come up a couple of different times throughout the presentation so you all can decide who you want to answer. There is a question about statistical significance and for those of us who aren’t statisticians in the audience, you mentioned that your study results weren’t statistically significant in one of your research study instances. The numerical differences seemed to be pretty large to the average person. Can you all put in layman’s terms what is statistically significant? What does that mean and why does it matter? 

Jeremiah Johnson: So just very simply statistical significance is measuring the likelihood that your results are being obtained due to the particular phenomenon and not by chance or other variables or factors. That’s the core synopsis. I’ve got an article in a publication that’s out there on the internet called Translational Criminology. It’s called key values for practitioners. I gave several examples of what statistical significance is and isn’t and I think in a lot of our research, the actual aggregate numbers are quite small and so small changes in those small numbers look like large percentages but in reality, when you run it through certain tests, statistical tests called hypothesis testing, some of those observed differences sort of disappear when you’re trying to make a connection the connection as to the success of the intervention.

Dr. Gary Cordner: If I could just add to that. There is actually quite a debate in the statistical and research world about whether we may be putting too much emphasis on statistical significance and we’re reporting studies and deciding whether they really mean anything or not. I think Jeremiah’s example from Shoreline is a perfect example of that because that is as your attendee pointed out. Some of the differences that we found between lights on and lights off were actually pretty big. Percentage-wise, because of the raw numbers were small, to begin with, they didn’t achieve that level of statistical significance. As somebody like me who’s not probably as much of a purist, we probably say that sure looks like something happened. You just gotta run that for a couple of months longer so the raw numbers would be a little bit bigger. It’s probably would’ve been statistically significant.



Audience Question: What are the cruise lights? Could you all define when you said you were measuring cruise lights? What does that mean again? 

Jeremiah Johnson: Sure. So the cruise lights term is something that maybe it’s just specific to my agency. I’ve heard it refer to as other terms “metro lights” or other things around the country so it basically refers to in my jurisdiction as static red and blue LED lights on the patrol. These are not the flashing red and blue emergency lights associated with traffic stops or accidents but it’s just a steady glowing light that’s observable from a great distance. I think in Jason’s jurisdiction it might be a little different. Like I said, not all law enforcement agencies have this particular set up.

Jason Potts: Ours are called Code-2 lights. So essentially, they’re flashing red, blue and amber lights. Obviously, there’s difference having those lights on in the daytime as opposed to nighttime but yeah, it’s different for us.



Audience Question:  How did you get your leadership to buy into the idea of doing research about your agency’s practices and running these experiments and maybe more importantly, how do you convince their supervisors that doing research like you’ve done is actually a good and important thing and isn’t a waste of time? Talk us through that element, Jason, Jeremiah, how did you approach your leadership, how did you convince them? What were the objections? How can others replicate your success in getting to do research? 

Jeremiah Johnson: I think that I mentioned that I have a Ph.D. but being inside my own organization, I don’t think that means a lot. What was particularly helpful with having an outside organization like BetaGov come in. The woman that loves BetaGov, the director, she kind of refer to it jokingly as the expert with a briefcase. So she kinds of functions as the consultant but they went off as paid consultant because they provide a pre-trial. I think a couple of things. Number one; there wasn’t any additional outlay in costs for this particular intervention that I was bringing in somebody from the outside who seem to have some expertise and knowledge was quite helpful. I wasn’t actually surprised because in graduate school that I’ve been told that police departments were resistant to research and originally my intervention was designed with just two departments and I have two additional chiefs in the area jump on the experiments so there is a group of four but interestingly we ended up dropping one out because there is implementation failure. We have support from the chief of police who’s excited about the project but that did not trickle down to other levels of the organization. They didn’t really play ball.

Jason Potts: I was just gonna add the same sentiments. I will also say you have to have some credibility currency when you bring forward these ideas and so yes. It’s free with BetaGov, and it’s on overtime. We want to look at it from a cost-benefit analysis too. This is all during overtime so lights on, lights out for us was during overtime same with the theft deterrence strategies. So you sell it that way and what’s in it for them. Kind of look at it inversely and think about it from a union perspective. Hey, these are other opportunities to test things, sleep deprivation, work conditions, you name it. there’s a ton of opportunities to test things. If you sell it that way and frame it that way, there’s a success. There was a lot of support for it. I think the younger cops actually understand it and get it. The older ones are a little bit more entrenched and they think it’s silly.



Audience Question:  Conversely, it’s not just the chief that you have to sell sometimes. Sometimes it is the lower levels of leadership. How did you get by in it? Sounds like Jeremiah you had an instance where you didn’t get implementation buy-ins from that one individual but they didn’t implement the studies the correct way. How did you approach it among the lower ranks? Was there a different process? Did you have to sell it? How did you do that?

Jeremiah Johnson: Well, in my agency I think they have one of the shift supervisors on the midnight shift so a good portion of the time I was physically there to make sure people were doing what they were supposed to be doing. It should really come from the above. One of the chiefs in the study he actually, during the four months, woke up in the middle of the night and took the time to drive down to his jurisdiction to make sure that his officers were doing what they were supposed to do. So yeah. We really see that level of commitment from the upper echelons making sure that message was pushed down through the chain of command. Strong support but I think that first-line supervisors level, it’s important and I just wasn’t able to be present in those other jurisdictions like I would’ve liked to have been.

Jason Potts: Chris I say that that’s the key to these things. Our Sergeants are the key to anything. If you don’t get Sergeant buy-in and none of this is happening so some of us, we empower our sergeants kind of made it theirs, their project and just you’re not going anywhere in policing without sergeant buy-in.



Audience Question:  Gary what’s the goal of LEADS? Is it to teach officers how to conduct their own research or is it to identify new and creative ways in policing? Is it both? 

Dr. Gary Cordner: It is definitely both. I think the larger objective is to try to develop the capacity within the police field to do its own research not to be so completely dependent on university professors or full-time researchers. I think it’s sort of the sense that we’re a developing profession and one of the things that professions do is produce their own knowledge and they don’t depend completely on somebody else to do it for them. NIJ LEADS scholars program in part is just trying to help build that capacity within the police field and then also help some really sharp individuals develop their own personal capabilities and get some professional development out of it. We get some cool studies completed that help produce some of that knowledge that then others can replicate or adapt in their agency.



Audience Question:  Where do things go here from each of you guys? Are your agencies implementing some of these practices? What’s the next study on the horizon all that good stuff?

Jason Potts:  As far as the lights on lights off, it’s probably gonna inform us to do it next year as well so yeah we’ve had some success but we don’t really have the capacity in our department to do this long-term again, these are on overtime details so that’s why they’re still short for me. I have 34 days and then it’s a 55-day study. They all form longer studies. The virtual reality what we’re working with BetaGov to come up with some other pilot studies or increased studies out there and maybe police academies or other jurisdictions to see if we can replicate the things we did and maybe to tweak a little bit to be a little bit more realistic.

Jeremiah Johnson: So the Cruise Lights study that I’ve originally conceived is currently being replicated in nine different jurisdictions, Jason’s one of them, in two different countries. They often contribute to the evidence-based. I’m hoping to get another study launched with looking a little bit at perceptions of legitimacy around press releases for officer-involved shootings looking at images that are shared with police departments and press releases when officers are named. Hopefully, we can get that trial up off the ground in the fall and that would be an RCT but done in sort of an experimental lab setting.



Audience Question: Gary, what should we be expecting from the LEADS program here in the coming months? 

Dr. Gary Cordner: So we’ll be taking on ten additional new LEADS scholars here probably. In a couple of weeks, they’ll be identified and as far as we know NIJ has every intention to continue this program on into the future adding ten each year. On the challenges for NIJ, I’ve mentioned that the three-year gig for the scholars, so NIJ is thinking real hard about that now how do they keep the alum. I think Jason and Jeremiah might both be alum. Certainly, Jason is. How do we keep them engaged and how can we continue to support them even though they’ve sort of rotated out of the official three-year program? A couple of those things I’ll mention is NIJ working on incorporating some civilian police employees into the program, up until now, it’s strictly sworn law enforcement officers. Also, looking into the possibility of supporting some early-career police researchers and that arises out of the observation that sometimes you know young PHDs that do have an interest in police research have never been taught how to go into a police department and talk to people and you know, build relationships and develop an understanding of the work and the profession and so, NIJ is looking for ways to try to help groom and help develop young researchers as well.


Click Here to Watch a Recording of Police Led Field Experiments: Lessons from the NIJ LEADS scholars Program.

Additional Resources
5 years ago
Contemporary Police Responses to Addiction: Lessons from NIJ’s LEADS Program
Opioid use and abuse had been declared an epidemic. It does not distinguish between race, gender, ag […]
5 years ago
Implementing Evidence-Based Policing: Lessons from the LEADS Program
Evidence-based methods have been utilized in medicine and sciences for a significant time and it has […]
5 years ago
Thoughts on Maximizing Human Resources from Gary Cordner
Love Gary Cordner's reminder about the importance of leveraging every employee to their fullest pote […]
5 years ago
LEADS Scholars and Agencies: an Interview with Gary Cordner
Evidence-Based Practices have become a significant part of conversation throughout the justice commu […]