J.D. Mosley-Matchett, PhD (00:40) Hello and welcome to another episode of AI Update brought to you by InforMaven. Today's guest is David M. DiSabito Jr., a Professional Educator of Business Analytics and Information Management at Western New England University, where he serves as the AI Liaison to the College of Business and chairs its AI Working Group. His extensive involvement in Assurance of Learning teams and AACSB accreditation efforts underscores his commitment to advancing assessment practices within higher education. In collaboration with cross-disciplinary research team at Western New England, David has explored the integration of generative artificial intelligence into academic assessment. Their findings, published in New Directions for Teaching and Learning, highlight the potential of human-AI partnerships to enhance assessment methodologies. The team has also shared their insights at various assessment conferences, engaging with professionals across diverse disciplines. Welcome to the podcast, David. David M DiSabito Jr (01:50) Well, thank you, JD. pleased to be here on InforMaven. J.D. Mosley-Matchett, PhD (01:54) been hearing about the AI tool you created and I can't wait to learn how it can revolutionize assessments in higher education. It's called Walter, right? David M DiSabito Jr (02:03) Yes, it's called Walter. J.D. Mosley-Matchett, PhD (02:05) So how has this AI tool, Walter, been developed? Can you share the journey behind how you created it? David M DiSabito Jr (02:12) Yeah, well, it's actually a fairly long story, so I'll try to keep it short. J.D. Mosley-Matchett, PhD (02:16) Okay, David M DiSabito Jr (02:19) The impetus came from the fact that I was teaching a Python programming course and an R programming course, and I was on our Assessment of Learning team to get prepared for our AAC SB accreditation. And I had been in conversations with our Dean, Dr. Sherri Ann Walker from Western New England University. And since November of 22, when the first Chat GPT became available to the general community. And our conversation continued. She basically kicked off the assessment team's 20-23 cycle. And she looked at me and said, it'd be really cool if David could write a program using generative artificial intelligence to do this assessment drudgery work for us. That's what spawned the idea of And a few weeks later, our provost, Maria Toyota, got a hold of the idea. So they really pushed me to develop this. And I was pushing myself because I wanted to develop it. So was kind of a lot of fun. J.D. Mosley-Matchett, PhD (03:34) That sounds great. So what challenges did you encounter during the development of Walter? David M DiSabito Jr (03:41) There was tons of challenges. One was just getting to talk to the OpenAI applications programming interface. So just making the communications. And the other thing that's kind of interesting is Walter can work with multiple APIs. I'll talk a little bit about that later. But just getting the communication process open. Then determining, you know, how do we want to prompt the artificial intelligence. At the time when we first started, it was text only world. And a lot of the student artifacts were coming in like Microsoft Word and PDF formats. So getting the prompting set up and then like trying to format the output because the AI at the time was not very good at formatting outputs. It was providing information not as quite a report card. So to get it to basically fill out a report card was, that was a the big challenge. So I think we've accomplished that. J.D. Mosley-Matchett, PhD (04:52) So, why did you call it Walter? David M DiSabito Jr (04:56) Oh, that's a very long One of our advisors board of directors is Polish, and there's a famous Polish bear part the bear rescued in World War II when Poland was occupied and the bear's Polish name, I'm not sure pronounce it right, was Wojcik and that translates I think to Walter English. And the bear was very cool. It was a morale booster. The bear, I believe it actually earned the rank of corporal. So it's kind of a crazy long story, but Walter helps build morale and that's how it got its name. And I wasn't even really that attached to it, but now I love it. It's become part of J.D. Mosley-Matchett, PhD (05:53) Okay, well, let's move on to the second question. How can AI enhance academic assessment? Does Walter actually enhance the process of assessment? David M DiSabito Jr (06:05) believe it does and it will. Currently at Western New England University, we have not used it for assessment, but what we have done is we've looked at prior assessment data, student evidence, and run test runs with Walter looking at how humans assessed the data and how Walter, the generative AI, assesses the data. And we've been able to build some confidence can do a good job. So, for this year in 2025, we're going to include Walter as part of our assessment process to basically remove the drudgery of having a second reader assessor read, 100 papers. Walter can read and assess 100 papers based on learning goals, learning objectives, 100 five-page papers in one minute. So, our idea is to have the assessment team look at the data and the results, not necessarily sit there and have to dredge through 100 papers and rank them. J.D. Mosley-Matchett, PhD (07:16) So what feedback have you received from educators and the students too, because they're important stakeholders? David M DiSabito Jr (07:23) Well, the main idea, Walter's really got two sort of audiences. The main audience that it was designed for is institutional The second audience is instructors. As far as I know, I'm the only one that's used it for student. I've been very transparent with my students. We actually got the students' permission to assess written work using Walter and done that in the fall of 23 and the fall of 24. So, I've got a data set of about 140 student artifacts that have been assessed with Walter. So the student feedback on it is the students loved it, but I have to sort of make a qualifier on that. And the qualifier is that I made a deal with the The students, didn't just want non-human, to evaluate their work. of their work and they wanted me to be in looking at and evaluating what they'd done. I used the rubric grade them. And then we had Generative Artificial Intelligence also grade them. So, see some discrepancies and overall the human was slightly more generous in the grading. But the deal that I made with the students was would get the best grade. So sometimes Walter would give a better grade and sometimes the human myself would give a better grade. And I awarded the students the higher of the two grades. So, liked that idea. J.D. Mosley-Matchett, PhD (09:01) All Yeah, I can see that. David M DiSabito Jr (09:06) Faculty on the other jaws drop. basically when they see the speed of what Walter can do. And then it comes to a matter of trust. Do they trust the generative artificial intelligence or not? the tricky part. That's the part that requires a lot of energy and effort and work. Now, just to share a really cool the first time that I set up the prompting, for that project that I just mentioned, took me a long time to get results back that I could trust. But the second time, it didn't have to do anything because I'd already done all that work. So the first set of about it took me hours and hours, probably as long as it took me to physically the papers. But the second time, it took like less than a minute. So pretty Pretty amazing. J.D. Mosley-Matchett, PhD (10:01) That's a great return on investment. David M DiSabito Jr (10:04) Yeah, and then if I do it again in the next semester, it'll even be more return. J.D. Mosley-Matchett, PhD (10:11) So with respect to this element of trust, how do you develop that with a tool like Walter? David M DiSabito Jr (10:23) Well, by playing with it. you know, unfortunately, we don't have enough people playing with it yet. So, it's actually such an important topic that we're going to tackle it in a paper with some of the other faculty in our College of Business at West New England University, where I think trust is going to be part of the title. J.D. Mosley-Matchett, PhD (10:30) Mm-hmm. David M DiSabito Jr (10:45) What we've done is we're expanding on a paper that's already been written by Janelle Goodnight and May Low. And what they did is they looked at the importance of a second reader in trying to conserve resources in the assessment process. What they had folks do was the instructors used the same rubric that the assessment team used on their student evidence. So they basically got two columns, the instructor and the... assessment team scores and they wrote a paper and did a study on that and we're extending that paper gonna add in the artificial intelligence using that same rubric and have assessed all of data that they had done in their study. gonna be eye opening to And rubrics are such an important part of the process. One of the superpowers of Walter is you can adjust the rubric and run 100 five page papers in one minute so you can see the results. So it's a matter of J.D. Mosley-Matchett, PhD (11:43) Yes. David M DiSabito Jr (11:54) getting an objective rubric versus a subjective rubric and basically spot checking and testing and making sure your results look appropriate. J.D. Mosley-Matchett, PhD (12:05) So when you said that it took you hours the first time you used Walter, and then just a matter of a minute or so the second time you used it, what took so much time the first time? And is this something that someone who's non-technical would be able to accomplish? David M DiSabito Jr (12:30) great question. And we're bumping up that sort of as we speak. I started with about a dozen, researchers and only about three of them are currently really digging in. They're on the cutting edge of investigating the generative AI. The fact that it is a bit of it difficult So that's one of the challenges. We're trying to come up with ways maybe create rubric wizards and things and rubric best practice things that we're gonna maybe stick into Walter later. What took me the time, the first time, was getting rubric together that was objective and not subjective. And actually, I'm still, I could still tweak it to make it even more objective. The thing is, between even humans don't like subjective language. And the lens that every single human, you know, looks through is different, their life experiences are all different. And the lens that the Gen AI looks its vacuuming up of the internet is so vast that subjective language be desired if you're looking for assessment results. J.D. Mosley-Matchett, PhD (13:49) It's true. But when you're talking about a high-stakes situation like grading, you have to be as impartial as possible. Even if it's hard. David M DiSabito Jr (14:02) Yeah, that's the thing that's very, very interesting is, Walter does not get tired. Walter is like a marathon runner and us normal humans, we get tired pretty quickly. You know, even, five or 10 five page papers, we're starting to feel it. Professors all have these the students, they're bright and early every single class, their assignments are turned in on time, they're attentive. And then you've got the other student who's absent 30 % of the who looks like they're are you know, has the bias of the data in its training set, which is sort of human but a lot of folks are looking at that. J.D. Mosley-Matchett, PhD (14:52) this notion of data privacy and compliance. Can AI really ensure that? Given the importance of data privacy, how does Walter ensure compliance with regulations like FERPA and HIPAA? David M DiSabito Jr (15:08) Well, a really good Data security is so important, but at the same time, you always hear of situations where data security has been breached. And truth is you can do a very good job of trying to protect data. There is like security measures that you can take to protect data. But in the end, if there's a really brilliant hacker on the other side of So that being said, there is no true security, I think, when it comes to data, unless you go into a hole, into a cave, and let no one else in the world or any electronics penetrate you're doing. that's kind of a qualifier. I think it's worth mentioning. the developer of Walter and OpenAI have entered into a business associate agreement that has a zero data retention policy on prompting. So the prompting is used by the GenAI to produce the result and then the prompts are deleted. So we've entered into that agreement. We haven't entered into that agreement with other providers of generative artificial intelligence like like Anthropic and Google Gemini, but we are looking into how that can be done. One of the problems is that the gen AI is so popular it's like next to impossible to get support for it. So we're very excited that we actually do have a business associate agreement with OpenAI that allows us to protect the zero data retention. Another side of this is we're actually considering developing our own AI models and protecting them at Western New England University. We have a FinTech incubator, which is a stack of high powered processing GPUs we can use and secure. So that's another avenue that we're looking at in So the bottom line is that at AI Integrated Concepts with our product, Walter, the end user, the administrator. So we set up schools and your school administrator has complete control. of the data. do not save the data. We do use the data. We do not share the data. school administrator and the end users themselves have full control. So that being said, from what I've understand about HIPAA and FERPA compliance, as long as they maintain control handle the security on their side, this be help clients maintain FERPA and HIPAA standards. J.D. Mosley-Matchett, PhD (18:13) Okay, that sounds reassuring. So moving on to the next question. How does Walter impact faculty workload and efficiency in grading and assessments? David M DiSabito Jr (18:26) Well, That's a really good question. The reason I'm sort of chuckling is I think the answer is we're going to find out. I can I can talk about myself and I've already kind of broached that subject with already in the program today. It does take a lot a lot of work up front, but after when once once you build that J.D. Mosley-Matchett, PhD (18:33) Hahaha David M DiSabito Jr (18:49) and have a prompting project set up can go from 105 page papers down to one minute, and you know how long it would take to read and assess and grade and score 100 5-page papers. So the idea is to really allow faculty to focus more on their students J.D. Mosley-Matchett, PhD (19:02) You David M DiSabito Jr (19:13) particular needs and be more student to allow the to think more about curriculum their pedagogy help students learn. J.D. Mosley-Matchett, PhD (19:26) Okay, when you're talking about cutting down the time that instructors have to spend in the assessment process and allowing them to redistribute that time into other more important elements of teaching and learning, one of the things that people are worried about is AI replacing them. Have you run into any concerns from faculty that grading is supposed to be something that's part of their understanding what the student does and doesn't understand? And if you turn that over to a machine, they're losing that element. What would you say to that kind of an argument? David M DiSabito Jr (20:16) off the top of my head J.D. Mosley-Matchett, PhD (20:18) Mm-hmm. David M DiSabito Jr (20:19) my own experiences. any professor would tell you, you know your A students grading is so much easier than your C or D students that make your head want to explode. Just being honest. J.D. Mosley-Matchett, PhD (20:36) yeah. David M DiSabito Jr (20:38) What Walter does is it virtually finds A's and it immediately will find those C's and D's. So then you can look at your rubric criteria and scoring because Walter will give you basically a report card how the student scored based on your rubric. And you can see what's going on with a student from there. And if you're concerned, which you should be, you could go back to the original artifact and look and see if Walter is correct. And like I said, it's about a trust thing. I'm at the point where I've done enough experimentation that I pretty quickly can trust the rubric scoring. And then it gives an evaluation based on the program actually goes through a second time and looks at the rubric scores and re-exams the artifact and makes recommendations. So it's pretty cool. think what it's doing is it's giving you more time to think about that. J.D. Mosley-Matchett, PhD (21:38) Okay, I like that. And it's almost like having a graduate teaching assistant. And it may take time to trust the teaching assistant, but eventually when you do, it's the same sort of opportunity for allowing the instructor to focus more on the teaching aspects instead of the assessment David M DiSabito Jr (22:01) Yes. J.D. Mosley-Matchett, PhD (22:02) Why is it important for AI developers to collaborate with higher education professionals? David M DiSabito Jr (22:09) think it's super important. It really requires all parties, you know, definitely stakeholders. And even going beyond that, up to the level. accreditors have stake in keeping their institutions moving forward in a continuous improvement cycle. The administrators are very interested in making sure that the students are learning and that there's evidence that they can look at their learning. The deans want to make sure that their colleges are meeting standards or trying to improve their standards, and then the faculty themselves and then the students. So it's important collaborate with each piece. Just defining what are the learning goals? What are the learning objectives? For instance, the employers that you're trying to get your students work ready for. So it's super important. And it's been a lot of fun demonstrating Walter to all of the stakeholders. J.D. Mosley-Matchett, PhD (23:21) Speaking of demonstrations, where might our audience members be able to check out Walter and see what Walter does? David M DiSabito Jr (23:31) Well, currently we have some ideas on board in regard to that. What folks have been doing is contacting me directly and I'm hoping that it gets out of hand J.D. Mosley-Matchett, PhD (23:43) Ha ha! David M DiSabito Jr (23:46) I have had lot of inquiries as late. I've been asked several times to sit on panels. I've been asked to go to conferences, have been asked to do virtual webinars. And I have done many personal demos for educational institutions across the US. Yeah, it does cut into my schedule, but at this point in time, since it's in its embryonic stages, I'm able to handle it. But I'm thinking it's getting kind of overwhelming even as we speak, and I will need some help. I'm thinking of doing maybe a YouTube channel some other stuff as J.D. Mosley-Matchett, PhD (24:26) Maybe you need to get one of those intelligent assistants. David M DiSabito Jr (24:30) You need all kinds of help, J.D. Mosley-Matchett, PhD (24:37) Don't we all? Don't we all? Any parting words for our audience that we haven't touched on? David M DiSabito Jr (24:44) want to thank you JD and thank InforMaven for this opportunity to talk about Walter. I think the general interest artificial intelligence and the topic is so hot. I'd be interested in working with members of your audience and to continue the conversation. J.D. Mosley-Matchett, PhD (25:05) That sounds great. Well, thanks so much for sharing your insights with us today, David. It's so exciting to see practical applications of AI that make a real difference in higher education. David M DiSabito Jr (25:10) Yeah, my pleasure. J.D. Mosley-Matchett, PhD (25:19) So for more information about AI news and trends that are directly impacting administrators in higher education, please follow InforMaven on LinkedIn and visit our website at informaven.ai. David M DiSabito Jr (25:20) Thank you.