ERP130 - Project Delays and Failures, Breaking the Cycle

Mike, welcome to the Evolved Radio podcast. Thanks for having me, Todd, great to. Have you back as well. A returning guest. I think your previous episode, Advanced Project Management, it was episode 104. If people want to go check out some of the fun stuff that we talked about there. I want to start us off with basically my favorite stat that I've been quoting kind of the last two years, two stats actually, because I still feel like this is something that people are not alarmed enough about. So I will continue to talk about this. So the first stat, one of my favorite stats in the industry, people have probably heard this because Paul Dipple has been talking about this for a long time, is that a full 25% of the industry and the MSP channel is break even or losing money. So alarming stat enough to begin with, right? Everyone's kind of heard this probably at some point. The really interesting part of this was service leadership dug a little deeper and in the data they found that in that 25% that is struggling, most of them actually have profitable services. The problem is, is they're so bad at project management, they're cannibalizing all of the profits from the other departments of the business. And when I saw this, I was like, this absolutely makes sense like this. Why are people not recognizing this? They continue to sort of fight through service and automation. Projects still tends to be a bit of an afterthought in a lot of immature MSPs. And I don't think people recognize the damage that it's doing. And you guys obviously have a bit of a front row seat to this. I'm curious, sort of your take on, you know, how true you find this to be and what do we do about this? Both in promoting the idea of projects is how you fix your MSP profitability and then how do you go about that, I suppose. Yeah, yeah, absolutely. And it's a great question. And we know that statistic well, see that and kind of advise organizations around that. I guess there are a couple of components and this can kind of be split into several areas of sort of why does this happen for businesses? One of them is organizational maturity, right? A lot of MSPs as you know, even self describe, are accidental entrepreneurs. The way their business grows up and what they're doing. And so this idea of sort of disciplined project management around that process wasn't necessary if they were a single engineer or growing from small process and kind of moving forward because, you know, the human brain is amazing at managing a certain number of tasks, process and sort of Quote, unquote, the critical path in your head. They know the things to do to execute process, and when they're doing it on their own, there's no repercussions. But there's a point that, that business scales to a certain level where that no longer can be maintained in someone's head, how they're tracking and managing that. And so that becomes an issue organizational maturity. But this still happens even in a larger business where organizations struggle because of the scale of what they're doing and how they're tracking and managing all of their data. And so one of the big issues that we talk to organizations about is as they go forward on this is managing, how are you managing the critical path, your projects and how are you structuring your data? And for most organizations, they're just, I guess there are a couple of things we'll talk about structurally, how they manage projects. So it's sort of, it's sort of like you're saying, why is this program running badly? Well, because the code is bad, it's got lots of bugs in it. Would you ever expect software to run well that had buggy code in it and incomplete code? You're like, no, that's an absurd notion. But running projects that have really buggy plans that weren't complete and so it's aren't predictable, they aren't accurate, they're managing that. And I don't think that for a lot of these organizations. One, the technology that exists in the PSAs that they use don't have super robust project management platforms. And so they're managing shared lists. And lots of times when they use third party tools, certainly non integrated third party tools, they're using like shared lists that aren't reconciling with their psa. And this can become a really dangerous, risky thing. And so once again, the surprise is why are these projects going badly? Well, then there's the next thing, right? So let's say you had, even if you were managing the data in a third party system, to overcome the limitations of your PSA and you had really capable PNs in there. The thing that nearly everybody doesn't realize is that statistically for your project you're going to have problems. So the statistic that we'll often quote is a project plan that has 40 tasks on the critical path. You know, the dependent chain of tasks. For those listening who are not familiar with it, the critical path is, you know, this task is dependent on this, is dependent on this. It's that longest chain that defines the minimum length of that project and there could be 100 or 200 tasks in the project. But if there were 40 on the critical path, if everybody got their work done on time, 90% of the time, that includes the customers and their responsibilities and distribution. Let's be real. And I think we all agree that's a ridiculously high number. Sorry, Mike, I cut you off like, oh, can you restate that? Yeah, yeah, I was just saying so. So I think we can agree that's a ridiculously high. Like I don't think any MSP would go, yeah, my engineers and everybody gets their stuff done on average 90% of the time. That's a high number. Self reporting in. But even if they did get it done, there's only a one and a half percent chance that that project would be on time. Even if it was beautifully built, beautifully structured, with all the detail in the world, the odds are horrible. And these are the same odds. To put it in the context that msps can understand, a task in a project is like an endpoint you're managing. It has a failure rate. And by the way, that task has a much higher failure rate than any endpoint you're managing. It just does. That failure is, oh, the parts didn't come in. I didn't know I was supposed to do it. I had to go to the customer site and it got rescheduled. So once again, you think about the ecosystem that you're managing. You have a lot of endpoints out there, a lot of tasks. And for some MSPs, they have as many tasks and projects as they do endpoints that they're managing over the course of a year. Right. But they get surprised and they think that everything's going to go off perfectly and the failure rate's higher. So they're like, you're telling an MSP, what are the odds you're going to go a couple of months with not a single endpoint problem? They would laugh at you. But they don't realize the failure rate is so much higher on the task. So that's the other part of this is that the, I'd say not just MSPs, the whole world doesn't realize all projects statistically are doomed to have problems. Now, what are you going to do about it? So hang on, let's pause on this because this sits on one of my other sort of favorite stats of project management in IT is that 65% on average IT related projects are a failure. Right? Which is a pretty shockingly high number based on what you're saying around like the failure rate of projects. Do you think that that's more global across projects in general? Like projects as a whole are fail. Are tend to be failures or is the inherent complexity of it projects that leads to a higher failure rate? Yep. So, and I'm going to differentiate between delay and failure. Right. So there are lots of projects that are delayed, that are ultimately successful, successful projects, but they were delayed. So when I say there's a one and a half percent chance that that project will be done on time as planned. So you've got to realize we're going in this, we're going to have delays. Now if there's a drop dead date, you better know that you're going to be scrambling and you know the term crashing that project because you're likely to have problems unless you're at the 99 percentile or 99. Like the probability is just bad. So, so that other statistic gets confounded by the fact that there are delays and risks and those projects get killed because confidence is lost. One of the big things that we always talk to a lot of our customers about and I think one of the interesting things that we learn improving to the point of the benefit to improving project management is not only the margins, which is really, really important atomically when you look at a project and go, hey, we were profitable on this project. But the other really super important thing that we learned over the last couple of years that We've heard from CEOs of a lot of our customers was the fact of I have better credibility now with my customers, which means they're a better reference and they're more likely to renew. In the past, when I was missing dates which they didn't know they were doomed to have problems with and they didn't know when they were missing the dates the customer would call him and go, feel like you should have known about this, feel like you should have known about this delay. And by the way, if, if you didn't know we were going to be a month late on this, what else are you missing? And a lot of these organizations lost customers and big customers because they were. The customer went, you're making me stressed because you don't seem like you're on top of this. And they're, they were totally on top of their managed services. But it gave the appearance of. So there's a, like to your point, there's two really important reasons why do you want to get your house in order on this. One, don't lose money. You don't need to lose. But Two, this is your reputation and you know, we're an AI vendor and AI project management, blah blah, blah, and automation. But I don't care how much AI and automation comes into the MSP industry will be relationship based. I think it will always be relationship based. And if it isn't relationship based, it goes, goes away. I mean, so that relationship is based on credibility. And one of the, what I would love to hear, and I don't know how we get to this statistic is how much credibility damage is occurring. You know, you're saying, oh, 25% lose money and that's based on bad project management. What percentage of churn is related to the fact that they dropped the ball in a project? Not, not race to the bottom on pricing per node, not other things, but damaged customer sentiment because hey, you mismanaged some really important projects to me was really frustrating to me and my team and you know, maybe if you'd communicated better, we wouldn't be so mad. And that was the other issue. We'll tell clients you're not going to eliminate delays. This idea that you think you're going to ever prevent a delay with the world's best project, but you want to know early, you want to know soon, so you can call your customer and go, hey, we're going to have a problem in six weeks. Not oh, we had a problem a week ago and it's going to create a six week delay. Like that's what happens for a lot of these organizations. And so like I said, there's a two hit thing. You lose money and then you lose customers and you lose reference of all customers with bad project management. So yeah, this is, this is huge because like this is, I'm pulling out all the stats today. Like this is the other thing that I found really fascinating, especially kind of two years ago. Similarly related, like I was doing this talk all year about sort of efficient project management. So I was talking about these things ad nauseam all year. The other piece was that more MSPs are losing customers due to project failures than service failures. And that's a switch. Like that's not always been the case. It was usually service desks that caused that loss of trust. Of like I submitted a ticket two weeks ago, no one ever even called me back. Right. Like that stuff is what eroded that trust. Most people, I think of most organizations have gotten to a place where their help desk is pretty good. You know, there's obviously like a spectrum of people's capability of maturity on delivery on that side, but projects again, like it's just, it's really dropping the ball. And I've heard multiple stories from MSPS where they got completely blindsided by customer customers canceling their contract because of project failures. And they were, they were, they were totally caught off guard. So I think again, like that's a really important reflection of like this means a lot because now, now not only you more, more likely to lose money because of projects, you're actually more likely to be fired because of projects and lose customers as a result. Yeah, it's a double hit. Yeah, it's a double hit. All right, so I want to understand a bit about what we do about this, right. Because I think there is a general baseline like, like what you're talking about, I strongly believe in is that there are just fundamentals that people are not putting in place in managing projects. Right. Like a lot of this just happens off the side of someone's. Because you know, the tyranny of now, you know, I do this whole band, busy campaign and everyone excuses the fact that the stuff doesn't get done because I'm busy. Right. And that because of these reasons, that's unacceptable. So like where do you start? Right. I guess is the important aspect of this because sure, like I'm always, I would say I'm the first to admit the psa. Most of the project modules in there are okay at best they're not great but as you say, don't go into a third party tool because that's not the problem. It's the fundamental systems that you're putting in place, the expectations of how these projects get scoped, implemented and managed. And that doesn't require a lot of rocket science. Like that's just the basic tasks of this stuff. And then you can layer on the complexity on top of that and make things, you know, spin tops and do amazing things and automate stuff and. But you need those fundamentals in place, right? Yeah, yeah. And I guess, I guess a couple of things in that regard and just at least from, and everyone has different sort of industry perspectives and I, and I hear what you're saying as far as, hey, what is the need? What is the baseline? It's, there's not one quick solve for it. And it depends on the size of the organization. Right. And what are their problems and their challenges. The larger organizations will say for sure it is a technology problem. The PSAs are inadequate to do that and manage the scale when you know, we have customers with 3,500 projects in their portfolio with projects of a thousand tasks, you know, individual Projects that have a thousand tasks in them, the PSAs cannot manage this in any way, shape or form and it becomes a dumpster fire. Now, I will say using a third party tool that is not integrated, you're. The costs and the chaos of that are a huge problem. Right? So having a third party with swivel. Chairing between two systems, you have no data integration, right? But like layering something on top of the psa, I'm still a fan of that, to be clear. Yep, yeah, yeah, we have customers that come in and go, oh, I don't want to do the integration. We're like, yeah, you do, you do want to do the integration. Because you're just not going to want to deal with that. And, and having the integration where it's not just the tasks and its completion, but the structure, the product process, the tickets, the time, the notes, all of that, the custom status is the role, all of that stuff has to go back and forth because you want your engineers looking at the PSA is a source of truth for their work, right? You have your PMs and you got engineers floating in between regular ticket work and project work. So, so having a place where accurate data can live is critically important. Once again, I'm not saying anything that. And I will say the PSA vendors and the heads of products have come to me and talked to me about this, going, hey, we know this isn't kind of great in our product, but that goes into resource management isn't just the project management because what's important to the MSPS dates isn't just about the project delivery dates, it's about effective resource management because that impacts the margins, right? So, yeah, automating scheduling. So I'll say a couple of things, right? People do things that they have to do and want to do. They don't do things they should do. So you and I can sit here all day long and tell people, you should do this, you should stop smoking or stop drinking or lose weight. And nobody will do that unless they want to or they have to do it. So the first thing that any MSP listening to this has to look at it and go, like, if they're listening to you and I and they see us as pontificating about best practice and what you should be doing is not going to do it. Don't waste your time. But if they go, hey man, I am tired of this, like, I have to move to the next level. This is painful, I'm tired of the embarrassment. I don't want to waste money unnecessarily there's a real want there. And there's also a risk for them to sort of say, hey, we have to do this. And you know, if someone's not willing to kind of go, hey, we have to do this, then I do believe anything we say is a waste of time and they should just turn this off. Sorry, I don't mean to like reduce the viewership or even watch the rest of it, but if they are willing to do it, there are two things that we always say, hey, maybe you're not doing this today and you have to add these two data items that maybe you're not doing today. Like, so they're describing what the task is today, right? Hey, we've got to go install a firewall or we have to do this or we have to do that. And they're assigning a resource and maybe they're putting a work estimate in there. Like we think this is going to take three hours, that's our budget, four hours. And they know their cost. But two things a lot of them aren't doing. We say the double D is a dependency and a duration. You know what has to happen before this? Sometimes there are different types of dependencies, but they don't even have to go down that path. They have to give it a duration. Like, hey, I want you to configure this firewall. I understand it's going to take an hour, two hours, whatever it is, I'm going to give you five days to do it. That dependency and duration, the minute you do that in any project plan, you've now programmed work that unlocks a ton of automation. It also means your work can be debugged. It means you can autonomously look at resource capacity conflicts. You can automatically schedule. Like it unlocks a universe of efficiency. But if you go, I can't be bothered with that. My team, they don't do it today, they're going to find it to be a hassle. Then you go, okay, but just so you know, you are abdicating your ability to ever get out of this hell of the unpleasant surprise of why were we late on that? What happened with that project? Why did we lose money on it? You're foregoing automation. And it's almost like you're saying we want to be a break fix business. No, MSP would go, we love being a break fix business. And people who aren't automating project management are running break fix projects. That's what they're doing. And so if they're willing to add those two Other details, hey, dependencies and durations. If the technology can support it, they now have the ability to automate a lot of functionality, accuracy and predictive capability in the platform. So we always say like, that's a, for us, that's a pre qualifier. We're like, look, if you're not willing to do these two things, you're just not going to reap benefits that AI and automation can provide you in project management and you're just going to continue to get blindsided. So you have to do that math, figure out whether or not that's worth it to you. Well, this is the part, like, one of the things that absolutely drives me crazy about the way that we do projects in the MSP industry because we found this business model that we get paid regardless of whether or not we're doing work, which allows scale, right? Like the MSA model versus time and material. That's why margins are 60 to 70% instead of 25. And then we pivot to doing projects and go right back to the TNM model where we're just paying for hours, right? Like, how many hours did we do? What are we billing for? What's the labor rate on this person? And therefore how much margin can we make? The only way you make more money is by adding more people and more hours. So you're like, why have we given up on the MSP model for projects when we recognize how great it is for our businesses, right? I think like this, this drives me nuts of like not being able to plan, resource plan and understand sort of how this project gets implemented so that it is, it is more predictable. I think it's a call to arms for people of like, work towards this. This is not something you can do tomorrow. But these are the basic things that you need to start to put in, can evolve your process and make it more mature over time so that you can move from 25% to, you know, 40, 50, 60% projects. Well, and this is one of the things that also, you know, we talk about limitations of the tech that they may be using in the PSAs is this ability to analyze the performance of their projects. Like, so if you're going to go to a fixed fee model and we talk about this, you're going to do a fixed fee model, you're not going to have the time to do a retro. Like large organizations, when they have large monolithic projects, they do a retrospective and go, how did we perform? What did we do? What do we do? But msps, even the large ones, don't have the Bandwidth to do these retros to understand. Where did we misquote this? Where do we need to modify this? So once again, coming back to listing dependencies, durations and work estimates and being able to automatically analyze that template that you're using and this idea of productizing services to your point, right? So, hey, what is our service delivery offering? Let's create a template. Let's make that a product that we're going to offer. We're going to estimate the work hours. But then being able to understand after the execution of that project, hey, we got it wrong. We thought that was going to be two hours. It's actually six hours. Okay. When we bid this out again, we're going to bid it at six hours and, and knowing that ahead of time and because they're not doing retros, to your point, this is one of the areas why so many of these MSPs lose money in their projects. They're scoping because they have no ability. They're not big enough. They don't have to. Even the big companies can't do the appropriate analytics to dial that template in. And the other component is scope creep, that they don't have a structured process to capture new cost and then move that into the customer and, and you can't tell a customer. And this is one of our favorite statistics too, that we talk about is they'll identify cost overruns 60 to 90 days after when, you know, if they don't have an automated way to do that retro or identify the risk automatically. Once again, this is the benefit of AI and automation during project execution is you identify when it's coming off the rail. So, you know, if I was renovating your bathroom, Todd, and I came to you midway through and said, hey, you have rotted floorboards. We pulled the tile up. I know you want a new tile. You got rotted floorboards. We're going to have to rip that out and put it in, blah, blah, blah, blah, blah. And it's going to be an extra, you know, six grand. Like, but we kind of need to do this. You're gonna go, yep. Okay, now imagine, finish the bathroom. I bill you, and 90 days later, after billing you $38,000 to redo your. Bathroom, by the way, I replaced your. Four, by the way, we had to rip out the floor. I forgot to tell you, we had to rip out the wooden floor to do this. It's another six grand. You're going to go, no, like you were willing to pay it in the middle of the project, but you're Now, a hard no, because you paid that bill 30 or 60 days ago. And so this is another major reason MSPs get stung on this. And once again, if they're willing to make some process changes, and you preach this a lot, and I know you're trying to develop and elevate their experience for project management, but there are big dividends that they're willing to, you know, one, change their process a little and get the right tech in place so they can live real time, identify these risks, the delays, real time, the budget overruns, real time, because then they can adjust in flight. And once again, and I'll shut up here in a second, it goes back to that endpoint model. Like, do you want to find out about that, that noisy node or potential problem early or do you want to find out after it blew up and took the whole network down? And right now, once again, for most MSPs that don't have discipline around this, it's blowing up and taking the whole network down, I. E. The whole project down, that they don't have any kind of early detection or warning about that. And this is like, I think, like, this is full circle, right? Because I feel like the part of the reason that people are not able to manage those costs on the fly is that they're uncomfortable having these conversations with the customer around a change of scope and cost. Right? But they're. So, they're just absorbing it because they're like, well, you know, the customer will probably say no, maybe they're going to be angry at us. They, they sort of justify the idea that they should eat the costs on the project or the scope change or the overruns without sort of having a transparent business conversation about that. The problem is, is I think most people don't recognize the impact of that, right? Because their P and L has a blended service between help desk and projects, right? So it's like, well, you know, we're, we're doing okay not recognizing your Service delivery is 60%, but your project delivery is like 5 or negative 15, right? Yes. And the thing is, some MSPs are like, well, we're just not going to do project work anymore. And you're like, okay, so now you're opening the door for a competitor to come in and do that project well and take your business. So they're like, this idea that you can ignore this problem and it'll go away or just give it to somebody else is chording churn, right? Yep. All right, so one other bit I wanted to get, get your, your Input on is how much detail is enough. And maybe this is kind of a continuation of what we're talking about here because there is sort of this happy medium somewhere between like it just a gory amount of detail and everything has checkboxes and you know, every little, little unit is tracked. You know, been working with this organization. They do a lot of physical deployments and it has a lot of individual devices in numerous places. So there's a huge amount of information to track here. And they're looking to improve their project delivery process from where they were and they're getting good gains as a result of that. But they're getting a bit of pushback from or from the staff of just like oh my God, like this spreadsheet is eight pages long of things that we kind of need to go through and verify in order to make sure the project is complete. And like I'm of two minds of this recognizing like look, that information is important. This is just the stuff we need to do on the project. But if you're coming from a low maturity state and you're asking for high maturity delivery, then there's going to be a lot of friction in sort of how you move that ball forward and get the buy in from the team. Any thoughts on sort of a circumstance like that of like we're meeting resistance on trying to do the right thing because maybe we're asking for too much too early. Yeah. So the ROI is sort of a step function on this about the changes an organization's willing to make and the benefits that they're going to receive. But and there's a little bit of this. Don't. If you're not going to do at least this much then don't bother doing anything because it's going to be a bigger problem. So let's talk about detail in two different structures. There's the detail of number of tasks and then there's the detail within an atomic task. Those are sort of two levels of detail that you have to contemplate in this. So one of the biggest mistakes that we see MSPS make and we actually are head of operations who speaks a lot in the industry has a. I think he's got a webinar on it called the Project which is basically the entire project is one ticket. Like they're so used entering tickets and they just load the whole. This is clearly an insufficient level of detail for a large scale project and you're once again courting disaster by loading everything into a ticket. So there's the issue of what is the appropriate level of detail and you could do a whole session on what is the level. And people now could use ChatGPT to find out the appropriate level of detail for the number of tasks on stuff. So I don't need to cover that. But, but, but certainly more than one for a large project. And that's going to vary on the scale of the project, whether it's 20, 30, 50 or without. Like I said, we have, you know, customers that have over a thousand tasks, MSPs, you know, that do that for their customers. The second thing, so, so look and get some level of detail. But the second thing is what do you have to do on an individual task to execute function? And when we talk about sort of bare minimum, you need to know what the task is. I don't think anyone would debate, like you have to describe what that task is and have detail associated with that as a bare minimum, you have to assign that to a person. Like, you gotta do that, you gotta assign that to a person. And then for msps, because of their financial models and their budgeting, you have to put a work estimate on it. We think this atomic task configure this Device typically takes 22 minutes or typically takes an hour and a half. Whatever it is, you got to do those three things. And then to unlock automation and accuracy, you gotta have dependencies and durations. Like you gotta have those because if you don't, then you're gonna be in the blind, you're gonna be kind of blind again. You're, it's gonna be difficult. Now there's a lot of additional detail that you can add above and beyond that, right? A lot. But when, you know, when we try to coach organizations and go, hey, we know today, you know what the task is, you've already got that. You're doing that in your PSA and saying like, we got to do this. And you kind of know that list of things you should do. You know the resource type at least that you want to assign it to, if not the person, you know the resource type that you're going to assign it to and you roughly know how long it should take and if you get it wrong, you can update that. But once again, if you're willing to say, I mean, how long do we think we'll give someone to do this? Three days, four days, and what is it dependent on? You add those two things, you can unlock, once again, automated timelines process and updating, automating your timelines, because it's going to change statistically. You know, we talked about you're going to have problems and that means the project's got to be recast. And once again you want to be integrated with the psa because the minute you recast that, you want all your engineers tasks to get their dates updated automatically to go, hey, this is the new normal. Just quickly on this. This again calls forward. One of my favorite quotes on projects from Patton is planning is everything and plans are nothing. Yeah. Because the assumption is this is going to go sideways so you're going to have to replan as you're going. Right. I think to this, like don't meet, don't recognize this as resistance or a failure. This is, this is just how things go. Like you've got to be probability. That's the point. We, you know, come full circle. Like you said in the very beginning of the conversation, that's probability. So but this is what you want. You want AI and automation to recast that reintegrated and updated. You don't want. And this is why project plans are terrible because when people deal to try to manually do this, it's not possible. And so they get garbage dates and they're communicating garbage dates to their customers. So this is why, you know, what we're passionate about is to say, look, we statistically know that this is 100% chance your 99.9999 chance you are going to have a problem. We talk about the portfolio of work. It is a septillion times more likely that the sun won't rise tomorrow morning then your project portfolio, if you have 20 of these 40 task projects is going to go off as planned. Like the odds are terrible. So what are you going to do? And that's where we step up. Like what do you plan to do about this? Hope that it doesn't happen. I mean, good luck with that. Yeah, let us know how your custom. Right. All right, so and this I think gets to, I think a really interesting space in this again like PM in general being somewhat underserved in our industry. And this is sort of an interesting segue because a big part of what you guys do and bring to MSPS is a level of AI and automation that is absolutely not available in other platforms. And I think it's the stuff that you guys add is super cool, especially because as you said, it's integrated, it's not a standalone platform. So the fact that you're, you're servicing the MSPS in this way is, is really interesting. That said, you know, we talked about this two years ago on, on a previous Episode two years ago in AI is a frigging lifetime. So I would be curious on what has changed with sort of the base capability of those platforms and you know, the back end mechanics of what you guys can bring to bear on projects. It must be somewhat radically different, I assume. Yeah, so, so I'll say this. So. So we rolled our own over a very long process using deterministic, symbolic and heuristic technologies. The new so and we also have other things where we use the latest gen AA stuff in hybrid fashion. Right. And so there are a couple of things that are really important about to know about this. So in our platform because it's deterministic in nature, it means it's repeatable, interpretable the results and why we got what we got. I think everybody now dealing with AI knows the idea of hallucination in the generative platforms and the probabilistic systems and you don't know why it's giving you the wrong answer. The thing that, that we know now is true because like, you know, obviously we went and certainly when ChatGPT came out we went oh gosh, is this going to be an existential risk to what we're doing? Are people just going to be able to throw garbage project plans or any project plan into an LLM and it's going to just tell you how to do everything and fix it? Right. And so we did research and fortunately through connections have access to people who are in research circles around this papers as recently as this year coming out of Harvard, Michigan State, Microsoft and Amazon, all of the LLMs. And this linearly probabilistic generative AI solutions are terrible with graph data. So and what I want to, I just want to sort of unpack when we talk about error rates around a solution. So for scheduling a project like hey, I want to schedule, we talk about what's an acceptable error rate. We have different error rates for that we as a consumer will accept around the task of automations and those can get expressed of acceptability of percentage rates of like a less than a half percent, 10%, 5%. You know, if you're writing a poem for one of your kids or whatever, like an error rate of not quite even getting. You don't care about, about that. Right. But if you're saying what's the error rate of my parachute opening? You're going to want that to be less than, you know, 100th of a percent or a thousandth of a percent chance that your parachute and that might even be high for the number of people go Skydiving, less than a half a percent would be range or quarter. What would be expected for automating project scheduling? Like getting that out and saying, oh, this is when this project's going to be due. This is what, here's your scheduling problem, here's your conflict, here's your resource issue. Right now, the error rates for teeny tiny graphs and projects, but is 40%. It is so extraordinarily high as to not even consider and there's no improvement. So these models, if you look at the drop in error rates for the models on on language in 2D and 3D vision, those error rates drop significantly because of the nature of what they're doing generatively. So I'm generating text and a letter for you, or a business plan, or I'm generating an image. That generative process, they've really been able to drop it. But the idea of observing a living dynamic system that is in a graph, it gets lost. It can't relate beyond its nearest neighborhood, it gets confused. And because it can't traverse a tree in a depth first search or breadth first search, it falls apart and gives bad data. And unfortunately, like some of the things we talked about before, you get one bad answer in that graph and that cascades because it's all connected. So, you know, we look at this and go, any of you hoping that you're just going to be able to plug your crappy management into an LLM and get the solve? And you're just waiting for that because it's AI and it gets better. There's no line of sight of that and there's no papers. So ChatGPT came out and it was amazing. I think 22 is when we started to go, wow, that's really something. I think that paper came out in 18. And certainly things move quickly and we can say things happen overnight, but there was line of sight. And we did know back when that paper came out that, oh my God, we can parallelize this computer. We just need to get more compute and more data. They, they weren't surprised when they were going down that path. They knew they were going to get better results. They knew those error rates were going to drop, which is why all that money flowed into it and why there was a hype cycle. And we as consumers saw the benefit and we're like, oh my gosh, this is amazing. Once that error rate dropped to a point where we trusted it once again, they're nowhere near that for managing projects. And so we look at this and go, you got to use traditional you need to have an expert system in place, deterministic. You can layer, we have customers that layer other AI stuff on top of our data because they're operating off of a foundation of accurate data. Like they're taking the guesswork and the hallucination work out of it. And then they're going, okay, summarize this. Yeah, this is interesting because, like, I play with enough AI that I recognize what you're talking about in the boundaries of this. Right? So I think a lot of people saw early on hallucinations where you would ask an AI about something and it would just completely falsify something. You can tell it's, it's, it's not even remotely true. My brother had a great example of this. He asked GPT a long time ago, like, who was, who is the, who ran his company, and it responded with a real person that had nothing to do with his company. I'm not the CEO of my own company. Interesting. Okay. But that was the early versions of this. And it seemed like a lot of that got solved because hallucinations dropped precipitously. But I still regularly see situations where, like, it has no idea what it's talking about, even in basic math, right to the point where it is trying to generate the next token. And unless it's, you're asking it to solve a particular math problem, it can get some of this stuff absolutely wrong, which is weird. And the other part that I see this show up and is, and probably relates a lot to projects is I am regularly fascinated by the fact that GPT has no idea what day it is even when I tell it, you know, and then I'll, I'll say the data is this, blah, blah, blah. And then we'll go through like not a huge context window and I'll, I'll try to ask it to plan for something else. And it completely misses the mark. Like it's like it's April 2023. You're like, what? No, where are you? Right, yeah, well, and you just hit on another really key point of why it's limited in projects. Right. Is this the larger. The context window, that performance rate decays in a non linear fashion, so it can perform really well with a paragraph or two of data. Goes off a cliff. Any large project plan blows up context windows. And so like I said, it's just, there's no line of sight that, that's gonna, you know, I think this idea of, oh, they'll figure it out, they'll figure it out. You know, like the figuring out is A hybrid solution. Like you're. And it is doable in a hybrid solution today, right? I mean that's, that's the big question. But you're spot on in saying like, and projects have a lot of data right? There is at the end of the day, a lot of data. When you're talking about what are the engineers on. Even on a smaller project that has 15 tests, well, who's working it, what are their schedules and what are their resources and what are their skills and what's the time like? Once you get to that level, it blows up. It might be able to generate a plan, but it can't observe it, monitor it, identify risks and problems. And, and that's, that's the issue right now that these MSPs face is, as you said, your patent quote, like, oh, we had it to start. Why did it blow up? Well, it was doomed to have problems. You didn't know that because humans suck at probability. But it was doomed to have problems. You just didn't put an early warning system in place through best practice, process structure and technology so that when that inevitably happened, you could adapt to it and adjust. You're doing that with your endpoints. You pivoted from being a break fixed MSP to a managed service provider that's actually observing through your noc. You got to kind of build a NOC for your projects. Yeah, yeah. I mean, I guess that's the project, right? Start, start working on, on these iterative improvements, get the basic mechanics in place, start working on resourcing dependencies, time windows, and then start to pick up steam from there. So really appreciate this, Mike. Thank you so much. If people want to get in touch with you or your team, what's the best place for them to check.com M. O O V as in Victor I L A dot com. Yeah. And we'll be at, we go to a lot of shows during the course of the year, so hope to see you, Todd, or other folks if possible, at those shows. And thanks everyone for taking some time to talk about projects because I know both you and I are interested. Yeah, yeah, we're pretty passionate about it, but, but glad for anyone who's also expressing interest and happy to answer any additional questions. Awesome. Appreciate it, Mike. Thanks so much.