Becoming an Engineer: the “Build or Buy” Question

[MUSIC] 
>> Hello, welcome to Picture Me Coding with Erik Aker and Mike Mull.

Hello, Mike.

>> Hey there.

>> Mike, this week I've been listening to this band Ovlov.

It's Volvo spelled backwards, Ovlov, and the album is called Buds.

It gives me a real indie rock vibe, but it's pretty fun.

I discovered it a while ago and I returned to it this week.

>> Does the name of the album imply anything about the subject matter or?

>> I don't know.

Unlike you and probably most people, I only notice lyrics when they annoy me and otherwise they're in the background.

>> It's not like a stoner rock or something though.

>> No, it sounds very like 90s, early 2000s, indie rock.

It's fun.

I like it.

East Coast, there's a kind of Massachusetts, Connecticut, Vermont, indie rockish kind of feeling to it.

>> Pretty sure I've heard this band.

I think they come up on my Spotify Rex fairly often.

>> I don't know where I heard of them.

What have you been listening to?

>> I have a pretty strange one this week.

I don't recall exactly where I initially found it.

But I've been listening to this album called Strings Attached.

I think it has a subtitle of something like The Sound of the Kannel.

The voice of the Kannel, sorry.

The chief performer is this Estonian woman named Annalisa Eller.

This Kannel, Kannel is a Estonian folk instrument.

It's a very interesting sounding instrument.

It's sort of like a tabletop harp.

It sounds a bit like a cross between a guitar and a piano.

I really like the sound of it.

It's more interesting than harp sound, I think.

She plays some really interesting music on this album.

It's just very interesting to listen to.

It's one of those nice albums to sort of put on your headphones and drift off for a while in the evenings.

>> Okay.

I will have to give that one a shot.

I'd like to see pictures of this instrument, the Kannel.

>> Yeah.

I guess it's very specifically Estonian.

I guess there are variations of it throughout the Baltic countries.

But this one in particular is specifically Estonian.

So, if you're into Estonian music, as many people are, I'm sure.

>> Mike, for our technical discussion this week, I have for you a very old question.

I was trying to go back into history and figure out the first time people asked this question.

And my guess is it goes probably as far back as ancient Roman history, where people like Pliny the Elder, they covered this question.

As software developers, should we buy a thing or should we build that thing?

Build or buy is the question.

And I have to tell you that this question strikes fear into me when it comes up.

I get a little nervous.

It's an existential question for me because my role is to build stuff.

I don't ever, ever want to buy anything.

I mean, that's not really true.

There's a lot of stuff I don't want to build.

But it is an existential question.

Where do you remember first encountering this in your career?

Do we build the thing or buy the thing where you had a role in deciding?

>> I think that probably the first time I had to make this decision was late '90s, early 2000s.

It inevitably comes up if you're doing engineering of a large enough scale because you will eventually find things that are, you know, you think are necessary for your project, but are beyond the capabilities of your current engineering staff and really not in your wheelhouse in terms of, you know, the things that you are used to building.

And so, you know, the question of do you train up your engineers to do this kind of work and spend the inevitable money and time that you need to build this thing?

Or do you just go out and acquire it from some existing vendor?

It's going to come into play eventually.

>> So cost and capabilities, those are a couple key components here early on.

>> What do you remember what the thing was?

You were debating whether or not to build or buy?

>> So the thing I remember, I was working on a project.

Well, the customers for this product were doing consumer product formulations.

There was a considerable amount of statistics and what they call experimental design that went into it.

But there was also kind of a complicated set of rules that they had to follow, you know, things having to do with cost and compliance and seasonal availability and stuff like that.

And so we talked about incorporating this rules engine technology into the product that we were building.

>> Those things used to be hot, the rules engine.

>> Yeah, it was especially late 90s, early 2000s, I think it was kind of what AI was back then.

So, you know, there's certain technologies that are used to build these things.

And it's one of those things that a lot of most mainstream programmers don't have a lot of experience in.

And it just didn't seem to make a lot of sense to try to do that from scratch because it would have involved a considerable amount of learning and there was risk to trying to build something that you didn't have a lot of experience with.

So we started looking at various vendors to provide that tool.

>> Some people used to charge and probably still do tons of money for those things.

>> You know, this is one of the dilemmas of the build versus buy thing is that very often the thing you're buying is a substantial upfront cost.

And I don't know if it's true in general, but there's sort of like a psychological barrier between the amount that you have to pay upfront versus the amount that you know that you will inevitably spend developing that thing.

So, you know, if the cost of the tool is $100,000, that seems much worse than the developer cost to build that thing for some reason.

>> Over a course of a year.

And often now it's not just buying it one time, right?

We're renting mostly.

Well, to frame our discussion, I wanted to go back to Fred Brooks, your friend Fred Brooks.

We spoke in our episode about complexity on Fred Brooks's paper, No Silver Bullet.

And in No Silver Bullet, he makes this suggestion.

Now he's trying to say there software is complex and we're not getting better at reducing its complexity.

We should have less complexity.

And so he pretty much says you should try not to build stuff.

And he raises this as one of his solutions.

I'm going to quote from his paper.

The most radical possible solution for constructing software is not to construct it at all.

Any such product is cheaper to buy than to build a fresh.

That's a bit of a high claim there, right?

It's cheaper to buy it than to build it.

Even at a cost of $100,000, a purchased piece of software is costing only about as much as one programmer year.

And delivery is immediate.

Immediate, at least for products that really exist.

The key issue, of course, he says is applicability.

Can I use an available off the shelf package to perform my task?

And he goes on a little bit, but there's some interesting claims in there.

It's always cheaper to buy than build a purchased piece of software cost about as much as a programmer year.

It's purchased, not rented.

Delivery is immediate.

He does say the key issue is applicability.

What do you think about Fred Brooks' comment?

You should always try to buy.

I think it's more or less sensible.

I think there are a few things in there that are maybe passed over a little too glibly, especially the issue of applicability.

I think it's probably true that if the thing you are buying is more or less identical to the thing that you would build, then buying it is probably cheaper.

The problems come up when is that thing identical to what you want?

How much do you have to adapt it and so forth?

So we'll talk about those things later.

But I think there are certain cases where that's obviously true and certain cases where it's debatable.

I think I keep thinking about renting, too, because a lot of software nowadays, you're going to pay a monthly yearly fee for it and you can commit.

You can get sucked in when we talk to an identity provider, vendor, and large SSO company, their salesperson wanted to get us into a contract that was vastly larger than we needed.

And he let slip at one point in our meetings with him.

We know that this solution is sticky.

Once you join with us, it's going to be very hard for you to leave.

I'm ever thinking at the time, I don't think I would say that as a salesperson talking to software engineers.

Yeah, that does seem like a saying the secret part out loud.

But yeah, I think that definitely has changed since the time that Brooks wrote his analysis as well as the idea of software as a service was not a thing then.

Although I think probably even then company I was working for in the early 2000s did charge an annual subscription fee for products that they sold.

And it was primarily so that you would get the ongoing upgrades to those products.

I watched a video this week about IBM unbundling software.

So they used to give software away for free.

It used to be just like, we are going to give you some software to incentivize you buying our hardware.

They were a hardware company.

They wanted to sell mainframe computers.

They wanted people to buy them.

In order to buy them, they had software needs and they would just kind of have like software engineers assisting the sale who were writing bespoke software.

And sometimes stuff became generic and they packaged it in and then along came these other companies in the 60s and 70s, well, 60s who decided to sell software independently.

And instead of buying that software, some of IBM customers were like, well, we'll just make, we'll wait for IBM to make that and give it to us for free.

And there was an expectation in the early days that software was just free, that it was like the machines you paid for.

And then eventually IBM unbundled in 1970 and they started selling their software separately.

And people kind of threw fit about this and came about because of some antitrust lawsuit that they saw coming and eventually did come.

And so then software became a thing that was sold.

I think open source complicates this a bit.

So I want to kind of push that to the end if we can.

I think build or buy is a hard question because there are certain cases where the answer seems obvious and a lot of cases where the answer is kind of fuzzy, where you can kind of go either way.

And you have to look into the future and determine, is it a better investment for us to build and own this thing?

Or should we rent it?

I'm going to say rent instead of buy.

Yeah, there's at this point in time, there's I think this general rule of thumb that if something is part of your company's core competency or it's something that you need to understand and be good at, it's probably something that you should build.

Whereas if it's something that you don't have deep expertise in or which is not giving you a competitive advantage, it's maybe something you should buy.

But as you say, there's a definite there's a spectrum there and there's a lot of stuff in the middle where the decision is not really very clear cut.

Yeah, that core competency thing, I usually refer to that as the Jeff Atwood argument.

That's where I first encountered it.

Jeff Atwood, one of the founders of Stack Overflow, he has a famous blog post called Let's Go Shopping.

And in the post, he talks about these core competencies.

He says you should build the core competency.

And his post is really a defense of the fact that he wrote a custom HTML parser and sanitizer.

And his argument was, this is a quote, "deeply understanding HTML sanitization is a critical part of my business."

And he's talking about Stack Overflow.

And he quotes Richard Feynman, "What I cannot create, I do not understand."

That's the core competency argument.

We're just going to accept that one.

If we can identify a company's core competencies and the company wants to build software, they should build those.

We're going to say that.

That's where we should start probably.

I generally believe that.

I think there's a little bit of a nuance there, which is your core competency also has to be something that is not extremely common.

If you are involved in a business where you are competing with 100 other companies doing exactly the same thing, maybe your core competency is enough of a commodity that you can find something useful in the market to do that thing.

I think the outward argument that you need to understand the thing that you are doing well is compelling.

But I'm not sure that it's applicable to every business.

I think there are probably some businesses that are the workflows are so well established that unless you figured out some way to do that thing better or disrupt that industry, you probably can buy something off the shelf if it's available.

So let's say I want to start a bank.

I'm going to go, I'm going to buy an off the shelf, cobalt, mainframe application, start it running.

Boom.

Bank is open for business.

Seems totally reasonable to me.

Yeah.

I think lurking underneath this discussion and I when I listen to two random dudes on the internet talking about stuff as we sometimes do here.

I like examples.

I like things I can wrap my head around.

I can really sink my teeth into specific examples.

So I want to frame our discussion using I invented sample companies here.

And there's a key distinction in sample companies.

One is a software company and make and sell software.

And the other is not a software company, but they want to be like a modern company.

I've got a website, you know, they want to do technically interesting stuff.

So we're going to imagine that we are co CTOs, two sample companies.

You OK with that?

Co CTO?

Yeah, I guess that's fine.

Who do we report to?

Do I report to you or do you report to me or do we report up to like the chairman of the board or something?

We report to the board.

OK.

Yeah, we could just go off completely like what we have our own like fiefdoms and stuff.

It could be an awful, awful organizational structure.

These are companies with co CTOs, Mike and Eric.

The first company is called the C around us.

They make a version of Excel that you can run underwater while scuba diving.

They sell licenses and hardware.

This is a software company.

Yeah, I kind of want to start this company now.

Underwater Excel.

I can't.

I haven't done a lot of market analysis here to know, you know, what the what the market is for underwater spreadsheet software.

But well, I mean, I wanted to make a company that I wanted to dream up a company here that definitely doesn't exist.

But there's obviously a market for that because Excel has dominated all terrestrial areas.

And really, they haven't gone into space.

Well, they probably are in space actually by NASA.

And I don't think they're underwater.

So obvious gap in the market for Excel.

OK, second company.

Go ahead.

Go ahead.

Oh, I was just going to comment that, you know, there's arguably a hardware component to this company as well.

So yeah, I'm cheating a bit.

Like they sell software and hardware.

They sell a box.

You can run underwater.

You could use your finger to do Excel through the box.

How does that sound?

Still a software company?

Still a software company.

I'm skeptical of the using your finger part, but we'll get back to that.

Well, this is how I sold it.

I'm a PM and a CTO.

This is what I want the engineers to make.

OK, the second company I call ILOF and invite my souls so far.

They make couches, boring couches.

I mean, maybe not boring.

They've got cool upholstery and stuff.

They do not sell software.

They have a website and factories and ordering systems and analytics and they can measure and predict the whims of their customers.

Let's imagine they're not massive like a Wayfair or a Zappos.

Right.

Those are arguably software companies.

They're middle tier, I would guess.

So software is not primarily what they do, but it's important what they do because maybe they get a lot of sales to their websites and they want to predict the market and things like that.

Now, what do you think?

Does it change the question of builder by if I'm an engineer at the software company versus if I'm an engineer at the not a software company?

The couch company?

I think it definitely makes a difference.

I think when you are at a software company, this argument about owning your core competency becomes considerably more important.

Because software is the product.

You should build that thing.

Software is the product and in the case of the sofa company, I know that our audience is far above normal intelligence, but should we mention the Walt Whitman reference there or just?

Oh, that was totally accidental.

Okay.

Anyway, so the sofa company, my assumption is that having better tools, having better access to data probably helps their business.

But at the end of the day, they're not making any money off the software.

They're making it solely on the profits from sales of their product.

And this is a thing that happens sometimes at not a software companies.

The software is a cost center.

This is something I've heard.

It's always like, how much do you guys want to spend to build this thing that we asked you to sort of build?

That's a lot of money.

Here's the questions I really want to get to.

Is are there categories in each of these cases of things you would definitely, obviously buy that building those would be a mistake?

Is there a set of obviously build?

And my underwater Excel example muddies this a bit, I think.

Obviously buy, obviously build.

And then here's the third category, which personally I hate.

It's the buy, but you got to do some work to shim it in to what you've got, which is build and buy, buy and then build a little bit.

And the thing I hate about this is you really don't know how much building you're going to have to do to be able to successfully buy.

And you cannot rely on the salespeople of the companies doing the selling to know how much building you need to do to successfully buy.

So it's sort of like fake buy.

OK, so three categories.

Definitely build, definitely buy, buy it and shim it in.

So my take is that there are certain things that are definitely buy for almost any company.

That's what I want to know.

What are those?

Any company, software, non-software, you are going to buy these things.

You should not build these things.

So in my experience, I think something like accounting software is probably going to be a buy for almost any company unless that company is, you know, an accounting software company.

OK, I buy that.

That doesn't make much sense to reinvent the wheel on that.

No, and I think, you know, Brooks talks about this way back in his silver bullet paper about how there was a time when you're paying so much money for the mainframe that the amount of money that you needed to spend to build an accounting package probably didn't really register.

But at this point in time, building your own accounting package would seem strange unless you had just very unusual accounting practices that did not fit any of the existing products.

I will say, though, that this is still not, even though I think it's an obvious buy, I don't think it's a trivial decision because I think there is still the need to evaluate what is best for your company.

So there is a cost here that comes with any buy decision where you have to go through looking at the options, doing research, evaluating them, figuring out, you know, what the cost benefit is for those things.

So it's not it's not obvious, I think, for all companies that there's a particular thing you buy just that you probably will buy some package.

Accounting software seems like a great example because who would run a company and say, I really want unorthodox accounting in my company.

If it's not a criminal enterprise, you know, like that's not a place where you you think about what other people are doing and think we should differentiate.

You don't do that, right?

You say, I want to know what everybody else is doing and I want to do that.

Yeah, I confess also that I have extremely little knowledge of practical accounting.

So I apologize to people out there who think we're oversimplifying this.

But yeah, maybe I mean, I've used QuickBooks.

Yeah.

And I think, you know, into its options are probably pretty common, at least in the small to medium sized business world.

Did you have other examples?

So you said accounting software.

What other examples obviously buy depending on the company?

I think things like, you know, there's a lot of companies now that provide HR support payroll stuff and PTO stuff like that.

Yeah, time tracking.

Yeah, that sounds lame to build and pointless to build.

Yeah, I mean, it doesn't seem like you're likely to come up with something.

I mean, maybe in really small companies, you just have an access database or something.

But I think in a sufficiently large company, it probably makes sense to to get it off the shelf or a software as a service solution for that type of thing.

Boring things by the boring things.

Yeah, boring to me at least.

But I again, I'm sure there are people out there doing totally fascinating things with HR software.

So it's another case where you're like, can you imagine the kind I would really like to do something unconventional in HR that probably gives the attorneys a little bit of the cold sweats.

I bet.

Right.

Yeah, I don't know.

Beyond that, I mean, the office suite type of stuff.

I think, you know, people probably don't build their own word processors anymore.

You're either using the Microsoft suite or using Google tools or something like that.

OK, Microsoft suite.

That sounds like Excel.

Brooks has a funny comment about Excel.

Maybe you saw it.

These powerful, he's talking about spreadsheets.

These powerful tools, so obvious in retrospect and yet so late in appearing, lend themselves to myriad uses some quite unorthodox.

I thought you would like that.

And articles and even books have now abound on how to tackle unexpected tasks with the spreadsheet.

I thought you were going to like that one.

But underwater Excel maker, they're going to buy the office tools or they're going to make their own brand of it.

I mean, maybe they can rent it or something and build their thing on top of it.

Maybe this is where my idea of this company is pretty farfetched.

Yeah, so the way I'm imagining this as Co-CTO is that I'm underwater.

I'm in a wetsuit with a tank on my back and those big gloves on your fingers.

Yeah, I've got those gloves on at least or maybe, you know, I've got like chain mail so I don't get bit by shark.

Yeah, OK, OK, all right.

So I'm probably got custom hardware and maybe it's some sort of embedded device that straps onto my arm or something.

Oh, like those cool wristwatches they wear.

Yeah, and it's resilient to pressure and temperature and salt water and so forth.

And so I probably am not just running Microsoft Excel on this thing.

I probably need to build something that gives it a user interface that can be used by somebody who has limited agility and with their hands and who needs to be able to see it in the murky depths and so forth.

So I think I've never scuba dive before, but you've gone pretty deep into imagining this far beyond what I had imagined.

I watched a lot of Jacuzzo growing up.

Couchmaker, do they build their own website or do they go on Squarespace or something like that?

I'm feeling like they probably farm out the e-commerce platform.

I think because it's basically pick out what you want and pay for it, right?

So there's not a lot of customers not providing a lot of information beyond their name address and payment method, right?

I guess, is there an inflection point in scale where the couchmaker becomes a massive concern and maybe they're selling all kinds of furniture?

And then at some point they become like a Zappos or I've been looking at construction sites lately.

Maybe that's why the couch site comes up, but like some of these sites are very large.

Wayfair, I met a guy at KubeCon who worked for Wayfair and he described some of their deployments and it was mind boggling.

Huge stuff.

Is there an obvious inflection point where it's like, oh, we're so big now.

We have to own those things because it costs so much.

Like these stories that Dan Lu talks about a Twitter where they did incredible hardware hacking to get chips and servers that suited the massive needs of Twitter at a more cost effective basis than just renting cloud compute.

Yeah, that's a good point.

There probably is a situation where the connection of your e-commerce system to things like inventory and planning software becomes important.

If somebody wants to order a particular couch in a particular color, you need to know whether or not that's in a warehouse or whether or not it has to be built for them and what the backlog is for building new things and is the cost going to change in the six weeks it takes to manufacture that thing and so forth.

So yeah, I suppose there probably is a certain point of scale where the build decision becomes more favorable than the buy decision.

I think I want to pitch to you some typical SaaS stuff that comes up in my day to day work that people talk about buying.

And I want you to tell me the underwater Excel company, Yay, Nay, if a salesperson comes pitching this to you and the self-accompany.

So one that I implemented the company I work for, but using open source tools, is an observability stack.

So we've got all this software and we want to watch it running.

We want to know are there errors in it?

Although I have to say we did actually rent, we do rent from Sentry error alerting.

But I've got logs, I've got metrics, I've got open telemetry traces.

These things are wildly expensive and very, very sticky.

If you go and you pay a service for these, you're going to pay a huge amount of money and it's going to be very hard to leave.

What do you think?

Yeah, for our particular example companies, I'm not quite sure about this.

You're saying those devs don't need to know what's going on in their software.

The underwater Excel makers, the guy's like, I'm going to die because I can't enter this formula into this cell in my spreadsheet.

Yeah, I'm unclear if the underwater Excel software has a cloud component that needs to communicate via satellite or something back to the home office.

So that's the case if there's some component of this software that maybe it backs itself up to the cloud, just in case your diver implodes or something like that, you want to still have the data.

So in that case, I think you need observability.

And again, I think it probably is a matter of scale.

My guess is this company would probably do something more like what we did or what you've done.

They'd build it themselves.

They built it themselves using hotel and Grafana and Prometheus and things like that.

Open source tools.

Open source complicates the builder by question.

I think it really does.

What about authorization?

You know, there's a common phenomenon these days used to be Auth0 and they got bought by Octa, also Octa.

Microsoft Office offers this.

There are different companies that offer authorization.

You want to have authorization on your website.

You go here and they'll handle multi-factor off and then you want to bring your own off into it.

Like this can also be very expensive.

What do you think?

I think this is in this day and age very likely to be a buy decision.

Buy?

For a couple of reasons.

It would depend on the type of the company, I think as well, because if you're in a company where compliance-- Please stick to the examples we have in front of-- No, go ahead.

I'm sorry.

Go ahead.

So let's say that the underwater spreadsheet company, one of their chief customers, let's say, is the DOD.

So they have some sort of compliance regime.

They have to follow to sell things to the Department of Defense.

Oh, yeah, that makes sense.

Like government contractor, all the regulations around that.

Yeah, so now you have issues of, am I going to store people's passwords on my systems?

Am I going to try to roll my own two-factor system?

And my guess is that unless your company is very large, that's probably not something you want to mess with, because there's an extra level of compliance and auditing you're going to have to go through to store that stuff and have that kind of stuff vetted for you.

Here's one that I felt was you'd probably-- this is a softball one.

Email.

The obvious choice is you spin up your own send mail server.

Yeah, that's an obvious one, right?

You don't want to run some TP servers that sucks.

OK, I probably shouldn't have even said that one.

Everyone knows you're just going to buy that.

All right, all right.

I kind of had on here-- this is not very well fleshed out.

Maybe I shouldn't give you this one, but deploying code, where do I deploy it?

Cloud, this is a little harder for our Excel thing, because we're kind of building a box, right?

So if a company-- it doesn't make sense for them to have their own data centers, probably, unless they get massive.

If they're going to do some software stuff, it's got to be cloud.

Yeah, I think so.

I mean, this was an issue, I think, more in the early 2000s, when cloud was sort of a new thing.

There wasn't as much infrastructure available.

So you might have had to-- for instance, I worked for media companies in the early 2000s, and they were still pretty much doing their own on-premises machines or data centers or things like that, because it was hard to keep track of all the licensed media that you were supposed to charge royalties for and stuff like that.

But now, yeah, for both of these companies, I can't imagine them having their own-- having a significant amount of their own hardware.

Both of these companies, interesting.

Even the hardware box underwater maker.

Yeah, again, it might come down to security issues or-- I don't know.

But yeah, I think-- Yeah, that's right, because we might be training dolphins as part of the Navy's dolphin training program.

Yeah, I mean, you might have to have floating ships with data centers on them or something like that.

This company is sounding more and more interesting all the time.

I wonder how big the audience is for underwater Excel.

I like to imagine there is one person who would be interested.

I suspect it would be somewhat useful for oceanographers and maybe people inspecting the hulls of ships or things like that, divers.

But yeah, I can't imagine it's a billion dollar market.

Here's the thing that comes up in my mind as an example of this discussion.

Salesforce.

Now, this is-- we will have to tread lightly talking about this one, I believe.

Salesforce massive, massive company.

When I look at Salesforce, I think, wow, OK, sales, customer management stuff.

I wouldn't want to build that.

Sounds lame to build for most companies.

It's cool to have a solution to do that.

However, it is very expensive.

And the interesting thing about Salesforce is they kind of pitch themselves as the do everything solution.

Yes, customer management, but also an entire software deployment platform.

You can build all kinds of apps.

You can build a simple crud database if you want.

I always describe it to people as a glorified Django admin.

Yeah, I think this is an interesting one because there's kind of two aspects to it.

One is the CRM.

Most companies probably don't want to build a CRM.

It's probably not a core competency for many companies.

And sales processes are-- well, maybe they're not universally the same.

They are enough the same that these things are pretty easy to adapt to most sales organizations.

But I think this gets a lot trickier when you start looking at Salesforce as a platform as a service type of thing.

I remember back in the old days reading the Kokomo 2 book.

Tell me about that.

Kokomo 2.

So this stuff seems kind of like-- I don't know.

It seems very old school now, but back in the '90s and early 2000s, there was this cost modeling stuff.

So I think the chief guy behind it is this computer scientist named Barry Bohm who did research on cost modeling.

And he came up originally with the Kokomo model and then the Kokomo 2 model.

And there's a lot of criticism of it.

There wasn't a huge amount of data available to them to do it.

So people have always questioned the accuracy of it and whether or not it's the work involved in it is worthwhile.

But I think there was some good ideas in it in terms of what are the drivers that actually cause software projects to cost money or to overrun their schedules and that kind of thing.

Am I wrong in remembering Kokomo as a way to look at a code base and say, here's what it would cost in programmer effort to build this?

Yes and no.

So the idea behind it was they'd done analysis of existing code bases.

And then the idea was that you could try to estimate the cost of a thing that you wanted to build based on estimates of things like lines of source code.

I think Kokomo is the thing that invented this thing called function points too.

But in any case, the idea was that you wanted to sort of estimate how much something was going to cost or how much time it was going to take.

And you could use these sort of proxy metrics as a way to do that.

And it was things like lines of code or equivalent things.

But one of the things I remember was that there was a discussion about the build versus buy decision and they talked about the cost drivers there.

And at the time it was kind of enlightening to me because I think I went into the idea that the equation was simply replacing the cost of the tool with the cost of developer time.

And in that work they sort of pointed out that there's more stuff involved.

So if you're going to buy something, there's a period where you have to do evaluation of it.

And that-- And you're not building at that time.

That's a cost right there.

Yeah, you're using probably the time of engineers to do sort of an initial filtering stage where you figure out the things that are possible and then a deeper evaluation on what you filter down to.

So there's some time there that you're paying for.

And there's, of course, the cost of the thing that you actually buy.

But then there comes in a period where you have to figure out how do you integrate this thing that you bought into the system that you're building.

And it's sort of a rarely the case that the thing is plug and play.

There's probably going to be some Galoo code that you have to write.

There's probably going to be some documentation you have to read.

OK.

And that right there is why I hate this question when it comes up to work.

Yeah.

I think it's hard to answer.

It's difficult to do this analysis.

And often once you've committed to the buy, it's like there's no going back.

You enter the door, you sign the contract, you're stuck.

You might find out things at that point that you wish you would have known before you signed the contract.

Yeah.

I've also found, at least in the last few years, that a lot of vendors seem to be more and more reluctant to let you do extensive trials.

The evaluations, yeah.

And I think that makes the decisions harder too, because you inevitably end up doing sort of the integration testing after you've already committed to buying something.

And I think it's sort of complicated still also by the fact that there's outside of the technical world, the decision makers who are primarily on the financial side still think of the buy as what's simply a comparison of what is the cost of the tool compared to the cost of the developer time.

Debilling it.

Exactly right.

So the reason I hate this question so much when it comes up, maybe hates too strong a term, but the reason why this question makes me nervous is, one, how much work am I going to have to do to integrate this?

I call that the shimming.

How much work do I have to do to shim this into our system?

And two, what don't I know?

What am I going to hit six months from now where the tool suddenly makes a thing that would have been hard but doable now possibly impossible, right?

What does the tool make easy?

What does it make hard?

What does it make completely infeasible?

And if you know those things going into it with your eyes wide open, you can make a more informed decision.

But you often don't know those things.

You commit to the software.

You start building your integration stuff.

You don't know how much integration stuff you have to build.

Sometimes you build so much integration stuff that you get to this point where you're like, we should have just built this stupid thing ourselves.

Those are the cases that are hard.

Yeah.

That's the middle category.

Yes.

And there are methods for looking at these things, but I think they're like all software estimation somewhat imprecise.

Yeah.

Imperfect.

And often the people we're dealing with are salespeople who are almost reluctant to have, sometimes they're not.

There are good ones out there who are like, no, no, we will tell you exactly what our customers do to integrate this.

We're very happy.

We're proud of our product in a way.

That happens sometimes.

But in other cases, they'll just kind of, you know, pat you on the back to go, this is going to be great.

It's going to be great.

It's going to be so easy.

It's going to be beautiful.

It's going to be a phone system that we switched to many, many years ago.

And the old phone system would archive our calls to S3, which we needed for auditing purposes and other purposes.

New phone systems didn't do that, but they were like, well, we have this great new API.

It's going to be great.

It's going to be beautiful.

It should be so easy for you to do that.

And I remember at the time, someone coming to me and saying, hey, how hard would it be to just run a script that archives our calls with their API up to, you know, an object store?

And I'm like, well, that doesn't sound that hard.

And there's no way we should be building that.

That is not a core competency.

I do not want to build that.

And we built it anyway.

And the API that they told us about that they sold us on, it was in beta.

So it changed.

And that broke a bunch.

And every time I just complained loudly, like, why did we build this thing?

This was not a thing.

We should have gone with a different vendor.

Yeah, this is why having an evaluation period or some period where you can test the thing and you're not involved in a multi-year commitment is kind of important.

I had this similar thing with a service that is used by, it's a thing that we built, but it's a service that's used through an API.

And it's a good service.

It does what we needed to.

But the initial sales pitch was, well, this would be a lot cheaper if you commit to a year or, you know, three years or whatever.

But it was one of those cases where we don't really know if this works.

We don't know how this is going to roll in.

So it was actually cheaper to pay on demand for several months to try to figure out how things work than to take the cheaper option of a multi-year commitment.

And thankfully it worked out in this case, but I can just see, you know, you're faced with this mostly financial decision of getting a pretty big discount on a month's year-long commitment or something like that.

And then you buy a tool that just doesn't fit into your needs.

I think it might have been in the Phoenix project.

Or maybe, I can't remember where I read this line, but it was like, you always will know things at the end of a software project that you wish you would have known at the beginning before you started.

But that's the nature of the beast.

You go through the exercise of building, implementing the thing, and you know the things you needed to know at the beginning to know how long it would take, how hard it would be, and how many compromises you'd have to make.

And this builder by question, very much a microcosm of that.

And with the addition of some maybe antagonistic oppositional pieces on the other side who really want these people, who really want you to buy, they're going to try to tell you things they think you need to hear to buy, or in the case of a certain observability company, which I won't mention, they have salespeople calling you on the weekend early in the morning, harassing you to buy.

Don't do that.

So I think if I could find out that a buy was super easy, slotted into place, do no shimming, and it will immediately solve a specific pain point, then I understand the cost.

And in a lot of cases, I'm going to say yes.

But in a lot of cases, I'm going to be like, I don't know.

It may be easier for us to build this because I don't actually know enough to know what we're going to commit to if we say buy.

And I think as a leader in a business, at this couch company, they're going to be like, this guy always wants to build stuff.

He's got not invented here syndrome.

Exactly.

Yeah.

Yeah, I think we were discussing this earlier offline, but I think this is a really important aspect of becoming an engineer.

You have to figure out these economic questions because engineering is really about operating within the constraints of your operating environment.

That sucks.

That's not why I got into this.

Yeah.

Yeah, it's not exactly my favorite thing either.

I would rather just write code and figure out what it's going to do later.

But I think it is an important part of the business.

It's just something that we haven't really figured out well yet.

I think the last thing I will say is, I feel like I'm always convincing business people, no, no, I really do like buying stuff.

I really, really do.

It sounds great to buy stuff.

I just have doubts.

It's as easy as the salespeople say it is.

It's not that I want to build everything.

I swear.

Yeah, I quite often make attempts to slip in mentions of things that we actually are buying so that I can-- Pad the credentials of being a guy who's not always building.

Yeah, I can sort of fend off these claims of you just want to build it yourself.

I can say, well, we're using this API.

We're using this tool.

All these things have worked out well.

Sometimes they haven't.

But I think if you-- one thing that the company I'm currently at does, which helps, is that very often we have to vet companies for security issues.

So there's kind of a process of vendor analysis in any case.

So even if you've chosen somebody that you want to go with, you still have to do a pretty deep dive into their business practices and whether or not they're going to be available for the long term and that kind of thing.

And sometimes you have to make decisions about if you do go with this vendor, what happens if they go down for two days?

What does that mean for the business?

So I think there's some advantage in being-- even though I don't like doing it, I think there's some advantage of having these processes where you do look closely at the vendors, and when you are relatively certain you're going to use them.

Yeah.

And then when something goes south, you get to finger point, which is pretty cool.

There's two aspects of this we haven't talked about, but we're pretty much out of time.

I'm just going to raise them really briefly here at the end.

One, we didn't talk about what it means to buy in the sense of open source, which is not a buy.

It's I'm buying for free.

I don't know.

That's implementation, I guess.

That's one thing.

I think we'll probably leave that one.

The last, last unknown when it comes to buy is some tools.

You buy them, and you got to learn how to use them, and you got to pay someone to operate the tool.

I'll give you an example, like a Power BI or a Tableau.

These are beautiful reporting solutions.

You can make incredible stuff with them.

It looks gorgeous.

It is not a buy it and immediately use it from day one.

It's a buy it and have a team of people or one person who gets very good at operating it.

That's a complicated decision too.

If I buy this thing, I need expertise, and I need a lot of learning just to be able to come up with a speed on it.

One thing we've managed to avoid so far is the total cost of ownership thing.

I think that figures into both of these questions.

In the open source case, the old school cost models would say those are free in the sense that you get all the code, and you don't have to build it.

You don't have to pay for it.

There's still the aspect of adapting it to your needs, the shims as you say.

And updates.

Updates.

There may be an evaluation process there too because there's more than one choice.

And you still have to have somebody has to set it up and run it, and you may have to pay for resources to keep that thing operating.

And then with the reporting tools, your total cost of ownership includes the people that you have to pay to learn and run those things.

Again, not strictly part of the build versus buy decision, but I think you do have to look at total cost of ownership when you're considering these alternatives.

This is so complicated.

So Mike, when we discuss these topics, often after our discussion, something related will come up at work or somewhere else, and I'll say, you know, in this paper or this essay or this discussion that Mike and I had, I learned this thing, and I don't get to do that as a result of this one.

The question is still as hard as it ever was.

The only thing I'll get to say is when Mike and I talked about building an underwater Excel company, we decided that these were the things we would definitely buy and these were the things we would definitely build.

And I just don't think people are going to take me seriously if that's my only thing to offer to the discussion.

Yeah, I'm very intrigued by the underwater Excel example though.

All right.

Well, that's what we're going to end it.

We're going to try to take it offline.

We'll start building a business proposal for it.

We'll find some investors.

And maybe in a year, you'll see us pitching this at conferences.

You need to work on spreadsheets while scuba diving.

I guess I would have to become a certified diver though.

Yeah, we do need to do that this year. 2024, that's the year we're going to do that.

All right.

This has been Picture Recoding with Mike Moll and Eric Aker.

Today we talked about build versus buy, and we just frustrated the conversation, and it still is annoying me this question.

Yeah, if anybody out there knows how to do it, drop us a line.

That's your new thing.

Yeah.

If you're listening to this and you have the answer that we're looking for, please send it to us.

Once again, if you want to reach out to us, you can reach us at podcast@picturerecoding.com.

Thanks so much, Mike.

We'll see you next week.