(upbeat music) 
- Hello, welcome to Picture Recoding with Eric Aker and Mike Moll.

Mike, we have so much stuff to cover this week that I don't even care how you're doing.

I don't even wanna ask you how you're doing.

I just wanna get right to it.

Are you up for that? - Let's do that. - Okay. - 'Cause I don't wanna talk about how I'm doing either. - I wasn't serious about not caring.

I do care.

I care.

Have you been listening to music?

We haven't talked about music in a little while.

This is a music podcast about software. - I've been listening to a lot of music as I always do.

I think my choice for this week is the latest album from Nilefer Yanya.

So-- - You've never heard of this artist. - She's a British artist.

She's had a series of really good albums.

She has this new one called My Method Actor, which I really like. - What's a style of music? - So it's kind of right on the borderline between pop music and rock music. - Like pop rocks, you put it in your mouth with soda and it blows up. - No, not exactly.

But it's very interesting lyrically.

She is a pretty decent guitar player.

It's a large describe, but I'm trying to think of somebody comparable, but-- - Meet Love. - No, no. - Okay. - But there's something about her voice that I really like too.

I mean, I think she's British by nationality, but I think her family came from Turkey, I think. - Nilifer Yanya. - Yeah, something really interesting about her voice.

It's kind of slightly deeper than you'd expect and I don't know.

Just the way she says words is interesting.

But yeah, this is like her third album in a row that's really good.

And so I just thought I had to mention her. - Wow, I gotta listen to this person.

Third album in a row.

I just can't stop listening to the Good Looks album.

Lived Here For A While is what it's called.

This is, I think, my favorite record that I've heard this year.

I can't stop listening to it, playing it over and over and over again.

The first song is called If It's Gone, and it has this line that always gets me right in the chest where the singer says, "I hope you find the courage to tell your parents "what they did when they left you all alone, "locked up in that room when you were a kid." (laughs) Well, he sings that, I just picture this poor little kid.

And just I hear that line, "Oof, ouch, geez."

It's a great record.

I love this one.

I can't stop listening to it. - This is the same band that did the-- - Bummer Year. - Bummer Year, right? - Best song from last year that I heard. - Yeah, I listened to this album too.

I liked it, but there was not a Bummer Year type of song on here that grabbed me. - Yeah, I know what you mean, but I know what you mean.

Bummer Year is like one of the greatest songs.

But I don't know, there's some stuff on here I really like.

I like the flow of it.

I like what they start with.

I like what they lead into.

I like how it changes and adapts.

It's an interesting record.

It's a little more, there's some sort of different tones and pacing and stuff in there than their first record. - Nice. - All right, so interesting thing happened for us, Mike.

We have been podcasting now for over a year.

We are on, I think this is episode 46.

And we have received an email from a fan for the first time in our podcasting careers.

This email comes from Randy Edwards.

I'm gonna read it to you, Mike.

Randy writes, "What have you been listening to?

I'm relying on you guys to discover new music."

Mike, I always feel like you're accusing me of killing our music description here.

Did you put Randy up to this?

(laughing) I did not put him up to it, but I appreciate that he pointed it out.

To be fair, we typically ask guests if they have music to share with us because we don't want to put them on the spot.

So on episodes with guests, we typically do not have music, but it is an important feature to me.

So I'm glad that he's pointing it out. - It's an important part of our lives, but we're kind of weird, I guess, right?

(laughing) There's not really many other music podcasts about software in the universe of podcasting.

Okay, let me keep going with Randy's email.

Randy writes, "I also take minor issue with your treatment of story points.

My understanding is that the reason story points exist as an abstraction from directly estimating time is that empirical evidence suggests that human beings produce more consistent and accurate estimations when doing relative estimation.

I think it's valid to convert from story points to time in the aggregate, though not to do the reverse."

What does that mean?

In the aggregate, it's valid to go for story points to time in the aggregate, but not the reverse from time to story points. - So I think he's saying that you wouldn't want to say, this is going to take three weeks.

How many story points is that? - Oh, okay, okay.

All right, so that's not valid.

He gives an example.

For example, Randy writes, "In my org, we estimate using story points, but capture our effort how long it took to resolve a ticket using hours.

The historic data then gives us a way to convert from story points to hours during project planning when we're estimating very large projects."

Of course, we're still considering this a rough approximation and adding a healthy buffer, but it's been pretty accurate for the past few years.

It helps if you keep the projects as small as possible.

Obviously, software development estimation is still a black art and I wouldn't swear by any of this, but it's a functioning mechanic for now.

Hope you guys are having fun.

I love the podcast, Randy writes.

Thanks so much, Randy.

That's a very nice email.

I appreciate it.

When he says, "We estimate story points, but capture our effort using hours, how long it took to resolve a ticket."

I think there's a detail in there, Mike, that I've heard you point out many times.

Most people don't do that, right?

They don't actually go back and say, "How long did this actually take me?"

Isn't that your critique?

That's one of the things I said, and clearly people do do it, but it's been my experience that it's one of those things that gets lost.

I suppose it depends on, like if you have full-time scrum masters and somebody who's really driving the retrospectives and so forth, people probably put more time into it.

That's interesting 'cause I'm too busy to go back and figure out how long it took me to finish the thing.

If you have somebody else do it for me, that helps me out.

You also have to track it too, which is a can of worms in and of itself.

If you start a task on Monday and finish it on Friday, it probably didn't take you five days to do, so you have to have a pretty good sense of how many hours you actually spent on that task.

It sounds like it's working for Randy, though, and I think part of that is that detail that you said a lot of people leave this out.

They don't actually go back and say, "How long did it really take us to do this thing?"

And if you have both of those data sets, your estimation using an abstract quantity, story point, and the actual, "How long it took you to do it?"

Together, these two things can inform how long it'll take you to do things in the future.

I buy that.

I buy what Randy's saying here. - Yeah, I should also probably point out that we happen to know that Randy is a manager of software engineering teams.

And that's-- - That's true. - Yeah, we do know him. - This is important to him, and so his take on this should probably be given more credence than mine should. - But do you think that, I wanna come back to that point, but I was curious about one of the first things he said here, "Story points exist as an abstraction "because humans produce more consistent "and accurate estimations when doing relative estimation."

I don't know how to weigh this claim.

What do you think about that? - I think that's fair.

I think we kind of touched on that in the episode.

We talked about how you can look at something and say, "Okay, this is a one-liner.

"I can give it one point, "or this is something I know how to do, "but it's gotta take me a couple days "so I can give this so many points."

And when you have a certain level of experience, you do have some sense of how relatively long two tasks are.

I think my problem with it is that you are doing the relative estimation typically within the context of one sprint planning session.

And so you're not comparing what a three in this session is to a three in the next session necessarily.

So I think his approach of trying to capture the hours after the fact is probably as good as we can do with this now. - Now, I just wanna remind our listeners, Randy sent us an email at the email address for the podcast.

Our email address is podcast@picturemecoding.com.

If you wanna send us feedback, please send us email at that address.

You can also do the things that we're supposed to recommend that people do who listen to the show, where they review and they give us stars and stuff on different platforms, but you don't have to do that.

We won't hold you to it.

Here's the thing, you mentioned Randy is a software engineering manager.

And when I was reading Randy's email, I thought, "Oh, I respect what Randy's saying.

"I respect his perspective here, "but sometimes Mike, I wonder if you and I, "we have been on the individual contributor path "for so long that we can't even really understand "and sympathize with the goals and desires "of someone in Randy's position."

It's kinda like that movie Mystic River where you've got these guys who grew up in the same neighborhood and some of them become, one of them becomes a police officer and the other ones become like gangsters and the police officer kinda can't sympathize with, or it's like heat, you're Robert De Niro and Randy is Al Pacino.

What does that make me?

I guess I'm Val Kilmer. - You're Val Kilmer. - Yeah, okay, I'm Val Kilmer.

And the problem is we're on opposite sides of that table.

He's the cop, the engineering manager and I'm the individual contributor.

And you just wanna pull off one last heist, one last big score, one last massive project. (laughing) You don't want the fuzz telling you, I need to know how long it's gonna take you to build that thing, Mike, right? - I really love this analogy. - Because it makes you Robert De Niro, is that why?

(laughing) - Among other things, but I have to confess, I did spend fairly large chunks of my career being a manager of various sorts. - Oh, so you were like one of those movies about a corrupt cop, right?

You were a bad cop. - Yes, both figuratively and literally in the sense that I was a pretty bad manager because I think my predilection is to be a guy who's cracking the safe, not the guy who's-- - Yeah, well I could see that about you.

Yeah, do you think it's like we're on opposite sides of the table and we really cannot sympathize with each other's concerns?

The criminal sees the desires and maybe respects the desires of the cop and the cop maybe respects the wildiness and skill of the criminal, but ultimately their goals are in contradiction to each other.

Widdershins. - I should also probably point out that knowing Randy, I also know that he's a talented engineer, so he's perfectly competent, safe record as well. - He could go the criminal path if he wanted to is what you're saying.

He didn't have to go.

He went straight because he was not a good criminal. - Yeah, I mean, when he was on the criminal path, he was a perfectly good criminal. - Yeah, he did some great heists. - Great heists.

So where was I going with this?

I kind of forgotten now.

Anyway. - Randy brought up time and we wanted to spend our time today talking about time and I wanted to use the word time as many times as possible when we talk about time.

I have a story for you.

This is, you ever see that Reddit thing?

I, M-I-T-A-S-HOL, A-I-T-A, am I the asshole?

You ever see that before?

People, maybe it's not just Reddit.

Maybe there's something. - Yeah, I see that all the time. - I have one of those for you.

This is true story.

What time at work?

I was working on a project.

It's an existing software project.

It's just a basic, like I've got a payload and I need to send it to an HTTP API.

That's pretty much it.

This is a banking related thing and it has to do with releasing funds for some of our customers, right?

And as part of releasing the funds, we had to say what day we wanted them to be released on and we send this over in this particular time format.

It was this like Windows server that they were running and I had to send it.

This is like Microsoft timestamp.

It's pretty much like epoch time to the millisecond with this MS prefix on it.

But that timestamp was gonna get interpreted on the server side as when do you want the funds released?

And they pretty much told us verbally, we don't care about the time, we care about the day.

And so this was running for a few years and then something happened where, I don't remember exactly, I don't think it was like as blunt as like a leap day type of scenario.

But one day they were like, well, the funds were released a day later.

I can't, I actually can't even remember it was a day earlier, day later.

And I saw it got in this discussion with their lead developer.

And I said, okay, well, what time zone do you want me to use when I produce this timestamp to produce a date?

And their lead developer said, and I'm not making this up.

He said, dates don't have time zones.

And I got kind of mad about that.

And I was like, you're out of your mind.

I am going to produce a timestamp at a certain point in the day.

And I have to pick the time zone.

I want to produce that time zone in timestamp in.

I have to pick the time zone, right?

Most of the time my programs emit timestamps in UTC.

And he was like, it doesn't matter.

That's not right.

Cause that's somewhere else.

Like if it's, I was like, yeah, but what you're saying is I have to figure out where your service running and produce a date that matches.

Like if I'm an hour off and we're at the end of the day and he just kept insisting dates don't have time zones.

Well, he's, you know, that's why everybody in the world celebrates New Year's Eve at exactly the same time. (laughing) Yeah, I guess that's true.

I got it.

I just like, I got so mad and so confused trying to explain to him that I just don't think it's viable.

I don't think it's correct for me to ignore the existence of time zones when I produce a date on my server, which is probably going to be using UTC time.

Am I the asshole in the story?

No, my charitable interpretation of what he meant is that if it's Monday and you want those funds to be released on Tuesday, you should send me a date which corresponds to Tuesday.

Yeah, I mean, it seems to me like his software should have been smart enough to understand what date at timestamp.

So you say to, so your charitable reading, if you say to me, send me a date that corresponds to Tuesday, then my question is, this is my honest question.

You could tell me, I'm crazy.

Do I need to know what time zone your computer is respecting when you say what is Tuesday, when is it Tuesday?

Well, yes and no.

I mean, I think, yes, I think you would because I'm trying to craft this example in my brain, but let's say that the Tuesday is going to be, say, the 21st of the month for every time zone.

I mean, there's no time zone in which it's not going to be the 21st.

So you have to sort of work backwards in your brain to know that if I send him a UTC date and he's in, say, mountain time and he interprets that UTC date as being, yeah, I'm just, even talking through it, I'm confusing myself.

Is that happening to me?

I got real mad and then I tried to explain why it was wrong and then I confused myself and here I am telling you and I'm confusing myself.

Why is time so hard?

That's my question for you.

Because it's an illusion.

It's not real. (laughing) That may be true, but that's not why.

We're talking about time on a couple of different levels today, but I think at this time, at this sort of concept of time.

In our relationship. (laughing) Well, when you're talking about things like time zones and dates and how those things relate, I've been doing this for a long time and I still have to think about this stuff and look up what it means and how my databases are storing timestamps and do I need to supply them with time zones?

And so I can't quite describe to lay people why this is so annoying, but it's just something that seems to be unsolvable.

It's not because we're doing it wrong or because somebody hasn't come up with a better solution.

It's just because it's a tricky thing.

I think my feelings about dealing with time in computer programs occupy a quantum superposition.

It's both my least favorite and most favorite topic at the same time. (laughing) Yeah.

I wanna read to you, this is a kind of a longer passage that I wanna break up.

I wanna read to you something I encountered in a distributed systems book.

It's actually a textbook about distributed systems called distributed systems.

It's by Martin Vonstein.

You can actually download this book if you go to the website, distributedsystems.net.

You can read it for free if you want to.

It's got some good stuff in it.

There's some stuff that we know, but there's a lot of really interesting sections in it.

It has a whole section on clocks and time, as you might imagine, for a textbook about distributed systems.

I think it's about halfway through the book, page 200, something like that.

So I'm gonna read to you a couple paragraphs, bear with me, and I'm gonna pause because when I read these, I was aghast. (laughing) Okay, here's how it goes.

Since the invention of mechanical clocks in the 17th century, time has been measured astronomically.

Every day, the sun appears to rise on the eastern horizon, then climbs to a maximum height in the sky and finally sinks in the west.

The event of the sun's reaching as highest apparent point in the sky is called the Transcendent Sun.

This event occurs at about noon each day.

The interval between two consecutive transits of the sun is called the solar day.

Since there are 24 hours in a day, each containing 3,600 seconds, the solar second is defined as exactly one, 86,400th of a solar day.

So there's 86,400 seconds in a 24-hour period.

This is from page 251 of the distributed systems textbook.

All right, you with me so far?

Solar day, there's a lot of like abouts and it's arounds.

It's pretty fuzzy feeling already, but we're on solid ground, right?

24 hours in a day, 3,600 seconds.

In an hour and 86,400 seconds in a day.

Okay, you're good so far? - I'm good. - Okay, here's where it gets fun.

In the 1940s, it was established that the period of the Earth's rotation is not constant.

The Earth is slowing down due to tidal friction, an atmospheric drag based on studies of growth patterns and ancient coral.

Geologists now believe that 300 million years ago, there were about 400 days per year.

The length of the year, the time for one trip around the sun is not thought to have changed.

The day has simply become longer.

In addition to this long-term trend, short-term variations in the length of the day also occur, probably caused by turbulence deep in the Earth's core of molten iron.

These revelations lead astronomers to compute the length of the day by measuring a large number of days and taking the average before dividing by 86,400, the resulting quantity was called the mean solar second.

All right, have you heard this junk before?

The days are getting longer, Mike.

Is that why I'm so tired all the time? - Yeah, it's the tidal friction.

I love that expression, tidal friction.

I read and I looked us up on Wikipedia about over a billion years ago, they estimate that the day was about 19 hours long.

I think those people who worked, went to jobs a billion years ago, they must have gotten so much better sleep than we do.

They only had a 19-hour day.

They probably only had like a six-hour work day. - Yeah, but they didn't have TV either.

So, I mean, it was like once it got dark, what were we gonna do? - Yeah, no smartphones.

He's pretty much camping all the time. - Yeah.

So this is from Wikipedia.

There's actually a group dedicated to measuring and reporting the Earth's rotation.

They're called the International Earth Rotation and Reference System Service.

And they have the thing called the Earth Orientation Parameter at International Celestial Reference System.

I didn't know if this stuff was so fuzzy that you had to have groups to measure the stuff, that they shift.

I was looking at day-length fluctuations and throughout the year, like if you had a sundial, the clock will be off of the actual day-length at varying intervals on these like sine waves.

I was looking at leap seconds.

Here's from Wikipedia.

"Title deceleration rates have varied over the history of the Earth, Moon system.

Analysis of layering in fossil mollusks shells from 70 million years ago in the Lake Cretaceous Period shows there were 372 days a year.

And thus the day was about 23 and a half hours long.

And then they go back 620 million years.

It was 21.9 hours long.

There were about 13 months a year and 400 solar days per year.

I just want to go back to this period 600 million years ago when we had 400 days in a year.

Because then the time when they start putting out all the Halloween and the Christmas crap in the stores is so much longer.

You can enjoy summer that much longer when your Earth's rotation is that much.

Wait a minute, faster, right?

The day's going by? - Yeah, my neighbor's already have out there Halloween stuff, so. - I just, I can't really abide by this Earth rotation slowing down.

Okay, so here's my question for you as a software engineer.

Let's say you go to work in a lab that does, I don't know, climate science or geology.

And they're like, "Mike, I want you to calculate the mean, I don't know, ocean acidity over this interval."

And it happens to be like a couple hundred million years or a million years.

Do you got to like take into consideration the fact that the day is going to change to figure out that interval?

Like, does that make time, which is already hard for us, just bonkers difficult to deal with? - Probably.

I can't think of an example off the top of my head, but I suspect this does come into play.

Although, you know, to be fair, if you look at the numbers here, like 620 million years ago, there's not a whole lot of geological record, 600 from, 620 million years ago.

I don't know what title rhythmites are, but. - I don't know what that is.

It's just like a weird rhythm.

That's a rhythm there.

There's a lot of funny words.

Actually, you read these Wikipedia pages, there's so many words and technical terms, I didn't understand. - I don't know how much like biological record there was at that point in time, but it is an interesting question, how much you have to take these things into account. - All I know is our days are getting longer and our government should do something about that. - Yeah, I think there should probably be rotation control limits placed. - Like we all have to run to the other side of the country to slow it down, to speed it up.

That's, I'm confused, I'm confused with myself.

All right, wait, let me continue reading from the distributed systems book.

So the day used to be shorter, hundreds of millions of years ago.

It's longer now.

We've talked about how there are 86,400 seconds.

We talked about the mean solar day.

What does this mean for us?

If I wanna know what time it is in my program, how do I figure that out?

So the distributed systems book, Martin Rensdien, continues saying, "With the invention "of the atomic clock in 1948, "it became possible to measure time much more accurately "and independent of the wiggling and wobbling of the earth "by counting transitions of the cesium 133 atom."

Well, this sounds fantastic, Mike.

We can measure time much more accurately.

This is like an unalloyed win, right?

There's no downsides to this. - That's a good voice for that setup. - I'm an optimist, okay?

I see they invent the cesium clock and you get extremely accurate time and I think, "Well, that sounds great."

I'll just ask that thing what time it is, win.

Okay, and the book continues, "The physicists took over the job of timekeeping "from the astronomers," which I wonder if the astronomers were bummed about that, "and they defined the second to be the time "it takes the cesium 133 atom to make exactly 9,192,631,770 transitions."

The choice of 9,192,631,770 was made to make the atomic second equal to the mean solar second in the year of its introduction.

So this is the 1948 vintage of the cesium atoms transitions.

Currently, several, that doesn't sound like a large number, several laboratories around the world have cesium 133 clocks.

Periodically, each laboratory tells the Bureau internationale de Lour, B.I.H. in Paris, how many times its clock has ticked.

The B.I.H. averages these to produce international atomic time, which confusingly is abbreviated to T.A.I.

I always forget T.A.I., international atomic time, since it's like a backwards.

Thus, T.A.I. is just the mean number of ticks of the cesium 133 clocks since midnight January 1st, 1958, the beginning of time, divided by 9,192,633,770.

Okay, I get a little confused.

Are you following this so far?

Is this making sense to you?

A little bit.

So two minor nitpicks here.

One is, I think, probably a large fraction of astronomers consider themselves to be physicists.

Okay, all right.

Yeah, that's kind of a diss.

The physicist hooked over the Java time cube from the astronomers.

Let's throw that sentence away.

Sorry, it should be just a sex book.

So I get what they're saying here.

I have a little bit of-- I don't like their description of the cesium clock.

Why?

Because I think it makes it sound like the cesium is like actively radiating something that people are counting.

And I don't think that's exactly how they work.

There's a better description of it, you think?

Well, it probably does not matter at all.

I read a little bit about these things and my interpretation is that this atom, cesium-133, has an interesting physics.

So it's the lowest energy state, which is called the ground state.

Actually, it has what they call a hyperfine structure.

It's got something to do with the quantum spins of the nuclei and the electrons or something.

So there's this small energy gap, what they call a hyperfine structure.

And so I don't think there's anything like being emitted from the cesium atom actively.

I think I probably gleaned about 20% of what you just said.

Yeah, what I said may be wrong too.

Is it like a long polling thing?

My alarm just went off.

Let's go check and see what the cesium-133 atom says as time is.

Is that how they're doing it?

No, I don't.

Well, yes and no.

So I watched this video about the guy at NIST, the National Institutes of Standards and Technology.

NIST, who apparently invented these clocks or was involved in inventing these clocks.

And I guess they call him the Time Lord at NIST now.

Time Lord?

That's a job title?

Yeah, I don't think it's his job title.

I think it's just his unofficial designation.

Wow.

My understanding is that, like it says in here, there's a bunch of laboratories that have these clocks.

And they constantly are trying to verify that they are synchronized.

And they're kind of like also the root clocks for these various time synchronization protocols that get used in server machines.

Here's what I want to do.

I want to go, hey, the physicists and astronomers, physicists/astronomers invented some super accurate way of telling the time.

I'm just going to ask them what time it is every time I want to know.

That'll work.

OK, so international atomic time is abbreviated TAI.

I'm going to continue to read you more from the Distribute Systems textbook.

Although TAI is highly stable and available to anyone who wants to go to the trouble of buying a cesium clock, Sidebar, can you buy a cesium clock?

If you had enough money, probably.

Probably a lot of money.

All right, so anybody could buy one.

There is a serious problem with it.

Well, who saw that coming?

I didn't.

86,400 TAI seconds-- those are cesium clock seconds-- is now about three milliseconds less than a mean solar day, because the mean solar day is getting longer all the time.

I bummed about this.

Three milliseconds is a lot, too.

It's kind of surprising.

That's a bummer, OK?

I thought it was just that time was confusing for me.

It's confusing for the universe.

OK, so here's what the Distribute Systems textbooks continues to say, using TAI for keeping time, which is what I wanted to do, Mike, would mean that over the course of the years, noon would get earlier and earlier until it would eventually occur in the wee hours of the morning.

People might notice this.

And we could have the same kind of situation as occurred in 1582 when Pope Gregory XIII decreed that 10 days be omitted from the calendar.

This event caused riots in the streets because landlords demanded a full month's rent and bankers a full month's interest, while employers refused to pay workers for the 10 days they did not work to mention only a few of the conflicts.

The Protestant countries, as a matter of principle, refused to have anything to do with papal decrees and did not accept the Gregorian calendar for 170 years.

OK, sidebar here.

We started talking about time, and now we're like in the midst of a Dan Brown novel.

What the hell's going on?

This is actually slightly more interesting than a Dan Brown novel.

I actually haven't read one.

OK, so let me just try to sum up.

We have a day that gets longer because of title variation, because the Earth rotates more slowly over time, over a long time, geological time.

OK, all right, all right.

We have the invention.

OK, I'm already saying something wrong.

You're saying?

Well, I mean, yeah, I mean, but that 3 millisecond discrepancy is just since, like, 1948, right?

Oh, OK, so we have pre-1948, pre-season clock time, post-season clock time.

But then the cesium clocks started marking time in 1958.

So that's when time started.

And then Unix time started in 1970, probably.

I'm already so confused, Mike.

And the cesium clocks don't match, as you pointed out, the 3 milliseconds.

How do we solve this?

I'm going to continue.

So the Paris Bureau of the Hour Bureau of Time, B I H, they solve the problem-- I'm still quoting from the Distribute Systems textbook.

They solve the problem by introducing leap seconds whenever the discrepancy between TAI and solar time grows to 800 milliseconds.

I wonder how they picked that.

Do you think they picked it because it's like, hey, let's just fix it anytime somebody notices?

Yeah, I don't know.

Other than the fact that it's 80% of a second take, I don't really-- I bet they went to a conference.

And they just went around and they were like, let's test how what the interval people tend to notice stuff.

And they settle that around 800 seconds, people notice that something weird is happening.

That's when we're going to fix the cesium clocks, the TAI.

So this correction gives rise to a system based on a constant TAI seconds, but which stays in phase with the apparent motion of the sun.

This time system is known as Coordinated Universal Time, abbreviated to UTC.

And for me, it's like, mind blowing gift, UTC.

This is where the leap's-- I didn't know this.

Maybe everybody knew this, but me.

Did you know all this?

Not in this detail.

No.

The next part, I think, is the part that was even more mind blowing to me, though.

OK.

So before we continue, I went to the-- Wikipedia has just mind boggling content about this.

They have this delta T timekeeping page.

They have day length fluctuation page, where the day will fluctuate throughout the year.

They have the leap second-- oh, sorry.

And not only does it fluctuate, but it fluctuates like a sundial.

It would be different than your clock.

They have a leap second page.

The leap second page is phenomenally astounding itself.

I read on the leap second page, there's a plan to abandon leap seconds by 2035.

And I was like, oh, shit.

What's the next thing?

I can't even figure out what the next thing is going to be.

Who knows what they're going to do?

So-- Leap seconds.

So I always had heard about leap seconds, and I didn't really get it.

I just thought, oh, yeah, they just introduced that to make sure that-- I kind of knew vaguely that the atomic clocks were somehow more accurate than what we think of as like, oh, what time it is based on where the sun is in the sky.

So I'm going to read to you a few more chunks of this book.

I-- hopefully this is not dull, me just reading chunks of this book.

I just-- I couldn't believe this when I counted this.

So more from Distribute Systems book.

The leap second was introduced in 1972.

Since then-- and I actually have a really nice graph of this on one of the Wikipedia pages.

Since then, 27 leap seconds have been added to UTC with the most recent occurring on December 31, 2016.

All have so far been positive leap seconds, adding a second to a UTC day.

While it is possible for a negative leap second to be needed, one has not happened yet.

Oh, I'm sorry.

I'm actually-- I messed up my reference.

This is from the Wikipedia page on leap seconds.

So this is what stunned me.

It's possible to have a negative leap second.

Mike?

Is that a leap, though?

It's a leap backwards.

It's actually like in the show, quantum leap.

They went backwards.

Right?

Yeah.

So that was from Wikipedia.

Introduced in 1972.

There's only been 27.

OK.

Most recent one, 2016.

And we've never had a negative one, but we could.

It's legitimate to have a negative one.

Because the Earth's rotation-- this is still Wikipedia-- Earth's rotational speed varies in response to climactic and geological events.

UTC leap seconds are irregularly spaced and unpredictable.

Insertion of each UTC leap second is usually decided in about six months in advance by the International Earth Rotation and Reference System Service.

Those guys who-- those people out there who are measuring the rotation of the Earth, they decide.

And the goal is to ensure that the difference between UTC and UT1 readings will never exceed 0.9 seconds.

So still about 800 milliseconds.

And here's what the Wikipedia page says.

This practice has proven disruptive, particularly in the 21st century, and especially in services that depend on precise time stamping or time critical process control.

Now, I have never worked on one of those, Mike.

I don't really care that much.

If we're within 900 milliseconds, I'm kind of exaggerating.

I usually do.

And since not all computers are adjusted by leap second, they will display times differing from those that have been adjusted.

After many years of discussions by different standard bodies in November 2022, at the 27th General Conference on Weights and Measures, it was decided to abandon a leap second by or before 2035.

So people just pursue a pissed about this.

I don't know what they replace it with, though, because you can't just stop adjusting time, right?

Yeah.

Well, I mean, if you go back to the distributed systems and it's like the book, if you just go by the cesium clock, eventually, it's your solar day slows.

It gets longer, which, again, is why I'm so tired all the time.

Then your clock eventually doesn't correspond to the reality that you see when you look out the window.

It's either night or day.

So I suppose since this is a music podcast about software, we should probably, at some point in time, try to explain why this matters to software.

I started off on this journey with you, talking about time.

And my goal was things will get clearer.

And this weird thing happened where they have not gotten clearer.

They've actually gotten more cloudy and confused for me.

Yes.

And also, we haven't even touched on the subject of a relativity yet.

Oh, maybe we save that for a future day.

Do you think it matters?

How does this stuff matter for my programs?

You mean the time stuff in general?

Yeah, time.

I could just do everything in UTC, right?

Problem solved.

Somebody else handles this.

I think for the most part, what matters in software is that things which are collaborating have the same time.

I think it's just the problem is that because a lot of the systems now that we're dealing with are so global that it's kind of hard to make sure that everything has the same global time.

I did one of the first programs that I wrote.

I was trying to send out text messages during a period of like, oh, it's in the morning for you.

And I had to have a database.

And they might be anywhere in the world.

And I was like, this will be easy.

I suppose that in most programmers' lifetimes, the difference-- having two servers where the clocks are different by a millisecond is probably-- or a millisecond, let's say, is probably inconsequential in most cases.

But there are quite a few situations where you do need to have all the systems in a distributed system on essentially the same time.

Well, can you give an example?

Well, I think there's some basic ones.

Like, some database systems seem to use time stamps as a way to decide when to commit things.

Actually, this distributed systems book, this whole chunk I've been reading to you, they talk about Google Spanner and the True Time Service.

Yeah.

Yeah, Spanner, I think, is one of the interesting examples.

Again, probably something-- like, if you're writing a typical credit application, something like Spanner is probably not something that you need to think about.

But there are systems where it's important.

Distributed logging is something that people might-- in almost any environment might run into.

There's cases where you need the time stamps to be consistent so that you can look at-- if you're trying to analyze a problem or something, you need to make sure that you're getting the log outputs in the right order.

That's a good point.

You need to know what happens before the other thing.

Echoes of Lamport there happened before.

Yeah, important causality concerns there.

I don't know too much about it, but I understand that it's extremely important for cell phone towers.

There's the cell phone system in general for a number of reasons.

I think part of it is because some of the transmission methods actually use-- they're actually using the same frequency sometimes for transmission and sometimes for receiving.

And so there's this timing so that you know whether or not the signal on that frequency is a transmission or a reception.

Wow, that's an interesting example.

What about the Cloudflare bug?

We talked about this in our-- I think it was going to take a lot of fireworks to clean this place up.

The bugs, massive bugs, episode that we did.

Maybe I'm wrong about when we talked about it.

But Cloudflare had a bug where their systems interpreted time as having gone backwards, I think because of the leap second, right?

Because they were just subtracting the moment before from the moment after.

And time looked like it went backwards when it paused.

And it just completely broke DNS for chunks of the internet, significant chunks of the internet.

I guess they needed a negative leap second there.

Here's-- there was this other thing that surprised me.

I'd never heard this before.

In the distributed systems book, they talk about how most power companies, they synchronize their clocks to UTC.

So when there's a leap second announced, the power companies-- actually, this is from the book.

The power companies raise their frequency to 61 hertz or 51 hertz for 60 seconds or 50 seconds to advance all clocks in their distribution area.

I was like, what?

[LAUGHTER] So I've got a clock plugged into the wall.

And the power company's going to help keep that in time.

Is that what they're saying?

I was like, what are you even talking about?

Yeah, I was unclear if this is only for their internal use or not.

But yeah, this thing about changing the frequency to 61 or 51 hertz was kind of mind blowing to me.

All of this is mind blowing.

There were 400 days in the year 600 million years ago, Mike.

I want to go back to 600 million years ago and have 400 days.

And that's what I really envy those people who got up in the morning, drank coffee, and had another 30 days in their year to just sort of relax and chill.

Yeah, I think the man would have just made them work more.

Probably.

The dinosaur man, you mean?

Dinosaur man, yeah.

So the basis, the services book for keeping global times called Coordinated Universal Time, but it is abbreviated as UTC.

UTC is the basis of all modern civil time keeping, and it is a worldwide standard.

To provide UTC to people who need precise time, some 40 shortwave radio stations around the world broadcast a short pulse at the start of each UTC second.

I'd never heard this either.

It's like, it is apparently impossible to do time on your own.

It's kind of like language in that way, right?

There's a weird big design reference for you.

The accuracy of these radio stations is about plus or minus one millisecond, but there's no free lunch in this conversation.

Okay, there's always a but, but due to random atmospheric fluctuations that can affect the length of the signal path and practice the accuracy is no better than plus or minus 10 milliseconds.

This is a brutal area of concern for us.

Yeah.

Okay, maybe satellites get better then.

Go ahead.

That's a lot of milliseconds.

I think one thing that I think maybe is not obvious to everyone is that 10 milliseconds is a pretty big, some software applications that's pretty significant.

It sounds like you're talking about one 100th of a second there seems like that should be insignificant, but that actually is a pretty big discrepancy in some applications.

I only get a hundred of those things in my CPU slice, in my CPU share that I can slice up.

That's significant.

I mean, remember Bob talking about high frequency trading and they're interested in microseconds?

Like how fast I could do things in microseconds?

This must be mind boggling for them to deal with.

And in fact, there was a podcast, it was too many years ago from standard charter where they were talking about how to keep their servers in sync in time.

And it was incredible the amount of engineering effort they have to go to to make the servers, to make them agree on what time it is.

So we could talk to satellites.

Satellites can provide UTC accurately up to half a millisecond.

Some other satellites do even better.

This is finally the last sentence from the distributed systems book I'm gonna read to you.

By combining receptions from several satellites, ground time servers can be built, offering an accuracy of 50 nanoseconds.

Okay, 50 nanoseconds, pretty good.

Why do I not feel totally comforted by this in the end? - I think there's something, there's something discomforting about the fact that even when the time difference is so minuscule, there's something unsettling about the fact that we can't just make it the same.

It seems like there should be a way to agree on the exact time down to the nanosecond, but. - You got that right.

There's something unsettling about it.

You got that right.

What have we learned so far?

We have learned that the day is getting longer, which kind of sucks.

We've learned that cesium 133 atom clocks, pretty cool, very accurate, but that also doesn't help us that much because they stop agreeing with the actual time it is when I look out my window and I see how light it is outside.

And the other problem is, I need to actually go out and ask one of those super accurate clocks how long it is, and that's gonna take some time.

Or I need to talk to a bunch of satellites, and that's gonna take time.

So just sending the messages around the world is gonna take time.

It takes time to figure out what time it is.

And Mike, I'm really not sure if any of this has gotten us any closer to determining if I was the asshole for telling that guy I need to know what time zone to put your date in.

I'm going to declare that you are not the asshole.

I'm going to accept your declaration and I'm going to overlook the fact that you are a friend of mine and so you will probably be kind to me.

And I definitely think that if anybody ever makes that claim again, you should give them a detailed lecture on tidal friction and leap seconds.

I should tell them.

You know what, my friend, 620 million years ago, there were 400 days in the year.

Okay, don't give me this crap about dates not having time zones. - That's right. - How many time zones do you think they had when it was 400 days in the year? - Zero, one.

I guess they had one. - You think they just had one big time zone?

That must have been so hard for those programmers 600 million years ago.

Mike, you did a little bit of reading about NTP and I also read a little bit about NTP in the Distribute Systems book.

Distribute Systems book makes a distinction between accuracy and precision.

If you have a group of servers and you want them all to agree to be consistent within like a Delta, within, oh, maybe Delta's the wrong term there, within an Epsilon, right? - Epsilon, yes. - Epsilon, yeah.

So that actually is called precision, the internal consistency.

If you have a group of servers and you want them to be consistent with some oracle, some known source of truth, like an atomic clock somewhere in the world, that's called accuracy.

So internal precision, external accuracy.

And ever since I started running Linux servers, I noticed, oh, there's this thing called NTP.

Well, that sounds great.

I could just ask them for the answer.

What will go wrong there?

(laughs) What do you know about NTP, Mike? - Not a lot.

Network time protocol is what it stands for.

I think the basic idea is that, so what you're trying to accomplish is you've got a number of servers and you want them to all have the same time.

And if you relied solely on the internal clock of those servers, that probably would not happen.

So most of-- - The internal clock of the server is using Quartz, right?

Quartz crystals? - Yeah, there's some sort of crystal oscillator that keeps time and those things, even though they're pretty accurate, they tend to drift due to, you know, variations in temperature and other things.

And so if you have no way to synchronize the machines, they will slowly get out of sync with each other in terms of their time, even if you set them to exactly the same time.

So-- - So basic dumb idea, I'm gonna run a server, you want to know what time it is, you ask the server, right, that's NTP. - Yeah, and I think there's sort of a hierarchy to it, as I understand it.

So obviously just asking, if you just have like five servers on the LAN and you say, hey, what time is it for you?

And you just try to build a consensus around that, I suppose that would work within the context of that network.

But I think with NTP, there's this sort of hierarchy where, you know, going up the, they call them stratums or something like that, you get more and more accurate time as you were describing accuracy.

And so this protocol attempts to, and I just keep the machines in sync with each other, try to keep them in sync with the actual accurate clocks. - I tried to understand NTP and I did get kind of confused.

The first problem I think is kind of obvious, right?

If I've got a server and I'm gonna send a message to the server to figure out what time it is, well, the sending of the message and the receiving of the response, that takes time.

And I don't, that's variable.

Like that NTP server could be in different places, my network request can hop a bunch of, bunch of different nodes that are variable.

And so there has to be some way of coordinating to sort of being roughly aware of that variability and factoring it into our time.

Okay, so that's the first problem.

The second problem is if my quartz crystal on my computer is running fast, then the time is really, I'm ahead of it.

I need to go backwards in time, but you can't just go backwards in time.

So we need some way to account for that.

And so what happens is that you can actually, you have to like gradually do this.

You'd be like, oh, I'm fast and I may have a timer running on my machine and it's set to do so many interrupts per second.

And every time I do one of these, I add a little bit of time.

And so if I wanna slow down, I add a little bit less time.

And if I wanna go faster, I can add more time, right?

Am I even saying that right?

This is how I read it in the book and I think I'm already confusing myself.

I cannot talk about this without getting horribly confused.

I really should not have taken on this topic with you, Mike.

I don't know what I was thinking agreeing to this topic. - Yeah, that's, I probably don't understand it well enough to explain either, but what you were saying is true is that this is, it's designed to work with variable latency networks. - There's no free lunch is what I've discovered in this conversation with you though.

You know what, Mike, we didn't even get to the fallacies.

There are famous lists of fallacies.

I read these more than 10 years ago.

We'll have to cover these in a future episode.

We didn't even get to one of my favorite posts ever about time, which is called falsehoods programmers believe about time zones.

I love this post.

In the post, I discovered that India is five and a half hours off of UTC and Nepal is 45 minutes of UTC.

Oh man.

Let's talk about this stuff in a future episode.

We didn't even talk about our friend, Lamport.

Lesson Lamport clocks, the best paper I've ever read about clocks.

And we also did not talk about special relativity, which I am a little relieved by.

I feel like I can still study for that test. - We didn't talk about time zones on the moon either. - There's times on the moon.

What? - There's gonna be time zones on the moon. - All right, Mike, how's about this?

I'm not prepared to talk about this with you next week, but in a future episode, can we revisit this topic and try to talk about the things we did not talk about? - I think we should at least try to determine whether or not dates should have time zones. - Right now or in the future.

Here's what I wanna do.

After having this conversation, I'm gonna go lie down for a while. - That sounds fair. - Yeah. - I mean, you'll have an extra few nanoseconds today to lie down because of the tidal friction. (laughing) It's so unfair that the days are getting longer.

All right, this has been a picture me coding with Eric Aker and Mike Ball.

Mike, I had a lot of, well, no, I didn't actually have a lot of fun talking about time with you today.

I got stressed out and exhausted talking about time with you today, but I did laugh a little bit and learned some things along the way. - I hope it was not as confusing to everyone listening as it was to us talking about it. - Yeah, me too.

Okay, I guess we should mention too, if you wanna send us a message, podcast@picturemecoding.com is the email address.

And Mike, I think you're gonna be out of town for a bit.

Is that right?

So we may not have consistent episodes for the next probably month and a half.

We'll probably have a few episodes, but don't worry, we're not going anywhere.

We will be coming back in a more regular cadence once Mike is back.

Is that sound right, Mike? - Yeah, I think we'll probably get a chance to record, but I'm going to be on a bit of a road trip.

So it may be sporadic. - Okay, all right, that's fine.

Just keep that in mind.

Listeners, we're glad to have you with us.

Thanks so much for joining us again.

And Mike, I will see you again sometime soon. - Sounds good. - Bye-bye. - Bye. (upbeat piano music) [MUSIC]