Mathias Hansen (00:10)
Hey everyone. Welcome back to Countdown to Laravel Live Denmark. Today I have Graham Campbell here. Graham, how are you doing?

Graham (00:17)
Hey, I'm doing good. How are you?

Mathias Hansen (00:19)
I'm doing great. Graham, you're of course speaking at Labour Live Denmark in August and we are so excited to have you.

Graham (00:27)
Yeah, it's gonna be fun. First time in Denmark.

Mathias Hansen (00:30)
That's exciting, but you don't live too far away, right?

Graham (00:34)
No, I'm in the UK. That was the day called York.

Mathias Hansen (00:38)
So that's like a what 45 minutes flight or something like that? An hour maybe?

Graham (00:42)
Probably

a bit longer. I think I'm flying from Heathrow, so it's like a good three hours to go to Heathrow and then yeah, the flight is shorter than the travel.

Mathias Hansen (00:49)
have to get to the airport first. Yeah, yeah, of course, yeah.

But the flight itself is probably pretty short. It's gonna take longer to actually get to the airport and security and all this stuff. But we appreciate you coming. It's gonna be a good time.

Graham (00:56)
Yeah.

Yeah.

Yeah, I'm looking forward to it.

Mathias Hansen (01:05)
So Graham, you have been in layerable community for very long time. And I was just looking a little bit into, basically looking at your GitHub profile too, because I feel like I keep seeing your name everywhere. Same thing I mentioned last time, last time I met you at layerable.eu, I see your name everywhere. So you have your hands in quite a few things in addition to the layerable framework itself. So do you wanna talk a little bit about...

Graham (01:19)
Mm-hmm.

Mathias Hansen (01:34)
some of the things you've built or have been involved with software-wise.

Graham (01:39)
Yeah, a lot of the packages I'm in today are ones that I inherited. So Gozo was originally built by Amazon because back in the time they ordered an SDK for PHP and there wasn't a PHP client library. It just didn't exist. It's kind of crazy to think today you have so many. They eventually decided to stop paying for the maintenance of that and the community took it on. And I'm one of...

Mathias Hansen (01:55)
Mm-hmm.

Graham (02:06)
I'd say two or three maintainers now that are like actively working on that. I also maintain php.env. So like when you have your .env files in Laravel, like that's running on php.env. So that's pretty cool. Yeah, I still maintain some Laravel packages. I have some Laravel packages from back in 2013 and I still, every year I'm like upgrades to the next version of Laravel, support the next version of PHP. Yeah, still going.

Mathias Hansen (02:36)
That's incredible. And especially, I thinking about the .env package. I feel like that's like a boilerplate thing that almost every single PHP project out there will have and use. It's like the golden standard of, well, of course you're going to have a .env file. Of course you're going to use that library. So that must be like millions of PHP installs out there using that library.

Graham (03:01)
Yeah for sure, mean it's probably got more downloads than Laravel, because every time you download Laravel you download that. So I get all of Laravel's downloads and then all the other ones. So yeah, it's kind of neat.

Mathias Hansen (03:12)
That's insane. How does it feel to have a library that's such a core essential thing that so many people are using?

Graham (03:21)
⁓ It sort of isn't a big deal. mean, a lot of people don't really think about the foundational libraries. It's sort of just taken for granted that they exist. ⁓ So yeah. So it just is there.

Mathias Hansen (03:30)
Right.

Yeah, and it's probably one of those things that as long as everything's working fine, you don't hear from anybody. But as soon as there's a tiny minor mistake or something, people will be all up in arms about this library they never heard of.

Graham (03:50)
Yeah, I do have to be

super careful about breaking changes, particularly in Guzzle. Like there's a lot of people like, like file issues and sending PRs, but I have to be super careful. ⁓ eventually there will be a Guzzle 8, but it's just how, how many years that take people to migrate from seven to eight. Like there's still people on five and six and three even. And yeah, I just, yeah.

Mathias Hansen (03:55)
Right.

Right.

yeah, you know the feeling of you have this old code that works great, but you don't feel like upgrading to a newer library version of Gosl for example, because then I have to figure out what the differences are and read the change log and all this stuff, right? ⁓

Graham (04:25)
Yeah.

Yeah, yeah like there are changes

I want to make for sure on Guzzle 8 but I don't want to maintain lots of major versions of Guzzle. So yeah I'll just hold off until we absolutely need it. I've been holding off for like five years.

Mathias Hansen (04:41)
Right.

Right.

Anything in particular you're excited about for Gosselaid? Any features or changes you want to do that you're excited about?

Graham (04:50)
It's mostly cleanups

and like more sane defaults. Like I think the default timeout is like 300 seconds, which of most people is way too long. Like if you have a, like a web app, you're doing an API call in an FPM request. Like if it's 300 seconds, that's not good. So yeah, there's just a few, a few things like that. like changing that could have an impact. Like if someone's using it to call an LLM and streaming data or downloading like a 10 meg file.

Mathias Hansen (05:02)
Right.

Graham (05:19)
10 gig file even, like, yeah. They forgot to change this stuff.

Mathias Hansen (05:24)
Yeah, we actually have lot of issues, or sometimes have issues with this for our geocoding API. We have a long running post request that sequentially geocodes up to 10,000 addresses. And it can take like 30 minutes to finish if you have a really big data set. And people don't realize that there is default timeouts in these libraries. And yeah, if you don't have a proper timeout set, you're going to have a bad time.

Graham (05:38)
Mm.

Yeah.

Yeah, for sure.

Mathias Hansen (05:53)
Cool. So Graham, you always been into technology? Have you always had a computer at home or some kind of ⁓ gaming platform or something like that?

Graham (06:06)
Yeah, I'd say I've always been into technology, always been into tinkering and programming. Even before I really knew it was programming. think probably one of my earliest bits of software, if you want to call it, would be inside of PowerPoint, you had like visual basic macros that you could write. So you could do stupid stuff, like programmatically.

Mathias Hansen (06:25)
yeah.

Graham (06:29)
The stuff I was doing was objectively making the slideshow worse, but like, it was just fun to automate things, even though they didn't need to be automated. So, yeah.

Mathias Hansen (06:33)
Ha

So do remember one of the first kind of things you coded? Regardless of the language really.

Graham (06:48)
Yeah, so that would have been inside PowerPoint. Like, probably stuff after that would have been like, like real Visual Basic. So you have like the .NET like studio, whatever they call it, Visio Studio. It was probably, yeah, that was a long, long time ago, but it was kind of easy to get started because you can...

Mathias Hansen (07:00)
Yeah.

Graham (07:06)
Well, it had the visual part, right? So they had this UI where can drag and drop components. So it was for like making like windows, like UIs. Yeah. And you could do really horrible stuff where you wanted to do like async or concurrency. They have like this timer component that you can drop in and do horrible stuff. So sure the code I write is really terrible, but you sort of have to learn to do it wrong for you. You can see how to do it right. So yeah, I a few years.

Mathias Hansen (07:12)
Yeah, like a UI builder. Yeah.

Ha ha.

Graham (07:36)
Same thing about that. I don't think I discovered PHP until 2012. Which, yeah, if I was time, I'd already been programming like .NET and Java and I guess like bash scripts for some time. Yeah. Yeah.

Mathias Hansen (07:51)
Yeah, yeah.

So you got involved with Level pretty shortly after learning PHP, I guess.

Graham (08:00)
Yeah,

it's pretty much almost at the same time. I probably used like vanilla PHP for like six months before I discovered Larvo. This was back in a time where like open source is very small. Like Composer wasn't really, like Composer almost didn't exist back then. Like, and like most open source code was just like, I guess if you want this, you're going to copy paste it into your, into your project. Like that's sort of how it goes. I mean, that's how Larvo 3 was as well, right? Like you.

Mathias Hansen (08:17)
Yeah, yeah.

Yeah.

Graham (08:30)
It's only now or before they used Composer. It's crazy to imagine like having to do that these days. Imagine that they don't know Joe's directory.

Mathias Hansen (08:38)
Yeah, remember there was

some kind of directory of libraries on the layerable website. Libraries are made for layerable, right? And you have to download the zip file or something like that and copy them in or something like that, like pre-composer. That was a different error. Yeah.

Graham (08:49)
Yeah.

Yeah, it's I'm lucky

that I came into it as composer, as I was getting adoption. But La Vava was one of the first to adopt. There was still a lot of resistance back in 2012. Symphony and La Vava were really the first.

Mathias Hansen (09:09)
Yeah.

Cool. ⁓ So you are gonna be speaking at Level Life, Denmark of course, that's why we have you here, which we're really excited about. And you're gonna give us a little bit of insight into some real world experience with scaling. Do you wanna talk a bit about your talk?

Graham (09:31)
Yeah, so essentially it comes from a problem of needing to scale PHP running on AWS Lambda. ⁓ So that has a bunch of unique problems that you get to quicker than you might get to if you're running a more traditional BPS-based approach. the major problem that I was running into was just the pure number of connections.

Mathias Hansen (09:50)
Mm-hmm.

Graham (09:59)
Most of the CPU on the database were just spent managing newer connections and shutting out connections, which is kind of insane that you would buy a bigger database and use more connections and you would get lower throughput. Like you'd buy the biggest possible database and just do more stuff and get less for it. So yeah, I'm sort of going to talk about the journey of how to fix that.

Mathias Hansen (10:13)
Hmm.

Graham (10:26)
and some of the other stealth figure for free that we didn't even necessarily want but it was awesome that we got it. So yeah, it's pretty, it's gonna be pretty interesting.

Mathias Hansen (10:37)
So this is the case of you're scaling up your app service essentially, your web requests, and then the first thing that's gonna hurt is your database, which is left behind, not being scaled in the same fashion, right?

Graham (10:45)
Mm-hmm.

Yeah,

it's often this fallacy where people put like, a serverless is infinitely scalable. Well, it's sort of not. Like everything has limits. Like if you've got 10,000 concurrent lambdas and your database has like four VCPs, like good luck connecting to that.

Mathias Hansen (11:10)
So what was the most challenging part in that journey of dealing with scaling that?

Graham (11:20)
On the whole, wasn't that challenging because luckily other people have thought of this before. So like, we weren't really solving like new problems. So we ended up using a product called Vitas, which was recently developed at YouTube. Like they were having like similar and different scaling problems. Like they also had like lots of connections, but they had different problems where like sorts of queries were getting run.

Mathias Hansen (11:33)
Okay.

Graham (11:46)
were really expensive and would take the database out and like back then they didn't have SSDs they had hard drives so like if you had queries doing lots of IOPS like you really would like make a big mess um so I think they had queries where they they had like a featured page um like on the home page of YouTube what like 20... 2009 I want to say and like they accidentally uh featured a channel that had 250 000 videos

And they had a query that would fetch and join like data to get to 15,000 videos because they had their patination. like, like, so, so like, the test was able to like, like stop those queries. It's kind of nice. so yeah, we have less of those sorts of queries because we largely, okay. We had sort of time to think about what the queries would.

Like, I don't know, we're sort of more careful and maybe moving slower. Like I guess at YouTube, if you have like a hundred people writing like one new query per day, like if someone went to bad query once a year, like you're getting bad queries like every day. So like we have less of that. Yeah. ⁓

Mathias Hansen (13:04)
different kind of scale right yeah that's

all I said so after after people have watched you talk what is the main thing you want them to walk away with

Graham (13:22)
That's a good question. ⁓ I guess databases are cool. think that's the main takeaway. Databases are cool and like MySQL can scale. It really can. You don't need Mongo or DynamoDB or Postgres. You can actually get very far with Dynamo. ⁓ There's still lots of massive companies that use MySQL at scale. Uber, GitHub, HubSpot, Shopify.

Mathias Hansen (13:23)
haha

I like that.

Graham (13:50)
My SQL is really great.

Mathias Hansen (13:51)
It's

a pretty common fallacy, That, ⁓ my SQL or whatever database doesn't scale, I need to use a different database and suddenly magically everything scales, right? And that's not really always the way to solve a problem.

Graham (14:02)
Yeah.

A pet peeve I have is like people say no SQL databases girl or like they're for reads or they're better for writes. I'm pretty sure I could write a no SQL database. really bad. Like, like the interface doesn't really the problem here, right?

Mathias Hansen (14:11)
Right.

Right. Yeah, and think that was a fad for a while where everybody was raving about NoSQL and that being the solution to everything. And then that seems to have died down a little bit. Expectations have been tempered a bit. Some people might have been burned a bit by just jumping straight into it without really understanding the implications of it. I certainly been there. I wrote the hype wave for a little bit there and...

Graham (14:35)
Yeah.

Yeah, I mean...

Mathias Hansen (14:48)
did not know what I was doing and got burned by it so now I'm a little bit more hesitant to like just you know jump ship and do something completely different.

Graham (14:56)
Yeah, I the flexibility you get from a SQL database is kind of crazy. Like you can write any SQL query and it'll run and maybe it'll be fast, but it will run. So like if you need to do some add-hard BI query, you can just go do it. Whereas like if you're on some SQL database, you sort of can't. Like you'd have to sort of download the entire database, put it somewhere else and then run a query on it. Yeah, I mean there is a whole industry of like CDC and to all of databases.

Mathias Hansen (15:19)
Hmm. All right. Yeah. Yeah, just like

Right. Yeah. Yeah. We got to a point where we've been tracking a couple of billion API requests per month in my SQL. And it was not, we got to that scale where they, a ⁓ simple setup with a single server that we started hitting serious scaling pains. So we are actually, we actually going ⁓ migrating now to Clickhouse specifically for this, but we are certainly not jumping ship from a SQL on

Graham (15:26)
Yeah.

Mm-hmm.

Mm-hmm.

Mm.

Yeah.

Mm.

Mathias Hansen (15:56)
on everything

else. It's just a matter of the use case, right?

Graham (16:01)
Yeah, Clickhouse is pretty great. I would actually recommend, Jess Arter did a talk at LavaCon US about Clickhouse. It's on YouTube, so you guys should go check that out.

Mathias Hansen (16:07)
Yeah, it was awesome.

So that was actually what inspired us to really look into ClickHouse. That was that talk at Lerica in US, but just Archer last year. Cause we were like, oh, this seems like a really good solution to the problem we're having. Let's look a little bit further into this. All credit to this, by the way, goes to my coworker, TJ Miller, who's actually did the research and all the testing for this. yeah, it's about.

Graham (16:15)
nice.

Yeah.

You

Mathias Hansen (16:39)
about choosing the right tool for the job sometimes and not trying to, ⁓ you know, go crazy with your infrastructure scaling before it's even necessary. Right. We've, we've gone with the MySQL option for almost 10 years without any issues. So, ⁓ yeah.

Graham (16:55)
Yeah.

Mathias Hansen (16:58)
Cool, so you mentioned this is your first time in Denmark. are excited to have you. So I'm not sure what your preconceptions of Denmark or Copenhagen is or if there's anything you've heard of, but is there anything particular you think you might want to see or do experience or eat while in Copenhagen?

Graham (17:04)
Yeah.

Yeah, I've got no idea. I'll need some recommendations. Yeah.

Mathias Hansen (17:23)
That sounds great. I

know there's a lot of people who are locals or close to locals who will be happy to help recommend places as well. ⁓ Are you into biking or are you into food or museums or anything particular?

Graham (17:45)
Yeah, I cycle less than I used to. I used to live in Cambridge for a year. It sort of got me into it. I work from home now, so there's no impetus to cycle. Yeah, but when you're commuting, it's easy to just extend the commute on the

Mathias Hansen (17:51)
Yeah.

It was a commuting thing, right?

Yeah, yeah.

Graham (18:08)
Yeah, so I don't know. Is Denmark good for cycling?

Mathias Hansen (18:12)
I would say so, there's definitely a big ⁓ biking culture in Copenhagen. People commute on a bike a lot. Also, yeah, because it's easier, I'm not having to deal with parking, it's very flexible, right? Just get from A to B. ⁓ One thing I would...

Graham (18:23)
Mmm.

I think being on

the wrong side of the road would be something to get used to.

Mathias Hansen (18:33)
Well

that's how we feel when we visit England, right? It's the same deal. It takes a few minutes to get used to. You should go do what Peter Suhm another speaker Peter Suhm, doing. He's driving a right-hand drive car in Denmark. So it's like double up on like everything is wrong.

Graham (18:37)
Yeah.

Yeah.

On the wrong way. Yeah. ⁓

Mathias Hansen (19:00)
I think he got used to it pretty quickly and you can pay some cool pranks having a kid sit in the driver's seat and so I've tricked people on the road.

Graham (19:06)
Yeah.

Mathias Hansen (19:12)
Cool, well it was really great talking to you Graham and really excited to hear you talk and learn some things about scaling my sequel. I know I could definitely ⁓ learn a thing or ⁓ two ⁓ to improve our stuff, so I'm really excited for it.

Graham (19:27)
Yeah, sounds good. I'll see you there.

Mathias Hansen (19:29)
I'll see you there.