🗣️ Ben Poland discusses Jenkins Kubernetes plugin scaling limits - the plugin consistently failed around 1500-2000 pods with progressive slowdown that couldn't be resolved by adding more resources Watch the full episode: https://ku.bz/klBmzMY5-
"You mentioned starting with a single Jenkins instance that eventually grew to three. What were the specific limitations you hit with Jenkins as you scaled? Yeah, so, you know, we, like, basically we overloaded the single Jenkins instance. So when I started in 2020, we had, like you said, a single Jenkins controller. And there was, we ended up October of 2021. It sticks out of my brain because, you know, it was, it was a rough time. We finally hit the limits and it was a little unexpected. You know, I wish we had been a little bit better prepared. We knew that it was going to happen. It just, it all kind of came to crashing down faster than we expected. So luckily we had been preparing to deploy more Jenkins controllers. We kind of had a game plan in the works. And so we ended up having to accelerate that pretty quickly because all of our developers were not able to push code for a day or two. It was, it was rough. And, you know, you get into a situation with CI where the longer it's down, the more code people have written. And so when you do get it back up, everybody's pushing all stuff, they're like re-triggering builds. And, you know, you end up kind of with a thundering herd issue, right? But yeah, so we had that rough period. We ended up deploying a second Jenkins controller. Then we ended up with a third. And, you know, up until like late last year, early this year, we had over 20 Jenkins controllers to handle all of our CI load. And so, I mean, the issues that we're..."
💬 Discussion
🗣️ Ben Poland discusses Jenkins Kubernetes plugin scaling limits - the plugin consistently failed around 1500-2000 pods with progressive slowdown that couldn't be resolved by adding more resources Watch the full episode: https://ku.bz/klBmzMY5-