Cover image of Stateful Workloads

Stateful Workloads

7 Podcast Episodes

Latest 6 Aug 2022 | Updated Daily

Episode artwork

DoK Specials - Why are Operators paramount to running stateful workloads on Kubernetes?

Data on Kubernetes Community

In this panel with Sylvain Kalache, Head of Content at the DoK Community, drives a conversation featuring Nic Vermandé- Principal Developer Advocate at Ondat, Julian Fischer- CEO at anynines, and Sergey Pronin- Group Product Manager at Percona.


20 Jul 2022

Episode artwork

Leveraging Running Stateful Workloads on Kubernetes for the Benefit of Developers (DoK Day EU 2022) // Arsh Sharma, Lapo Elisacci & Ramiro Berrelleza

Data on Kubernetes Community

https://go.dok.community/slack https://dok.community/  From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE) Kubernetes comes with a lot of useful features like Volumes and StatefulSets, which make running stateful workloads simple. Interestingly, when combined with the right tools, these features can make Kubernetes very valuable for developers wanting to run massive production databases in development! This is exactly what was seen at "Extendi". The developers at Extendi deal with a large amount of data in their production Kubernetes clusters. But when developing locally, they didn't have an easy way of replicating this data. This replication was needed because it allowed developers to test new features instantaneously without worrying if they would work as expected when pushed to production. But replicating a 100Gb+ production database for development wasn't turning out to be an easy task! This is where leveraging Kubernetes + remote development environments came to the rescue. Running data on Kubernetes turned out to be way faster than any of the traditional approaches because of Kubernetes' ability to handle stateful workloads exceptionally well. And since Extendi already used Kubernetes in production - the setup process was fairly simple. This talk will cover practical steps on how leveraging Kubernetes based development environments allowed dev teams at Extendi to run production data on Kubernetes during development using features like Volume Snapshots, having a huge positive impact on developer productivity. Arsh is a Developer Experience Engineer at Okteto. He is an active contributor to the upstream Kubernetes project and was awarded the Kubernetes Contributor Award for his contributions in 2021. Arsh has written blogs and spoken about different topics in the cloud-native ecosystem at various conferences before, including KubeCon + CloudNativeCon + Open Source Summit China 2021. He has also been on the Kubernetes Release Team since the 1.23 release. He also serves as the New Contributor Ambassador for the Documentation Special Interest Group of the Kubernetes project and continuously mentors new folks in the community. Previously, he worked at VMware and was an active contributor to other CNCF projects, including cert-manager and Kyverno. Lapo is a Software Engineer currently leading the development team of a Social Listening and Audience Intelligence platform. He started coding at the early age of 14 and since he turned his passion into a real job, he has always been looking for boosting his knowledge by constantly researching for newer and newer technologies. Active on Ruby Open Source projects Ramiro Berrelleza is one of the founders of Okteto. He has spent most of his career (and his free time) building cloud services and developer tools. Before starting Okteto, Ramiro was an Architect at Atlassian and a Software Engineer at Microsoft Azure. Originally from Mexico, he currently lives in San Francisco.


27 May 2022

Similar People

Episode artwork

Autoscaling Stateful Workloads in Kubernetes (DoK Day EU 2022) // Mohammad Fahim Abrar & Md. Kamol Hasan

Data on Kubernetes Community

https://go.dok.community/slack https://dok.community/ From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE)   Managing stateful workloads in a containerized environment has always been a concern. However, as Kubernetes developed, the whole community worked hard to bring stateful workloads to meet the needs of their enterprise users. As a result, Kubernetes introduced StatefulSets which supports stateful workloads since Kubernetes version 1.9. Users of Kubernetes now can use stateful applications like databases, AI workloads, and big data. Kubernetes support for stateful workloads comes in the form of StatefulSets. And as we all know, Kubernetes lets us automate many administration tasks along with provisioning and scaling. Rather than manually allocating resources, we can generate automated procedures that save time, it lets us respond faster when peaks in demand, and reduce costs by scaling this down when resources are not required. So, it’s really important to capture autoscaling in terms of stateful workloads in Kubernetes for better fault tolerance, high availability, and cost management. There are still a few challenges regarding Autoscaling Stateful Workloads in Kubernetes. They are related to horizontal/vertical scaling and automating the scaling process. In Horizontal Scaling when we are scaling up the workloads, we need to make sure that the infant workloads join the existing workloads in terms of collaboration, integration, load-sharing, etc. And make sure that no data is lost, also the ongoing tasks have to be completed/transferred/aborted while scaling down the workloads. If the workloads are in primary-standby architecture, we need to make sure that scale-up or scale-down happens on standby workloads first, so that the failovers are minimized. While scaling down some workloads, we also need to ensure that the targeted workloads are excluded from the voting to prevent quorum loss. Similarly, while scaling up some workloads, we need to ensure that new workloads join the voting. When new resources are required, we have to make the tradeoff between vertical scaling and horizontal scaling. And when it comes to Automation, we have to determine how to generate resource (CPU/memory) recommendations for the workloads. Also, when to trigger the autoscaling? Let’s say, a group of workloads may need to be autoscaled together. For example, In sharded databases, each shard is represented by one StatefulSet. But, all the shards are treated similarly by the database operator. Each shard may have its own recommendations. So, we have to find a way to scale them with the same recommendations. Also, we need to determine what happens when an autoscaling operation fails and what will happen to the future recommendations after the failure? There can be some workloads that may need a managed restart. For example, in a database, secondary nodes may need to be restarted before the primary. In this case, how to do a managed restart while autoscaling? Also, we need to figure out what happens when the workloads are going through maintenance?  We will try to answer some of those questions throughout our session. ----- Fahim is a Software Engineer, working at AppsCode Inc. He has been involved with Kubernetes project since 2018 and is very enthusiastic about Kubernetes and open source in general. ----- MD Kamol Hasan is a Professional Software Developer with expertise in Kubernetes and backend development in Go. One of the lead engineers of KubeDB and KubeVault projects. Competitive contest programmer participated in different national and international programming contests including ACM ICPC, NCPC, etc


27 May 2022

Episode artwork

#4 DoK community: The problem of stateful workloads - balance of keeping data HA vs. costs // Ren Lee

Data on Kubernetes Community

Balancing redundancy and HA with costs: did you really need all N replicas?AKA We were running what and it cost us how much?! With Ren Lee SRE at Arista Networks Key takeaways: “Lazy but Simple” vs. “Proactive but Expensive” methods of scaling: knowing when to pay the seemingly scarier price of running infrastructure than costing engineering time, and vice versa Hidden costs: cost of bad deployments and things that just don’t work When autoscaling becomes the demon: especially in public cloud environments when access to pools of resources is no longer your barrier Abstract: In an engineer’s ideal world we would love all the resources and redundancies we can possibly get for our services and infrastructure that supports them for sanity and of course, HA. However, how do you balance between “enough” redundancy and the actual operational costs of supporting such engineering choices, and what are some of the tough engineering decisions that need to be made? This talk focuses primarily on services being run on Kubernetes (or public cloud offering of Kubernetes), but the principles can be extended to any infrastructure environment. Key Topics: capacity planning, cost management, distributed services Bio: Ren is an SRE at Arista Networks for CloudVision services team. Deeply passionate about fixing broken things without anyone noticing and using effective monitoring to preempt potential disasters. Wrangler of services that run on Kubernetes to keep the zoo running any day, every day. Join our slack: https://join.slack.com/t/dokcommunity/shared_invite/zt-g3ui5r0g-jDKz5dhh2W1ayElqwKYYAg Follow us on Twitter: @dokcommunity Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Ren on Linkedin: https://www.linkedin.com/in/therendeye/ This meetup is sponsored by MayaData, which helped start the DOK.community and remains an active supporter. MayaData sponsors two Cloud Native Computing Foundation (CNCF) projects, OpenEBS - the leading open-source container attached storage solution - and Litmus - the leading Kubernetes native chaos engineering project, which was recently donated to the CNCF as a Sandbox project. As of June 2020, MayaData is the sixth-largest contributor to CNCF projects. Well-known users of MayaData products include the CNCF itself, Bloomberg, Comcast, Arista, Orange, Intuit, and others. Check out more info at https://mayadata.io/


13 Aug 2020

Most Popular

Episode artwork

Stateful workloads, OpenEBS and Kubera with Uma Mukkara, MayaData


Uma Mukkara, co-founder of MayaData, joins us this week to talk about OpenEBS and Storage in Kubernetes. We discuss why you need to bring storage inside kubernetes. Uma explains the CAS (Container attached storage) model and how Kubernetes can become the control plane for data. We also discuss Kubera: MayaData's newest product that adds a number of features on top of Kubernetes to better help us manage data on Kubernetes. OpenEBS is open source and a CNCF sandbox project https://openebs.io/ https://mayadata.io/ https://docs.openebs.io/docs/next/cas.html https://thenewstack.io/how-openebs-brings-container-attached-storage-to-kubernetes/ Intro Music: "Dj Quads - It just makes me happy" is under a Creative Commons license (CC BY-NC-SA 3.0) https://creativecommons.org/licenses/by-nc-sa/3.0/


21 Jul 2020

Episode artwork

How NetApp Can Help Run Stateful Workloads on Kubernetes

The New Stack Context

On this week’s episode of The New Stack Context podcast, we discuss Kubernetes and cloud native storage with NetApp's Business Director for its Cloud Infrastructure Business Unit, Dale Degen, during NetApp's annual user conference this week in Las Vegas, NetApp Insight 2019.Since its acquisition of StackPointCloud last year, storage giant NetApp has become a leader in Kubernetes management. The NetApp Kubernetes Service provides a control plane for running any commercial or even homegrown Kubernetes distribution. Users can deploy a complete Kubernetes distribution within three clicks. When used in conjunction with NetApp's OnTap storage operating system and Cloud Volumes, users can set up a multicloud Kubernetes deployment.Then later in the podcast, we chat about the Linux Foundation's Open Source Summit + Embedded Linux Conference Europe 2019, which was attended by Alex Williams, TNS founder and editor-in-chief. We also discuss some of the top news from the TNS site, including Pivotal's recent Spring Platform conference, some tips for running Kubernetes in production, and the future of serverless and cloud computing.


1 Nov 2019

Episode artwork

How NetApp's Trident Helps Kubernetes with Stateful Workloads

The New Stack Podcast

A question that may come to mind for those managing enterprise workloads is: What is Trident? This is the main question that George Tehrani, Director of Product Management, Open Ecosystem answered on this episode of The New Stack Makers, hosted by TNS founder and Editor-in-Chief Alex Williams.
 "By and large, any workload of importance to the enterprise needs some sort of persistence. So what Trident is, is an open source storage orchestrator that NetApp developed and continues to maintain. It greatly simplifies the creation, management, and consumption of persistent storage for enterprise workloads in Kubernetes, as well as the other major distributions of Kubernetes such as OpenShift."


9 Oct 2019