From Struggling to Mastery: A Practical Guide to Data Pipeline Operations

Speaker 1 [00:06]

I will quickly start with some questions to you. How many of you have some data pipelines in production at the moment? Oh my god, it's nearly everybody. Very good. Okay, so if I ask you, if you assess your maturity of your data pipelines from the scale to one to five, I will ask the scale and then can you please raise your hand? Scale one is the, let's say, like struggling and the five the mastery, right? So you can imagine something. And for one, can you raise your hand? Okay, three, four, five, two. Oh, okay, interesting. So around ten. Okay, three. Oh, it's nearly half of the audience. Okay, four. Okay, around ten. And five? No? Come on. Okay, let's start then. Okay, we are going to talk about something I think doesn't get enough attention and the operational maturity of your data pipelines. Your products and your decisions are only as reliable as the data pipeline feeding them and especially in that AI stage, right? Look at what depends on your data pipelines. On the left, your mission critical products, AI agents, personalization engines, and the forecasting models, and all consuming the features from your ML platform, right? And on the right, every dashboard and the report which steering the business decisions. So two completely track, but they share the same foundation, your data pipelines, right? Okay, when that foundation cracks, what happens? So with the stale data, broken transformers, or silent failures, you are not just getting a bad dashboard. Your recommendations go off, your forecast drift, and also your AI agents start serving the confident nonsense at scale, right? And today, we are going to focus on what foundation, on that foundation layer, and try to give you a framework to make it rock solid. Same Monday morning, different channels, two different teams, and the root cause is the same. So the pipeline broke over the weekend, and nobody on the data team until that the business told them, right? And your stakeholders are basically your monitoring system. And here is the uncomfortable truth. A lot of teams never really live to that stage, right? Just they are just getting faster in the five-fighting. So the core problem is, I think not that teams are bad at this. The problem is there is no agreed open standard. So if I ask the five data engineers, how mature are your pipeline operations on scale one to five, as we did, you would get different scales probably, right? And without the shared vocabulary, you cannot measure the progress, you cannot set the goals, and you definitely cannot justify the investment to leadership. So that's what we are trying to fix here. Here's what I am proposing, a five-level maturity model, struggling at the bottom. That's the Monday morning we have just saw, right? And the mastery at the top, that's where your pipelines run themselves. And this model applies across the three pillars. So first is the orchestration, how your pipelines are scheduled and recover from failure. And the data quality, how you prevent the bad data, right, from reaching out to your consumers. And also the third one is the data SLO. So how you measure the guarantee and the delivery with the targeted quality. So the most teams are, I think, somewhere between the level one to three, but I saw that quite a bit level three and a bit more level four here. So as well, that's good. The call today is to give you a shared language so you can pinpoint exactly where you are and then what the next step looks like. So let's start with the pillar one, orchestration. How your pipelines are scheduled, right? And this is about, and also how they handle the dependencies, what happens when things go wrong, and things always go wrong as well. And I'm going to walk you through each level and see where your team fits. Okay. Level one is where many teams start, right? So the cron jobs, manual triggers, no dependency management. If a job fails, nobody knows until someone downstream complains and there is no retaliation. You just rerun it and then by, of course, hand and hope for the best. And the knowledge how to fix the things lives just, you know, like in one person's head. The level two feels like a big step forward, right? You have adopted airflow or similar one, but look closer. The ducks have grown organically massive and copy pasted, of course, and then no one can read them, right? And yeah, there are some like rhetoric logic, but it's inconsistent. And here's the trap. The pipelines were never designed to be backfillable. So when you need to reprocess the 30 days of the data and there is no parameterized date range, right? And then, yeah, no partition, aware of overwrite, of course, and then, yeah, every backfill becomes a custom project, which is good for the project management and the product. But yeah, having an orchestrator doesn't mean that you are well orchestrated. So level three is where things start click. So the single biggest unlock even like here, item potency, I would say. So your task can be safely retreat and backfilled because they were designed away from the start, right? And ducks are modular. The whole team can operate them. And when something fails, the others tell you what's happened and the links to runbook. So if you are at the level two, this is your target. Level four is when you start creating the pipelines like a software, right? And instead of handwriting the each stack, pipelines are generated from the config and add a new data source, just add another YAML entry, pipeline appears automatically. So your entire platform is defined as a code, like with cloud formation, terraform, nothing is clicked configured anymore. So you are not just reacting to the failures anymore. You have dashboards, which is showing the duration trends, resource usage and bottlenecks. So you see the degradation before, it becomes an outage. Level five is the North Star, right? Okay, people are generally like things like it's mastery, but of course it can be achievable as well, I think. So pipelines are event-driven here and data arrives and the pipeline runs and actually the airflow of trees, assets, watchers make this more accessible now. And of course, that you can do in another ways as well. And the retari are smart. So not just try again the three times, but the system knows why something failed and the spot instance lost. Okay, then retari automatically. If it's out of memory, then yeah, retari with the more resources. If it's a bad query, then yeah, don't retari, just alert the human, right? Here is the key mindset shift. I would say your team is only gets paged for the failures that the system hasn't seen before. So non-problems are just handled automatically through the sensible retari policies and also the platform resilience. And also, yeah, I want to be honest that, yeah, kind of a few teams can be here or, but that's fine. I mean, that's the, the goal is not level five overnight. So the goal is to keep the just climbing and yeah. Yeah. A quick look ahead. Level five, I mean, can be at today's ceiling, but AI is pushing it further, right? So imagine asking, asking an AI agent that why did that pipeline fail and then getting the root calls analyzed in seconds. So, or saying backfill user events for the last two weeks and in plain language or pipelines that learn their own normal behavior from the, your historical data and fix issues before you even notice. Yeah, this is coming. So we can start thinking about that one. And here's the key point. AI doesn't fix the broken fundamentals, unfortunately. So it amplifies whatever you have already. And if your pipelines are not idempotent, AI cannot make the retari safe, right? And also, if you have no observability, then yeah, AI has nothing to learn from. So the maturity model still can apply to get the level tree first, I would say. And then the AI feature becomes possible as well. So let me tell you the why these two things, item potency and the automatic retari is matter together. So I was on call in Zalando, right? And one of the spark job pays on the data bricks and the lost an instance. So there was no automatic retari and nobody knows until I checked that one as well. So I wake up manually trigger the rerun and I was just thinking, okay, it will be just okay and that but fails again. Why? The job was not idempotent and the partial rights from the first run are sitting and looking at me from the S3 and the rerun cannot handle them. So yeah, I was manually listing the S3 objects, finding the corollab partitions and deleting the files by hand. And yeah, what happened then? So the problem is solved but actually it should have resolved itself and it turned out to our nightmare. So the fix was two things. The first is to design the job to be safely rerunable and the configured automatic retari is. And after that, yeah, the job failed as well but retari itself and succeeded. So I just saw the success notification in the morning, not the 2 a.m. alert. So that was the jump from the level two to level three, I would say as an example. So pillar two, data quality. If our consideration about is whether your pipelines run and the data quality is ever about whether your data quality is correct, right? And here's the things about the data trust. So it takes like maybe months to build that trust between your consumers and once the stakeholder finds the wrong numbers in a bad dashboard, then they stop trusting this data. And when the trust is gone, yeah, they start building their own spreadsheets. And we have all seen that happen, right? Okay, so level one, as you can imagine here, like no checks at all and the bad data flows the production and the first person to notice is a kind of stakeholder. Level two is the feels better. Someone maybe has written assertions in a notebook. So maybe yeah, SQL query that checks the row counts. But look closer here as well that the checks are scattered and not version controlled. So there is no consistency. Yeah, every check is treated as the same severity and the critical problem failures get locked. But the pipeline still keeps running. Bad data still ships. Yeah, it is kind of like our Monday morning slide, right? So with that one. Level three has two key unlocks here. First checks as a code. So your quality validations live in the same repo as your pipeline and version through able in pull requests or reusable across the pipelines, no more scattered assertions in the notebooks. And the second is the validation rules, right? You're explicitly defined what correct looks like. And primary keys are unique, for example, or revenue is positive or date makes sense. You classified each checks as a warning and critical so you can distinguish them. And you validate the schemas on the ingestion so you catch the source chains immediately. You monitor the freshness. Is this data actually from today? And every table has a clear owner, right? And the track the coverage. So level three actually is here, catch every problem you thought or you assumed. But the level four, catch the problems you didn't think of, also stops spreading the bad data. Write, audit, publish kind of a pattern is your circuit breaker. So data writes first, gets out it, then only publish the consumers if everything pass. So if checks fail, the production table stay untouched and the consumers always see the last known good data. Yeah, the anomaly detection adds another layer here, right? So instead of only handwriting rules, the system learns what normal looks like from history, null rates. Yeah. For example, imagine that the null rate is 1 to 3% for the last 90 days and today it's the 20% and it will be automatically a flag and nobody wrote that rule, right? And data lineage gets you trace the issues end to end and also the data contracts determine what is agreed on. The pillars are connected here strongly, data quality detects and the orchestration response automatically based on that one. So the level five here, the quality becomes an organizational capability. So every data product has a quality score like an uptime percentage. This table has been 99.7% reliable over the last 90 days, for example. And this score is visible to everyone. Any user of that consumer can check that one from the platform without asking the data team. And yeah, every producer publish a contract. Every consumer knows what they can rely on. And when something goes wrong in the system with the full lineage of this data, we can see like the problem stated in the source system X in the column Y at the 3 a.m. So no more hours of the detective works. At this level, you don't vote the quality on after building a pipeline, but it's a product feature from day one. And the same way uptime is for a web service. So is my data quality story. I will go a bit quickly due to the time. Yeah, we had some legacy flat file for the mapping the reference for our training data for the some machine learning pipelines and our pipeline just read the from from the for the position for off to that mapping file, but somehow the the owner team just at another column and or key shifted the position five, right? And they didn't tell us. So here's what made this so hard to catch. So this mapping file only affected a subset of rows and most of the training data was perfectly fine. Just a small percentage rows got null feature values. And overall, not null rates barely moved. So everything looked healthy at the glance. And then but over the days, the model get trained on the that quietly degraded data and the performance of the model didn't cliff. It slowly slid downward. And so yeah, nobody can could pinpoint that one, but it's just realized in a coincidence. So three levels of the defense would actually count that one, right? So the first one from the level three a validation rule on the specific join that null rates, the feature must stay below a threshold. And the level four is the anomaly detection. The system notice even a small, a small drift on the null rate compared to the 90-day baseline. And the level four is the data contract. So if we have the data contract with the schema change, then it should be notified that that consumer. So pillar three data SLOs, if the orchestration is about whether your pipelines run and the data quality is about whether data is correct. So SLOs are about whether the data arrives when it's supposed to be and with the expected quality. So it is the key insights from the SRE. You cannot improve the what you don't measure, right? So we can take that and apply here as well. So without the formal targets, every failure looks the same and you cannot justify investing the reliability because you have no data on how unreliable you are and the impact on the business. So level one is like no targets at all, right? When somebody asks you like, okay, when my dashboard will ready, you can just say, okay, when the pipeline finishes, maybe somebody knows that should be ready in the some, some, something, some, yeah, some times. But it's so basically you don't have any data to continue. So level two is the has a alerts on the pipeline failure. So you know, something crashes, but it's the binary run or didn't run. And for example, like somebody knows that the pipeline generally would finish at the 8 AM, but it's not documented anywhere. And if it is finished at 11 AM, nobody escalates, right? So there is no definition of the late rate yet. So the level three is the biggest jump here. So you go from the no targets from no targets to from SLIs, formal SLIs and SLOs. So you define what you measure, freshness, completeness, set the targets, for example, table X updated by 8 AM every day and with that expected quality targets. You can answer is this table on time with the data, not guessing. So every critical table has a clear owner, but the level three is reactive, right? So you find out that one is the 8.30 that you missed the target and the damage is already done. So level four flips the data from reactive to proactive. So your system now is from historical baseline. That table is usually finished by the 7.15 and it's 7.30, so still running. So then the on call should be get called and the heads up before the bridge, not after. There's the difference between the sorry we are late and the the heads up like we are working on it. It's changed completely the the perception definitely. And for example like error budgets make liability quantifiable here as well. So if your SLOs is 99%, that's about like three times per year missing the data. So you track the pipeline success and the SLOs compliance separately here because the job can succeed but it can be still late. So level five is like a bit raised the bar and the SLOs became an organizational tool for the data landscape. And burn rate alerting tells you how fast you are consuming your software budget. The real power is actually the SLOs for the connect your engineering investment to actual business impact with that kind of alignment. Yeah, if you try to define an SLO or a pipeline and no consumer can articulate that why they need it, then maybe that pipeline probably shouldn't exist as well. Right? So by the several pipelines might be decommissioned or relaxed firefighting. And the the stakeholders can see like real-time liability per data product here and also like SLOs are embedded in the contracts. So I can go to like the one of the my example here as well which we had. Liability starts with really knowing the what matters. So imagine a team that's running more than 50 pipelines every day and nearly every day at least one failure and firefighting like some same kind of failures about the one FTE per week and every single week. So once a time personalization service on running the stale data until the Tuesday. Yeah, that sounds urgent, right? But when someone finally asked the consumer team what's the actual business impact? All weekend stale data and there was no so much impact and then we started like talking around that SLOs and then we realized that actually we can separate some pipeline. It's not like a street firefighting needed and some of them maybe even can be decommissioned. So yeah, the SLOs they are not just a measurement tool but they align the your investment with the actual business need. So it's the full maturity scorecard for the three pillar. You can take a picture and you can use it Monday morning, sit down with your team and score yourselves on each pillar. So you will probably find you're at different levels for each, maybe level three on orchestration but level one on the SLOs. So that's normal. The point is not to be level five everywhere. The point is to know where you are and what the next step looks like. So here is the practical part. For example, if you are for the orchestration, if you are at the level two, just configure the automatic criteria on your top five tags and write. And for the SLOs, if you are at level two, talk to your consumers, ask them the freshness they actually need and you might be surprised really by the answer. So this is the like when you have that scene, your scorecard and also see your status. Like the most important thing is the look at the left side, right? So a team at the level three orchestration but level one on the quality and SLOs. So their pipelines run beautifully. Retailers work, backfields are clean but bad data still ships the production on schedule. So nobody knows if it's actually on time or correct. So that's the fragile setup, I would say. And you might be delivering the garbage reliably. So the first move is always to invest your weakest pillar and get balanced. And yeah, after that, based on your business need and when it is justified, you can go even further like level five or further. And we are hiring in Zalando. So happy to get all of you in Zalando if you like. Yeah. And yeah, please welcome to look at that one. And thank you very much for your listening. So this is the three QR codes for you, LinkedIn, if you want to connect. And I would like to hear where your team scores and my blog, mldatacraft.com. And also I put that here the downloadable PDF version of that complete scorecard and with the detailed version with all criteria. So you can use it in your next team meeting. Know where you are, know where to go to next. Thank you. Thank you.

Speaker 2 [24:42]

Thank you so much. Thank you for the insightful presentation. We move on to the QA. First question, when you implemented this, this framework, where the technical or the organizational obstacles dominant?

Speaker 1 [25:05]

Yeah, thanks. Good question. Where did we start implementing that one? Actually, we have some GUAIT in Zalando, like not as structural as that one, which actually we have a kind of relation to that kind of levels and also the or technical tooling, right? And try to establish all the that one like across the company with the GUAITs and also like the educating the or teams onboarding them the tools. And also like we have the Zalando Data Foundation that with empowering our data capabilities, which we are talking about how can we leverage those kind of things with the existing capabilities or what is the gap for that capabilities and try to fill them. And of course, that organizational problems is always as a topic, right?

Speaker 2 [25:58]

So we all know that. It continues with, can you elaborate on how you introduce the framework in your organization?

Speaker 1 [26:09]

So the first time we get together with a group of people from the, as I said, like Zalando Data Foundation, we have really good colleagues like on top of that one from the infrastructure and also from the data side as well. And also like volunteer engineers get together and write some, try to produce a guideline with the practical examples that has been already implemented in Zalando and then publish that one. And then based on the feedbacks, we are like improved that guideline and also try to scale the other teams as well.

Speaker 2 [26:46]

Yeah. Next question. Can you upload your slides? I really like your concept and would like to recheck my current state back at work. So that would be really useful.

Speaker 1 [26:58]

Yeah, sure. I will do that one. I think I forgot the upload. So sorry for that one. I will.

Speaker 2 [27:05]

Thank you for that. Next question. You mentioned audits. What do you think about data frame validation, early failure in pipelines?

Speaker 1 [27:16]

I mean, this is exactly the idea here, right? So the right audit publish as a circuit breaker, like from the software engineering. So you don't stop. I mean, you don't split that bad data to downstream. So you must stop the bad data when you realized and it should be as much as near the source. Like the, I mean, if it's a so a raw event from the source, which is flooding the data lake, first, they should be the detector. If it is not possible, and if you are a consumer, you can request that one. But based on the request, if it is not possible, then you need to find, because the more getting that bad data separate in the pipelines, it's getting really the higher cost to fix that one. And also like all of that incident management, everything is getting like really challenging.

Speaker 2 [28:09]

Yeah, makes sense. Next, you mentioned, no, we answered this one. Okay, now next. Are there platforms or tools that you recommend to implement this framework? Which ones to avoid?

Speaker 1 [28:23]

Yeah, that's a good question. Basically, when I tried to present that model in the Zalando, I was trying to reference the or tooling as well, right? But since if I reference that there's some tooling there, it's the my aim was that to just generalize that framework that can be established with such a like any tools that which you like to and you trust that one. So if you ask me like, of course, that we can maybe connect later and then take a look at like which tools can be used. But the open source like the Grafana for the dashboarding, the alerting for yeah, there are like, for example, we use the Anamolo in the Zalando. It's a good platform, but it's yeah, of course, that it's not open source. But you can find a similar one like open source for the data quality orchestration is, I mean, Airflow is the I think that the biggest player here for to be honest, as I say, that doesn't mean that you cannot use the other orchestration tools as well. So this is why I try to make it generalized and generic for the any platform.

Speaker 2 [29:31]

I guess we have time for one more question, one last one. I thought my organization was in level three, but after this talk, I realized we are not even in level two. Now I feel bad how to level up.

Speaker 1 [29:45]

Eleven three?

Speaker 2 [29:46]

Level they talk that it's in level three.

Speaker 1 [29:48]

Can you ask a question, please?

Speaker 2 [29:49]

Yes, I thought my organization was in level three, but after this talk, I realized we are not even in level two.

Speaker 1 [29:58]

Yeah, I mean, to be honest, if I had a more time, I was going to make some Slido here and then you can we can see the like the like all the distribution before the talk and after the talk. But I realized and I removed that Slido slides because it takes a lot of time and as you see like it's intense the slides. So yeah, I mean, it's normal. This is why actually the I'm really happy to hear that one because if you know like where you are exactly and then you can see like what would be the next steps, right? And you can go like incrementally there and like make it rock solid.

Speaker 2 [30:33]

And also there were some instructions in the presentation so you can use that. All right, let's thank our speaker once again.

Speaker 1 [30:41]

Yeah, thank you for your time. Thank you.