Large Shops: Developer Experience at Scale, Rina Artstain, Google, and Anton Drukh

Episode Transcript תמליל הפרק

Hebrew, then English* בעברית ואז אנגלית:

(*Translated with Google Translate - so there may be some errors.

Please also bear in mind that Hebrew is a gendered language and therefore anything that is noted in a specific gender is intended as neutral and not gender-specific.)

 (opening music)

Welcome to the second season of Dev Interrupted, the Hebrew version of Dev Interrupted, LinearB's podcast for engineering managers. Where we will host leaders in the industry and talk with them about everything that interests engineering managers, those who work with them and those who might want to manage an engineering organization some day. This season we will place a special emphasis on developer experience. I'm Yishai Beeri, CTO at LinearB, let’s get started.

(transitional music)

Yishai: In this episode I am happy to host Rina Artstain, Tech Lead at Google, and Anton Drukh, Tech Executive Mentor. Hello, what a pleasure you came.

Rina: Really fun.

Anton: Thank you very much, good morning.

Yishai: So as always, we will start by telling each one a little bit about your journey to date, so we will start with you Rina, tell us about your journey, where you started and where you ended up.

Rina: My journey is spread all over the internet, so anyone who wants to know can just search but,

Yishai: Let's give a summary though,

Rina: That's it, the summary of the previous chapters, so I always like to tell that I served in 8200, but not in the way you think, because I was a liaison officer there, and somehow I ended up studying computer science by mistake, and it was a successful mistake. My first years I worked at Intel, like everyone else, as a student, and then I worked for one company and then another company and so on. I didn't really think about what I wanted to do when I grew up. Then after having my third child I had a crisis and then I decided that I wanted to go to the big leagues, and it was a hard failure. It was very embarrassing, you can also read extensively about this. Then I got to Dropbox, which is my first international corporate, and after 3.5 years there I moved to Google and that's it, I've been there ever since.

Yishai: So you made it to the big league.

Rina: I did it, I did it.

Yishai: Okay, and today you, I mean you started as a developer,

Rina: Yes.

Yishai: So where are you today? What role?

Rina: So I said, today I'm a bit between roles, but I actually joined Google as a Tech Lead, I had a period as a manager and now I actually moved to a new team where someday I'll be a tech lead again, but for now I'm in onboarding so I don't turn my nose up too much .

Yishai: Excellent. Anton, tell us a little about yourself.

Anton: Probably. So I'm also a graduate of 8200, yes in technological areas, I was Atudai [SIC: a program for studying and then completing military service]. My service in the unit, I was surrounded by people a thousand times more talented than me in many things related to technology and somehow I managed to notice that the ability to maybe move things, talk to people, understand what someone else is going through and drive others, is something I am a little better at. And that opened the way for me to more managerial positions. I managed engineering in several places big and small, and the most essential thing in terms of career was actually joining Snyk, right at the beginning, as the first employee, about 7 years ago today, and growing together with the company as an engineering manager for 5 years and a bit. Two years ago I spread my wings, so to speak, and finally did something that was closest to calling it my personal entrepreneurial episode and decided to try my hand at self-employment. And since then I actually work with engineering managers and mainly try to save them from the mistakes I made on my way, and help them with the intricacies of the roles.

Yishai: Excellent. Today we will deal with the topic of developer experience and especially DevEx at scale, so I would love to hear from you a little bit about where you saw or where you first encountered this world or thoughts or even the name, to start calling these things by name, in a growing company, like Snyk, or in large companies like you were In Dropbox on Google, where did you first start to think that there is something here that is worth giving a name to or a phenomenon or an interest that should be paid attention to?

Rina: So I think that really at Dropbox it was the first time I came across this topic. I think it's also a matter of time, it's just something that gained momentum about 5, 6, 7 years ago, and really caught on. I think that Dropbox had a team, if they didn't call it DevEx then it was something like it was related to it, I think it's something that is very noticeable in large companies precisely in the sense that there is an expectation that there won't be chaos, but there is still chaos. So, for example, sometimes tests run for a whole day and then it fails, or all kinds of things like that that cause lots and lots of frustration, and developers everywhere - they want it to be easy and convenient and fast and they just feel it in some of these satisfaction surveys and everyone whines all the time about the tools, and all the time about the experience and too many meetings or too few meetings or too many code reviews or too few code reviews, it's never okay, like it's never good, so,

Yishai: But the feeling is that because the company is large, cumbersome, there is perhaps an excess of procedures, therefore it is difficult to fix? Or is it that even though the company is big, you still encounter the mess that exists even in small companies.

Rina: I think there is an expectation that it won't be. That somehow everything will be resolved, and that is actually the gap.

Yishai: Anton, what did you see like this in Snyk, that you started in the very first stages and grew quickly, how did this topic come up and

Anton: I will start even maybe before Snyk and with my personal perspective. I think that developer experience is something that in the beginning we experience it as some kind of collection of little secrets that people around us have and we kind of want to see how they work, maybe we can adjust some method here, some kind of something. Some 20 years ago, in the army, it was a choice whether to go after the people who used Emacs as their main work tool or those with the VI? I chose VI and you see them doing amazing things in no time and you say - shit, it takes me half a day, like how can it be? I have to catch up on this gap. This personal perspective expands to looking at how a team works, certainly with managerial positions, and you simply see where people get stuck and where things that should take a little time take a lot of time, testing is always a good example, and where things move quickly. And you say wait, how can I do more than that? I associate this very much with some kind of managerial mantra that I was very comfortable adopting, that basically the role of the manager is mainly the enablement of helping other people to be as if satisfied with themselves, everything else will come. And then we start hearing about these friends who work at some company that has a staging system that's critical to CI, but 80 percent of the time it's down and no one wants to take responsibility for it because it's really hard, but everyone suffers from it. And you say no, no, this won't happen to me, it can't be like this for me, but it happens to you, so you say shit, how did I get here? What to do?

Yishai: Like everybody.

Anton: Exactly, as if what appears from the side as good, they don't know what to do, suddenly you find out that, well, nobody knows what to do. So I come to Snyk with mainly a lot of experience about what I don't want to happen. I can't tell you that it's as if I had some kind of good playbook, in a month we'll be doing this and that. But a very, very clear example that I keep repeating in my head, one of the first things I did at Snyk, after quite a lot of management positions in larger organizations, where I don't have any hands-on time at all, I seem to have to peel off the rust and get back to coding because what are we, 5.5 people , like how much management is there already? And I say okay, it's really, really important to me that when we grow to be a bigger team, that we don't bother bringing in logs from production. Because I remember the episode of asking the ops guy, yes? He doesn't even call himself DevOps, it's before the branding, the IT guy almost, to tell him please bring me logs from yesterday, so he says what range of hours? And I say to him why do you need a range of hours? Bring the logs, I'll look for them, he says, you don't understand what you're asking for, it's like catalina.out from some 16 Tomcats, it's like let's focus on me. And the overhead of waiting for it and making graphs for it is like what, what is it? It's prehistoric. So one of the first things I do at Snyk is to make sure we have a centralized logging in the system and for about 5 years it's as if we picked the fruits of this tree and continued to work with it and fix it, because it wasn’t perfect, and a thousand things. But this is my experience from a developer experience and above all I like the transition from such a personal view of what makes me feel good at work, what puts me in flow, how I work easier, and a team or organizational view of how I want my organization to feel. Do I want to be that company that on Friday night everyone complains that the staging isn't working, with their co-workers or I don't know what, or not.

Rina: I think it's something I noticed, that whenever a startup is launched, you can see what hurt the founders in the previous place because it's the first thing they do in infrastructure, the first thing, so here it's logging, it can be all kinds of things.

Anton: Totally, we bring our scars with us and it's perfectly fine to surrender to it, you have to feel it. I think I've never worked in Big Tech and this whole concept of good, where it's organized, they have procedures and as if they know everything, looks really good from the outside. But when building something then I'm very much in favor of devoting myself to, how shall I say it? The things I don't want to happen can be a work plan. It's like, you have to translate it into something constructive, but to prevent the things you don't want to happen on your shift, is really cool.

Yishai: So these pains have actually always been with us, since the days of Emacs and VI and I think that the very fact that we mention these tools allows us to bring here, to create flame wars here forever. When did we start calling it by name or saying there is a person in charge that is his responsibility and it is not, at the end of the day every day of development is developer experience. But at some point you say okay, it has a name, it has a function or someone who maybe has another responsibility to take care of these things, and not just for it to be organic. When does this thing happen? When have you seen, whether it's throughout history or in a specific organization, where the dime dropped and they said okay, I need somebody on this. And not just someone who is tired of the tests so he goes and fixes them.

Rina: In general, I think it's not good to have someone in charge of it, except in a really huge company already, which isn't scalable, which is really worth investing in people. But the proportion of how many people are needed for DevEx compared to how many developers there are, it's orders of magnitude, it's one to a hundred, one to a thousand. You don't need more than that because it is impossible to centralize it, because every team has its own challenges and problems and the specific flow of how it works and that is also true in a huge company. They are very, very different from each other.  If SaaS the “norm” in double quotes is that you can literally deploy every few hours, it's not for example the product I'm working on which is a storage product that takes weeks to deploy because that's the architecture. So you can't project the same thing, as if it’s the same experience from here to here. So the tools, it's good to have uniform tools and it's good to have someone who looks at it from above, but in the end my group has to build its own DevEx, it can't copy paste from another group because it will not work.

"It is impossible to centralize Developer Experience, because every team has its own challenges and problems, and their specific flow of how they work, and that is also true in a huge company."

Yishai: So you're actually saying that DevEx is like a snowflake? Each team has its own needs and in the end it too will build its own solutions? Or you can say okay, there is a collection of things, pains that are quite repetitive, for example how long do the tests take or how do I make sure that if they fail they fail quickly. And to do that, at least the measurement, at least the awareness of, listen, the fact that you wait half an hour for the tests does not make sense.

Rina: So I think the pains are the same pains, but sometimes the gap is in expectations, or for example there is an inherent complexity. I was in a crazy Twitter thread about this just about two weeks ago. To me, waiting an hour for the tests is super fine, like it's even very good. Again, in the product I'm working on and also in Dropbox, 6 hours is completely normal and that's fine, and the thing is what is your expectation? You're working on a very, very complex system, whose tests really take so much time, so you can try to shorten it, but it won't really solve your problem because then you'll encounter it later when you get to an end-to-end test, so obviously you need some sort of quick feedback loop so I can develop and see I haven't broken anything really huge. But to really do the integration, it's fine to wait 6 hours, and then you have to look at where the real pain point is, and I know complex systems, the real pain point is flakiness. Nobody cares, if I do a routine and forget and in 10 hours it comes in and everything is fine, then I don't care that it took 10 hours, who cares? I keep working. But if it's 6, 10 hours, and then it fails, then it's already a catastrophe, and even,

Yishai: Failed not because of a bug I introduced.

Rina: Also because of a bug that -, no, as if it's obvious, if it fails it still frustrates you but it's part, what to do, that's the role of the testing. This flakiness that you don't know and you just press push again and again have to wait 10 hours, it's really unbearable.

Yishai: So send three and you'll have another 10 hour majority.

Rina: Yes, that's it, exactly, so I also think that even for me, when I think about my flow, a 6-hour test is better for me than a 20-minute test. Because in 20 minutes I won't have time to do anything, and I'm waiting for the tests to finish, and it's not great, really, I prefer a longer cycle.

Yishai: You're saying there's some stage where it's better if it takes longer so I can multitask.

Rina: If it's over a minute or so, an hour is better.

Yishai: Well, that's interesting, but even if the expectations are different, there is something universal that says shorter tests are better, except for what you just mentioned, a soft stomach that might be better, you shouldn't shorten from an hour to 45 minutes, or it's not that helpful. Everyone agrees that flakiness is a bad thing, I think no one will say give me some flakiness, it will enrich my day. The question is whether these things, where there are agreements, it is not a place to produce, even if not a central responsibility but at least some kind of method in the organization. Let's say okay, I care about it, I measure it, I deal with it, and it doesn't always stay number 17 on the list of the backlog to do one day when we have time. How do you get there or when do you see the beginning of this situation where someone says I deal with this on a regular basis?

Anton: Oh, I'll go back as well, I'll tie this question to your previous question about when it became a thing. So like many things in life, things become what we call them because of really good marketing efforts. Here, to see, everyone says DevEx, DevEx, what is it, two or three years ago no one would have understood what you were talking about, and I don't think it's something unusual that suddenly landed on us from somewhere, but you can look back at “revolutions”, in other quotes, of fields responsibility of developers, and to see how such things actually happen. Everything comes from a place of perhaps more technological progress, which makes our way of working unnecessary, and the most prominent example is, say, the worlds of DevOps, where once the technological complexity of writing code and the technological complexity of running systems were quite separate spheres and justified separate roles, but things moved forward and suddenly it is more often expected to take a wider responsibility for your code, even after you have pushed it to the main branch, and not to let someone do another job for you that you can actually do better yourself. Also look at the worlds of QA, it used to be clear that it was a separate role and required different training and a different mindset and - how can you let the cat watch the cream? What, for the developer to write tests for his code? Who has heard of such a thing? And there were changes there too. By the way, a marketing effort that perhaps caught on more abroad and I personally hear less of it in Israel, is the transition from quality assurance to quality engineering, from QA to QE. That is exactly the same person who knows how to do quality, but does not do the quality themselves but teach,

Yishai: Automation engineer?

Anton: No, no, no, really not writing the automation but helping people spread their wings on it. He says if you write the code and you also think that you should be measured on the quality of the code you write, let's make the writing easier for you, I will teach you. And for me it really connects to the DevEx worlds where people don't have to expect someone else to come and improve their flakiness, fix their staging or shorten their tests, as clear examples of developer experience, and there are other worlds like that, but it's really good in my opinion. A ratio of one to ten, a hundred or more, that there is a person who tells you that they care about your development experience while doing your development work, let me help you be efficient with your time when you take responsibility for it. I'm not your address for flaky tests, you don't wake me at night because the tests are flaky, but I'll tell you voila, it's true, it bothers you for a good reason. It's not a tax that needs to be paid, there's nothing to be done, you know how it is here, the system. Great, the tests are flaky, but here is what can be done to solve it, here are the ways to move forward.

Yishai: So what is this person? Is it mainly a source of knowledge? This is how you should handle it, here are methods that work? Or is there tooling?

Anton: To me it connects more to the platform team in corporate worlds. Yes, if you first, again look at titles and DevOps engineer then SRE then platform engineering, actually making accessible what's important, giving the confidence that you're not wasting your time when you're dealing with flakiness, a thousand, it's important and proper. Second, that way you won't go around in circles around yourself but move forward, but you are the one who does the work, you are responsible for the code, you are responsible for the tests, you are responsible for their flakiness, in fact in a constructive way to put in front of the people a show that shows them that it is actually them, without blame. Without saying move, move, I'll solve it, you don't know how to write tests that aren’t flaky, I'll do it.

Yishai: Sometimes even knowing to say it is really flaky and there is a number next to it, not just a gut feeling. Because in the end flakiness is something amorphous, it's hard to grasp it, the developers complain but is it very flaky, is it a bit flaky, is it getting better than last month?

Rina: That's what's nice, for example in an organization like Dropbox or Google, that there really is a group somewhere that develops these tools, everyone needs infrastructure for tests, everyone needs infrastructure for deployment, everyone needs flakiness detection, everyone needs these things. Someone else takes care of it. I plugin into these tools and it's amazing, it's amazing.

Yishai: And this is the platform group, in various incarnations.

Rina: Exactly, yes, and they give me these tools and it's great. I don't have to think if this test is flaky or not, I don't know, you give me statistics, I trust them that it's true, okay? Then I can reflect that to my managers and show them here, this is what we suffer and here are the places we need to improve.

Yishai: I want to justify investing in improving the tests because here, my flakiness is high and I can lower it.

Rina: And I think that the targets we set are also really subjective, they are a bit of a subjective matter. We can say we want the flakiness under half a percent, we'll never get there, and that's just demotivating. Then we can put something maybe a little more realistic, of say 5 percent, which is not amazing, it is not good, but it is much better than where we are now, after that we will talk about going down to even less.

Yishai: Where will it be determined? Where are we today and what is realistic? Or from benchmarks of, let's say in an organization like Dropbox or Google you can say hey, there are a thousand teams, there is a benchmark, I am in the percentile, among the best, I have nowhere to advance further.

Rina: Right, so I think that if you're among the best, then you really don't have to progress any further, but if you're not one of the best, then you still need them, I think there's also a lot of talk about this, about this subjective matter. When talking about developer experience, it's often a gap from expectations It's not just a dry index, it's how I feel where I could be or what hurts me the most and how I get there, you need these dry KPIs, but you also need to say where it really hurts me and that's subjective. So if now I have 40 percent flakiness and I can get to 5, that's crazy, that's amazing. When I get to 5, we'll talk about the rest, okay?

Yishai: It might not hurt so much anymore.

Rina: Yes, exactly, there may be something else that is better to invest in.

Yishai: You talked about the subjectivity of pain. If I connect to this, I think that developers, people in general also have recency bias. Yesterday I suffered from flakiness, that's what hurts me, and sometimes there are different sensitivities, even between people on the same team, to the same problem. There is some way to try to even it out or say we improved the flakiness from 40 to 20, we all feel it helped. How much did it help? Is there a way to measure the ROI on the investment in flakiness? How will I even check that it helped me other than thinking or feeling better? Or in the end only the feeling is important?

Rina: In my opinion, in the end, only the feeling is important. Yes, and it also has another side of, for example, just, now I work exactly in these areas and before, in order to set up, for example, a new test suite, there was a process where you really had to go to some kind of system like Jira and then set up all kinds of things there and then run some job of Jenkins, from something terribly terribly complicated. And now more have entered the Google ecosystem and I'm getting complaints that it's too difficult to define, really, 4 lines of code, I don't know, something like that. And I respect that completely, really, as if it's legitimate, because if they come and complain then something is probably bothering them and we'll work on improving it, but it's sometimes a little funny to me this gap of like two months ago it would have taken 10 tasks and sometimes you also have to give some sort of proportion. And to say okay, we understand, we've heard, maybe it's 10th place in our priorities right now. So as if you always have to balance the objective reality and the subjective feeling and that's why you have to measure both all the time.

Yishai: Anton, how do you relate to the truth of the difference between the subjective feeling and finally there may also be objective measures. Listen, the experience or the reflection of the experience in the indicators of the development process, overall your situation is good, why are you complaining?

Anton: So I agree with what Rina says about the fact that subjectivity ultimately matters because in terms of developer experience the word developer speaks of the audience and the word experience speaks of what is important. And in my opinion, the reason it has so burst into our consciousness now and it was difficult for it to happen before, is because this subjectivity was a kind of ceiling that was difficult to break. Someone who has a clear agenda and has time to invest in it and doesn't ask for favors from anyone, they just do what they think is right. And there are those that just roll with it or say like that is how it is here, works with what is available and that's where it ends. And perhaps the progress also around, again, work methods and work tools and the adoption of more I don't know, rapid work methods, that justify this need for simplicity and that it is not a matter of luxury but of efficiency during development is really critical. This helped to adopt the worlds of developer experience as something that managers really want to reflect upwards, justify with ROI, etc. Here, the transition from subjective success to objective measurement is very, very important. It helps to explain why it is important upwards, it helps to synchronize the people who take part in it, so that they understand what they are walking towards, and it also helps to show the long way we have come. Saying flakiness is problematic, but let's not pull out some figure from somewhere and get upset that we're not there, let's now measure something that quantifies our pain, reduce the pain and then see what we want to measure again. It is a very common practice in sales teams, to always give a quota to salespeople, so that they have somewhere to aim, and every time to change the incentive, okay? A quarter like that, now you get a bonus for it, whoever closes deals in the first month will get, each dollar will be counted as two dollars towards their quota. Now you can't stay with it for too long, people crack the system. Developing is the same thing, you give numbers, you measure, you promote but you have to decide, okay, where did it get us and where do we continue from here. I also wants to change the example a bit, we are constantly talking about flakiness of tests, obviously we have mutual scars here but this will not be a podcast episode about flakiness of tests. Let's talk about time to get a code review, okay? Experience, another scar, yes? There are teams where every retro,

"Subjectivity ultimately matters because in terms of developer experience the word developer speaks of the audience and the word experience speaks of what is important."

Yishai: It's personal, now it's personal.

Anton: Exactly, flakiness you have to do git blame to see who is responsible for the flakiness and they will always be able to say it was before me anyway. While waiting for the code review you have a person you are waiting for, like “what’s going on?!”? And it can destroy teams from the inside which is a waste of time. And I know a lot of organizations that in retrospectives in sprints the issue always comes up. I ask myself where I was most wrong in the estimation of how long things will take, very easy, yes? Make a mistake in the estimation of how long it will take you to apply for my code review, as if it's not my fault either. But to put it as a number and tell the team then you know what? Let's measure it, it's not that complex, no one is under any kind of police and that, but if that's what hurts us then let's put it on the table. It doesn't matter so much what the number is, what matters is the trend, where it moves,

Yishai: It's the awareness.

Anton: With what effort we have to get it, move it. Because this effort also comes at the expense of something else. Larger platform  teams in organizations that depend on this shared responsibility, very quickly discover the need for a product perspective to guide them. Because it is terrible to neglect, in quotation marks, this “internal body” for development, whose customers are actually the other people in the engineering organization. The VP Product has basically endless other roles, jobs to staff before he hires a product person for the platform organization. But a good platform organization must grow it from within and say I want to know how I manage to move the needle, that is important to me. How much would this cost me? At what effort? Also measure the investment of the body of the platform itself in front of the value it brings, and also quantify the time wasted on waiting for the code review and the flakiness of the tests and whatever it is in the other teams. To show everyone listen, this, speaking of recency bias, this thing annoys you but it is a drop in the ocean compared to this big thing that is simply harder for you to look at because it is more painful.

Yishai: So here you are actually talking about measuring the ROI, of the investment. I invested in taking care of flakiness or code reviews or anything else that bothers the developers, but in order to justify this investment the platform organization or the development organization has to say I made an investment, apart from the fact that I feel better. There is also ROI in outputs, in productivity, because if everyone feels much better but productivity does not move, then maybe it was not worth the investment or was only worth a part. So how do you measure?

Anton: Simply developer time, spent on it, don't need to enter dollars,

Yishai: Time wasted.

Anton: Can you come and say okay, how much time did you waste on the fact that you had to restart the test suite 3 times until it passed? How much time have you wasted doing less important tasks because the urgent thing was stuck on a code review you didn't get? Just measure this time, it's a super rigid figure, move it, reduce it.

Rina: It's also relatively easy to measure it, because if you use the systems then it's just okay, the comment goes in and then you wait for the review and there's a gap in time and that's it, so it's pretty easy to get these numbers out.

(transitional music)

Yishai: So I want to dive a little into the question of the Israeli perspective in an international company, it can be as big as Dropbox or Google or a company that may have Israeli DNA but very quickly it becomes international also in terms of who the developers are. It's no longer just “the guys”, and sometimes where the management is located, and where the product is, is there any impact on this side? On this international view on the developer experience? Or even on the developer experience of the Israeli developers?

Rina: For sure yes, there are two aspects here, one is that we are Israelis and we are a bit half-assed, like that, and we constantly try to circumvent the rules. And then for example I feel, and it may not be true, but I feel that Americans have it much easier, like okay, this is what the system does and that's it, and I flow with it and with all its limitations and everything. And the Israelis are constantly looking for how to get around it and how to do something a little better and a little more than that and not wait for someone else to do it, but do it alone and that's it. And sometimes it's a waste of time because really sometimes the ROI is very low, especially really in very mature organizations like for example Google that seem to “leave it” okay, well, then the ticket is not perfect, let it go, not bad, as if really. So as if from this side.

Yishai: So the Israelis feel as if these gaps hurt them more? And they are looking for a way to solve them in places where perhaps Americans come to terms with it and the gap between expectations and reality is smaller?

Rina: Yes, I think they really are, we're just not, we have chutzpah. We don't take things for granted and we constantly try to change and improve and bypass the system, constantly bypass. And the other side is that many times, and it very much depends on the company, but the Israeli site is a bit peripheral, and then we are not counted. Then there are things like, for example, receiving a page at two in the morning about the fact that the rollout was stopped because of something you did that they could have solved it on their own, and they don't think about the fact that for you it's like two in the morning, and maybe they didn't have to call you. No that-, I’m not very effective at two in the morning and when they call me and say listen to me, this, then I say ok, do revert, turn it off, I don't know, I don't care, we'll talk about it in the morning. And then there are really things that you don't think about, that really when there is a global rollout and you don't take into account that there are people in other parts of the world that this will regularly screw up their night. So these are also things to consider, when you are not the center of the

Yishai: That sometimes in a large organization the Israeli team is the one who gets screwed because of the hours, which is convenient to do a rollout during the hours when he sleeps.

Rina: Yes. They start the rollout at say six in the evening Israel time, and the problems start after the canary, everything really starts at one in the morning the problems.

Yishai: Anton, what did you see in this transition from Israelis to Israelis and more people? To the rest of the world?

Anton: So the truth is that even before the move and growth of Snyk, for those who don't know, the company started in a split way, with an office in Tel Aviv and an office in London. Right from the first day with every organization within the company, development, marketing, customer success, with a presence in both offices. I really liked this challenge and it really attracted me compared to all the companies I've worked in before that hmm-, it would be nice to work with people who speak a slightly different language and grew up in other places, they don't have 8200 there and I don't know, oh, and they're also in some two hours time difference, it's like it's not Palo Alto, what fun. And we start working like this and I discovered in these mental differences some sort of inexhaustible well of lots of things to work with and on. Let's take a slightly more positive take on what Rina mentioned as the Israeli lack of formality. I call it action bias, that is very, very big, okay? Like if we see a place where we can bring value, the ability to come and say "yes, but late, it's the weekend", it doesn't come easily to us, we see value that can be spent? We take it out here and now.

Yishai: Thursday night.

Anton: Completely. And it reminds me of a very interesting conversation I had with our chief architect from London, he came to visit the country, we sat for a beer in the evening, he had been with us for a few months and already understands right and left and I tell him tell me, what do you think of this story, where do you see the mental differences between the average British developer at Snyk and the average Israeli developer? He told me listen, I'm glad you're asking this question because 3 months ago I would have turned over tables here but now I see it a little differently. I told him what, why would you turn a table, about what? So he tells me what about pushing code to production on Thursday at nine in the evening? What is this, we are here a matter of life and death? Like what, why do you require me to come into my office on Friday morning, to deal with all your ricochets when you're not around at all? What was so important? I have a feeling that it is sometimes childish, what are we YOLOing it? Like there's no next week? Is it make or break? But over time I realized that it is actually based on a lot of action bias and a desire to promote things and not settle for good, okay, it's like this, it will stay like this next week as well. And I was so glad he said that because I immediately told him back, I told him listen, what's the deal with finishing the day at six no matter what? As if it's okay, personal life, I think we have a good and important work life balance. But the guys here seem to be working and we are all in such teams divided between Tel Aviv and London, everyone depends on everyone else, and I really feel that a wheel in the car stops turning when there are still a few hours left. These differences are something that must be constantly talked about, and my warmest recommendation for people, whether they are aiming at the big tech areas or small startups, is to experiment with teams that are as heterogeneous as possible, as different as possible in terms of the types of mentality they have. It could also be Israeli teams, not all of whom exactly served together in the same years, in the same unit, or other things. And simply let other people reflect to us other definitions of professionalism. Because pushing code for production at nine o'clock on a Thursday is a certain kind of professionalism, I'm not saying that it's not, knowing how to go home at six o'clock and to pace yourself over the very strenuous work you have at a startup, that's a kind of professionalism, that's fine, you have to know how to combine the both things.

Rina: I just want to say that I didn't mean it negatively.

Anton: (laughs) Sorry.

Yishai: So the Israelis have an action bias and perhaps also a slightly larger expectation gap compared to what is desired and what is found. Where is the developer experience of the developers abroad affected by the fact that they are working with Israelis? You talked about the pushovers for production at the weekend, does our action bias create noise? Does it create disturbances for other teams?

Rina: Sometimes yes, and sometimes you really have to play some kind of game that is almost unbearable, of making a long-term plan in order to convince them that we are not some barbarians who don't understand what we are doing when we don't really want this long-term plan and it doesn't interest us, but simply to promote things sometimes you need a little

Yishai: Create slides.

Rina: Yes, exactly, to put slides or write some kind of document. I don't know how much of this is like an inherent problem in a job that is cross-cultural and how much is really, for example at Dropbox, that it was a very, very peripheral site. And I don't feel it,

Yishai: And apparently it “was”.

Rina: Yes (laughing). I don't feel it at Google as much, because it really is a much more multicultural organization. I mean, there is always the issue that the US is the center and everything else is less, but in Dropbox it was really, like everyone is in the US, and there are some barbaric guys, sorry, in Israel, who make a mess of them, yes? And you need to reach some kind of organizational maturity for everyone to really be like this in the same line of how the organization behaves and how things are moved and how things are done. And it takes time, I mean obviously there are cultural differences, there are always cultural differences, how much it interferes depends very much on the maturity of the organization and the people who are there.

Yishai: And you, Anton, said that you saw success in the fact that the teams themselves were heterogeneous, not a team of Israelis,

Anton: Completely.

Yishai: And a team of Englishmen or Americans but a heterogeneous team that overcomes the time zone and cultural differences at the team level.

Rina: I think it is very difficult, but to make a heterogeneous team that is Israeli and American because the time differences are not, it is impossible.

Yishai: They are hard, I agree. We talked about the subjectivity, we talked about the complaints, about the developers who talk on Friday among the guys, I have it worse, no, I have it even worse, everyone has their own stories and scars. You have seen these developer experience questions affect, for example, hiring ? How do I bring in talent, he asks me, he interviews me about developer experience at my place? I don't come if there are no tests? Things like that?

Rina: They ask, they ask about on-call, they ask about tests, they ask about CI/CD, they ask about these things, definitely, I haven't heard that they ask about flakiness, but yes about the way work and the pipeline and how often they do deployment, like things like that, I heard questions such

Anton: Yes, these questions are more dependent on the level of knowledge and perhaps the amount of scars that candidates come with. But it's a very, very powerful tool for hiring managers to actually sell the position and the team by what's important. And talk about not only how good we are at X but what we're investing to be good at X and what we're ready for, put on the table to get these things because they are really, really important to us. I would talk a lot about how I see Snyk not as a security company as it might want to be portrayed in the eyes of its users, but as a dev tooling company, because that's how I wanted it to be portrayed in the eyes of the developers who join Snyk. Then the ability to come and talk about the product mindset, about how we want the user experience of the users of our product to be, and what this means for our user experience ourselves and all the concepts of dog feeding, etc., etc., very quickly open the door to , so let's not work half-assed, as if let's be proud of what we achieve. It doesn't come easily at all, I really wanted to avoid defining these roles, let there be someone. No, they’re in charge, go talk to them. And this expectation of lots and lots of autonomy from the teams, increases very much the level of expectations from them. Suddenly there are a lot of things that can be done, so what do you choose to do? And how do you focus? Grinding your teeth at the things that you suffer from and you don't take care of them, in order to earn something that you set for the sprint or for the next month. It relies a lot on people's motivation and a desire to simply pick up gloves. We talked about long tests, okay? I remember big services that start small and grow quickly, and suddenly tests that ran in less than 5 minutes take 25 minutes, and it happened in such a very, very creeping way for everyone, we were all there when it happened, it was like that yesterday too. Why would today suddenly be fed up like this - can't we continue? But someone should come and say, I want to take some time in the next sprint, to divide the test suite into several parallel builds, to bring us back under 15 minutes, what did I ask for? Does that sound reasonable? Requires a lot of support from the managers who will come and explain now, wait a minute, at the expense of which feature does this come? After all, this reduces the product scope in the upcoming sprint, and on the other hand, how do we hold ourselves accountable and the person does not go on some kind of self-searching journey in the intricacies of the our CircleCI - and surely returns home and says here, look, I got it for us, or we all got it together. These are very important things.

Yishai: How do you really get the buy-in from the product team? In the end, it's a non-functional investment that someone needs to be convinced that it will do good going forward for velocity and productivity. It's a constant fight for all the non-functional, is there any difference that fighting for developer experience is different from other non-functional investments? By getting the capacity?

Anton: What I like to do is to give some kind of guideline to the teams, which divides their time between investing in the product roadmap and investing in what we called in Snyk internal requirements. All the technical debt conversations of various kinds, etc., etc., just give it some name that says here the team is helping itself. My analogy for this is the difference between chopping down trees, which is creating the user value of our product, and sharpening the axes. Speaking of another such perspective from international teams, you understand how much our analogies border on militarism and all that,

Yishai: Axes and chopping are so un-Israeli.

Anton: Axes and a knife between the teeth and,

Yishai: No one cuts trees here in this country.

Anton: No, every time I talk about this ax I cringe a little.

Rina: That's it, an ax is the most foreign, as if no one here chops trees.

Yishai: There are no forests here, there are no more here,

Anton: But in our case, so the guideline that I give as a manager is some kind of justification to talk about it. We have time for internal requirements, let's not be at zero and refuse this investment because it has no legitimacy. It has legitimacy. Some? What is the number? I am comfortable being in a situation where I want the organization as an organization to be in some kind of split which is good in terms of benchmark, I ask friends what is going on with them and everything. But between teams it can change, there is a team that has more responsibility for legacy systems, a team that is more like blue ocean, some kind of new area, etc. But instead I don't want to dictate to my teams well, now you have to take care of it, what's going on with the flakiness, it's killing you without you feeling it. But I ask them what is the value you want to achieve in your internal requirement investment? And how do you quantify it for yourself? Here are the strongest product counterparts we've had the chance to work with as development teams, don't look at it as oh, nice, there's about 20 percent of the team's time that I don't have to provide them with work, I don't have to write specs for them, not that, come on, finally I can maybe take some time off, who knows. But instead they come and say let me be your accountability partner. Are you now rewriting this service from Javascript to Typescript? Some kind of internal requirement? Excellent, how will you know that this sprint was successful? Is it just some files that now have a TS extension instead of JS? Or is it something else? What are you looking for? How will you know you succeeded? These are questions that make all the difference for the team and make them actually understand what is important. What is the purpose this is trying to serve, what is the outcome we are  trying to achieve? And not good, all our code is point ES but all the functions are colons any colons any, we are in a typescript, not really.

Yishai: So here you call on the product people to be partners in accountability and sometimes even in the definition of the internal requirement of the non-functional,

Anton: Don't rush to say voila, take care of it yourself, I have enough of a headache here with the business and I don't know what,

Yishai: No, I can contribute my skills in these areas.

Anton: And with the product roadmap, except to come and say a moment, I can help you sharpen the axes because in the end I want to see really great axes when chopping trees, how can I help?

Rina: I think this perspective is very important because sometimes the individual developer does not know all the considerations and has not heard all the complaints of all the others. So many times I get the same complaints from 10 developers and they don't know that we are aware of it and we decided not to deal with it now because XYZ and it's a bit difficult to communicate this decision, and like what Anton said is perfect and great and I wish all organizations functioned like this, sometimes you have to let things go to crash and burn and say we have reached a place where we must take care of tech debt. Because it's very easy to always sweep it under the rug and say good, not good and it's not measurable and we don't know how it will affect that, and sometimes there's a really huge explosion and then you can say guys, that's why we came here, we have to invest in it and then reach To this steady state we say 20 percent of the time and start measuring things and invest in it. Not everyone has the scars from the previous place, they need something to explode,

Yishai: You say it has to hurt in order for us to accept the justification for

Rina: Exactly, yes.

Yishai: Which may be a sign of the maturity of managers or those who already know the material, and knows it will eventually happen, this pain will come, I can avoid it if I dedicate time to it in advance,

Rina: Sometimes you need a really, really good justification, and then you sometimes really direct the organization to a place where it's going to explode, in order to receive this investment, because the product is not always smart enough to understand,

Yishai: We stop the testing and -, the business explodes, what's the problem? Or random.

Rina: (Laughing) And I think that sometimes you really really need it, to convince, because it's hard to see, it's hard to see all those 5 minutes that go by there and the whole hour that goes by here, it's hard to measure it and it's hard to justify the ROI when there's a features that need to go out to customers.

Yishai: So this perhaps brings back to the stage the questions of how I measure and show lost time, externalize it so that in the end if I show the senior managers you will hear, percent, two percent of your developer workforce is wasted, lost time is full of money and it is full of value, if I return them, this time to productivity, the ROI is amazing.

Anton: My recommendation here is to be creative and innovate with which indicators we want to have on the internal dashboard of our team, which will help us quantify the lost time or the “pain”, in quotes, in one way or another. But not to be so creative when it comes to steady state, I mean we decided to focus while we wait for code review, and we got to a better place, let's leave this number there, let's see if we deviate from it, it becomes our benchmark. Now, if we want to justify investing in this thing again, let's wake up earlier in case this index starts to drop.

Rina: I think it's necessary at the management level to keep an eye on it all the time. I mean, it's not enough to look at it once and say oh, well, we'll improve it, but we really have to sit on people's heads with this, because it's very easy to let the tests turn red and to say well, it's not important, it's not important, it's not important, you can go out anyway. And it's very easy to really let all these measures of time, of the code review and that, deteriorate. Because it's really, it needs this red line of the target and not go above it, as if it doesn't matter what this target is, just keep an eye on it all the time because it goes up from 5 minutes to 7 minutes to 10 minutes,

Yishai: It is never shortened to slow, alone.

Rina: No, it never gets shorter, doesn't get better on its own (laughs)

(transitional music)

Yishai: Anton, today you deal a lot with mentoring, actually helping development managers, development managers who are starting the journey or who are already progressing, where do these issues of developer experience come up in the processes you go through with the development managers?

Anton: My mentoring is something I've been doing for two years. And in these two years I saw a very different climate between, somewhere in the middle of 2021, when everyone thought there was no such thing as a ceiling and there was no such thing as no money to what our market experienced in the last year and six months. What is interesting is that in this period the concepts of the developer experience are repeated over and over again, but in somewhat different perspectives. At that time, during the peak period, a good developer experience is an excellent selling card for organizations that are really competing for talent. Come to us because with us it will be easier for you to be effective. Well, a statement that if you know how to back it up with hard data, it's worth a lot. In the last year, in the last six months even more, I have the feeling that a lot of engineering managers are under a different magnifying glass of “show us that your organization is efficient”. Look at what's happening outside, we need to extend the runway, we need to lower the burn rate, let's make sure that our organization is efficient, and if it's not efficient then we'll reduce it. It's like fine, we don't need to be afraid of it, it's time. These two things, these two different perspectives, in my opinion confront development managers with a very, very non-trivial question, which is, is my organization effective? And this is something in the worlds of experience that we talked about, go now and wave engagement services and say that everyone feels engaged and it's quite fun here, that means we are efficient, doesn't it? It's not that convincing. To break down this question into which indicators are relevant, is the goal of the indicators internal visibility of the team itself regarding how it works, or rather the visibility of the management into my organization, of what we achieve for the company while we invest, difficult questions that everyone faces.

Yishai: Can we agree that developer experience and productivity are very, there is a large overlap between them, are they not contradictory? Because it's different goals, it's goals, or the management wants to save or be efficient, and on the other hand a happy developer, one could think that they are sometimes conflicting goals.

Rina: Developer experience is not the ice cream in the fridge, it really isn't. And it goes hand in hand. Developers really like that their code goes into production, they like it, it's the dopamine, it's the fun part, that it goes out and it progresses and brings value, they don't really enjoy writing code for nothing,

Yishai: Into the drawer.

Rina: Yes, for the drawer. And I really think this, this is the breakthrough of the DORA reports and these things, which showed that there is simply such a high correlation between efficient organizations that release fast and good, high-quality code, and developer satisfaction. In the end this is what we want, we want to release code for production, quickly and well.

Yishai: Maybe the first step in the developer experience is to tell me that my code has gone into production. Sometimes developers don't know that. I did a merge, I may have an additional role, but I'm not always the one responsible for getting the code out and arriving, maybe it's also in production but it didn't reach customers, feature flags and rollouts and such. Let me know that the piece of code I did, the bug fix or that, arrived, it does the job.

Rina: Right. I think the power of you, your team opening the feature flags, is an amazing thing. The fact that I'm following this thing and no one else is, okay, they gave me the tool that allows me to make feature flags for me to develop this infrastructure myself. But the fact that I open it and I decide to which target audience it goes and to what percentage and how quickly I open it and do the rollout plan in general, this is something that is amazing to me, it is exactly this feeling of the capabilities,

Yishai: What I do affects the world and I see it.

Anton: In my opinion, the potential difference between the developer experience vector, how much fun it is here to be in flow and be, I don't know, effective with my time, and the productivity vector of how much value my work manages to generate for people who care less about what happens under the hood of the car. This is an incredible breeding ground for the administrators to come and say wait, I want these to be two orthogonal vectors of yes, me, come on, okay, you want to upgrade the React version? Let's call it developer experience because we will be in the newest version, well, do it, even though it doesn't bring any value to anyone but I was told that developer experience is important so here we take care of it? Or I want to put these vectors on top of each other and say we can achieve much more in each of these things if we do them together. And then it also corresponds with people's motivation, it's great that you want to upgrade the React version, we really want to be on the leading edge of technology, where does it help us in the product roadmap? And if not, then maybe we will upgrade to the next version and not to this version, as if let's do things for the right reasons and not just the things that seem right to us.

Yishai: So the intersection of, or the places of developer experience and productivity come together, it is also easy to justify the investment and I can also produce some kind of sustainable developer experience.

Anton: Completely.

Yishai: In conclusion, I would be happy if each of you would give some recommendation to a manager in a development organization or even to a developer who wants to take the next step, whether it is to enter these worlds of dealing with developer experience or what to look at, what to think about, what to happen. One recommendation for those who are like this, for the next step.

Anton: I have a recommendation for managers, especially those who find it difficult to say goodbye to their hands-on contribution and they don't understand what they should do between all these meetings, look at the way your team works, look for those air pockets in areas that affect everyone, that are very much in correspondence with worlds the developer experience, and turn it into your hands-on time. It won't jeopardize anything critical because you'll be in the meeting and everyone will benefit from it. And the same advice can also be taken to the areas of developers who want to level up. The fastest way to increase the impact is to spread your wings on the people next to you, it's not only to solve your problems, in quotes, faster, but to look at our problems. Anyone who decides that tomorrow he or she will look at his or her development work, less as an individual sport and more as a team sport, is doing himself a very, very big favor and there is nothing like asking this question. Wait, what hurts us, all of us? Is that the Flakins? Is it time to review? Is it the ability to push something to production and remove the feature flag in a reasonable time? What can we do to improve it?

Yishai: I liked it, so actually the way to progress or increase impact is to get involved, in developer experience and investments around it are investments of this type that allow me to break through the ceiling of I only make features, that's how I progress.

Anton: Completely.

Rina: Anton said it perfectly, so I'll just add one thing that there really is the DevOps bible, it's called Accelerate, and it's, you don't need to read everything. There's an unnecessary introduction, there's the end of the research methods, unnecessary, you can only read the middle, it's something like 50 pages, just don't reinvent the wheel, today there are a lot of indicators and a lot of research around what makes a good developer experience, just start with that and then you don't have to say like what hurts me, what is it, you have to ask these questions but no You have to ask them open questions, you can come up with familiar indicators,

Yishai: There is already a frame.

Rina: Exactly, there is a framework, well-known indicators of how to measure and what to measure,

Yishai: We are talking about the DORA metrics, and then SPACE, and now there is,

Rina: There's more, I don't know

Yishai: These frameworks are advanced, but yes, DORA has become a very familiar name.

Rina: Right, so now I actually built these KPIs recently in my organization and it was like it was very easy to say here, this is our framework, not everything we have to take, there are things that are less suitable, more suitable, but this is where we start, and just really don't invent the wheel, use the knowledge already found and start from there. Those who want to level up and that, just don't try to invent, I've said it about 5,000 times (laughing).

Yishai: Rina, Anton, thank you very much for coming. It was fun.

Anton: Thank you.

Rina: Was really fun.

(transitional music)

Visit devInterrupted.com to subscribe, you can also find all our episodes in English there. I remind you that we at LinearB are growing rapidly, recruiting for a variety of positions in all fields. Come to linearb.io/careers to find your next challenge. I'm Yishai Beeri, we'll hear from you in the next episode.

(closing music)

Links to the nifty tools and resources mentioned in the episode:

• Google
• Snyk
• Dropbox
• DORA Metrics
• SPACE Metrics
• Accelerate