Article originally published by VictorOps<\/p>\n
Raleigh Schickel, DevOps Manager, has seen uShip evolve from a small team with a few developers, to a larger company with a dev team size of 60 and growing. Initially, everyone was always on-call for their own code. But this culture of ownership changed as the company grew.<\/p>\n
\u201cAs we hired more people, code ownership centralized, though not intentionally,\u201d says Raleigh. Developers started expecting Operations to be responsible for the running system, and started working on features and moving on.\u201c<\/p>\n
uShip<\/a> is a continuous deployment shop and developers are empowered to deploy code at any time. As Operations became more centralized, it was becoming more difficult to determine cause of application issues.<\/p>\n \u201cProblems that could have had a quick easy fix would lag,\u201d says Raleigh. \u201cThis made our Time to Identify and Time to Resolve unnecessarily long. The question became: How can we decrease time to identify and resolve?\u201d<\/p>\n uShip\u2019s developers also expressed their desire to own their own infrastructure and not wait on others. But they didn\u2019t want to be on-call. Raleigh says, this didn\u2019t add up.<\/p>\n \u201c[The developers] are looking for ways to speed up their development processes and time to market, but there are security and operational problems with that. If they change a setting and go to sleep and the service breaks, who deals with it? Who knows the most about it? I don\u2019t know what they did.\u201d<\/p>\n In response to their request, Raleigh convinced the developers to take on-call responsibilities. \u201cOur rationale was this,\u201d says Raleigh. \u201cIf you are willing to be responsible for the code you are delivering today, then we can expand access to the infrastructure that we are creating for tomorrow.\u201d They agreed.<\/p>\n Now (more than ever) with a team of 60 and growing, uShip needed a better way to manage on-call. uShip had handled incidents and log communication via email, with Nagios paging the team directly. This process was unwieldy.<\/p>\n They chose VictorOps for intelligent alerting, routing, and incident management. Now this 60+ person team could intelligently and humanely handle incidents that might occur anytime. Raleigh explains:<\/p>\n \u201cBefore VictorOps, we were limited to the same four or five people who were on-call all the time and that was a burnout gig. VictorOps allowed us to democratize the on-call process. We spread out the on-call load, which helps build empathy among developers about what other people go through. It allows those people who have traditionally been on-call to step back for a moment and catch their breath.\u201d<\/p>\n The VictorOps developer discounted pricing program enabled uShip to affordably provide accounts for the entire development team.<\/p>\n To manage their on-call schedules, development teams work on a three-month cycle in which each team spends two weeks on call. They are on-call from 6pm until 9am, at which time the Ops team takes over.<\/p>\n uShip\u2019s developers use the VictorOps team set up and scheduling features extensively. Since each team sets its own schedule for its members, they have used their creativity to design complex rotations. For example, they used VictorOps to put themselves on-call in two-hour chunks.<\/p>\n Raleigh especially loves the scheduled override feature because if there is a last-minute schedule change, it\u2019s no problem. If someone on-call gets sick or something happens, they can just create an override instead of having to tweak the on-call schedule.<\/p>\n Raleigh explains that uShip\u2019s development teams are primarily on-call to monitor application health. They respond to incidents related to questions such as, How many exceptions do we have? Is the marketplace healthy? Do we have enough new listings? Do we have enough new users?<\/p>\n Developers are also on-call for infrastructure issues that have well-defined, simple fixes. \u201cAs long as the alert is clear and tells them what is going on, they can go push a button and easily fix a problem,\u201d says Raleigh. \u201cIf they have to go reset app servers, we have buttons for that.\u201d<\/p>\n However, if a Linux server is broken and requires intensive troubleshooting or if a telemetry system is down, an Ops team member handles those incidents and are not a developer\u2019s responsibility to solve.<\/p>\n For the most part, developers aren\u2019t part of a time-based on-call rotation. Instead, they are always on-call for their code in their area of expertise. Via the VictorOps Incident Automation Engine, Raleigh set up routing keys that send each alert to the right expert. During feature releases, the responsible dev team goes on-call until everyone is comfortable that the deployment was successful.<\/p>\n \u201cDevelopers get to think about and understand the whole system in a way that they were not able to before,\u201d says Raleigh. \u201cTheir mindset was: of course my code works. Actually, there is a giant system out there that interacts with your code.\u201d<\/p>\n The dev teams self-organized to have one person from each team on-call at all times in case of a problem. They wrote an app called the Victorbot that allows anybody in Slack to create a manual incident and page the appropriate team via Slack in case of an emergency. \u201cThis is another way that VictorOps has helped us only page the right team when we need a response,\u201d says Raleigh.<\/p>\n Raleigh explains why putting devs on-call has been so great for the team. He says, \u201cThe devs get a little taste of what it\u2019s like to wake up in the middle of the night and handle the platform. They have shown great desire to make sure the launch of new code is healthy and for being the primary person on-call for it at launch. The best part is that we\u2019re shifting back to the ownership culture.\u201d<\/p>\n Ultimately, owning code isn\u2019t just a nice-to-have. It enables uShip to put its resources toward development rather than toward supporting increasingly complicated infrastructure, especially as microservices proliferate and require specialization. Raleigh says:<\/p>\n \u201cIf you believe in democratizing operations, then developers need to be on-call. Otherwise, if you have 20 microservices and five go down, how many Ops people would you need to put that fire out? It’s a choice. Are you going to pay for developers or are you going to pay for an Ops team? We think our company benefits more from delivering products. The more developers you have, the more you can develop product. It’s just kind of the reality.\u201d<\/p>\n With uShip\u2019s culture shift and devs now available to take on-call, Raleigh\u2019s DevOps team has more time to focus their work on helping the company innovate faster, which means writing code and developing and improving infrastructure. Raleigh says, \u201cAt some point in your company\u2019s life, you\u2019re going to take another look at code that was written with ideals in mind, and realize that as volume and traffic increase, it doesn\u2019t always perform very well. Right now, for example, our team is currently focused on writing code to reduce load on the database.<\/p>\n \u201cWe\u2019re turning two of my senior dot net developers into SREs so we can focus on this work,\u201d Raleigh continues. \u201cIt\u2019s good for everyone involved to be doing this type of work as a means to improve platform performance and mitigate future issues and alerts. We would rather have the team working for the future than firefighting today\u2019s application issues.\u201d<\/p>\n","protected":false},"excerpt":{"rendered":" Article originally published by VictorOps The original all-hands-on-deck culture faltered during growth. Raleigh Schickel, DevOps Manager, has seen uShip evolve from a small team with a few developers, to a larger company with a dev team size of 60 and growing. Initially, everyone was always on-call for their own code. But this culture of ownership… Read More<\/a><\/p>\n","protected":false},"author":28,"featured_media":7562,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[295,2],"tags":[297],"class_list":["post-7507","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-shipping-code","category-company-news","tag-shipping-code"],"acf":{"blog_post_content":null},"_links":{"self":[{"href":"https:\/\/ushipblogsubd.wpengine.com\/wp-json\/wp\/v2\/posts\/7507","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ushipblogsubd.wpengine.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ushipblogsubd.wpengine.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ushipblogsubd.wpengine.com\/wp-json\/wp\/v2\/users\/28"}],"replies":[{"embeddable":true,"href":"https:\/\/ushipblogsubd.wpengine.com\/wp-json\/wp\/v2\/comments?post=7507"}],"version-history":[{"count":0,"href":"https:\/\/ushipblogsubd.wpengine.com\/wp-json\/wp\/v2\/posts\/7507\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/ushipblogsubd.wpengine.com\/wp-json\/wp\/v2\/media\/7562"}],"wp:attachment":[{"href":"https:\/\/ushipblogsubd.wpengine.com\/wp-json\/wp\/v2\/media?parent=7507"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ushipblogsubd.wpengine.com\/wp-json\/wp\/v2\/categories?post=7507"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ushipblogsubd.wpengine.com\/wp-json\/wp\/v2\/tags?post=7507"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}\nThe challenge: how to recreate that early culture of code ownership.<\/h2>\n
\nVictorOps helps democratize the on-call process.<\/h2>\n
\nCreative approaches to on-call rotation schedules.<\/h2>\n
\nDevs on-call handle application health and well-defined issues.<\/h2>\n
\nAlways on-call in their particular area of expertise.<\/h2>\n
\nUsing Slack to create manual incidents eliminates even more noise.<\/h2>\n
\nDevs on-call feel empathy and build even better code.<\/h2>\n
\nChoosing to build features rather than building a huge operations team.<\/h2>\n
\nThe DevOps team has more time to innovate.<\/h3>\n