SDN Journal Authors: Pat Romanski, Elizabeth White, Yeshim Deniz, Destiny Bertucci, Liz McMillan

Related Topics: @DevOpsSummit, Linux Containers, Containers Expo Blog, SDN Journal

@DevOpsSummit: Blog Post

Logging and Continuous Delivery By @FloMotlik | @DevOpsSummit [#DevOps]

Continuous Delivery is the future of building high class software

Guest blog post by Florian Motlik, Cofounder & CTO of Codeship Inc.

Why Great Logging Is Key to Continuous Delivery

Over the last years Continuous Delivery has gained a massive following with many development teams embracing the style. Companies have chosen (as with many other modern developer tools), to either build their own embrace a hosted service likeCodeship. In the end though, no matter if you go with a hosted service or roll it on your own, the goal is to move faster and build a product that your customers really love. For that you need to iterate quickly, get feedback and iterate again.

Successfully rolling out that process depends on many variables. Proper logging is one of those variables, and can be a helpful tool to remove fear.

Why great logging is key to continuous delivery

Thou shalt not be afraid
As we've moved into the age of cloud software development, it is all about team productivity. Getting started on infrastructure is virtually free, so every team plays on the same level playing field and you need to constantly increase productivity to win.

By far the biggest killer of productivity is fear of moving faster because you might break things.

Then your team, processes or technology are not built with constant change in mind and you decrease the speed in which you release to have better control. This is a downward spiral that will only lead to slower processes, less innovation and you losing your market.

Fear stops experiments and promotes stagnation.

Having a repeatable and easily automated process can drastically reduce that fear. The more you execute that automated process the safer you feel.  Over time this will become stronger as all potential issues get discovered and fixed.

A second very important improvement is getting deep insights into any processes and workflows happening in your application.

When you continuously deploy changes to your application, being able to trace any step your application makes becomes your main tool to debug your production system.

The insight you can gain quickly from looking through your logs will often show you the problem a recent deploy brought into your infrastructure immediately.

This is indispensable with Continuous Delivery.

While Metrics are a great and very important part of getting insight into your infrastructure they only represent the state of the system. Additionally to understand the state you need to be able to trace and deeply understand anything happening in your infrastructure.

Integration with Pager systems provide an additional level of security on top of that to always be aware of problems happening.

Let there be light
We've grown accustomed to have full insight into our testing and deployment process as we are and have been using Codeship to build Codeship for a very long time. We understood we needed that same insight into our application as well to build the kind of infrastructure that supports the quality we want to deliver.

A good logging strategy and overview on your most important workflows is necessary.

Defining a graph of all the states that a workflow can have in your system makes it easy to add logging to each of those states and the transitions between them. At that point logging is not an afterthought, but integral part of your software development effort.

This needs to be clearly communicated to your team so everyone follows it thoroughly.

For example we test and deploy code for thousands of companies with many different language and infrastructure requirements. Those companies connect to GitHub or Bitbucket as their source code management system and various hosting providers like Heroku or AWS to deploy to. There are many moving parts in that system so we need to be able to detect and debug problems anytime without effort. A well thought out logging strategy can help tremendously and make it easy to fix issues when they come up. We can follow any build through all of our infrastructure and correlate issues between builds or our infrastructure anytime.

You can read more about how we use Logentries at Codeship in an earlier post.

Grand Central Logging
Being able to follow any workflow means collecting logs of various services in one place. If your developers have to look through various sources for correlating potential problems productivity and the speed to resolve an issue become very low.

Continuous Delivery is the future of building high class software. Automated testing and deployment build the basis of the workflow, but many other tools like centralized logging and error reporting are important building blocks of that workflow as well.

When your system feels like a black box you will hesitate to release changes to that infrastructure. Make sure you're not stuck with that black box and build a workflow that makes your team more productive and increases your products quality.

If you want to learn more about Continuous Delivery you can also take a look at our crash course which can be found on the Codeship homepage.

More Stories By Trevor Parsons

Trevor Parsons is Chief Scientist and Co-founder of Logentries. Trevor has over 10 years experience in enterprise software and, in particular, has specialized in developing enterprise monitoring and performance tools for distributed systems. He is also a research fellow at the Performance Engineering Lab Research Group and was formerly a Scientist at the IBM Center for Advanced Studies. Trevor holds a PhD from University College Dublin, Ireland.

@CloudExpo Stories
@DevOpsSummit New York 2018, colocated with CloudEXPO | DXWorldEXPO New York 2018 will be held November 11-13, 2018, in New York City. From showcase success stories from early adopters and web-scale businesses, DevOps is expanding to organizations of all sizes, including the world's largest enterprises - and delivering real results.
DXWorldEXPO LLC announced today that Dez Blanchfield joined the faculty of CloudEXPO's "10-Year Anniversary Event" which will take place on November 11-13, 2018 in New York City. Dez is a strategic leader in business and digital transformation with 25 years of experience in the IT and telecommunications industries developing strategies and implementing business initiatives. He has a breadth of expertise spanning technologies such as cloud computing, big data and analytics, cognitive computing, m...
DXWorldEXPO LLC announced today that Kevin Jackson joined the faculty of CloudEXPO's "10-Year Anniversary Event" which will take place on November 11-13, 2018 in New York City. Kevin L. Jackson is a globally recognized cloud computing expert and Founder/Author of the award winning "Cloud Musings" blog. Mr. Jackson has also been recognized as a "Top 100 Cybersecurity Influencer and Brand" by Onalytica (2015), a Huffington Post "Top 100 Cloud Computing Experts on Twitter" (2013) and a "Top 50 C...
Cloud-enabled transformation has evolved from cost saving measure to business innovation strategy -- one that combines the cloud with cognitive capabilities to drive market disruption. Learn how you can achieve the insight and agility you need to gain a competitive advantage. Industry-acclaimed CTO and cloud expert, Shankar Kalyana presents. Only the most exceptional IBMers are appointed with the rare distinction of IBM Fellow, the highest technical honor in the company. Shankar has also receive...
Enterprises have taken advantage of IoT to achieve important revenue and cost advantages. What is less apparent is how incumbent enterprises operating at scale have, following success with IoT, built analytic, operations management and software development capabilities - ranging from autonomous vehicles to manageable robotics installations. They have embraced these capabilities as if they were Silicon Valley startups.
Poor data quality and analytics drive down business value. In fact, Gartner estimated that the average financial impact of poor data quality on organizations is $9.7 million per year. But bad data is much more than a cost center. By eroding trust in information, analytics and the business decisions based on these, it is a serious impediment to digital transformation.
Daniel Jones is CTO of EngineerBetter, helping enterprises deliver value faster. Previously he was an IT consultant, indie video games developer, head of web development in the finance sector, and an award-winning martial artist. Continuous Delivery makes it possible to exploit findings of cognitive psychology and neuroscience to increase the productivity and happiness of our teams.
Predicting the future has never been more challenging - not because of the lack of data but because of the flood of ungoverned and risk laden information. Microsoft states that 2.5 exabytes of data are created every day. Expectations and reliance on data are being pushed to the limits, as demands around hybrid options continue to grow.
The standardization of container runtimes and images has sparked the creation of an almost overwhelming number of new open source projects that build on and otherwise work with these specifications. Of course, there's Kubernetes, which orchestrates and manages collections of containers. It was one of the first and best-known examples of projects that make containers truly useful for production use. However, more recently, the container ecosystem has truly exploded. A service mesh like Istio addr...
Digital Transformation: Preparing Cloud & IoT Security for the Age of Artificial Intelligence. As automation and artificial intelligence (AI) power solution development and delivery, many businesses need to build backend cloud capabilities. Well-poised organizations, marketing smart devices with AI and BlockChain capabilities prepare to refine compliance and regulatory capabilities in 2018. Volumes of health, financial, technical and privacy data, along with tightening compliance requirements by...
Business professionals no longer wonder if they'll migrate to the cloud; it's now a matter of when. The cloud environment has proved to be a major force in transitioning to an agile business model that enables quick decisions and fast implementation that solidify customer relationships. And when the cloud is combined with the power of cognitive computing, it drives innovation and transformation that achieves astounding competitive advantage.
Evan Kirstel is an internationally recognized thought leader and social media influencer in IoT (#1 in 2017), Cloud, Data Security (2016), Health Tech (#9 in 2017), Digital Health (#6 in 2016), B2B Marketing (#5 in 2015), AI, Smart Home, Digital (2017), IIoT (#1 in 2017) and Telecom/Wireless/5G. His connections are a "Who's Who" in these technologies, He is in the top 10 most mentioned/re-tweeted by CMOs and CIOs (2016) and have been recently named 5th most influential B2B marketeer in the US. H...
DevOpsSummit New York 2018, colocated with CloudEXPO | DXWorldEXPO New York 2018 will be held November 11-13, 2018, in New York City. Digital Transformation (DX) is a major focus with the introduction of DXWorldEXPO within the program. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive over the long term. A total of 88% of Fortune 500 companies from a generation ago are now out of bus...
DXWordEXPO New York 2018, colocated with CloudEXPO New York 2018 will be held November 11-13, 2018, in New York City and will bring together Cloud Computing, FinTech and Blockchain, Digital Transformation, Big Data, Internet of Things, DevOps, AI, Machine Learning and WebRTC to one location.
Cloud Expo | DXWorld Expo have announced the conference tracks for Cloud Expo 2018. Cloud Expo will be held June 5-7, 2018, at the Javits Center in New York City, and November 6-8, 2018, at the Santa Clara Convention Center, Santa Clara, CA. Digital Transformation (DX) is a major focus with the introduction of DX Expo within the program. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive ov...
DXWorldEXPO | CloudEXPO are the world's most influential, independent events where Cloud Computing was coined and where technology buyers and vendors meet to experience and discuss the big picture of Digital Transformation and all of the strategies, tactics, and tools they need to realize their goals. Sponsors of DXWorldEXPO | CloudEXPO benefit from unmatched branding, profile building and lead generation opportunities.
Dion Hinchcliffe is an internationally recognized digital expert, bestselling book author, frequent keynote speaker, analyst, futurist, and transformation expert based in Washington, DC. He is currently Chief Strategy Officer at the industry-leading digital strategy and online community solutions firm, 7Summits.
The dynamic nature of the cloud means that change is a constant when it comes to modern cloud-based infrastructure. Delivering modern applications to end users, therefore, is a constantly shifting challenge. Delivery automation helps IT Ops teams ensure that apps are providing an optimal end user experience over hybrid-cloud and multi-cloud environments, no matter what the current state of the infrastructure is. To employ a delivery automation strategy that reflects your business rules, making r...
Digital Transformation and Disruption, Amazon Style - What You Can Learn. Chris Kocher is a co-founder of Grey Heron, a management and strategic marketing consulting firm. He has 25+ years in both strategic and hands-on operating experience helping executives and investors build revenues and shareholder value. He has consulted with over 130 companies on innovating with new business models, product strategies and monetization. Chris has held management positions at HP and Symantec in addition to ...
Digital transformation is about embracing digital technologies into a company's culture to better connect with its customers, automate processes, create better tools, enter new markets, etc. Such a transformation requires continuous orchestration across teams and an environment based on open collaboration and daily experiments. In his session at 21st Cloud Expo, Alex Casalboni, Technical (Cloud) Evangelist at Cloud Academy, explored and discussed the most urgent unsolved challenges to achieve fu...