Posts in Continuous Delivery

Cloud Native

CI/CD Continuous Delivery Deployments DevOps system architect

How does your team is working with cloud infrastructure?

Cloud Native Topics

  • Development process
  • System design
  • Builds and packages
  • Deployments
  • Release
  • Cloud infrastructure

Development process

When working with cloud systems the proven methods to develop and run applications in production is DevOps

DevOps is the practice of code > build > test > deploy > release > repeat

DevOps is about bridging the gap between development to production

System design

You can use different methods of designing applications but if your applications are stored in the cloud then Microservices might be better to implement, otherwise why use on-premise approach when you can enable the benefits of using cloud infrastructure?

Builds CI and packages

Containers by now are the default approach to build applications as its comfortable to ship and deploy

Its easy to start local development environment using containers and work on your application

CI is an integral part as this will initiate your committed code to get build and get deployed

Deployments

Deployments to production is easier when the application is packaged as a container image

This is the next step after builds as this is the confirmation that the tests are ok and the container image with the latest code already pushed to the image repository and ready for deployment to production

Release

This is the step after the deployment is successful and the new image is used in production

Now the choice is when to enable new features using feature-toggle

Once the new features are enabled the release step is complete

Cloud Infrastructure

To manage the containers and successfully implement Microservices you’ll need Kubernetes cluster to orchestrate the containers runtime

Other services like sending email, databases, load balancers and more, can be integrated with your Kubernetes cluster to be used for the entire stack

Summary

Cloud Native is proven to have better results and happy developers

but Hey! you can always start a long running VM and install some stuff on it

How to deploy production systems

CI/CD Continuous Delivery Deployments DevOps

In some situations development teams are deploying some of the production systems from local, yes from their local laptops. that’s is a not recommended practice as it causes issues in prod.

Why developers modify from their local laptop?

There can be a few resources for why deploying new code and modifying production config is done from local laptop, here are a few examples

  • No CI pipelines
  • Failed CI pipeline
  • Just bad practice

In any case it is not recommended to modify production systems from local laptop and it’s better to use CI tools.

How to modify production systems?

When you update production with new code/config or new services the downtime should be zero.

for a successful deployment to production you’ll need to adopt a few approaches

Git Ops

In summary this means that every modification will be committed to your Git repository.

Infrastructure as code

In summary this means that the creation and update of the infrastructure should be declared in code/template.

CI Pipeline

When a commit is made to the relevant repository Git sends a hook to the CI system to start the CI pipeline, this will initiate: build > test > deploy

when the CI pipeline is complete we know that the build, tests and deploy successfully as expected. (DevOps outcome should be expected for every pipeline)

Note: there’s another step of release that “enable” the actual new code, and this is done via feature toggle.

Deployment to production

This is where the deployment strategy comes, obviously force-deploy will shutdown the services and start the services again (downtime)

we’ll need a better approach like blue-green that will create another group of resources with the new code and only after its active the current requests/traffic will be redirected to the new group of services.

after the deploy is OK the old resources can be deleted.

Debugging

To verify that the deploy is OK

  • check your metrics and logs
  • check that the service is operational (can be done via automated QA)

Summary

Do not be tempted to deploy from local laptop as this will cause issues and it will not be registered in logs or as a commit.

Use CI pipeline!

6 Rules For Cloud Architect

Cloud Security Continuous Delivery Deployments DevOps system architect

Are you a cloud architect? How do you plan a new infrastructure for a product?

How do you build workflow in the cloud?

What are the considerations of cloud security, costs and automation?

All are relevant questions when planning a new cloud design for a product runtime, so let’s discuss it.
Also, this is my approach and it serve me well in all of my designs and cloud operations in production.

Preplan

Preplan is not part of the 6 rules and just a starting point.

It’s better to plan before starting to build any project, in order to plan you’ll need to understand the product, ask these questions:

  • What problem is the product solves?
  • Who’s going to use it? (demographics)
  • What are the business risks of downtime?
  • What is the expected or current revenue?
  • What is the technical flow of the product? (user login, integrate with API, consume data from database, etc..)

The more you ask the more information you’ll have in the design process, so don’t skip this step.
It’s easy to just go into building stuff and not ask for need and requirements.

Costs

If your design will cost more than the revenue the product won’t justify itself, this is very important as a bad design in costs perspective can have a significant affect on the entire business operations.

So, in every step of the planning consider costs!

Cloud Security

In every product there’s a risk factor in term of business risks, what if the application is down for 1-hour? what is the affect in term of reputation and revenue?

What if some services and data are exposed to unauthorized parties?

So, ensure to include security measures in the design to make sure your product is protected, but don’t overdue it as it can cause issues with workflow and runtime.

Balance is key here.

Automation

Building and working without automation means spending time on repetitive tasks, this is not efficient and will cause slow delivery.

Try using IaC (Infrastructure as code) approach, this means you can deploy and modify entire infrastructure in minutes.

Also, you can find out the current stack components by checking the IaC files.

Combine IaC and Immutable infrastructure to get maximum results.

Decouple Dependencies

When building software and infrastructure it’s easy to tie components and hard-code stuff, the more hard-coded and dependency there is between different components the more issues it will cause.

Let’s say that you designed the infrastructure with hard-coded IP addresses, this means those IP cannot change, the same for other config files.

Another example is start-up of a service that is deepened on other services, for example application that’s require the monitoring agent to start, monitoring is nice but should not affect production services.

Continuous Software Updates

Software freeze is a risk in my opinion, this approach will lead to more work that needs planning and the longest the freeze the hardest it is to upgrade.

Let’s say you’re using Python3.6 and are using pip packages in your code, this means you cannot upgrade your OS because new OS comes with latest Python version and that python version uses the latest pip version.

So now you can’t upgrade your Python, pip or OS, just because you did not integrated updates in the regular operations.

Keep your code and system up to date!

Remove Single Point Of Failure

Similar to couple dependencies that can cause issues, relying on a single endpoint or component is risky, let’s say you’re using one load-balancer, what happened if that load-balancer is overloaded?

Single database? the same issue

Those are simple examples but in your product there are probably more components that are defined as single point of failure.

The less single point of failure the better!

How software update freeze can make your stack obsolete

Continuous Delivery continuous update DevOps system architect

Do you update your software frequently?

Is software update part of your CI/CD pipelines?

What is continuous update?

The issue with hard-coding software versions and not updating

Just to clarify this post is relevant to 3rd-party software and packages you import to your application (via apt, yum, pip, gem etc.. or downloading binary .jar etc..) also for OS versions

The wrong approach in my opinion is to statically add version numbers to imported packages and use the same OS version throughout your infrastructure and code

Why you ask?

Once your code work with other software (3rd-party) and tests are ok, you assume that the process is complete and resume working on your code

everything works until it doesn’t !

Scenario 1

let’s say you’re using java and a vulnerability is discovered and fixed with a new release, now you need to upgrade to new release but your runtime version is too far behind the latest version and cannot be upgraded

or better yet, you can upgrade but other components that communicate with your code is not compatible with the latest version

what do you do? oh yes, revert !

Scenario 2

the operating system is a few versions behind the latest, let’s say Ubuntu 18.04 and now you want to use Ubuntu 22.04 and your code is python 3.6

guess what Ubuntu 22.04 does not ship with the same python as Ubuntu 18.04

now you need to compile python 3.6 from source and install it to Ubuntu 22.04 and make sure to update the PATH to use python 3.6

Backlog

so now you decided to use python 10 instead of python 3.6 but what about pip packages? they are probably not compatible, why? because pip use the python version too

now go over your entire code and make sure every function works with new python version, then you’ll probably decide it’s too much work right now and not to upgrade

Solution

Simple, don’t freeze software updates!

if you keep your software up to date (including your OS) it forces you to adapt as you go! no need to upgrade or schedule upgrades because it’s a mindset, your software is evolving

stopping your software from evolving does not make sense, in fact it’s the opposite from what your job description is… developer

How to keep your software with latest version?

The answer is CD (continuous delivery)

CD means how fast, reliable and how frequent you deploy your code to production

So the goal is to deploy to production whenever you want and a few times per day, if you do that you know your code is in a releasable state

So using the latest release software while keeping a releasable state will make your job easier and your product better

Deploy vs. Release

Continuous Delivery Deployments DevOps

What is the difference between deploy and release?

when you’re working on servers and operating a runtime of application you need to make sure the deliver of new software to the users

doing so there are two option

  1. just deploy the new software to the servers
  2. deploy the new software but don’t make it available yet

Deploy

Deploy means you put the new software on the servers

Release

Release means that new software is now available to users

Two step process

making the deploy and release a two step process will ensure a smooth integration of new software into production servers, yes you can still can deploy after testing and make the two step process a one step process

How can you deploy and not make the new software operate instead of the current software?

using feature flags

with feature flags you can deploy the new software but not make it available yet, after you made a decision on when to activate the new software you can make the feature flag active and expose the new software to users (release)

a feature flag is just an if else statement on the new function of the new software

Continuous delivery in a bite

CI/CD Continuous Delivery DevOps

Continuous Delivery AKA CD is a set of practices that aim to build > automate > release software cycles.

CD is about delivering software to the end-user as fast as you can it’s a challenge due to best practices and other software concerns.

Benefits of CD

Let’s go over a few benefits you can achieve in your CD process/workflow.

Faster time to deliver software

Once your development team agrees on a CD “plan” something that works for your product and company and you have “field-tested” it, you’ll have faster build and release cycles.

Better teamwork in the development

Again let’s use the word “agreed” which means your development team has a plan and a system to accomplish a version release cycle, your development team will work better towards that goal of developing and releasing new software, why? fewer arguments, people know the workflow, and stuff gets done.

Lower costs (in theory)

Let’s assume you have the best optimal workflow for your development team, now that your development cycles are working fairly faster the costs may be lower, less compute resources, fewer meetings, and more efficiency.

Faster reach to market

So this is more of a business aspect but is depended on product development, if your product development is fast meaning you can release versions of your product (in this case software) faster then you can deliver faster features and products thus giving you the advantage over competitors that have a slower rate of product release.

How to automate jobs using Jenkins agent

CI/CD Continuous Delivery DevOps

Do you maintain Jenkins agents for your jobs? does your Jenkins agents running while in idle?

What if you could start a Jenkins agent for every job you run at Jenkins, and the best part no need to maintain and keep a Jenkins agent like a long-running server.

So how do you automate your jobs?

  • Start with installing the EC2 and Cloud plugins in your Jenkins server.
  • then configure Jenkins to run jobs only at agents and not on the master.
  • start a new job and a new instance will start (I’m using AWS so an instance will be initialized on that EC2 cloud)
  • wait for about 5-10 (it varies) until the agent is connected and watch the job run at that specific agent/node.

also you can configure the Cloud plugin to terminate the Jenkins agent once the agent is in idle for “x” amount of time, depends how many jobs you run every hour.

The outcome

If you use this approach you can save on cloud costs and automate jobs

P.S. you can configure spot instances to run as Jenkins agents.

Do you want to automate your Jenkins agents?

I can automate your Jenkins agents for you, contact me today HERE