Posts by SpinningOps Editor

Build, Test, Deploy. the hello world of CI/CD

When you start at programming the traditional first program you learn to write is: hello_world() and it became quite common on every programming language you choose to learn.

The hello world of CI/CD

In CI/CD you probably noticed the term build > test > deploy > repeat and this is the hello world in DevOps.

But what does it means?

so let’s assume your code is in Python, then there’s nothing to build unlike Go (golang) so why do you need build stage?

Let’s start from end to start

let’s assume your app is already deployed and running at your cloud of choice, how do you update your app to a newer code?

  • first build your code, why build? because it’s easy and convenient to use and deploy on different environments and can be reused or rollback if needed.
  • second test your code, why test? because you want to ensure your app runs without problems or bugs (making sure code does what it needs to do)
  • third deploy your new code (new version), so the users that use your app can have the new features or the newer bugs (that’s a joke) available for use.

How do you write your first CI/CD hello world?

  1. write code in your prefer language of choice
  2. build it via docker, and have a docker image from it (use tag for versions)
  3. run tests on the functions of the code using the built image (from step 2)
  4. deploy the new image after a successful test via docker with new image tag

so have fun with CI/CD and solve this problem for your company

How many stages do you need for a Jenkinsfile

You need to build a new Jenkinsfile / pipeline to build and test an application or a feature, so how many stages / steps do you use?

Also do you need a few stages or one stage is enough?

Here is where your approach comes to play, there are two main approaches I noticed in workflows from people

1st option: one stage to build and deploy

let’s assume you need to build an artifact from your code so one stage can be enough, something like: build your artifact > deploy (upload to your repository)

this approach will complete a couple of goals

  1. your code is built automatically via scripted job and if the build passes you know part of the code is OK, will it deployed successfully on your server? maybe, so you may want to add install stage before deploying.
  2. once your code built now it’s the easy part to upload the artifact to your repository or registry if it’s a Docker image.
    so it’s easy to understand and because it only two stages it’s faster to complete but you might want more options to integrate in your build.

2nd option: many stages to build, test and deploy

this approach will probably cover most of your software needs

  1. you build your code and once it’s done you start the second stage and start testing, now you don’t need a test to build because that already being done so you can add the installing stage and just deploy your artifact on a docker machine to verify it can be installed as predicted.
  2. once your artifact is installed you can continue to the next stage and add more test in a separate stage, here is where you might want to add commands to check some functionality that your code needs to do.
  3. after the first phase of your tests is completed you can add another stage to deploy your artifact after install to a light environment like Docker or a testing server (Jenkins agent) and make an API call to test some functions to verify that your code is working as expected.
  4. Now it’s where the parts come to play together, add another stage of tests to make sure your code can integrate with 3rd parties like Email, Slack or external API

once your tests are OK it’s deploy time

Time to deploy your code / artifact to your production server, it is recommended that you deploy it first to a staging server, just to make sure you didn’t miss anything, staging will run with your new code just a few minutes before deploying to production.

once everything is OK with your staging deploy to prod, a few options here:

deploy your code instead of the current version, until your deployment will complete your service will probably be unavailable temporary.

add another server to your servers group with the new version and once is OK remove the old version from the old server.

if your app have considerable traffic you might want check deployment tools like Spinnaker to deploy a canary approach of your app.

P.S.

use your approach based on your app needs and traffic

Continuous delivery in a bite

Continuous Delivery AKA CD is a set of practices that aim to build > automate > release software cycles.

CD is about delivering software to the end-user as fast as you can it’s a challenge due to best practices and other software concerns.

Benefits of CD

Let’s go over a few benefits you can achieve in your CD process/workflow.

Faster time to deliver software

Once your development team agrees on a CD “plan” something that works for your product and company and you have “field-tested” it, you’ll have faster build and release cycles.

Better teamwork in the development

Again let’s use the word “agreed” which means your development team has a plan and a system to accomplish a version release cycle, your development team will work better towards that goal of developing and releasing new software, why? fewer arguments, people know the workflow, and stuff gets done.

Lower costs (in theory)

Let’s assume you have the best optimal workflow for your development team, now that your development cycles are working fairly faster the costs may be lower, less compute resources, fewer meetings, and more efficiency.

Faster reach to market

So this is more of a business aspect but is depended on product development, if your product development is fast meaning you can release versions of your product (in this case software) faster then you can deliver faster features and products thus giving you the advantage over competitors that have a slower rate of product release.

Immutable servers do not accept changes

Do you have long-running servers? yes, keep reading.

What is a long-running server?

A long-running server is a compute resource that is hardcoded to perform a certain task, the caveat is that this resource can be changed over time and re-configured the state of the runtime.

Why do you care about changing the runtime state?

A few issues may pop when using mutable servers:

  • who knows the current state of the servers?
  • how do you know if things are still working as intended?
  • how do you know who logs in?
  • how do you keep track of who did things on the servers?
  • how do you keep the consistency of your servers?

Wait! don’t answer those questions because you will probably have a sophisticated answer unless you have a system that is working for you and is proven to manage a large number of servers.

Enter Immutable servers (compute resources)

Now that we have established the issues with long-running servers let’s talk about Immutable servers AKA immutable compute resources. the core concept is that immutable servers don’t accept changes thus making the state of the current runtime precise to what you wanted to be deployed.

by using this method of immutable servers you can expect the same result of operation across your servers or compute resources.

How to automate jobs using Jenkins agent

Do you maintain Jenkins agents for your jobs? does your Jenkins agents running while in idle?

What if you could start a Jenkins agent for every job you run at Jenkins, and the best part no need to maintain and keep a Jenkins agent like a long-running server.

So how do you automate your jobs?

  • Start with installing the EC2 and Cloud plugins in your Jenkins server.
  • then configure Jenkins to run jobs only at agents and not on the master.
  • start a new job and a new instance will start (I’m using AWS so an instance will be initialized on that EC2 cloud)
  • wait for about 5-10 (it varies) until the agent is connected and watch the job run at that specific agent/node.

also you can configure the Cloud plugin to terminate the Jenkins agent once the agent is in idle for “x” amount of time, depends how many jobs you run every hour.

The outcome

If you use this approach you can save on cloud costs and automate jobs

P.S. you can configure spot instances to run as Jenkins agents.

Do you want to automate your Jenkins agents?

I can automate your Jenkins agents for you, contact me today HERE

How to deploy S3 hosted website

When you have a website that is hosted at S3 updating new version of this website can interrupt with your regular tasks so how can you automate this repeating upload of new website version?

Jenkins as your automation

Jenkins can do many things including automation for this specific task of deploying new version of your website, thus making the operation of new version upload automatically.

how to automate your new version deployment?

  1. Create a git repository for your website
  2. Create a new pipeline in Jenkins using the Jenkinsfile here
  3. Start the job or commit a new version to the website repository

P.S. don’t forget to check the option in the job:

GitHub hook trigger for GITScm polling

Example of deployed website

Check out this example of a website that is automated via Jenkins here

Summary

Now you can add more tasks to your automation and create new automation jobs

What is your Deployment Strategy

As a DevOps and a cloud administrator you need to deploy apps to your production cluster.

How do you deploy your apps?

There are many options and there is no perfect one because every app and every company has a different approach so in this blog g post we’ll talk about a few so you can choose the deployment strategy that is suitable for you.

The purpose of the deployment is to deploy a new version of your app and images.

Rolling Red / Black or Green / Blue

In this approach, we will be Rolling the new version per instance or per group of instances.

Canary Analysis

In this approach we will do the following steps:

  • Replace a small number of instances
  • A certain percent of traffic is sent to the new version
  • Meant to test a stable version

Red / Black or Green / Blue

In this approach we will do the following:

  • Starts a duplicate server group
  • When the new group is OK traffic is sent to this new group

Choosing your deployment strategy can help you with deploying every new version of your software.

How to install Sensu

Monitor your servers is highly important to ensure that the service you offer is accessible to customers, so you need a monitoring tool, but which one you choose? in this blog post, we’ll talk about Sensu.

What is Sensu?

Sensu is a monitoring tool and observability pipeline that enable Monitoring-as-Code

What does Monitoring-as-Code mean?

With Monitoring-as-Code you can manage your monitoring the same way you manage your code, for example, keep versions of monitoring configuration pipelines and commit them to a code repository, etc.

What does observability mean in this context?

Sensu handles the discovery of new compute instances and containers like Docker and starts monitoring them.

How to install Sensu?

Let’s install Sensu using Docker

sudo docker network create sensu
sudo docker volume create sensu-backend-data

Start the Sensu backend

sudo docker run -v sensu-backend-data:/var/lib/sensu \
-d --name sensu-backend \
--rm --network sensu -p 3000:3000 -p 8080:8080 -p 8081:8081 \
-e SENSU_BACKEND_CLUSTER_ADMIN_USERNAME=admin \
-e SENSU_BACKEND_CLUSTER_ADMIN_PASSWORD=admin \
sensu/sensu:latest \
sensu-backend start --state-dir /var/lib/sensu/sensu-backend --log-level debug

How to install Sensu Agent?

Again let’s use Docker

sudo docker run -d --rm --network sensu -p :3030 \
  sensu/sensu sensu-agent start \
  --backend-url ws://sensu-backend:8081 --deregister \
  --keepalive-interval=5 --keepalive-warning-timeout=10 --subscriptions linux

After the install of the Sensu agent, the agent registers itself to the Sensu platform and subscribes to the configured topic, and starts collecting metrics.

In this case the metric is keepalive

How to log in to the Sensu dashboard?

Browse to http://127.0.0.1:3000/dashboard

Sensu Subscriptions

  • Sensu backends “publish” service-based monitoring topics
  • Sensu agents “subscribe” to service-based monitoring topics
  • All monitoring configurations are managed via the control plane (Sensu backend REST API)
  • Sensu agents can be deployed at scale with minimal configuration

For more information go to Sensu website and Sensu Docs

ClusterIP vs. NodePort

What are ClusterIP and NodePort? (in K8S)

First, let’s describe a service, A service is a way to expose an application running on a pod or a set of pods.

so to connect to that application we need a service.

What types of services can you use?

The “types” of services can be:

  • ClusterIP
  • NodePort
  • LoadBalancer
  • ExternalName

let’s focus on ClusterIP and NodePort for this blog post.

What is ClusterIP?

ClusterIP exposes the service for cluster internal communication, only within the Kubernetes cluster.

What is NodePort?

NodePort exposes the service for the cluster external communication.

to access the application you’ll need the IP of the node and the port.

<NodeIP>:<NodePort>

What is a cloud administrator

these days it’s very easy to confuse job description especially when you just starting learning about an occupation that you’re interested in.

adding to the problem it’s very interesting how every company has different requirements for the “same” role, so now it’s more confusing.

So what a cloud administrator does?

take the following explanation with a pinch of salt, why? because each company wants different things and have different approach so don’t get into specific topics and technologies.

A Cloud Administrator is responsible for computing resources and SAAS accounts the company is using.

Example of Cloud Administrator Responsibilities

  • Managing cloud accounts, AWS, GCP, Azure, or any other cloud provider the company is using.
  • Managing online SAAS accounts, G Suite, Office365, Slack, and any other software provider online.
  • that description includes backup, working on different OS like Linux and Windows servers, and even some coding.

P.S.

A Cloud is just someone else’s computer so if you’re a system administrator getting into cloud should be easy for you.