Kubectl

Create docker credentials as K8s secret

 kubectl create secret docker-registry secretname --docker-server=my.docker-registry.com --docker-username='my-username' --docker-password='mypassword' --docker-email='myemail@domain.com' -n mynamespace

Scale a deployment

kubectl -n my-ns scale --replicas 0 deployment my-deployment

kubectl -n my-ns scale --replicas 1 deployment my-deployment

Dependency check pom plugin

A mvn plugin to generate a report on dependencies:

mvn   site:stage  -DskipTests=true

https://maven.apache.org/plugins/maven-site-plugin/

Pom.xml

<distributionManagement>

<site>

      <id>mojo.website</id>

      <name>Mojo Website</name>

      <url>scp://localhost/scratch/kpokkana/workdir/simple-http-server-py/index-root/kpokkana/target2/</url>

    </site>

</distributionManagement>

<build>

        <plugins>

        <plugin>

        <groupId>org.owasp</groupId>

        <artifactId>dependency-check-maven</artifactId>

        <version>6.0.2</version>

        <executions>

        <execution>

        <goals>

        <goal>check</goal>

        </goals>

        </execution>

        </executions>

        </plugin>

<plugin>

    <groupId>org.apache.maven.plugins</groupId>

    <artifactId>maven-site-plugin</artifactId>

    <version>3.7.1</version>

 </plugin>

        </plugins>

        </build>

Docker

Basic commands

$ docker login -u username -p mypassword this-is-my-docker-registry.com # Log in to a docker repo

$ docker images # list images

$ docker ps # list all running docker containers

$ docker ps -a # list all containers, including stopped.

$ docker rmi image-name #remove an image

$ docker pull image:tag #pulls an image 

$ docker build -t image:tag --build-arg arg1=value --build-arg arg2=value -f myDockerFile.alpine context-dir/ #build image from DockerFile

$ docker save -o image.tar image:tag  # save the image as a tar file




$ docker load -i /image-tar-file.tar.gz # loads the image from tar file

$ docker ps --size # get size of container

$ docker system prune $ prune dangling images/containers

$ docker image prune -a $ pruner all dangling images

$ docker system df #The docker system df command displays information regarding the amount of disk space used by the docker daemon.

Change docker root dir

vi /etc/docker/daemon.json

From <https://stackoverflow.com/questions/32070113/how-do-i-change-the-default-docker-container-location/50726177> 

 

{

  "data-root":"/scratch1/docker"

}

Kafka: listing consumer groups

List  the consumer groups:

#kubectl -n kafka exec -ti my-cluster-kafka-0 -- bin/kafka-consumer-groups.sh --bootstrap-server my-cluster-kafka-bootstrap.kafka:9092 --list

Delete a consumer group:

Bring down dispatcher and controller, if any.

#kubectl -n knative-eventing scale deployment kafka-ch-dispatcher --replicas=0

#kubectl -n knative-eventing scale deployment kafka-ch-controller --replicas=0

Now remove the group by name:

 kubectl -n kafka exec -ti my-cluster-kafka-0 -- bin/kafka-consumer-groups.sh --bootstrap-server my-cluster-kafka-bootstrap.kafka:9092 --delete --group kafka.dx-test.default-kne-trigger.05719b03-3117-4633-b504-dc4020423ce0

Delete all consumer groups:

kubectl -n kafka exec -ti my-cluster-kafka-0 -- bin/kafka-consumer-groups.sh --bootstrap-server my-cluster-kafka-bootstrap.kafka:9092 --delete --all-groups

Describe a consumer group:

#kubectl -n kafka exec -ti my-cluster-kafka-0 -- bin/kafka-consumer-groups.sh --bootstrap-server my-cluster-kafka-bootstrap.kafka:9092 --describe --group kafka.dx-test.default-kne-trigger.3a796074-afb6-4862-a5f3-0db6c76b41d0

#xargs -0 -n 1 kubectl -n kafka exec -ti my-cluster-kafka-0 -- bin/kafka-consumer-groups.sh --bootstrap-server my-cluster-kafka-bootstrap.kafka:9092 --describe --group < <(tr \\n \\0 <consumer-grps-dangling-2.txt)

Find the ISTIO Ingress port and Gateway URL

export INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].nodePort}')

export INGRESS_HOST=$(kubectl get po -l istio=ingressgateway -n istio-system -o jsonpath='{.items[0].status.hostIP}')

export GATEWAY_URL=$INGRESS_HOST:$INGRESS_PORT

echo $GATEWAY_URL

HTTP proxy

In our work place, the internet traffic is accessible only via the org's HTTP proxy.

To set the proxy env var in a Linux VM use the eg:

export http_proxy=http://www-our-proxy.company.com:80

export HTTPS_PROXY=$http_proxy

export HTTP_PROXY=$http_proxy

export https_proxy=$http_proxy

export NO_PROXY=.us.domain1.com,.companyvcn.com,localhost,.companycorp.com

Here it is an HTTP proxy. So to establish an HTTPS connection also its using the HTTP proxy here.

Whenever an HTTPS connection is requested to the HTTP proxy by the user, the proxy creates a TCP tunnel between the destination and the user host. And thus the user make the HTTPS calls via this TCP tunnel.

Gradle

Gradle commands

you could specify a single test with: 

gradle clean test --tests "org.gradle.MyFirstTest.testA"

Or all tests in a class: 

gradle clean test --tests "org.gradle.MyFirstTest" 

Or all tests in a package: 

gradle clean test --tests "org.gradle.*"

Golang dep

sample commands

install : #curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh
#dep init
#dep ensure -add github.com/foo/bar github.com/baz/quux
#dep status
#dep ensure
#dep ensure -update github.com/foo/bar
#dep check

Markdown syntax

Text   

It's very easy to make some words **bold** and other words *italic* with Markdown. You can even [link to Google!](http://google.com)

It's very easy to make some words bold and other words italic with Markdown. You can even link to Google!

Syntax guide

Here’s an overview of Markdown syntax.

Headers

# This is an <h1> tag
## This is an <h2> tag
###### This is an <h6> tag

Emphasis

*This text will be italic*
_This will also be italic_

**This text will be bold**
__This will also be bold__

_You **can** combine them_

Lists

Unordered

* Item 1
* Item 2
  * Item 2a
  * Item 2b

Ordered

1. Item 1
1. Item 2
1. Item 3
   1. Item 3a
   1. Item 3b

Images

![GitHub Logo](/images/logo.png)
Format: ![Alt Text](url)

Links

http://github.com - automatic!
[GitHub](http://github.com)

Blockquotes

As Kanye West said:

> We're living the future so
> the present is our past.

Inline code

I think you should use an
`<addr>` element here instead.

GitHub Flavored Markdown

GitHub.com uses its own version of the Markdown syntax that provides an additional set of useful features, many of which make it easier to work with content on GitHub.com.

Note that some features of GitHub Flavored Markdown are only available in the descriptions and comments of Issues and Pull Requests. These include @mentions as well as references to SHA-1 hashes, Issues, and Pull Requests. Task Lists are also available in Gist comments and in Gist Markdown files.

Syntax highlighting

Here’s an example of how you can use syntax highlighting with GitHub Flavored Markdown:

```javascript
function fancyAlert(arg) {
  if(arg) {
    $.facebox({div:'#foo'})
  }
}
```

You can also simply indent your code by four spaces:

    function fancyAlert(arg) {
      if(arg) {
        $.facebox({div:'#foo'})
      }
    }

Here’s an example of Python code without syntax highlighting:

def foo():
    if not bar:
        return True

Task Lists

- [x] @mentions, #refs, [links](), **formatting**, and <del>tags</del> supported
- [x] list syntax required (any unordered or ordered list supported)
- [x] this is a complete item
- [ ] this is an incomplete item

If you include a task list in the first comment of an Issue, you will get a handy progress indicator in your issue list. It also works in Pull Requests!

Tables

You can create tables by assembling a list of words and dividing them with hyphens - (for the first row), and then separating each column with a pipe |:

First Header | Second Header
------------ | -------------
Content from cell 1 | Content from cell 2
Content in the first column | Content in the second column

Would become:

First Header	Second Header
Content from cell 1	Content from cell 2
Content in the first column	Content in the second column

Declarative Jenkinsfile

Some snippets:

steps:

------

pipeline {
    agent any
    stages {
        stage('Build') {
            steps {
                sh 'echo "Hello World"'
                sh '''
                    echo "Multiline shell steps"
                    ls –lah
                '''
            }
        }
    }
}

timeouts:

----------

pipeline {
    agent any
    stages {
        stage('Deploy') {
            steps {
                timeout(time: 3, unit: 'MINUTES') {
                    retry(5) {
                        powershell '.\flakey-deploy.ps1'
                    }
                }
            }
        }
    }
}

stages:

-------
pipeline {
    agent any
    stages {
        stage('Build') {
            steps {
                echo 'Building'
            }
        }
        stage('Test') {
            steps {
                echo 'Testing'
            }
        }
        stage('Deploy') {
            steps {
                echo 'Deploying'
            }
        }
    }
}

when

-----
pipeline {
    agent any
        stages {
            stage('Build') {
                when {
                    expression {
                        "foo" == "bar"
                    }
                }
                steps {
                    echo 'Building'
                }
            }
        stage('Test') {
            when {
                environment name: 'JOB_NAME', value: 'foo'
            }
            steps {
                echo 'Testing'
            }
        }
        stage('Deploy') {
            when {
                branch 'master'
            }
            steps {
                echo 'Deploying'
            }
        }
    }
}

when allOf:

----------
pipeline {
    agent any
    stages {
        stage('Build') {
            when {
                allOf {
                    not { branch 'master' }
                    environment name: 'JOB_NAME', value: 'Foo'
                }
            }
            steps {
                echo 'Building'
            }
        }
    }
}

parallel stages:

------------------
pipeline {
    agent any
    stages {
        stage('Browser Tests') {
            parallel {
                stage('Chrome') {
                    steps {
                        echo "Chrome Tests"
                    }
                }
                stage('Firefox') {
                    steps {
                        echo "Firefox Tests"
                    }
                }
            }
        }
    }
}

Docker

--------
pipeline {
    agent {
        docker {
            label 'docker'
            image 'maven:3.5.0-jdk-8'
        }
    }
}

One of the advantages of using containers is creating an immutable environment that defines only the tools required in a consistent manner.
Rather than creating one large image with every tool needed by the Pipeline, it is possible to use different containers in each stage and
reuse the workspace, keeping all of your files in one place.

pipeline {
    agent {
        node { label 'my-docker' }
    }
    stages {
        stage("Build") {
            agent {
                docker {
                reuseNode true
                image 'maven:3.5.0-jdk-8'
                }
            }
            steps {
                sh 'mvn install'
            }
        }
    }
}

Envs:

-----
pipeline {
    agent any
    environment {
        DISABLE_AUTH = 'true'
        DB_ENGINE        = 'sqlite'
    }
    stages {
        stage('Build') {
            steps {
                sh 'printenv’
            }
        }
    }
}

Credentials envs:

------------------
environment {
    AWS_ACCESS = credentials('AWS_ KEY')
    AWS_SECRET = credentials('AWS_SECRET')
}

post-actions:

------------
pipeline {
    agent any
    stages {
        stage('No-op') {
            steps {
                sh 'ls'
            }
        }
    }
    post {
        always {
            echo 'I have finished'
            deleteDir() // clean up workspace
        }
        success {
            echo 'I succeeded!'
        }
        unstable {
            echo 'I am unstable :/'
        }
        failure {
            echo 'I failed :('
        }
        changed {
            echo 'Things are different...'
        }
    }
}

Notify post actions:

-----------------
post {
    failure {
        mail to: 'team@example.com',
            subject: 'Failed Pipeline',
            body: "Something is wrong"
    }
}

slack:

------
post {
    success {
        slackSend channel:'#ops-room',
            color: 'good',
            message: 'Completed successfully.'
    }
}

Python

setups and commands

pipenv

pip install --user pipenv
python -m site --user-base //add 'bin' to that and append to PATH.
pipenv install <module-name>
pipenv shell

pip

pip freeze
pip list

venv

python -mvenv  myvenv

source myvenv/bin/activate  # Activate the venv

deactivate # Command to deactivate the venv

Vim

To comment out blocks in vim:

Press Esc (to leave editing or other modes)
hit ctrl + v (visual block mode)
use the up/down arrow keys to select lines you want (it won't highlight everything)
Shift + i (capital I)
insert the text you want, i.e. %
press Esc Esc.

Devops tools and useful stacks

Here is a list of tools and stacks that will be useful in DevOps project.

terraform

https://www.terraform.io/ 
Infrastructure defined as text files. Product by HashiCorp.
Terraform is an open-source infrastructure as code software tool created by HashiCorp. It enables users to define and provision a data center infrastructure using a high-level configuration language known as Hashicorp Configuration Language, or optionally JSON.

sonarqube

https://www.sonarqube.org/
Continuous code quality checks tool.

kibana

https://www.elastic.co/products/kibana
A data visualization plugin for Elasticsearch. It provides visualization capabilities on top of the content indexed on an Elasticsearch cluster.

mesos

http://mesos.apache.org
Distributed OS. Uses for orchestrating clusters.

zipkin

https://zipkin.io/
Zipkin is a distributed tracing system

helm

https://helm.sh/
Helm helps you manage Kubernetes applications — Helm Charts helps you define, install, and upgrade even the most complex Kubernetes application.

quarkus

https://quarkus.io/
A Kubernetes Native Java stack tailored for GraalVM & OpenJDK HotSpot.

gradle

https://gradle.org/
Gradle is an open-source build automation system that builds upon the concepts of Apache Ant and Apache Maven and introduces a Groovy-based domain-specific language instead of the XML form used by Apache Maven for declaring the project configuration.

groovy

Apache Groovy is a Java-syntax-compatible object-oriented programming language for the Java platform. It is both a static and dynamic language with features similar to those of Python, Ruby, Perl, and Smalltalk.

jenkins

Jenkins is an free and open source automation server written in Java. Jenkins helps to automate the non-human part of the software development process, with continuous integration and facilitating technical aspects of continuous delivery.

gitlab CI

GitLab CI (Continuous Integration) service is a part of GitLab that build and test the software whenever developer pushes code to application.

prometheus

An open-source monitoring system with a dimensional data model, flexible query language, efficient time series database and modern alerting approach.

ansible 

https://www.ansible.com/

Ansible is the simplest way to automate apps and IT infrastructure. Application Deployment + Configuration Management + Continuous Delivery. Ansible is an open-source software provisioning, configuration management, and application-deployment tool. It runs on many Unix-like systems, and can configure both Unix-like systems as well as Microsoft Windows. It includes its own declarative language to describe system configuration.

kubernetes

Container orchestration tool.

docker

Enterprise container platform. Docker is a set of platform-as-a-service products that use operating-system-level virtualization to deliver software in packages called containers

NiFi

Designed to automate the flow of data between software systems.

docker-compose

Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application's services.

cypress.io

E2E testing framework. Fast, easy and reliable testing for anything that runs in a browser.

salt

https://www.saltstack.com/
Salt is Python-based, open-source software for event-driven IT automation, remote task execution, and configuration management.

vault

https://www.vaultproject.io/
Secure, store and tightly control access to tokens, passwords, certificates, encryption keys for protecting secrets and other sensitive data using a UI, CLI, or HTTP API.

Bazel

https://bazel.build/

In software development, Bazel is a free software tool that allows for the automation of building and testing of software. The company Google uses the build tool Blaze internally and released and open-sourced part of the Blaze tool as Bazel, named as an anagram of Blaze.

Jaegar

open-source, end-to-end distributed tracing

Monitor and troubleshoot transactions in complex distributed systems

Dream keyboards

Few of best keyboards I found on google.

1.

Microsoft 5KV-00001 Sculpt Ergonomic Wireless Keyboard for Business

 

https://www.amazon.in/Microsoft-Ergonomic-Keyboard-Business-5KV-00001/dp/B00CYX26BC/ref=sr_1_2?tag=guru99-21&ie=UTF8&qid=1540935383&sr=8-2&keywords=sculpt+keyboard

2. Kinesis Advantage2 Keyboard KB600

https://www.amazon.in/Kinesis-Advantage2-Ergonomic-Keyboard-KB600/dp/B01KR1C5PY/ref=as_li_ss_tl?tag=guru99-21&ie=UTF8&qid=1479099603&sr=8-1-fkmr0&keywords=kinesis+advantage+2+keyboard&linkId=ce855438d358e648afa10f77b8f91195