Rohit's Blog's RSS Feed

Jira CI/CD Integration

Sun, 08 Aug 2021 05:00:00 GMT

Ever wanted to get build and deployment links directly in Jira? Now you can, just add this small step to the last stage of your pipeline and get all the relevant information right in the ticket details page.

Generate Credentials for OAuth App

Generate new OAuth Credentials and copy

See: https://support.atlassian.com/jira-cloud-administration/docs/integrate-with-self-hosted-tools-using-oauth/

Use with Any CI/CD Provider

As long as your CI/CD provider supports running Docker images you may use this integration. Tested in Drone.io, Gitlab CI, and Google Cloud Build.

Docker Images are available from:

Github Container Registry: ghcr.io/rohit-gohri/jira-ci-cd-integration
Gitlab Container Registry: registry.gitlab.com/rohit-gohri/jira-ci-cd-integration
Docker Hub: boringdownload/jira-integration

Pick whatever you want and is convenient for you.

Configuration is through Environment Variables. Read more in the repository Readme.

Jira Drone.io Example

Add secrets for JIRA_CLIENT_ID and JIRA_CLIENT_SECRET and then add this to your pipeline:

steps:
  - name: jira-integration
    image: boringdownload/jira-integration:v0
    environment:
      BUILD_NAME: drone-pipeline
      JIRA_INSTANCE: companyname
      JIRA_CLIENT_ID:
        from_secret: jira_client_id
      JIRA_CLIENT_SECRET:
        from_secret: jira_client_secret

Jira Gitlab CI/CD

Add a CI/CD Variable to your project for JIRA_CLIENT_ID and JIRA_CLIENT_SECRET (remember to mask them) and then add these steps to your pipeline (we use .post stage so it runs last)

jira-integration-on-success:
  stage: .post
  when: on_success
  image: registry.gitlab.com/rohit-gohri/jira-ci-cd-integration:v0
  script: jira-integration
  variables:
    BUILD_STATE: successful
    BUILD_NAME: gitlab-pipeline
    JIRA_INSTANCE: companyname

jira-integration-on-failure:
  extends: jira-integration-on-success
  when: on_failure
  variables:
    BUILD_STATE: failure

Usage With Github Actions

You can also use this as a Github Action if you want to send release information to Jira.

First add OAuth Creds as secrets to Github (See: https://docs.github.com/en/actions/reference/encrypted-secrets)

Add Client ID as JIRA_CLIENT_ID
Add Client Secret as JIRA_CLIENT_SECRET

Use in Builds Pipeline

- name: Jira Integration
  if: ${{ always() }}
  uses: rohit-gohri/jira-ci-cd-integration@v0
  with:
    event_type: build
    state: ${{ job.status }}
    jira_instance: companyname # Subdomain for Jira Cloud
    client_id: ${{ secrets.JIRA_CLIENT_ID }}
    client_secret: ${{ secrets.JIRA_CLIENT_SECRET }}

Use in Deployment Pipeline

- name: Jira Integration
  if: ${{ always() }}
  uses: rohit-gohri/jira-ci-cd-integration@v0
  with:
    event_type: deployment
    state: ${{ job.status }}
    issue: JCI-3, JCI-6 # Comma separated list of issues being deployed/released. You are expected to generate this yourself in a previous step
    jira_instance: companyname # Subdomain for Jira Cloud
    client_id: ${{ secrets.JIRA_CLIENT_ID }}
    client_secret: ${{ secrets.JIRA_CLIENT_SECRET }}

Result

That's all it takes. You'll get Builds information like you do for Branches and Pull Requests:

Image: continuous deployment by Shocho from the Noun Project

My Favourite Helper Types in Typescript

Wed, 27 Jan 2021 19:00:00 GMT

Over the past few years of using Typescript and slowly getting comfortable writing complex custom types, I have amassed a collection of small helper types from around the internet. Here are a few of them that I end up adding to almost all my major projects and how I use them:

Unpromise

Sometimes you need to get the resolved value of a Promise<T> wrapped type, mostly when using along with the inbuilt ReturnType type:

type Unpromise<T> = T extends Promise<infer U> ? U : T;

type a = Promise<number>;
type x = Unpromise<a>; // number

Use with ReturnType to make ReturnTypeResolved:

type ReturnTypeResolved<T extends (...args: any) => any> = Unpromise<
  ReturnType<T>
>;

function test(): Promise<{ weird: "value" }> {
  return Promise.resolve({ weird: "value" });
}

type WeirdType = ReturnTypeResolved<typeof test>;

PlainObject

I know we have the inbuilt Record but this is such a common and frequent use case that I like having this around:

type PlainObject<T = any> = Record<string, T>;

type NumMap = PlainObject<number>;
const a: NumMap = {
  key1: 1,
  key2: 2,
  keyx: "x", // ERROR: expected number
};

You can also create a reverse of this, ValueOf<T>:

type ValueOf<T> = T[keyof T];

type x = ValueOf<NumMap>; // number

RecursivePartial

Partial but for nested objects, most useful when declaring defaults or merging some overrides for a config type:

type RecursivePartial<T> = {
  [P in keyof T]?: T[P] extends (infer U)[]
    ? RecursivePartial<U>[]
    : T[P] extends object
    ? RecursivePartial<T[P]>
    : T[P];
};

const config = {
  a: {
    x: {
      p: 1,
      q: 2,
    },
    y: "nested",
    z: "value",
  },
  b: "root",
  c: "value",
};

const configOverride: RecursivePartial<typeof config> = {
  a: {
    x: {
      p: "wrong", // should be number
    },
  },
};

Also See

Go over the inbuilt utility types in the Typescript Docs. And checkout this great library, utility-types on Github for other common types people tend to use.

Credits

Links to the various places I originally found these from:

Unpromise: Sourced from this StackOverflow Answer: https://stackoverflow.com/a/53196435/9485498
RecursivePartial: Sourced from this StackOverflow Answer: https://stackoverflow.com/a/51365037/9485498

OpenAPI for Docusaurus! 📘

Tue, 19 Jan 2021 17:30:00 GMT

Introducing "Redocusaurus": Redoc for Docusaurus

Redocusaurus

What is it?

It is a preset for docusaurus, with 2 main components:

docusaurus-theme-redoc

Using Redoc as a React component, this provides 2 theme components, @theme/ApiDoc and @theme/Redoc. You can use them directly in js pages in Docusaurus.

docusaurus-plugin-redoc

This creates routes from your OpenAPI files or URLs and renders with the components provided by the theme.

How to set it up?

Install the preset
```
npm i --save redocusaurus
```

Add it to your docusaurus.config.js:

module.exports = {
 // ...
 presets: [
   // ...
   [
     'redocusaurus',
     {
       // Plugin Options for loading OpenAPI files
       specs: [
         {
           spec: 'https://redocly.github.io/redoc/openapi.yaml',
           route: '/api/',
         },
       ],
       // Theme Options for modifying how redoc renders them
       theme: {
         // Change with your site colors
         primaryColor: '#1890ff',
       },
     },
   ],
 ],
 // ...
};

Rebuild your site and you will have your OpenAPI docs available at the /api/ route.

Extra features over directly using Redoc

The theme includes redoc settings and some custom css to make it match the @docusaurus/classic theme. Along with support for dark mode.

Read more in the docs - https://redocusaurus.vercel.app/

API Icon by mikicon from the Noun Project

The Hidden Cost of AWS ELBs

Tue, 01 Sep 2020 13:00:00 GMT

This past month we had a really high AWS bill, and it was supposed to be high but not THIS high. The culprit - "Data Transfer Out". I'm sure I'm not the only one who has had to deal with this.

Some Context

We had a NodeJS service running on EC2 in an Auto Scaling Group (ASG) behind an Application Load Balancer (ALB) that was supposed to handle a huge volume of requests. We were expecting a consistent increase in traffic this month, so we had made estimates on how it would affect our costs. We had expected some high data transfer out of our Application Load Balancer (ALB) but this was a consistent 4-5x of our estimates. Time to investigate.

Analysing the Data

After filtering Cost Explorer data with tags, it was clear that the ALB was the source of these costs. I immediately went to CloudWatch and checked the "Processed Bytes" metric of the ALB.

Cloud Watch

Processed Bytes is supposed to be "the total number of bytes processed by the load balancer over IPv4 and IPv6 ... includes traffic to and from clients", so ideally Processed Bytes should be greater than the "Data Transfer Out". But it was not..?

Our stats showed Processed Bytes to be around what our estimates had predicted for this amount of traffic. But Cost Explorer was showing a number that was 4-5 times that. Time to contact support.

AWS Support

We raised a support request to AWS, but we didn't have high hopes because we were not on the premium support plan. But it was worth a shot.

ELB Access Logs

Next up, we enabled ALB Access Logs to an S3 bucket and let it generate some logs for some time. Upon checking the logs, we were certain that this wasn't a problem at the application level. The access logs showed response size consistent with what we had estimated, no inflation due to extra headers or something like that.

VPC Flow Logs

We wondered if there was some other service running that might be communicating over an Elastic IP or public IP instead of a private IP. To check that, we enabled VPC Flow logs to another S3 bucket and let it generate some logs for some time.

We didn't find any services that might be communicating over Elastic IP but we did find some services that could be moved to the same Availability Zone (AZ) and reduced some Regional Data Transfer costs.

Frantic Googling

After eliminating all possible sources of Data Transfer we still had no clue what was the source of the Cost other than that it was from the ALB. This led to frantic googling and sending/explaining the problem statement to various people.

Then over the weekend, trying to think of keywords that would not lead me down to more documentation, I finally found this StackOverflow answer:

Q. AWS ELB: How can I see how many bytes of outbound network traffic an SSL-enabled ELB generates

A. The handshake of a TLS connection is pretty much constant and do not depend on your application. So your function is really : number of connections * ( size of http responses + constant)

And it finally clicked, the missing piece of the puzzle. Since HTTPS was handled directly by the ALB, it wouldn't show in the Access Logs as the instance didn't have to deal with it.

Here is an article that estimates the payload to average ~6.5k (depending on your certificate size). Our response sizes were really small, and this could be a major part of the response size if the user was visiting the site for the first time. Since a lot of the traffic was new, it would be a significant chunk.

That led to even more Googling on how HTTPS might affect AWS ELB costs and we came upon this very good article that also suggested increasing the default timeout for idle connections to 10 minutes for the ALB along with changing the certificate.

Testing Different HTTPS Certs

We were currently using the default certificate generated by Amazon Certificate Manager (ACM) with the Load Balancer. We tested the size of the TLS handshake with Wormly Test SSL Tool and then replaced it with one generated with Let's Encrypt. To be thorough we also tested TLS handshake sizes of different websites with different Certificate Authorities.

Certificate Issuer	Certificate Type	SSL Handshake Size (Bytes)
ACM	Wildcard (RSA 2048 bits)	5971
Let's Encrypt	Single (RSA 2048 bits)	3753
Let's Encrypt	Wildcard (RSA 2048 bits)	3702
Let's Encrypt	Wildcard (ECC 256 bits)	3323
DigiCert	Single (ECC 256 bits)	3311
Sectigo	Wildcard (RSA 2048 bits)	6720

DigiCert was the best with respect to size, but Let's Encrypt was close enough (& free!). ECC certificates are not compatible with some older browsers, so we generated both the smaller ECC and RSA certificates through acme.sh and uploaded them through AWS CLI. The ALB will automatically select the best one depending on the client (read more here).

After adding them to the ALB listener and letting Cost Explorer catch up for a day, we saw a significant decrease in our Data Transfer Out Costs for the same number of requests.

It was still not equal to the estimates we had, but it will never be because we never considered the TLS handshake size in our estimates. AWS Costs really are a mystery until you actually get the bill.

CI/CD for your Resume!

Sat, 29 Aug 2020 18:30:00 GMT

After ending up with countless PDFs of my resume and never being sure of which one is the final version. I decided to start versioning it with the tool I already knew, Git. After finding a good enough LaTeX template, a git init, and some commits later, I had a good enough resume template with a working Github Action that would publish the generated PDF as a release artifact on every tag. I also added generated PDFs as build artifacts for every commit to preview changes without having access to a working LaTeX env, so that one may be able to make edits from their phone.

My Workflow

I simply edit the tex file using the Web Editor and commit my changes. The Github workflow will build and generate a pdf that I can preview using the build artifact.

Once I'm satisfied with the edits, I will create a new release, this will trigger the "Release" pipeline. It will build and publish the pdf as a release artifact.

The release allows me to always have access to the latest version of my resume at the /releases/latest/download/Preview.pdf path of my repository.

Download latest version

More than that, I can preview it directly in my browser using Google Docs by using https://docs.google.com/viewer?url=https://github.com/<username>/<repo>/releases/latest/download/Preview.pdf. I can create a short url of this that I can give to anyone knowing it'll always point to the latest resume without having to worry about updating it.

Preview latest version

Motivation

When I first realized I could basically code my resume, it finally stopped becoming something I dreaded doing and became something I actively wanted to improve. Adding CI/CD to it was the next natural step.

Link to Code

This is a Github Template Project with a LaTeX resume template and Github Workflows. You may use this template or add your own LaTeX files to build those.

resume-template

Additional Resources

My Resume that is using a similar workflow, but with some added integrations to sync my blog with the latest version of my resume:

resume

How to get ESlint and Prettier to play nice in VS Code?

Tue, 28 Apr 2020 18:30:00 GMT

Do you have your project setup with both Prettier and Eslint? Do you want them to run in a specific order instead of using ESLint to run prettier? Here's how to set your own order in VS Code with this extension:

Visual Studio Code Market Place: Format Code Action

This extension will register a new "codeAction" which will run 'Format Document' using the default formatter (prettier in our case) in VS Code.

source.formatDocument

Using this and the editor.codeActionsOnSave setting in VS Code you can specify the order in which to run the eslint codeAction, source.fixAll.eslint, and format document.

Install

Requires: VS Code 1.44+

Install through VS Code extensions:

Visual Studio Code Market Place: Format Code Action

Can also be installed in VS Code: Launch VS Code Quick Open (Ctrl+P), paste the following command, and press enter.

ext install rohit-gohri.format-code-action

Usage

You will need both the prettier-vscode and vscode-eslint extensions installed for this config to work. Disbale formatOnSave and use the source.formatDocument codeAction in whatever order you want with VS Code settings:

  "editor.formatOnSave": false,
  "editor.defaultFormatter": "esbenp.prettier-vscode",
  "editor.codeActionsOnSave": [
    "source.formatDocument",
    "source.fixAll.eslint"
  ]

This runs 'Format Document' with the default formatter (in this case prettier) and then eslint --fix for all document types.

Or for a specific lanuage only

  "editor.formatOnSave": true,
  "editor.defaultFormatter": "esbenp.prettier-vscode",
  "[javascript]": {
    "editor.formatOnSave": false,
    "editor.codeActionsOnSave": [
        "source.formatDocument",
        "source.fixAll.eslint"
    ]
  },

This would run prettier by default, but for javascript files would run prettier and then eslint. If you want to reverse the order then just reverse the array.

Motivation

I wanted to use prettier and eslint together, to override some of prettier's mpre opinionated style rules with a eslint config (particularly brace-style).

Instead of using workarounds like prettier-eslint which don't work for all eslint configurations as it hasn't been updated to work with eslint v6 properly. We can simply run prettier and eslint one by one like one would do on the command line. Here is how that looks like in my package.json:

  "scripts": {
    "format": "npm run prettier:fix && npm run lint:fix",
    "lint": "eslint src --ext=js",
    "lint:fix": "npm run lint -- --fix",
    "prettier": "prettier -c",
    "prettier:fix": "npm run prettier -- --write"
  }

The npm run format command allows me to fix some of the prettier formatting with eslint by running them both in an order. I wanted to replicate this same behavior with my editor's "Format on Save" functionality.

VS Code only allows setting one default formatter. So I could either run Prettier or run ESLint on save. That was until it introduced "codeActionsOnSave". This separated formatters and "Source" fixers like vscode-eslint. This is how one would enable both prettier and eslint in VS Code:

"editor.formatOnSave": true,
"editor.defaultFormatter": "esbenp.prettier-vscode",
"editor.codeActionsOnSave": {
  "source.fixAll.eslint": true
}

But this would often lead to prettier being run after eslint, and eslint errors still being present as this defined no order for formatters and coder actions.

Since the March 2020 (v1.44) update, VS Code allows setting codeActionsOnSave as an array. This allows setting the order of the code actions.

This extension uses the CodeActionProvider api to implement a simple code action that runs 'Format Document' on the current file. This allows us to disable formatOnSave and use it as a codeAction instead in a specific order with other extensions.

DangerJS Plugin - TODOs

Mon, 16 Mar 2020 18:30:00 GMT

What is It?

A plugin for danger-js that collects all the "TODO" comments introduced in a pull request (PR) and posts them as a comment on the PR.

It's a pretty simple plugin that just scans your diff for matching comments, it supports different language style comments including JS, C, C++, Ruby, Java, Elm, HTML, Bash, etc. (open an issue if something is missing). You can also specify custom keywords other than "TODO" or "FIXME".

How to set it up?

Set up danger-js and add it to your continuous integration (CI) pipeline
Add danger-plugin-todos to the "dangerfile" as mentioned in the README

Next time your CI pipeline runs for any PR this plugin will scan the diff to get all the To-Do comments and post them through danger as a comment on the PR. The comment will update on consecutive runs.

Possible Workflows

Scenario A

Sometimes when you start work on a feature you see things along the way that need to be fixed and you end up adding a To-Do or Fix-Me comment for them. The feature branch then gets merged and the comments quickly gets forgotten if they were not related to a part of the code base that sees frequent updates.

To track such To-Dos this plugin can be used, as a list of them will be added as a comment to the PR. This list of To-Dos can then be moved to a new issue before the PR is merged and the issue added to the backlog.

Scenario B

This plugin can also be used to create an actual auto-updating To-Do List.

Just add all your tasks for the PR in the code as To-Dos and open a Work In Progress (WIP) PR, this will create a list of To-Dos in the PR comments.

Now, as you work on them and remove them the comment will be updated and your list will become smaller. Once you are done with them the PR will be ready to merge.

More Features Incoming

There are some new ideas that are being worked on (check the issues here), like:

Generate a link to create a new issue with a list of To-Dos automatically: This would be pretty great for our Scenario A above. Link can be customized to support different issue trackers
Show completed To-Dos: PRs that complete and remove old To-Dos should be celebrated. This would add a success message on removing old To-Dos.

Anyways, I hope this is useful for you. If you have any ideas on how to make this better feel free to message me on Twitter or open an issue on the repo.