Engineering - Reference

README

• Keep Learning Everyday - learn fundamentals and use tools with long shelf-life
• Developing - Error Handling In Oneview
• Documentation
  • add order of operation (inspired by Network Protocol)
  • order is not obvious to derive from reading a bunch of functions/methods
• API Design
  • GraphQL API Design, 3 options -> consider using Enum
  • REST API Design, see my own Github REST API for Gists
• Graphene
  • use nonnull and required=True
• Django
  • use foreign key when I use xxx_id and xxx is an existing model
  • when adding constraints, also add DB data cleaning migrations
  • use select_related at first resolver to speed up further model property access based on GraphQL query to reduce total amount of sql call and thus improve performance
  • use Celery when a job makes sense to be done in the background
• Curo/Oneview Gotcha
  • inactive curo entities aren’t synced to oneview by default, because validate_state_code=True by default
  • inactive curo entites / soft deleted oneview model entities aren’t return from oneview ORM by default, need to use xx_deleted instead
• Work
  • track tickets raised in Product & Dev Chat in confluence
• Production deployment checklist
  • see if secrets and associate aws IAM policies, service are setup in prod
  • read each migration files to understand the risk of this deployment

Error Handling

Reminder

• use with transcation.atomic()
  • put multiple DB table changes in one with transcation.atomic()
  • put a single DB update/save with Curo call using submit_entity_unhandled
  • don’t use decorator @transaction.atomic to avoid future changes that introduces errors in error handling logics
• validate first, client facing logics last (e.g. sending emails)
• in Submission, avoid duplicate oneview and curo entities by using get_or_create?

Principles

• Error Handling means what are the correct business logics when errors happen
• consider curo as extension of oneview DB, so wrap curo call and DB changes in one transaction
• in general, one cannot revert a third party API call, because of possible side effects of the API call

Example: Curo

with transaction.atomic():
    task = Task.objects.create(...) # or update some properties then task.save()
    submit_entity_unhandled()

Example: Third Party API Call

# if response data is needed
foo_id = third party POST or PATCH
save `foo_id` to local DB

# if response data is not needed
with transaction.atomic():
    local db changes()
    third party POST or PATCH

• if third party call failed, not bad state, because there will be no local db changes.
• if save to local DB failed, bad state happened, maybe the solution is to verify first to avoid saving to local DB failed? because of the stable network or retry mechanism, unless this is DB constraint, save to local DB shouldn’t failed, so maybe verify can avoid the failure to local DB.

Regarding argument just put it in a queue and retry?

Answer:

• Putting some logics in a queue doesn’t solve the real problem of what should happen if this logics failed?
• and being async introduces complexity such as how to inform the user if this logics failed?
• when errors happen, which errors can be retried? which errors cannot be retried?

Github REST API for Gists

Create a Gist

Method: POST

PATH: /gists

JSON Data: {"description":"Example of a gist","public":false,"files":{"sample.md":{"content":"Hello World"}}}

Response Status Code On Success: 201

Example Response:

{
  "url": "https://api.github.com/gists/2decf6c462d9b4418f2",
  "forks_url": "https://api.github.com/gists/2decf6c462d9b4418f2/forks",
  "commits_url": "https://api.github.com/gists/2decf6c462d9b4418f2/commits",
  "id": "2decf6c462d9b4418f2",
  ...

Curl

curl -L \
  -X POST \
  -H "Accept: application/vnd.github+json" \
  -H "Authorization: Bearer <YOUR-TOKEN>" \
  -H "X-GitHub-Api-Version: 2022-11-28" \
  https://api.github.com/gists \
  -d '{"description":"Example of a gist","public":false,"files":{"sample.md":{"content":"Hello World"}}}'

Read a Gist

Method: GET

PATH: /gists/{gist_id}

Response Status Code On Success: 200

Example Response:

{
  "url": "https://api.github.com/gists/2decf6c462d9b4418f2",
  "forks_url": "https://api.github.com/gists/2decf6c462d9b4418f2/forks",
  "commits_url": "https://api.github.com/gists/2decf6c462d9b4418f2/commits",
  "id": "2decf6c462d9b4418f2",
  ...

Response Status Code On Resouce Not Found: 404

Curl

curl -L \
  -H "Accept: application/vnd.github+json" \
  -H "X-GitHub-Api-Version: 2022-11-28" \
  https://api.github.com/gists/GIST_ID

Update a Gist

Method: PATCH

PATH: /gists/{gist_id}

Response Status Code On Success: 200

Example Response:

{
  "url": "https://api.github.com/gists/2decf6c462d9b4418f2",
  "forks_url": "https://api.github.com/gists/2decf6c462d9b4418f2/forks",
  "commits_url": "https://api.github.com/gists/2decf6c462d9b4418f2/commits",
  "id": "2decf6c462d9b4418f2",
  ...

Response Status Code On Resouce Not Found: 404

Curl

curl -L \
  -X PATCH \
  -H "Accept: application/vnd.github+json" \
  -H "Authorization: Bearer <YOUR-TOKEN>" \
  -H "X-GitHub-Api-Version: 2022-11-28" \
  https://api.github.com/gists/GIST_ID \
  -d '{"description":"An updated gist description","files":{"sample.md":{"content":"Hello World from GitHub"}}}'

Delete a Gist

Method: DELETE

PATH: /gists/{gist_id}

Response Status Code On Success: 200

Example Response: No return Data

Response Status Code On Resouce Not Found: 404

Curl

curl -L \
  -X DELETE \
  -H "Accept: application/vnd.github+json" \
  -H "Authorization: Bearer <YOUR-TOKEN>" \
  -H "X-GitHub-Api-Version: 2022-11-28" \
  https://api.github.com/gists/GIST_ID

See reference

• JSON API CRUD
• create a gist
• get a gist
• update a gist
• delete a gist

Engineering - How To

write

Scrupulous writers, in every sentence that they write, will ask themselves at least four questions:

1. What am I trying to say?
2. What words will express it?
3. What image or idiom will make it clearer?
4. Is this image fresh enough to have an effect?

Two more that they may ask are:

1. Could I put it more shortly?
2. Have I said anything that is avoidably ugly?

Consequently there are ways to write well. The following rules will cover most cases:

1. Never use a metaphor, simile or other figure of speech which you are used to seeing in print.
2. Never use a long word where a short one will do.
3. If it is possible to cut a word out, always cut it out.
4. Never use the passive where you can use the active.
5. Never use a foreign phrase, a scientific word or a jargon word if you can think of an everyday English equivalent.
6. Break any of these rules sooner than say anything outright barbarous.

give consistent good PR review

• add doc string if there is anything unclear
• understand all code and tests added/modified
• handle network and other errors gracefully?
• read and test my comments carefully
• add ideas about incremental improvements

best practices: behavioural

• There is No Such Thing as Over Communication!
• Look for opportunity for continuous improvement
  • code base
  • toolings and personal toolings (vim, shell alias etc..)
• Read documentation/manual/user’s guide before using a new tool/cli, technology
  • this implies prefer using tool with good doc e.g. fish instead bash
  • for new technology, it’s important to understand how to use them before we use them
• Be more curious about how things work internally
• Be more courageous to try new things
• Be prepared because I am bad at ac-hoc talk and thinking
  • ask for agenda for meetings so that I can be prepared
• Validate assumptions earlier
• For meetings,
  • gather questions before meeting
  • set out the purpose of the meetings
  • write meeting minutes and action points

best practices: documentations

• the overall idea for documentation is to capture information that was in the mind of the designer but couldn’t be represented in the code
• add order of operations (function calls/rest api calls) to documentation
  • inspiration: a network protocol defines the format and the order of messages exchanged between two or more communicating entities, as well as the actions taken on the transmission and/or receipt of a message or other event.
• write present tense with full sentence
• write code comments, documentations before implementation
• four types of documentation
  • Tutorial e.g. get started with working code
  • How-To e.g. Step by Step Guides like Recipes
  • Explanation e.g. experts deep dive
  • Reference e.g. function docstring or python official documentations

best practices: design

• First Thing to Consider
  • what is the goal? ask why we are doing it?
  • does it bring value to business?
  • don’t just use technology because it’s cool!
• Design Goals
  • meets requirements
  • reliable
  • maintainable and operable (sustainable delivery of valuable, working software)
  • scalable
• Design it twice
  • write down two designs, each with pros and cons, then perhaps there will appear a third design with combined pros and less cons
• Architect great design rather than just working codes, then writing comments should be fun since that’s how you identify the best design
• Reduce complexity by adding or removing abstraction
• Start with writing documentation, ask for feedbacks as early as possible
• Designing new process that is easy to sell to the Team!
  • familiar technology
  • easy to pick up
  • improves dev experience, should improve the overall developer experience rather than damaging developer experience

Engineering - Explanation

Process v.s. Threads

Main Difference

• processes don’t share memory (process has its own virtual CPU and virtual memory)
• threads share memory (share the same address space, and thus can access the same data)

Why use threads?

Parallelism

single-threaded program can only use one CPU core where a multi-threaded program can use multiple CPU Cores.

Avoid blocking slow I/O

Some threads waits for slow blocking I/O, while other threads can perform useful computation.

Threading enables overlap of I/O with other activities within a single program, much like multiprogramming did for processes across programs.

Many morden server-based application such as web servers, database management systems, make use of threads in their implementations

One could still use multi-process instead of multi-thread, but threads share memory (an address space) thus make it easy to share data. multi-processes are better choice for logically separate tasks where little shareing of data structures in memory is needed.

ascii control characters

hexadecimal dump of ascii charaters

Control characters may be described as doing something when the user inputs them, such as code 3 (End-of-Text character, ETX, ^C) to interrupt the running process, or code 4 (End-of-Transmission character, EOT, ^D), used to end text input on Unix or to exit a Unix shell. These uses usually have little to do with their use when they are in text being output.

The control characters in ASCII still in common use include:

• hex: 00 (decimal: 00, null, NUL, \0, ^@), originally intended to be an ignored character, but now used by many programming languages including C to mark the end of a string.
• hex: 07 (decimal: 07, bell, BEL, \a, ^G), which may cause the device to emit a warning such as a bell or beep sound or the screen flashing.
• hex: 08 (decimal: 08, backspace, BS, \b, ^H), may overprint the previous character.
• hex: 09 (decimal: 09, horizontal tab, HT, \t, ^I), moves the printing position right to the next tab stop.
• hex: 0A (decimal: 10, line feed, LF, \n, ^J), moves the print head down one line, or to the left edge and down. Used as the end of line marker in most UNIX systems and variants.
• hex: 0B (decimal: 11, vertical tab, VT, \v, ^K), vertical tabulation.
• hex: 0C (decimal: 12 form feed, FF, \f, ^L), to cause a printer to eject paper to the top of the next page, or a video terminal to clear the screen.
• hex: 0D (decimal: 13, carriage return, CR, \r, ^M), moves the printing position to the start of the line, allowing overprinting. Used as the end of line marker in Classic Mac OS, OS-9, FLEX (and variants). A CR+LF pair is used by CP/M-80 and its derivatives including DOS and Windows, and by Application Layer protocols such as FTP, SMTP, and HTTP.
• hex: 1A (decimal: 26, Control-Z, SUB, ^Z), Acts as an end-of-file for the Windows text-mode file i/o.
• hex: 1B (decimal: 27, escape, ESC, \e (GCC only), ^[). Introduces an escape sequence.

from man ascii

use od to see the hex value for characters

od -t a -t x1 -t c

• -t a - prints in ascii format
• -t x1 - prints in hex format
• -t c - prints in c escape format

e.g.

> printf '\0\a\b\n\r\t\v ' | od -t a -t x1 -t c
0000000  nul bel  bs  nl  cr  ht  vt  sp
          00  07  08  0a  0d  09  0b  20
          \0  \a  \b  \n  \r  \t  \v
0000010

A web tool to find unusual characters quickly

https://www.soscisurvey.de/tools/view-chars.php

Understanding the color escape code

\033[XXXm is Select Graphic Rendition subset of ANSI escape sequences, \033 is actually ESC character in ascii octal format, \e is equivalent in zsh shell, however, don’t use \e, \e is not recognised in awk nor python.

> echo "\033[31mRed Text\033[0m"
Red Text
> echo "\e[31mRed Text\e[0m"
Red Text

• \033[ - Begin the color modifications
• CODEm - CODE + m at the end
• \e[0m - End the color modifications

> for i in {1..111}
do
    echo '\\e['$i'm' "\e[${i}mtext\e[0m"
done | column

See also: a comprehensive explanation

Authentication vs Authorization

Authentication is the process of verifying that “you are who you say you are”

Authorization is the process of verifying that “you are permitted to do what you are trying to do”

Architectures

Major Architectures

• Monolith
  • Shopify is a Monolith, but a modular Monolith
  • Creator of Ruby on Rails On The Majestic Monolith
• Microservice
  • Martin Fowler Microservice
• Event-Driven Architectures
• Event-Based Architectures

event driven v.s. event based architecture

Event Driven

A key feature of true event-driven architectures is that the producers and consumers are completely decoupled – a producer shouldn’t know or care who is consuming its events and how the consumers use those events in their service..

Examples

• Kafka + Kubernetes
• IoT + MQTT pub/sub

Event-based

Two key characteristics of event-based compute

1. the existence of a compute instance is intimately tied to the occurence of an event to be processed.
2. the compute acts on a single event at a time.

Examples

• API Gateway + Lambda

See also origin of this idea

full development cycle

• backlog refinement meeting and estimations
• planning meeting
• business requirements epic and tickets (Happy Path, Unhappy Path)
• FE & BE contract: REST API Specs, GraphQL Schema
• Infrastructure As Code Setup, feature flags,
• code logics implementation
• clarify requirements in ticket
• code documentations
• lints
• unittests
• code reviews
• developer manual e2e tests
• QA / functional testings (deploy to test env, setup test accounts)
• deployment and rollbacks
• monitoring
  • engineering: errors, performance, retrospective debugging logs,
  • operation: CPU, Memory, number of invokes, latency, throughputs,
• iteration on released feature
• software support

SaaS business model

SaaS is a business and software delivery model that gives organisations the ability to offer their solutions in a low-friction, service-centric model that maximises value for customers and providers. It relies on agility and operational efficiency as pillars of a business strategy that promotes growth, reach and innovations.

Explain Public and Private Key Cryptography like I am 5

Typically,

• Public Key is used to encrypt messages.
• Private Key is used to decrypt messages.

Importantly, if some text is encrypted with private key, only public key can decrypted it.

If some text is encrypted with public key, only private key can decrypted it.

For example, HTTPS handshake is basically, I get the Public Key of a website, generate a random password, encrypt it with Public Key and then send it to the website server and then website server uses Private Key to decrypt the password, so this password is shared. And further communication is encrypted using that shared password.

But really, public key isn’t just for encryption nor private for decryption, the otherway around works too, private key can be used for encryption and public key for decryption.

One example that uses private key for encryption is digital signature, take some text/file, create a digest of test/file, encrypt the digest with private key and send the encrypted digest together with the text/file. Then if the public key successfully decrypt the encrypted digest and the digest matches then we know the test/file is the original, because the only key that can encrypt something that can be decrypt with public key is the private key.

The only reason public and private keys are not interchangeable is because you can recreate the public key from private key.