Database Migration Systems: An Overview

I have spent a lot more time than I should be looking into solutions on database migrations for RDBMS. Sure, on the surface, there may seem only a few ways to hammer a nail, but dig a little more and it is an endless rabbit hole. I figured in this writeup I will put together all the various techniques and ideas on handling database migrations.

These ideas are mainly assembled from 7 different database migration systems πŸ₯Š:

  1. Flyway πŸ•Š – SQL file-based database migration system
  2. Liquibase πŸ’§ – flexible migration system with a monitoring dashboard
  3. Django 🐍 – Web framework for Python
  4. Active Record πŸš‚ – ORM for Ruby on Rails
  5. Entity Framework 🌏 – ORM for .NET
  6. Alembic βš—οΈ – database migration tool for SQLAlchemy (Python).
  7. Skeema πŸ“ – pure-SQL schema management utility.

Now, just because I didn’t mention them here doesn’t mean they are not good. There are simply too many, and I’ve only looked into these 7 in detail. You are welcome to comment your favourite below and how it stacks up against the competition.

Introduction: The Evolving Database Problem

In production systems, there is typically no such thing as a final database schema. Similar to your app releases, this “final” database is a moving target, and we only develop with the most recent database version (or state). New tables, columns are added, dropped, renamed over time; data gets shuffled to different places – a database tends to evolve from one version to another. It gets messy real quick with a team of developers, so a database migration system eventually gets introduced, and it is not uncommon that companies will opt to build theirs in-house.

There is a great article by Martin Fowler introducing evolutionary database design, as well as techniques and best practices.

Essential Components

Although database migration systems may have a plethora of ideas of how to go solve the evolving database problem, practically all of them shares 4 essential components:

  • Migration/revision/changeset
  • Rollback
  • Repeatable migration
  • Version tracking

Migration

A migration (flyway) or revision (alembic) or changeset (liquibase) is a single unit of change that brings a database from one version to another. One key problem database migration systems need to address during a migration run is execution order. I will go through 4 strategies on how to solve this:

  • Filename
  • Appearance order
  • Relative revision
  • Topological order

Filename. This is the simplest and most intuitive strategy. Chances are if you implement a migration system yourself this would be the first approach you would consider. In flyway, you would organise your migrations as such ('V' here stands for versioned migrations):

πŸ“‚ migrations/
β”œβ”€β”€πŸ“„ V1__create_table_foo.sql
β”œβ”€β”€πŸ“„ V1.1__add_column_bar.sql
β””β”€β”€πŸ“„ V2__create_audit_tables.sql

When you execute flyway migrate, the files will be executed in numeric order. Note that by default, flyway does not execute migrations that are slipped in between executed migrations (e.g. if V1 and V2 have already been executed and you add V1.1 later, V1.1 does not get executed). This can be changed via the -outOfOrder parameter.

One problem with this approach is nothing stopping multiple developers from submitting the same version more than once; flyway will not accept ambiguity in versioning. A common solution for this is to have a CI run flyway migrate in pull requests. Combine this with only merging pull requests that are up to date with master, and you prevent these conflicts from entering your source code in the first place.

A simpler solution to solve duplicate versioning is using a timestamp. This is the most hassle-free way to resolve this and is the approach that Rails Active Record and .NET Entity Framework uses. After all, what are the odds of multiple developers creating a migration at precisely the same second right?? πŸ˜‰

Here’s an example of Active Record migration folder:

πŸ“‚ sample_rails_app/db/migrate/
β”œβ”€β”€πŸ“„ 20190822013911_create_users.rb
β”œβ”€β”€πŸ“„ 20190822021835_add_index_to_users_email.rb
β””β”€β”€πŸ“„ 20190827030205_create_relationships.rb

Note that you could also do this in flyway (e.g. V20210617115301__create_table_foo.sql) – but unlike flyway, both the migration filename and contents in Active Record and Entity are generated, where the prefix is the timestamp of migration creation. I will elaborate more on this later.

Appearance Order. A migration in liquibase is referred to as a changeset. Multiple changesets are stored in changelog file. When you first configure Liquibase you need to specify a single master changelog file. In addition to changesets, a changelog file can reference multiple changelog files. Liquibase then executes all changesets and referenced changelogs in order of appearance within the master changelog file, from top to bottom. This makes changing execution order trivial as compared to changing around filenames.

Liquibase recommends the following directory structure, where the changelog is named after the app release version (not the migration version):

πŸ“‚ com/example/db/changelog/
β”œβ”€β”€πŸ“„ db.changelog-master.xml
β”œβ”€β”€πŸ“„ db.changelog-1.0.xml
β”œβ”€β”€πŸ“„ db.changelog-1.1.xml
β””β”€β”€πŸ“„ db.changelog-2.0.xml

The order of execution is then dictated by the master file db.changelog-master.xml:

<?xml version="1.0" encoding="UTF-8"?>   
<databaseChangeLog  
  xmlns="http://www.liquibase.org/xml/ns/dbchangelog"  
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"  
  xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
                      http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.8.xsd">  

  <include  file="com/example/db/changelog/db.changelog-1.0.xml"/>   
  <include  file="com/example/db/changelog/db.changelog-1.1.xml"/>   
  <include  file="com/example/db/changelog/db.changelog-2.0.xml"/>   
</databaseChangeLog> 

You can check out how this is used in practice in keycloak’s github repo (the master file is called jpa-changelog-master.xml, which is referenced here).

Note: It’s a pervasive misunderstanding that liquibase only supports XML. Hardcore DBAs can rejoice to know that you can also write your changesets in SQL. Doing so would deprive you of magical features, but liquibase does allow you to use both file types in the same project!

As with using filenames, conflicts do exist when multiple developers write to the same file, but liquibase claims that this is much simpler to deal with using version control systems like git.

Relative Revision. This is used in Alembic, where migrations are called revisions. The idea is that execution order is not dictated by appearance or filename, but relative from one revision to another, beginning from the initial revision. This can be confusing at first, for example when you see the migration folder in Apache Airflow it looks like this unordered mess:

πŸ“‚ airflow/airflow/migrations/versions/
β”œβ”€β”€πŸ“„ 004c1210f153_increase_queue_name_size_limit.py
β”œβ”€β”€πŸ“„ 03afc6b6f902_increase_length_of_fab_ab_view_menu_.py
β”œβ”€β”€πŸ“„ ...
β”œβ”€β”€πŸ“„ f2ca10b85618_add_dag_stats_table.py
β””β”€β”€πŸ“„ fe461863935f_increase_length_for_connection_password.py

The first 12 characters prefixed to every migration file is its hash, this is called a “revision ID” – it identifies a migration, but tells nothing about the execution order. To query the past execution order, you need to run alembic history.

Alembic determines ordering from 2 attributes in every revision file. Consider an initial revision 1975ea83b712_create_account_table.py that is generated by the alembic generate command:

# 1975ea83b712_create_account_table.py
revision = '1975ea83b712'
down_revision = None

The revision attribute is the same as the migration’s revision ID. Where down_revision is None, we know that this is a root node; the first revision that gets executed. Say another revision ae1027a6acf_add_some_column.py gets added (also generated by alembic generate):

# ae1027a6acf_add_some_column.py
revision = 'ae1027a6acf'
down_revision = '1975ea83b712'

This declares that revision ae1027a6acf revises 1975ea83b712, and therefore requires 1975ea83b712 to be executed first. So each subsequent revision from the root knows which revision it modifies, and from there you can build an execution order 1975ea83b712 β†’ ae1027a6acf.

To run migrations, we use alembic upgrade head, where head will point to the most recent revision.

Though we would be tempted to think of this as a linked list, where each node has a reference to another node, in cases where there are multiple developers it is more like a tree. This is because Alembic allows multiple revisions to “branch” from a shared parent (i.e. there is more than one revision file that has the same down_revision). In such cases, alembic upgrade head will fail as there are multiple head‘s. To resolve this, a developer would use alembic merge to merge all heads into one. The Alembic branch docs have a nice ASCII art to illustrate this:

The merge command will create another revision file that fuses more than one revision to a single timeline. This is simply a revision file with multiple down_revision:

revision = '53fffde5ad5'
down_revision = ('ae1027a6acf', '27c6a30d7c24')

As you might be able to tell, these concepts on branching, merging, and head revision is very similar to git. And similar to git, you don’t have to create a merge commit; another strategy is “rebase”, which means you change the parent revision (i.e. down_revision). Assuming there are no conflicts, you can do away with the merge revision and get a clean timeline with no branches.

In the case of our above example, we would change the down_revision of 27c6a30d7c24 from 1975ea83b712 to ae1027a6acf (notice that we also removed 53fffde5ad5):

Topological order. This idea is unique in that it does not see migrations as a linear path with a start and end, but a directed acyclic graph (DAG) where the nodes are migrations and the edges are dependencies. An example of a dependency would be if you have a foreign key constraint, the table it points to must first exist. Now you can run migrations in any order you like, as long as dependencies are met; the execution order is thus defined by its topological order, and there can be many viable paths.

In Django, every model has its own migration folder. Take this example from Thinkster‘s Django Realworld Example App (simplified for brevity):

πŸ“‚ django-realworld-example-app/conduit/apps
β”œβ”€β”€πŸ“‚ articles
β”‚   β””β”€β”€πŸ“‚ migrations
β”‚       β”œβ”€β”€πŸ“„ 0001_initial.py
β”‚       β”œβ”€β”€πŸ“„ 0002_comment.py
β”‚       β””β”€β”€πŸ“„ 0003_auto_20160828_1656.py
β””β”€β”€πŸ“‚ profiles
    β””β”€β”€πŸ“‚ migrations
        β”œβ”€β”€πŸ“„ 0001_initial.py
        β”œβ”€β”€πŸ“„ 0002_profile_follows.py
        β””β”€β”€πŸ“„ 0003_profile_favorites.py

Every migration file has a dependencies attribute, which may contain zero or more dependencies. If a migration has no dependencies, it can be executed prior to other migrations. For example in 0003_profile_favorites.py:

dependencies = [
    ('articles', '0002_comment'),
    ('profiles', '0002_profile_follows'),
]

To make the concept more visually apparent, here is its DAG along with a viable topological order (from 1 to 6):

Blue nodes are profile model, yellow nodes are article model. The directional edges can be viewed as “β€”required by β†’”. Red circles on the corners of each node mark execution order.

Read Django’s official documentation on this in “Controlling the order of migrations

As pointed out earlier, there can be multiple viable execution orders for as long as the dependency conditions are satisfied. For the above example, the following is also a valid execution order:

  1. profiles:0001_initial.py
  2. articles:0001_initial.py
  3. articles:0002_comment.py
  4. profiles:0002_profile_follows.py
  5. articles:0003_auto_20160828_1656.py
  6. profiles:0003_profile_favorites.py

This seems complex, but it provides a lot of flexibility for developers, and greatly reduce the chance of conflicts. Also, developers working with Django are usually insulated from migration implementation details as it is generated via django-admin makemigrations command.

Rollback

A rollback reverts a specific migration. A simple case is when you add a column in a migration, its corresponding rollback would be to drop the column. Typically rollbacks are added within the same migration file, where a migrate function has a corresponding rollback function, as is the case for Entity Framework (Up and Down), and Alembic (upgrade and downgrade).

Some implementations (e.g. Django, Active Record) decide to go a step further by not even requiring an explicit rollback function; the migration tool automagically figures it out for you πŸͺ„! In Liquibase, if you use their XML/YAML/JSON syntax, it can also automatically figure out how to rollback a changeset.

In flyway, rollbacks are called undo migrations; these are just SQL files that map to a corresponding migration (e.g. a V23__add_column_bar.sql is reverted by U23__drop_column_bar.sql). In addition, undo migrations are a paid feature in flyway; not only do you have to write them out by hand, but you also had to pay for a license to do it.

This is not all bad, since you may not always be able to generate or write a rollback. For example, although you can reverse adding a column, you can’t undrop a table or a column, and you can’t undelete rows; once data is gone, it is gone for real. In practice, DBAs tend to only go forward. If you regret some design decision in the past and want to revert it, you write another migration. However, rollbacks are still handy as a plan B in situations where deployments go horribly wrong despite meticulous planning.

Repeatable Migration

There are special cases where you would want to run a migration more than once. These are often stored procedures, and views – and in cases where business logic is part of the database, they are often changed.

Typically repeatable migration work by comparing the checksum of migration of past execution and current execution. Flyway prefixes repeatable migrations with R__, and they are always executed last in any migrate run. Liquibase supports a runOnChange changeset attribute that reruns the changeset when changes are detected.

ORM focused migration tools like Active Record, Entity Framework, Alembic, Django would presume you will never need repeatable migrations. In fact, if you come from ORM land, stored procedures and views are a foreign concept, if not frowned upon (as the rails purists would say, “It is not the rails way!” πŸ”₯).

Although Alembic does not support repeatable migrations out of the box, there is a community contribution Alembic Utils that adds support for it… for Postgres only. Otherwise, you are left with implementing it yourself via this ReplaceableObject recipe in the Alembic cookbook.

Version Tracking

Version tracking is a technique to guarantee that migrations only execute once, or in the case of repeatable migrations – only when the migration has been modified.

Practically every database migration system will implement this as a tracking table in your database. Storing tracking information in your database guarantees atomicity during a migration run (i.e. if a migration fails the version is never tracked). Migration systems may also place a table-level write-lock on a migration table to assert such that only one migration run can happen at any given moment; this is particularly useful when you have multiple copies of your app connecting to the same database in parallel.

Version tracking tables go by many names: alembic_version (Alembic), __eFMigrationsHistory (Entity Framework), schema_migrations (Active Record), flyway_schema_history (Flyway), DATABASECHANGELOG + DATABASECHANGELOGLOCK (Liquibase), and django_migrations (Django).

Typically, tracking tables store only the migration version, identifier, or filename. Execution timestamp may also be stored, allowing developers to roll back to a specific point in time. In some implementations like Flyway and Liquibase, tracking tables also store checksums. These are used for 2 things:

  1. For normal migrations, checksum asserts that an executed migration has not been tampered with.
  2. For repeatable migrations, checksums are used to determine if migration needs to be re-executed.

Workflows

Although database migration systems share a lot of essential components, their workflows tend to differentiate them from the rest.

  1. Revision based workflow
  2. Code first workflow

For each workflow, there are 2 paradigms: either you write SQL that the database directly runs, or rely on source-code generation. In source-code generation, you write your migration in some other language than SQL (XML, C#, Ruby DSL,.. etc) that generates SQL code. It’s an extra step – yes, but developers love this approach with religious vigour. There are a few key reasons:

  1. It can support different database vendors
  2. Can do automatic rollbacks
  3. Allows developers to write in their favourite programming language

Alright, let us walk through the different workflows.

Revision Based Workflow

In a revision based workflow, you write your migration script first, and that determines the next database state. Then continue to add subsequent migration scripts continue to evolve the database over time.

In Flyway, you write raw SQL files the follows a specific naming convention. In Liquibase, you typically use XML, and that enables vendor-independent migrations as well as automatic rollbacks. Active Record takes this a step further with a command that generates migrations for you; use rails generate model command to create a migration file. For example:

rails generate model Post title:string link:string upvotes:integer

This creates db/migrate/20210627134448_create_posts.rb, written in Ruby DSL (Domain Specific Language):

class CreatePosts < ActiveRecord::Migration[5.2]
  def change
    create_table :posts do |t|
      t.string :title
      t.string :link
      t.integer :upvotes

      t.timestamps
    end
  end
end

To make changes to a model you use rails generate migration. For example to add a downvotes column to the posts table do:

rails generate migration AddDownvotesToPost downvotes:int

More than just being descriptive, the migration name you provide in the generate migration command is used to create the migration itself (i.e. it figures out you want to add a column to the posts table from the name AddDownvotesToPost).

Code First Workflow

A code first workflow can also be thought of as a declarative workflow: you first model out how you want your database to be, and let the migration system generate a migration.

Take this Entity Framework model (C#) from their getting started guide:

namespace ContosoUniversity.Models
{
    public class Student
    {
        public int ID { get; set; }
        public string LastName { get; set; }
        public string FirstMidName { get; set; }
        public DateTime EnrollmentDate { get; set; }
        
        public virtual ICollection<Enrollment> Enrollments { get; set; }
    }
}

Then you execute add-migration <DescriptionMigrationName> and a migration file (also in C#) is automagically generated. When you make changes to your classes, simply run add-migration to create a new migration file. In short, your C# classes determine the current state of your database. In Entity Framework lingo, this is known as “Code First Migrations“, and when I first saw this as a young developer 4 years ago, it blew my mind.

As I eluded to earlier, Django also generates its migration files from its models with its makemigrations command. Alembic supports this as well via an additional --autogenerate flag to the alembic revision command.

For folks that disdain source-code generation, Skeema allows you to declare the final schema in SQL, and it generates a SQL diff that changes the target database state to the declared state. In fact, it is so confident in this approach it does not even do version tracking. There are no replaying historical migration files; whether it be a new development instance or production database, we are always one migration away from the declared schema.

Even though it uses plain SQL, Skeema does not parse SQL code itself; it runs the SQL code in a temporary location called a workspace and compares it with the schema of your target database. The generated SQL can be reviewed via skeema diff, and deployed via skeema push.

Some caveats: as cool as code first workflows may sound, they would not cater to ambiguous cases. For example, did you rename column color to colour, or did you drop column color and added a new column colour? The simple answer is that there is no way to tell apart renames. Skeema details this drawback in its product announcement, but assures that renames rarely happen πŸ˜‰.

Closing Thoughts

Finding the right approach is more than just performance or features. You need to think about developers in your team whether they want to work with some custom migration solution you found in a comment in Hackernews, that points to some GitHub repository with barely any activity or stars. Rarely would developers want to learn something just for one project. Any database migration system you introduce puts a learning curve for your team. Most likely those developers would know little SQL and would hardly have any interest in being good at it. Typically the database migration system they would opt to use is simply the one that comes bundled to the chosen web framework.

Database migration is a messy conundrum without perfect solutions. As I would come to learn – we simply need to decide what is best on a case by case basis.

Testing PubSub Locally with Python + Docker

Sometimes you just want to quickly prototype a streaming system that uses pubsub. You don’t need the full power of the production version in Google Cloud. You don’t want to go through all the long prerequisite setup steps as with the official pubsub emulator guide. You just want to summon pubsub with a single command. And you don’t want to write verbose Java code.

What do you do? Well, an underrated (or perhaps just not really well-documented) alternative is to just spin up a docker container:

docker run -it --rm -p 8085:8085 gcr.io/google.com/cloudsdktool/cloud-sdk \
    gcloud beta emulators pubsub start --host-port=0.0.0.0:8085

As simple as that – you have a fake pubsub emulator running in your machine at port 8085 (the default port). Now we want to use python to start sending and receiving messages. We can use google-cloud-pubsub (API reference is also in the link) by installing it as such:

pip installΒ google-cloud-pubsub

By default, the google-cloud-pubsub plugs itself to Google Cloud’s endpoint. We can instead instruct google-cloud-pubsub to connect to our emulator with the PUBSUB_EMULATOR_HOST environment variable:

export PUBSUB_EMULATOR_HOST=localhost:8085

Now let’s have a publisher up and running. With the emulator environment var set we launch an interactive python console and punch in the following:

from google.cloud import pubsub

publisher = pubsub.PublisherClient()
pubsub_topic = 'projects/dummy-project/topics/test-topic'

publisher.create_topic(pubsub_topic)

# Use publish to start shipping messages...
publisher.publish(pubsub_topic, b'Hello?')
publisher.publish(pubsub_topic, b'Is anyone there?')

I just sent over some byte string payload, so let’s print it in a client. Spin up another interactive python console for the client:

from google.cloud import pubsub

subscriber = pubsub.SubscriberClient()
pubsub_topic = 'projects/dummy-project/topics/test-topic'
sub_name = 'projects/dummy-project/subscriptions/subslot123'

subscriber.create_subscription(name=sub_name, topic=pubsub_topic)

# Blocking call to start listening. Passes all messages to print method
subscriber.subscribe(sub_name, print).result()

Now, if you forget to set the environment variable (as you normally would), don’t worry – Google will not fail to remind you to use their production-ready version:

Traceback (most recent call last):
  File "", line 1, in 
  File "/home/bruce/.local/lib/python3.8/site-packages/google/cloud/pubsub_v1/publisher/client.py", line 114, in __init__
    channel = grpc_helpers.create_channel(
  File "/home/bruce/.local/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 195, in create_channel
    credentials, _ = google.auth.default(scopes=scopes)
  File "/home/bruce/.local/lib/python3.8/site-packages/google/auth/_default.py", line 321, in default
    raise exceptions.DefaultCredentialsError(_HELP_MESSAGE)
google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started

And that’s it. Happy experimenting!

Interactive p5.js Sketches in WordPress via CodePen

Today I have been doing some experiments in embedding interactive content into this blog:

It is a basic example; useful as a template to get started with. In the remainder of this post, I detail how I got here.

Codepen GIF - Find & Share on GIPHY

A quick search on the internet shows that various people have attempted to embed p5.js content in the past. All of the ones I’ve seen so far either embed an iframe manually, slip in php code, or some use some plugin. If you are under wordpress.com, the first 2 options is not possible and the last option requires a business plan.

The approach I use here needs none of that.

Prologue

For the past few weeks, I have been researching on creating interactive “sketches” that could provide a more engaging pedagogical tool for me to intuitively understand technical ideas. I figured that it would be great if it were also convenient to share it to everyone – no other platform achieves this better than the ubiquitous nature of the web. And, is it not even more convenient if I could just embed it to my blog?

Hence begins my search into web rendering engines like pixijs, web game engines like phaser, playcanvas, ctjs and Unity’s Project Tiny (I recommend gamefromscratch’s YouTube channel for more).

While these are great tools, they don’t exactly fit the definition of a “sketch”. These are more of a base to build complex systems, not something you can get up and running with a bit of code.

And so I found myself in p5.js, which takes after the core principles of Processing:

Processing is a flexible software sketchbook and a language for learning how to code within the context of the visual arts.

As I browsed p5.js examples, I noted it comes bundled with math libraries, 3D capabilities, audio + image + video processing, simulation systems – the whole enchilada. The best part about p5.js is that no install is required to get started. You could go to https://editor.p5js.org/ and an editor is up, with examples ready to be loaded with a few clicks

That blew my mind. How have I in all my years of meandering the internet have I not known this existed?

Fun fact: Processing also has a Python mode https://py.processing.org/

CodePen-Wordpress Integration

So that was p5.js. How did I integrate it to this blog? Well, as you can probably already see, I use CodePen and the magic of iframes. Most frontend developers would recognise CodePen (or its competitor JsFiddle) as a CSS, HTML, javascript playground.

While CodePen does give you a few options to embed into WordPress in their blog, the only one that will work under wordpress.comΒ is when you copy-paste any CodePen Pen link on a paragraph on its own:

WordPress will then auto-magically convert it to some widget where your visitors can interact with. This behaviour is also documented in WordPress.com CodePen support page, which has this adorable gif:

OMG Wapuu is so cute!

CodePen & p5.js

While you can find p5.js CodePens, those are woefully outdated. It uses v0.2.13, which does not have touch support – the latest version is v1.0.0. Though you could add/change the js dependencies from the HTML block, I prefer to set these imports under Settings > JS > Add External Scripts:

As of this writing, v1.0.0 is the latest version – you should use whatever the latest version is at the time you are developing. Setting to always use the latest version can potentially break your pen in the future if there are breaking changes.

For some reason, web browsers add some margin space of 8px to the body element. This will appear as white space in your sketch. You can disable this by adding the following in the CSS block:

body {
  margin: 0;
}

Aside from this, there is practically no difference from the example sketches you see in p5.js except for these additional lines in the setup function that is tailored for touch devices:

// Fit canvas to available space factoring codepen's margin
let body = document.documentElement;
createCanvas(body.clientWidth, body.scrollHeight - 5);

// Prevent page pan when you drag about the canvas
body.getElementsByTagName("canvas")[0].addEventListener(
  "touchstart",
  (e) => { e.preventDefault(); },
  false
);

That little bit of code just stretches the canvas to whatever space CodePen has available, which varies when you use a phone or tablet or fridge. It also ensures that while you are interacting with the pen you do not also accidentally pan the web page.

Of course, instead of doing all of this, you could simply start off on your own by forking my pen.

The Learning Curve

Wow! That looks exciting – but do you need to know javascript, CSS and HTML?

IMHO, just javascript would be enough. And they do teach you javascript via the examples in p5.js – covering all the bare basics as if you never wrote a single line of code before. That bit of CSS you saw earlier is probably all you will ever need (until you decide to style DOM elements or whatnot).

There is also a nice beginner-friendly introductory video by Cassie Tarakajian, who is also the lead maintainer of the p5.js editor (source code on Github):

Closing Thoughts

So that is a brief tour of adding interactive sketches to WordPress – I hope you find it useful, or at least entertaining. In future posts I hope to create and embed more of these interactive widgets, so stay tuned!

How to Setup LUKS2 encrypted Ubuntu 20.04 With Dual Boot

This post is a guide to setup disk encryption on Ubuntu 20.04 using LUKS2, while still being able to dual boot to Windows 10. Unlike most guides out there, I intend to keep the setup as simple as possible:

  • One partition for boot, and another for everything else (no separate data partition)
  • Boot partition is unencrypted
  • No swap
  • No LVM

I will not convince you why you should encrypt your hard disk – plenty of resources out there already.

Though I try to keep explanations simple, I presume you are already a little familiar with Linux to begin with; this is not a guide to be following if this is your first time installing Linux.

I am using Kubuntu here, but there should not be much difference in the setup procedure across different Ubuntu variants.

Rationale for Setup

You can skip ahead to “Prologue”Β if you do not bother to know.

I have used file based encryption via eCryptfs on my home folder for a few months. A bit of setup was needed since I did not encrypt during installation, but it was tolerably simple and it had so far worked really well. I opted to switch to disk based encryption mainly because Ubuntu team does not intend to support file based encryption moving forward. Disk based encryption is also more performant when dealing with many files (see performance comparison on phoronix.com).

Now although it sounds good to switch in theory, the setup is a big pain, especially for a GUI lover like me. Ubuntu and Pop OS offers disk based encryption out of the box, but the moment you need to customise your disk setup (so as not to wipe your Windows installation), the convenience is thrown out the window. I went through Full_Disk_Encryption_Howto_2019 from Ubuntu Community WikiΒ and Encrypting disks on Ubuntu 19.04 from Isuru Perera and found it too much work.

So I came up with my own guide.

I have only 110GB of space allocated for my linux setup, so dividing up between data partition and the OS is a tough call. I do not keep a lot of data inside my laptop anyway.

Encrypting my boot partition and putting decryption keys to root partition inside makes no sense to me. Even the default Ubuntu setup does not do this. Having said that, GRUB very recently supported LUKS2, in case you want to attempt to encrypt anyway.

I have 16GB of RAM and the concept of swap is foreign to me.

LVM is good if you want to grow your partition space across multiple hard disks, even while your OS is running. I am stuck with the single disk slot in my thinkpad, so this is a little unecessary.

Prologue

Disclaimer: many things can go wrong when you customize your setup and fiddle with the terminal as root user. It is recommended to keep a backup of valuable data and do a few trail runs first.

My disk setup as is below.

I have formatted my devices as follows (your device names may vary; do not just copy what you see!).

  • For my root parition I use a 110Gb EXT4 partition (/dev/nvme0n1p5)
  • For my boot partition I use a 300Mb 500Mb EXT4 partition (/dev/nvme0n1p6)

Update 9/5/2020: 300Mb boot partition is too small – should be 500Mb. Turns out Ubuntu stores a few versions of the initrd in the boot partition and I bumped to “Error 24 : Write error : cannot write compressed block” when updating. You could resolve this by removing older kernels (refer here), but it is not pleasant to bump to this issue. apt does offer to cleanup old kernels for you, but you need to explicitly do so after updating:

apt autoremove

I have also a 260Mb EFI partition (/dev/nvme0n1p1), which is mandatory for me because my laptop uses UEFI – you may not have this. Everything else belongs to Windows.

Be sure you jot down the device name of your designated root partition (/dev/nvme0n1p5 in this example). You will use it a lot later.

Partition managers are simple to use, so I skip the steps for setting up the partitions here.

Important: partition manager allows you to encrypt your partition as an option when you format. Do not enable this. As of this writing it defaults to LUKS1; we want LUKS2.

Setup Encryption

Now we setup encryption on the root partition.

$ sudo -i # proceed the entire guide as root user
# cryptsetup luksFormat /dev/nvme0n1p5

WARNING: Device /dev/nvme0n1p5 already contains a 'ext4' superblock signature.

WARNING!
========
This will overwrite data on /dev/nvme0n1p5 irrevocably.

Are you sure? (Type uppercase yes): YES
Enter passphrase for /dev/nvme0n1p5: 
Verify passphrase:

This will replace our ext4 partition with an encrypted LUKS partition. To verify that we are indeed using LUKS2, check the version (I have omitted the output for brevity):

# cryptsetup luksDump /dev/nvme0n1p5

LUKS header information
Version: 2
Epoch: 3
...

Next, we open the encrypted partition as rootfs so that Ubuntu can be installed inside:

# cryptsetup open /dev/nvme0n1p5 rootfs

Enter passphrase for /dev/nvme0n1p5:

This will create a new device /dev/mapper/rootfs where we can read and write freely, while the encryption and decryption is performed underneath. It is currently unformatted space, so we need to format as ext4:

# mkfs.ext4 /dev/mapper/rootfs

mke2fs 1.45.5 (07-Jan-2020)
Creating filesystem with 501760 4k blocks and 125440 inodes 
Filesystem UUID: 31dcba0e-ec56-4321-9578-0abcd162de2f 
Superblock backups stored on blocks: Β 
Β Β Β Β Β Β Β 32768, 98304, 163840, 229376, 294912 

Allocating group tables: done Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β 
Writing inode tables: done Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β Β 
Creating journal (8192 blocks): done 
Writing superblocks and filesystem accounting information: done 

Now we can install Ubuntu into /dev/mapper/rootfs.

Supposedly you can also format to ext4 in the graphical installer, but last I tried it seemed to think it needs to create a partition table, and I cannot get Ubuntu to boot properly afterwards.

Install Ubuntu

I will skip ahead a few steps to the disk setup. Here you may notice that there is no guided option where you can have encryption and keep your dual boot setup. You will need to do manual disk setup:

Select “Manual” for disk setup

In the “Prepare Partitions” page, let us start with the boot partition /dev/nvme0n1p6:

  • Use as: ext4 journaling file system
  • Format the partition: βœ“Β 
  • Mount point: /boot

Setup boot partition with parameters as shown

Now it gets a little confusing when it comes to the encrypted partition. For some reason, the Ubuntu installer displays it like it is a separate device from my hard disk. So select the partition under /dev/mapper/rootfs, which is also called /dev/mapper/rootfs, and set as follows:

  • Use as: ext4 journaling file system
  • Format the partition: βœ“Β 
  • Mount point: /

Setup root partition with parameters as shown

After setting both, you will see something like this when you select /dev/nvme0n1 and /dev/mapper/rootfs:

Lastly, if you have an EFI partition (/dev/nvme0n1p1 in my case), there is where you want to install the boot loader:

The steps after this is straightforward, so I will skip ahead to the part where the installation completes. Here click “Continue Testing” instead of restarting.

Crypttab

If you restart your Ubuntu installation now, your kernel would not be able to mount your root partition because it is encrypted. To circumvent this, you need to tell your kernel that the hard disk is encrypted. This is where crypttab comes in.

crypttab is like fstab in that it tells your kernel which hard disks to boot during startup. The key difference is that it is only for encrypted drives, and it loads before fstab.

Before we can setup crypttab for our fresh installation, we need to first understand that we are currently in the testing image. Any config changes we do in /etc/ therefore affects not our fresh installation (installed in /target/), but the testing image.

To change our terminal environment to the fresh install, execute:

# for n in proc sys dev etc/resolv.conf; do mount --rbind /$n /target/$n; done # change mount points to target
# chroot /target # change root directory to target
# mount -a       # mount all devices in fstab

Now, to inform the kernel to identify your LUKS encrypted root partition /dev/nvme0n1p5 as rootfs, execute the following:

# echo "rootfs UUID=`blkid -s UUID -o value /dev/nvme0n1p5` none luks" >> /etc/crypttab

Verify that your UUID has been added to crypttab:

# cat /etc/crypttab

rootfs UUID=aacc905d-beef-baba-a477-88aa12345fb2 none luks

Now you may be wondering: how is the kernel going to know what you set if the config is set in /etc/crypttab when it is in an encrypted disk?

Well, the kernel does not read it from there. You need to execute:

# update-initramfs -u -k all

update-initramfs: Generating /boot/initrd.img-5.3.0-46-generic

As you can see, it updates a file in /boot/, which is unencrypted.

Now restart and boot up your Ubuntu installation and you will be asked to key in a password to decrypt your root partition:

Post Installation

A fairly common issue you can expect is that Windows 10 time will be out of sync. You can fix it temporarily in Windows 10 and toggling setting the time automatically, but a permanent fix is to execute this in Ubuntu:

timedatectl set-local-rtc 1 --adjust-system-clock

It is important to note that your encryption password is different from your login password; changing one does not change the other. Having said that, it is convenient to keep them both passwords the same and disable login at startup – unless you want to key in your password twice during startup.

Though it is easy to change your login password from your window manager, you need to use the command line to change your LUKS2 password (this process does not re-encrypt your partition, so do not hesitate to change the passphrase on a whim):

# cryptsetup luksChangeKey /dev/nvme0n1p5

Enter passphrase to be changed: 
Enter new passphrase: 
Verify passphrase:

The wording “Enter passphrase to be changed” is very deliberate – LUKS can be configured to allow multiple passphrases to be configured for the same device.

I now end this post with a screenshot of my desktop. Cheers!

Additional Notes

Crap I formatted /boot/efi and now Windows 10 cannot boot! What do I do?

It is possible to restore the boot record to a new /boot/efi partition, but you will need a bootable USB stick with Windows 10 inside. To do this you’ll need a windows machine and a USB drive with about 8GB of space.

  • Create bootable USB drive using the Windows 10 Media Creation Tool (download option will not show up in a Linux machine).
  • Boot from the USB drive
  • Select Repair Your Computer.
  • Select Troubleshoot.
  • Select Advanced Options.
  • Choose Command Prompt from the menu:
  • Now use diskpart to assign a drive letter to new /boot/efi partition:
diskpart   # Start diskpart
sel disk 0 # Select first physical disk
list vol   # List all volumes. Search for your EFI paritition
sel vol <EFI_VOL_NO>         # Select EFI partition
assign letter=<EFI_LETTER>:  # Assign an unused drive letter
exit
  • Now you can recreate the boot record with <EFI_LETTER>:
bcdboot c:\Windows /l en-us /s <EFI_LETTER>: All

Now you will be able to boot Windows 10 again.

Setting Up Vscode for Single-file C++ Builds

So I have these self-contained C++ files and with a shortcut key I want to automatically build and run active file, and then use a debugger as well if I want to. Who would want to do this? Either you are just learning the language and want a quick way to prototype, or you are doing competitive programming (UVa, Codeforces, etc). The latter reason is why I’m writing this; me and my colleagues just started an “Algo Party” where we gather together to solve coding puzzles.

I’ve only tested this on osx and linux, but if you are using MinGW and you have GDB installed it should work.

Build and Run Any Random C++ File

When you open any C++ file, vscode is smart enough to suggest a bunch of extensions you should install. But after installing them you will realize that you can’t just run immediately start runningΒ hello world from your editor. To run the file in your active editor, installΒ Code Runner.

Now you can run the active file with the shortcut cmd+alt+n or ctrl+alt+n. There are some nice settings to have, like running in terminal, saving the file before run, and that juicy C++14 goodness. You can configure them via extension settings:

"code-runner.runInTerminal": true,
"code-runner.saveFileBeforeRun": true,
"code-runner.executorMap": {
    // ...leave as is
    "cpp": "cd $dir && g++ $fileName -std=c++14 -o $fileNameWithoutExt && $dir$fileNameWithoutExt && echo ''",
    // ...leave as is
}

Note that echo ''Β is just to print a new line after running the program. If you know you will always execute the program with an input file, simply add <input.txtΒ to the command above (before && echo '').

Press Ctrl+Alt+n to build and run the active file

Now if that’s all you want then you can stop here. The next section let’s you use a debugger, but requires a different setup.

Using Vscode Tasks To Build The Active File

This one is a project level configuration, so it assumes you keep your C++ files in one folder, like how I organize my solutions to my CodeForces problems in this git repository. To get started you can simply copy the json files from my .vscode folder over to yours.

Now on any C++ file in that project, simply hit cmd+shift+b to build the file. In addition to building the file, I’ve also added a -g argument to the compiler to build debugging symbols as well. You can remove this by tweaking the tasks.json. The file itself is fairly straightforward to understand. I already have a task to both build and run the file at once called “build and run cpp” (I always run my solutions with a file “input.txt”; if that’s not what you want you can simply tweak the command).

You can assign this to a shortcut key by going to File > Preferences > Keyboard Shortcuts and click the link in the message “For advanced customizations open and edit keybindings.json”. There you can assign the task to a shortcut you want (I use ctrl+9):

    {
        "key": "ctrl+9",
        "command": "workbench.action.tasks.runTask",
        "args": "build and run cpp"
    }

Build and run the active file using vscode tasks

Setting Up The Debugger

Hit f5 (or click the green arrow if you are in debug mode) to launch the debugger. This will launch an external terminal instead of using the integrated one.

Using the debugger in vscode

If you don’t use an input file then all is good, otherwise it is a hassle to constantly keying in inputs into the program. Unfortunately for us we can’t pass in the input file as an argument where the debugger is concerned (at least, for now).

Fortunately there is a simple workaround.

In STL, both the console input and file input are input streams. So what we can do is override cin to replace it with an ifstream. Because of the scope, our custom version will always take precedence:

#include <iostream>
#include <fstream> // std::ifstream

using namespace std;

int main()
{
    ifstream cin('input.txt'); // replace cin with our version

    // rest of the code remains the same:
    int p, q;
    cin >> p >> q;
}

Don’t forget to comment out that custom cin before you submit your solution to the online judge!