https://shawnliu.me/post/Recent content in Posts on Siyuan's BlogHugo -- gohugo.ioen-us© 2020. All rights reserved.Sat, 22 Aug 2020 15:01:17 +0800https://shawnliu.me/post/journey-with-kubernetes-1-kubernetes-rpi-cluster/Sat, 22 Aug 2020 15:01:17 +0800https://shawnliu.me/post/journey-with-kubernetes-1-kubernetes-rpi-cluster/Container technologies have gained huge popularity in the past few years. First there is Docker, and later comes Kubernetes. Kubernetes, simply put, allows you to build a cluster of servers that run containers. It makes decision where to schedule containers, maintains high-availability of containers, manages related services like network routing, DNS resolution, and storage, and handles many more things. I recently bought four Raspberry Pi 4 (4GB model) to build a toy Kubernetes cluster at home.https://shawnliu.me/post/how-colorschemes-work-in-terminal/Mon, 23 Mar 2020 20:23:25 +0800https://shawnliu.me/post/how-colorschemes-work-in-terminal/As a daily user of the terminal, I find coloring my terminal very useful and aesthetically pleasing compared to the usual black and white default theme. Personally, I’ve been using the default dark base16 theme. In particular, I’m using scripts from the base16-shell repo because I can simply add the setup to my rc file and make it work for whatever machine I’m using. I never really bothered to understand how the coloring works, it’s been pure magic to me.https://shawnliu.me/post/smart-and-transparent-routing-with-policy-based-routing/Thu, 19 Dec 2019 17:07:18 +0800https://shawnliu.me/post/smart-and-transparent-routing-with-policy-based-routing/Usually when you setup a VPN connection, you notice that all your traffic is routed through the VPN. In some cases, this may not be what you want. For example, I wanted to setup a VPN server which is used only when requesting certain IPs. For other traffic, it’s routed normally with my ISP. To achieve such purpose, I found that I could use policy-based routing with WireGuard (or any other L3 VPN).https://shawnliu.me/post/configuring-sr-iov-for-mellanox-adapters/Wed, 18 Dec 2019 10:07:47 +0800https://shawnliu.me/post/configuring-sr-iov-for-mellanox-adapters/SR-IOV is a virtualization technique which allows a physical PCI-E device to spawn many virtual functions. These virtual functions appear as normal PCI-E devices and could be passed through to virtual machines. This allows the VMs to benefit from direct hardware access while allowing the underlying device to be shared. In my case, I needed to enable SR-IOV on the Mellanox InfiniBand adapters in order to build a HPC cluster on top of VMs.https://shawnliu.me/post/resource-management-in-programming-languages/Mon, 09 Sep 2019 17:14:12 +0800https://shawnliu.me/post/resource-management-in-programming-languages/Resource management is very important if you want to write computer programs that behave properly. Resource here refers to anything that you would need to acquire and release inside your program, such as memory, file, network socket, etc. If the resources are not managed well, it may lead to resource leakage. Therefore, we need to ensure that resources are properly released under all circumstances. In this blog, I describe a few patterns used in different programming languages for resource management.https://shawnliu.me/post/managing-dotfiles/Fri, 06 Sep 2019 11:58:42 +0800https://shawnliu.me/post/managing-dotfiles/If you ever used the command line, you probably have heard of dotfiles. Although the word dotfile could refer to hidden files in general in the *nix world, it is usually used to refer to the configuration files of various command line programs, such as bash and vim. When you start to seriously use the command line, you will find that you need to change configurations for programs. For example, you need to update your PATH inside .https://shawnliu.me/post/creating-shared-ptr-from-raw-pointer-in-c++/Fri, 26 Apr 2019 16:03:30 +0800https://shawnliu.me/post/creating-shared-ptr-from-raw-pointer-in-c++/Recently I was writing some C++ code and one thing I needed to do is to pass a pointer of an object within itself to another function. At first, I was using raw pointers and everything worked fine. However, later as I decided to use std::shared_ptr instead, I encounter error where the same pointer was freed twice. After some search online, I discovered that I need std::enable_shared_from_this. What’s causing the problem?https://shawnliu.me/post/obtaining-wildcard-certificates-from-lets-encrypt/Mon, 26 Nov 2018 15:52:16 +0800https://shawnliu.me/post/obtaining-wildcard-certificates-from-lets-encrypt/In 2017, I wrote a post on how to set up Let’s Encrypt certificates together with NGINX. Back then, Let’s Encrypt hadn’t added support for wildcard certificates. Althought it’s not a big issue, it does mean you have to generate a new certificate each time you want to add a new subdomain to the certificate. Earlier this year, Let’s Encrypt finally added support for wildcard certificates. Therefore, I recently changed to a wildcard certificate.https://shawnliu.me/post/a-few-use-of-macros-in-c-c++/Fri, 26 Oct 2018 20:58:47 +0800https://shawnliu.me/post/a-few-use-of-macros-in-c-c++/If you’ve written C/C++ code, you must know and probably have used macros. You may have defined handy macros to compute min and max. However, there are more ways macros can be used in C/C++ projects. I will try to describe a few which I frequently use in this post. Header guard The most frequent use of macro for me is definitely acting as the header guard. In C/C++ code, you cannot redefine the same function or variable twice.https://shawnliu.me/post/passing-reference-with-std-ref-in-c++/Wed, 24 Oct 2018 22:27:56 +0800https://shawnliu.me/post/passing-reference-with-std-ref-in-c++/When you first learn C++, you probably know about reference. It allows you to pass parameter as if you are passing by value while having a similar effect as passing a pointer. Sometimes, however, this may not work as you expect. In this post, I will discuss how to pass a reference when the normal way doesn’t work. Passing parameters to std::thread and std::bind If you have used std::thread or std::bind, you probably noticed that even if you pass a reference as parameter, it still creates a copy instead.https://shawnliu.me/post/manual-network-configuration-in-a-kickstart-install/Mon, 01 Oct 2018 21:16:45 +0800https://shawnliu.me/post/manual-network-configuration-in-a-kickstart-install/In a previous post, I described how to automatically install a CentOS system with the help of Kickstart. In that post, I provided a link to the Kickstart file I used back then. It is truly automatic, including the network configurations. However, sometimes we may want to manually configure certain things, e.g. the network, while performing an “automated” installation. In this post, I use network as an example to demonstrate how to configure things manually during a Kickstart installation.https://shawnliu.me/post/cuda-as-a-language-in-cmake/Sun, 26 Aug 2018 06:48:54 +0800https://shawnliu.me/post/cuda-as-a-language-in-cmake/If you have ever created or built a CUDA CMake project in the past, you have probably heard about the FindCUDA CMake module. Prior to CMake 3.8, it has been providing this module to support compiling CUDA code and linking with CUDA libraries. This approach is not perfect though. Unlike other libraries, CUDA provides not only some libraries but a compiler as well. To let CMake use nvcc to compile something, the FindCUDA module provides macros like cuda_add_executable (similar to add_executable).https://shawnliu.me/post/setting-up-a-kvm-virtual-machine-on-centos-7/Thu, 09 Aug 2018 16:43:36 +0800https://shawnliu.me/post/setting-up-a-kvm-virtual-machine-on-centos-7/I’ve been running virtual machines for many years. In the past, I mainly used VirtualBox as my hypervisor since it’s free and it can be installed on Linux/macOS/Windows. Recently, I was asked to install a VM on a remote server at work. This time, I decided to set up a KVM-based VM instead. This post describes the steps to set up KVM and create a VM. Terminologies Before we dive into the steps, let me first introduce and explain a few terms.https://shawnliu.me/post/night-light-mode-for-monitors/Tue, 26 Jun 2018 20:01:03 +0800https://shawnliu.me/post/night-light-mode-for-monitors/Recently I switched from Ubuntu 16.04 to Fedora 28 on my desktop PC. After the installation, I noticed that newer version of GNOME has added a night light feature which is the same as night shift on macOS and iOS and the f.lux software. The feature is great. However, it can only adjust the color temperature of the video output and the monitor’s brightness stays constant. As I work besides a big window, I need higher brightness during the day and lower brightness at night.https://shawnliu.me/post/fixing-a-strange-grub-related-filesystem-error/Thu, 21 Jun 2018 17:37:02 +0800https://shawnliu.me/post/fixing-a-strange-grub-related-filesystem-error/I’m running a dual-boot desktop system with Fedora 28 and Windows 10. Today a strange error happened to my GRUB2 installation. This post documents how I fixed the error. Here is what happened: I was installing the Logitech software for my mouse in Windows because I need it to tune a new mousepad. After installation completed, it asked me to reboot the system. I did that and when the system boots up again, the normal GRUB boot menu didn’t appear.https://shawnliu.me/post/mixing-c-and-cpp-code/Sat, 03 Mar 2018 19:28:53 +0800https://shawnliu.me/post/mixing-c-and-cpp-code/C and C++ are two closely related programming languages. Therefore, it may not come as a surprise to you that you can actually mix C and C++ code in a single program. However, this doesn’t come automatically when you write your code the normal way. In this blog post, I will describe what makes it possible to mix the two languages and how to achieve that in code. C and C++ linkage When you compile a source file, the function names are mangled.https://shawnliu.me/post/linux-shared-folder-management-done-right/Fri, 22 Dec 2017 01:07:09 +0800https://shawnliu.me/post/linux-shared-folder-management-done-right/If you’ve ever collaborated with someone on a Linux machine or worked as a system administrator on a multi-user Linux system, chances are you’ve set up shared folders for groups of people to share data. It’s quite easy on Linux. Well, kind of. I mean, you can simply create a group, add relevant users in, and set the shared folder’s owner and permission correctly. However, this approach has a problem: new files and directories created inside a shared folder doesn’t inherit the owner and permission of the shared folder itself.https://shawnliu.me/post/vertex-centric-graph-processing-the-what-and-why/Sun, 17 Dec 2017 01:41:55 +0800https://shawnliu.me/post/vertex-centric-graph-processing-the-what-and-why/Vertex-centric graph processing is a new programming abstraction/model for writing graph algorithms. I got to know vertex-centric graph processing because my final year project is centered around this topic. I find it to be an interesting idea that enables large-scale graph processing with ease. In this post, I would like to talk about what vertex-centric graph processing is and why it is useful. What is vertex-centric graph processing? Vertex-centric graph processing, as the name suggests, is a new programming abstraction for processing graphs that is centered around the vertices.https://shawnliu.me/post/nvidia-gpu-id-enumeration-in-linux/Sun, 10 Dec 2017 14:17:28 +0800https://shawnliu.me/post/nvidia-gpu-id-enumeration-in-linux/I’ve been working on a server with 3 NVIDIA GPUs for my internship work. A few months ago, I noticed that the GPU IDs are different in different situations for the same GPU. Therefore, I decided to take a look at how the IDs are enumerated. What is the issue exactly? There are potentially two different GPU ID orders that we can get from nvidia-smi and the CUDA library. Below demonstrates two different ID enumeration schemes observed on the server.https://shawnliu.me/post/io-buffering-mechanisms-in-c-cpp/Tue, 29 Aug 2017 21:31:01 +0800https://shawnliu.me/post/io-buffering-mechanisms-in-c-cpp/Today I was fixing a bug for the logger in a project that I’m currently working on. The bug is that when the process is killed or it crashed, the logger sometimes doesn’t write all the logs into the log file with the fprintf function call. I knew this is an I/O buffer problem, but I didn’t know too much about how to control it and what are the I/O buffering modes besides flushing the buffer every time.https://shawnliu.me/post/migrating-to-a-new-mac/Sun, 02 Jul 2017 22:19:15 +0800https://shawnliu.me/post/migrating-to-a-new-mac/I recently bought a new MacBook Pro 13-inch (2017, w/o TouchBar) to replace my aging MacBook Air 13-inch (mid-2013). My old machine only has 4GB memory and 128GB SSD, and it’s slowing things down sometimes. When I was setting up my new Mac, I decided to have a fresh install of macOS. This allows me to think about what should be migrated from an old Mac to a new Mac.https://shawnliu.me/post/linux-like-key-binding-in-iterm-2/Sun, 21 May 2017 16:41:50 +0800https://shawnliu.me/post/linux-like-key-binding-in-iterm-2/I work regularly on both Linux and Mac. When I work in terminal on Linux, I use the Alt key very often. Alt is the prefix of many terminal shortcuts and I also use the Alt key in tmux to switch between panes within a window. This key, however, behaves differently on Mac. In this post, I will describe how to have Linux-like key binding in iTerm 2 for the Alt key.https://shawnliu.me/post/using-lets-encrypt-for-nginx-on-centos-7/Mon, 01 May 2017 11:37:33 +0800https://shawnliu.me/post/using-lets-encrypt-for-nginx-on-centos-7/Recently, I set up several web apps on my VPS. As these apps requires user credentials to access, I have to make sure they are secure. Therefore, I decided to enable HTTPS on these sites. Also, since Chrome shows HTTP sites as “Not secure” starting from Chrome 56, it’s a good idea to enable HTTPS support on every website you are running. In this post, I will describe how to enable HTTPS support in NGINX with SSL/TLS certificate from Let’s Encrypt on CentOS 7.https://shawnliu.me/post/initialization-forms-in-c++11/Mon, 03 Apr 2017 18:50:53 +0800https://shawnliu.me/post/initialization-forms-in-c++11/Recently I was learning C++ and I got to know the new brace initialization in C++11. Therefore, I decided to take a look at various initialization methods and the reason behind this new syntax. According to my research, in C++03, the initialization methods are a bit tricky and have certain restrictions. Let’s take a look at how you initialize objects in C++03 and how it could be improved with C++11.https://shawnliu.me/post/c-c++-project-organization/Fri, 24 Mar 2017 12:30:24 +0800https://shawnliu.me/post/c-c++-project-organization/In the past few years, I’ve written some very small C/C++ programs. Back then, I didn’t pay much attention to project organization as one project is just a few source files, headers, and a Makefile. Recently I’ve been doing a lot of development with C++ in a larger project. This got me into thinking how a C/C++ project should be organized. In this blog post, I wanna talk about how a moderate size C/C++ project could be organized to keep things in order.https://shawnliu.me/post/how-to-use-cmake/Tue, 21 Feb 2017 20:06:08 +0800https://shawnliu.me/post/how-to-use-cmake/Today I started working on an existing CUDA project and the first thing I did was to try to build the project. However, there were some problems with the CMake system which is used in the project. Therefore, I decided to learn more about CMake and fix the problem. I would like to share what I’ve learnt here in this blog post. Specifically, I wanna talk about how CMake should be used, which can be a bit confusing for someone using CMake for the first time.https://shawnliu.me/post/shadowsocks-on-centos-7-minimal/Wed, 15 Feb 2017 16:50:40 +0800https://shawnliu.me/post/shadowsocks-on-centos-7-minimal/I have a cheap VPS in the U.S. running a shadowsocks service. When I first bought the VPS, it was running Debian 7 and the Shadowsocks implementation I picked back then was its Python implementation. I used this combination mostly because I’m familiar with Debian and Python. However, since the VPS only has 64MB memory and limited disk space, the Python implementation wasn’t ideal for my server. Therefore, I deleted the old system and installed CentOS 7 minimal.https://shawnliu.me/post/kickstart-centos-7-installation/Mon, 06 Feb 2017 10:13:22 +0800https://shawnliu.me/post/kickstart-centos-7-installation/Recently I was playing around with kickstart files. Kickstart files contains specification on how a CentOS (or RHEL) OS should be installed. For example, it may be used to specify the keyboard layout, the system language, the time zone, the disk partition, the network, etc. Basically it can set anything that you could set with the graphical installation interface. If a kickstart file is referenced during OS installation, then the OS can be installed automatically without any human intervention.https://shawnliu.me/post/list-of-ip-in-ansible-templates/Wed, 01 Feb 2017 19:19:09 +0800https://shawnliu.me/post/list-of-ip-in-ansible-templates/I was developing a simple Ansible role for deploying elasticsearch today and I encountered a problem with Ansible’s Jinja2 template. Basically, in the configuration file for elasticsearch, I need something like discovery.zen.ping.unicast.hosts: ["192.168.0.1", "192.168.0.2", "192.168.0.3"] At first, I cannot extract IP addresses for all Ansible hosts. Later I figured out a way to do that, but I don’t know how to turn a list of IPs into a string with the IPs all quoted.https://shawnliu.me/post/how-domain-name-works/Mon, 16 Jan 2017 00:06:22 +0800https://shawnliu.me/post/how-domain-name-works/It has been about 5 years since I bought my first domain name to set up my old WordPress blog. Over the years, I’ve bought several domains and set up some common services with my domains (e.g. website, email). I understand how hard it is to understand the concepts related to domain name when you don’t have much network knowledge. Therefore, I would like to explain some of the important concepts related to setting up your own domain names in an easy-to-understand way.https://shawnliu.me/post/ansible-101-for-configuration-management/Tue, 20 Dec 2016 21:52:19 +0800https://shawnliu.me/post/ansible-101-for-configuration-management/Ansible is a tool for configuration management over SSH. I gave it a try recently and it has proven to be very useful since I need manage a set of servers. Here I would like to briefly introduce Ansible to you. Installation Ansible is written in Python, on macOS it can be installed with $ brew install ansible Getting read for using Ansible SSH keys Since Ansible talks with your machines with SSH, it’s recommended that SSH keys are preconfigured on target machines for password-less login.https://shawnliu.me/post/creating-and-publishing-python-modules/Mon, 19 Dec 2016 13:55:19 +0800https://shawnliu.me/post/creating-and-publishing-python-modules/I once developed a simple script to fetch course schedule and plan a timetable for myself. Later, I turned it into a Python module and published it to PyPi. Now, the module can be installed with just a simple pip install command. I would like to share with you how I created a Python module and successfully uploaded it to PyPi. Creating Python modules Creating Python modules are easier than most of you might think.https://shawnliu.me/post/gaining-knowledge-from-the-internet/Sun, 18 Dec 2016 15:09:17 +0800https://shawnliu.me/post/gaining-knowledge-from-the-internet/Internet has been around for more than 20 years. It connects people from all over the world and makes knowledge sharing easier than ever. However, it doesn’t mean all the knowledge just come to you. Over the past few years, I’ve been thinking and learning the ways to gain knowledge from the powerful Internet, and I would like to share some of the best ways with you. Two ways of gaining knowledge I think there are 2 ways of gaining knowledge.https://shawnliu.me/post/homebrew-without-xcode/Thu, 15 Dec 2016 22:38:03 +0800https://shawnliu.me/post/homebrew-without-xcode/Recently I was cleaning up the disk space of my MacBook Air. I realize that Xcode takes up around 10G space and I’m not doing any iOS/macOS development. I know that the only thing I need from Xcode is its command line tools for homebrew. Therefore, I would like to uninstall Xcode while keeping the command line tools. Remove the Xcode application First thing to do is to remove the Xcode.https://shawnliu.me/post/controlling-interactive-process-in-unix/Thu, 01 Dec 2016 20:58:47 +0800https://shawnliu.me/post/controlling-interactive-process-in-unix/In Unix systems, interactive processes refer to processes which are created from terminal sessions. This blog post aims to explain the basics of controlling interactive process in Unix systems. Foreground and background process There are two types of interactive process in Unix systems, namely foreground and background process. Foreground process is a process that runs continuously and only returns control to the shell after it finishes. Background process, on the other hand, refers to a process that runs in the background while the user has the control of the shell.https://shawnliu.me/post/qr-code-for-2-factor-authentication/Wed, 30 Nov 2016 11:36:44 +0800https://shawnliu.me/post/qr-code-for-2-factor-authentication/I was migrating all my 2 factor authentication keys to my new phone the other day, and I realize that the representation of different keys in Google Authenticator is different. In Google Authenticator, there are 2 places for labels for each key. One is above and one is below. Some of the keys I have doesn’t show the label above. Therefore, I decided to take a look at the reason behind it.https://shawnliu.me/post/using-makefile-for-task-automation/Mon, 17 Oct 2016 20:03:07 +0800https://shawnliu.me/post/using-makefile-for-task-automation/If you’ve written some C/C++ code, or you are familiar with *nix systems, you’ve probably heard about Makefile. Makefile is a file that contains some commands for use with the GNU make build automation system. A typical use would be $ ./configure && make && make install However, the usage of Makefile is not limited to just build automation. You can actually use Makefile for many kinds of task automation.https://shawnliu.me/post/configuring-openmp-and-mpi-on-mac/Tue, 30 Aug 2016 14:59:10 +0800https://shawnliu.me/post/configuring-openmp-and-mpi-on-mac/I’ve been taking the course Parallel Computing this semester, and I’m also participating in some student cluster competitions. Therefore, I wanted to configure OpenMP and MPI on my Mac to run some OpenMP and MPI programs. This post describes how I successfully configured OpenMP and MPI with Homebrew. Pre-requisite Make sure you have Homebrew installed on your Mac. Homebrew is a package manager for macOS. It is in some way similar to package managers in Linux distributions, such as apt-get and yum.https://shawnliu.me/post/my-thoughts-on-knowledge-management/Sun, 03 Jul 2016 12:53:03 +0800https://shawnliu.me/post/my-thoughts-on-knowledge-management/As a person who browses the Internet everyday, I have to say that there are so many contents created every day. As a result, I have to come up with ways to handle all these information without overloading my brain. In this post, I would like to talk about the tools I use on a daily basis and my workflow. RSS RSS, which stands for Rich Site Summary, is a way to summarize contents on the Web.https://shawnliu.me/post/reverse-proxy-with-custom-domain/Thu, 02 Jun 2016 11:16:59 +0800https://shawnliu.me/post/reverse-proxy-with-custom-domain/Have you ever run into the situation where you want to access a server behind multiple layers of firewall and with no public IP? If so, please read on. In this post, I’ll be sharing how I can successfully access my server with a custom domain. Pre-requisite You are gonna need a few things before we move on, they are listed below: Your server (of course!) A publicly accessible server A domain name (optional) Here let’s assume the public server has an IP of 1.https://shawnliu.me/post/building-mxnet-in-docker/Mon, 30 May 2016 16:27:53 +0800https://shawnliu.me/post/building-mxnet-in-docker/This blog shows how I built the MXNet deep learning framework in a docker container. If you follow this guide, you should be able to build your own. You can find the Dockerfile I created at the end of this post. Pre-requisite Since I’m building a GPU-enabled MXNet, the following pre-requisites are needed: nvidia-docker NVIDIA GPU (with support for Compute Capability >= 2.0) Building MXNet To create a docker container, you start with a base image.https://shawnliu.me/post/philosophy-for-everyone/Thu, 26 May 2016 17:30:08 +0800https://shawnliu.me/post/philosophy-for-everyone/What is philosophy? Have you thought about this question before? Do you feel that it’s a mysterious subject? Personally, that’s what I thought philosophy was before I took the Introduction to Philosophy MOOC from University of Edinburgh on Coursera. I think it’s partly because I never considered philosophy that useful in daily life. However, I think the course completed changed my mind about philosophy. I realized that philosophy is indeed important, fundamental and everywhere.https://shawnliu.me/post/student-supercomputing-competitions/Sun, 08 May 2016 22:53:50 +0800https://shawnliu.me/post/student-supercomputing-competitions/Hey guys. I’ve been participating in student supercomputing competition related events for about half a year now as a team member of the Nanyang Technological University Supercomputing Team. And I would like to share with you some information about the available competitions. There are in total 3 international supercomputing competitions for undergraduate students. Some of them are even available for high school students or postgraduate student. They are ASC Student Supercomputer Challenge (ASC for short), ISC-HPCAC Student Cluster Competition (ISC for short), and SC Student Cluster Competition (SC for short).