SMALLPEARL

Setting up pure Angular 21 Workspace as a Monorepo

2026-01-28T00:00:00+08:00

This document outlines the steps required to setup a pure Angular CLI (that is, without nx or other higher level tools) monorepo project. These instructions are for Angular 21, the current version at the time of writing this. The document is divided into two halves -- the first half covers setting up the workspace for multiple projects which are a mix of applications and libraries. The second half lists the steps required to configure vitest, the default test framework that Angular now uses.

Workspace configuration

Install @angular/cli globally.
Create an empty Angular workspace.
```
$ ng new <workspace-name> --no-create-application
```
This will create the folder <workspace_name>. All future commands are to be run from this folder.
Generate the required app projects.
```
$ ng g application <application_name> --project-root apps/<application_name>
```
Our monorepo has a folder structure where all applications are stored in separate subfolders under the parent folder apps.

Generate the required library projects.

$ ng g library <library_name> --project-root libs/<library_name>

Update <workspaceRoot>/tsconfig.json adding the paths to the libraries created. This will allow you to refer to the libraries you generated using a friendly name rather its entire path. For example, if one of the libraries you generated was named shared, you can update tsconfig.json as:
```
{
  compilerOptions: {
    "baseUrl": ".",
    "paths": {
      "shared": ["libs/shared/src/public-api.ts"]
    }
  }
}
```
If you generated other libraries, add them to the paths as above. This will allow you to import a component from the library as:
```
import { SharedComponent } from 'shared';
```

Also note that we added baseUrl. This directs tsc to refer to the paths relative to the value of this setting (It would assume paths are absolute otherwise).

Vitest Configuration

Angular 21 uses the quicker vitest for running its unit tests. However it's not set up when the workspace is created perhaps because we created an empty workspace to begin with. You're expected to do this explicitly.

Vitest is setup primarily by creating two files in each project in the workspace -- vitest.config.ts & test-setup.ts. The latter filename is just a convention, you can give it any name you want as long as you refer to the correct file in vitest.config.ts.

Also, if you would like to run and debug specific tests from your IDE (such as VSCode), you need to install a few additional packages. Let's assume that that's the environment that we're aiming for here. Adding these packages and tweaking the configuration to use these, would not in anyway degrade ng test command line. So we might as well do this.

$ npm install @analogjs/platform @analogjs/vite-plugin-angular @analogjs/vitest-angular -D

AnalogJS has published a few packages that makes the process of configuring the Angular's TestBed testing workhorse for invocation from the ng command line and the IDE's testing plugin seamless. Once you install this do the following (You need to repeat this for every project in the workspace):

Update angular.json such that the builder for test architect's point to @analogjs/vitest-angular:test rather than the angular native unit test builder.
```
"test": {
  "builder": "@analogjs/vitest-angular:test",
}
```

Now when you issue ng test, it will use the @analogjs package as the default builder. But for @analogjs/vitest-angular:test to work we need to go through a couple of more steps listed below.

Create vitest.config.ts in each project's root folder. It's content should look like this:

import { defineConfig } from 'vite';
import angular from '@analogjs/vite-plugin-angular';

export default defineConfig(({ mode }) => ({
  plugins: [angular()],
  test: {
    globals: true,
    setupFiles: ['src/test-setup.ts'],
    environment: 'jsdom',
    include: ['src/**/*.{test,spec}.{js,mjs,cjs,ts,mts,cts,jsx,tsx}'],
    reporters: ['default'],
  },
}));

Note how test-setup.ts is referred to in setupFiles. We'll create this file in the next step.

Create the file test-setup.ts in project's src/ folder. It's content should look like this:
```
import '@analogjs/vitest-angular/setup-snapshots';
import { setupTestBed } from '@analogjs/vitest-angular/setup-testbed';
import '@angular/compiler';

setupTestBed();
```
setupTestBed() will call TestBed.initTestEnvironment(), which readies the TestBed for running the individual suite of tests.

Create tsconfig.spec.json in the project's root folder.

{
  "extends": "../../tsconfig.json",
  "compilerOptions": {
    "outDir": "../../out-tsc/spec",
    "types": ["vitest/globals"]
  },
  "files": ["src/test-setup.ts"],
  "include": ["src/**/*.d.ts", "src/**/*.spec.ts"]
}

Pay attention to the files key. It's value explicitly adds test-setup.ts to the list of files to be transpiled.

If you're using VSCode, install the Vitest official plugin. This will allow you to run the tests from within the IDE by clicking on the green play button on the left margin.

Spectator Support

If you would like to use Spectator for your unit tests, you need to tweak your vitest.config.ts like this:

  ...
  test: {
    ...
    server: {
      deps: {
        inline: ['@ngneat/spectator'],
      },
    },
  }

The additional configuration is server -> deps -> inline. This informs vitest that the specified modules should be transformed and resolved by Vite as part of the test running environment. This is important as this will ensure that Spectator will use the same Angular TestBed environment that is initialized by the setupTestBed() call in test-setup.ts. The relevant docs are here.

Evolution of Web Frameworks

2024-01-18T00:00:00+08:00

This article was originally published in Acadamia Sinica's LSL Blog.

What is a Web Frameowrk

Web framework is a software framework designed to support the development of web sites and applications. They provide a standard way to build and deploy web applications and provide software tools to automate the overhead associated with common activities performed in web development such as database access, user management, access control, session management, etc. The fundamental objective of a web framework is to promote code reuse by reducing repetitive tasks into a common set of reusable components.

In this article, we will trace the evolution of web frameworks from its early days to its current form, while identifying the reasons for the transition in technology. As there are literally hundreds of web frameworks, covering all of which is impractical, we will only be using major transition points in the technology trends.

Early days

In its very early days, the Web consisted of a web server (CERN's httpd and later Apache) that listened for requests from web browsers and returned an HTML response. The servers could be configured such that based on different request paths, it would return different responses. This was fine for static content such as product information, documents or chapters of a book. For example different documents could be indexed by by different request paths and similarly different chapters could be index by the chapter number, thereby allowing the client to retrieve the document or chapter of interest directly. This path organization also helped reduce the amount of data being transferred resulting in better response times.

But as the web became interactive, whereby it allowed users to input data and the response dynamically generated based on the input, this mechanism of serving statically defined pages became quite limiting.

CGI

The response from the server designers to accommodate this requirement was to allow external programs to be executed from the server. Such external programs are passed certain arguments (via the standard program command line) and its response, written to stdout, is returned back to the client as the response. Arguments included data that is input by the user in the browser (supplied as form data) alongwith additional standard arguments such as cookies, session identifier, request origin, browser language, etc. The response would be complete HTML text that the browser can render as if it came from a static content. The type, content & structure of the arguments that were passed to the external programs and their response was standardized and this standard was referred as CGI or Common Gateway Interface.

CGI brought dynamism to the otherwise static web. Also, since these were pure processes, they could be written in any language. C, C++, Java, Python, Perl, you name it. Even as a shell script! As long as the language allowed you to parse the command line arguments and write back to the stdout, it could be used to write a CGI script. This made it immensely flexible and powerful.

CGI, however, also had two major drawbacks.

Since each script was executed as a separate process which was created as and when the request was received, it became quite resource intensive. This is because creating a new process is a very resource intensive operation -- both in terms of memory and CPU. It was observed that more often than not the initial setup expense of a CGI script far exceeded the cost of the actual computation done by it. As the web grew exponentially and consequently the number of HTTP requests handled by a server grew, CGI as a means of providing interactivity to the web proved to be not a scalable solution.

Another big drawback of CGI was security, or lack thereof. Since the CGI programs are processes launched by the web servers, they inherit the security context of its parent process, the web server. So if a web server is running with the highest OS privileges (as was often the case) the CGI script will also be run as a high privilege process. Since user input (typically form input from the browser) was passed directly to the CGI programs, attackers found ways to embed commands in this input and cause these commands to execute at the server resulting in serious security breaches.

Apache Server Modules (mods)

One of the solutions to address the scalability issues that plagued CGI was to extend the server via Multiprocessing Server Modules or server mods. Mods are dynamically loadable modules to which requests can be delegated from the main server. Being dynamically loaded, modules run within the Apache server process. Mods can handle the requests either by launching new processes or separate threads as per the mod's design. Once the request is processed, mods would send the response back to the core server which then sends it back to the client.

Though MPM sounds like a generic term, it was first developed by Apache and one of their first implementations was mod_perl, a module that allowed Perl scripts to be executed from within the Apache server without having to start a Perl interpreter for each incoming request. The same design approach was later adopted in other servers such as NGINX.

Note that by convention MPMs are named with the prefix mod_ followed by the mod's name.

Another benefit of mod_perl was that it could emulate a CGI environment, which allowed existing Perl CGI scripts to be executed resulting in a performance boost while help retain them without having to be re-written. Since PHP, one of the first dedicated web framework languages, was written in Perl, mod_php could be developed leveraging on the embedded Perl engine and that too resulted in achieving great scalability improvements over a pure CGI approach.

While running within the server process brings efficiency, it also raises a significant issue. An error in a module, can bring the entire server down. Also, modules cannot have isolated security restrictions that are stricter than the core server. Finally, modules have to be predefined in the server configuration to be loaded and cannot be dynamically restarted (though later versions of Apache added this capability) for maintenance activities such as updates & upgrades.

It merits mentioning that in spite of these shortcomings, Apache modules have proven to be popular and continue to be in use even today. However, their use has been limited more towards administering the Apache server via embedded scripts than act as a server for a production web application.

FastCGI

Another solution to address the scalability issues, was FastCGI.

In FastCGI, instead of creating a separate process for each external script, it uses a persistent process to serve multiple requests. The HTTP server and the FastCGI process communicate via a Unix or TCP socket. The incoming request details, that were earlier passed as arguments to the CGI script, are now passed to the FastCGI script via the socket connection. The response from the script, received via the same socket connection is then relayed back to the browser client.

Note that using a socket connection allows multiple requests to be handled simultaneously by making multiple connections to the FastCGI process or by launching multiple FastCGI processes (each bound to a different socket, conforming to predefined range of port numbers) or a combination of both.

Since new processes need not be created for each incoming request, this addressed the scalability issues faced by the CGI mechanism. Besides, being a socket connection, the FastCGI process could be run in a different security context than the webserver thereby improving the security risks of the CGI approach.

Another benefit of separating the FastCGI process from the web server is that these processes can be independently restarted after an update (or to reclaim memory caused by leaks) without resulting in total outage to the static portions of the website.

Unfortunately FastCGI also has its limitations, which are quite similar to that of Apache modules. Though comparable to MPMs, FastCGI did not quite gain the popularity of modules. Perhaps because it was a binary protocol and developed by a private company whereas Apache & its modules was community driven and therefore an open standard.

Frameworks that apply design patterns

The idea behind design patterns is to divide software application's various elements into specific contexts so as to bring clarity to the overall application design. Once an element's context is clear, its goal can be clearly defined which in turn will help define its inputs & outputs.

As the web grew, web applications became more and more complex. Consequently, software design patterns that helped manage the complexity of large software application inevitably made its way into web applications too. Design patterns such as Model-View-Controller that were used to bring clarity into large-scale software applications were applied to web frameworks.

As web applications started being architected & designed at a high level using design patterns, it became evident that languages such as Perl, which hitherto were being used to provide dynamism to websites, are inadequate. More expressive programming languages were required to translate the design into code that captured the pattern as closedly as possible. This is the genesis for the many modern web frameworks that we see in use today. Some prominent examples(just a random sampling and by no means exhaustive) are ASP.NET, JBoss(Wildfly), Ruby on Rails and Django.

While some of these frameworks use the MVC pattern, others such as Django employ what is called a MVT (Model-View-Template) pattern. But the idea is the same -- provide clear context to different portions of the code and assign specific roles & responsibilities to each context, resulting in consistent design throughout a large application codebase.

Besides scalability, the advantage of these frameworks is the minimum security that they guarantee. By forcing the application to fit within the framework, web applications inherit the minimum security guarantees that the framework offers. For instance, most of these frameworks, by default require production sites to be served over HTTPS. Also, operations that result in changes to the backend data require additional precautions (often provided by default by the framework) in the code all but eliminating risks posed by common low-cost attacks by hackers -- Cross Site Scripting & Cross-site request forgery being two of them.

Note that all these frameworks run on their own application servers, which is shown in the diagram. These application servers communicate with the HTTP server using a clearly defined protocol, which is conceptually not too different from how CGI & FastCGI works. But all the application servers support network transport as the communication medium, which allows application servers to be hosted in separate machines and often replicated across many machines for massive scalability.

AJAX & jQuery

While neither AJAX nor jQuery are true frameworks, they merit a special mention in our discussion on the evolution of web frameworks.

As web applications became more interactive, the browser side application UI became more complex. Some of these UI interactions required context sensitive data related to user input to be retrieved from the server. For example a user searching for a street should only have to enter the first few letters of the street name and the UI should respond by listing all streets that match the letters input by the user.

Delivering such complex & time sensitive UI interaction by re-rending the entire webpage would be a wastage of server resources. A more efficient approach would be to query the server for the relevant data (in this case street names matching the user input) and use the values returned to modify the HTML page already rendered by a previous full page request's response.

In order to facilitate this, browsers provided a dedicated JavaScript API to send a background request to the server and process the results. Browsers also allowed such JavaScript code to modify the displayed HTML page contents in the background by adding new HTML elements or removing and amending existing HTML tags.

This dedicated API is called AJAX, which is an abbreviation for Asynchronous JavaScript and XML. As the name suggests AJAX requests to server typically returned JavaScript objects or XML data. JavaScript objects are encoded in JSON, allowing them to be directly loaded into JavaScript scripts and worked upon.

Whereas AJAX provides a standard way to retrieve data from a remote in the background, adding & modifying the HTML page using the plain JavaScript API bundled with the browser was proving to be a pain. Enter jQuery. jQuery provided a fast & small library of wrapper functions that made the job of finding, modifying & adding HTML elements a breeze.

Together, these two key innovations suddenly made web applications interactively ricker and closer than ever before to statically installed desktop applications.

However AJAX & jQuery were not without their drawbacks.

Server logic became increasingly complex as it now consisted of two kinds of response types. Some requests resulted in full HTML pages being returned whereas other requests returned pure data encoded as JSON (or in some rare cases XML). Of course application servers returning pure JSON data simplifies applications server code as it does not have to deal with composing messy HTML page fragments. I call it messy as HTML being a declarative language is not easy to be broken down into components and maintained independently.

At the same time client computers became more & more powerful. Power that was not being fully utilized by the browser rendering a web application. Also, JavaScript became an omnipresent capability in web browsers which made writing web applications that rely on it less and less risky to being rendered non-functional.

What if all the work required to construct the frontend of a web application UI be delegated to the client thereby restricting the server code to just managing the data and the business logic? This would simplify the server code which would help achieve even greater scalability while making maitenance that much easier. Moreover, since UI is managed entirely from the browser, it can be made even more responsive.

This line of thought resulted in the next major evolutionary phase in web frameworks.

JavaScript Frontend Frameworks

JavaScript based frontend frameworks render the entire HTML page locally at the browser. They do this by running JavaScript code, which is retrieved from the server. This is the contemporary evolutionary phase that we are in right now.

Example for JavaScript based frontend frameworks are Angular, React & Vue.js, amongst others.

In these frameworks, the application server is responsible for primarily two functions:

Provide access to the data & manipulate them through a well defined interface
Deliver the latest application code as JavaScript bundles

The JavaScript bundles contain the entire application code and run in the browser fetching essential data from the server and building an HTML user interface for the user. This HTML is then injected into the browser window resulting in the application UI. When user invokes an application feature or triggers an action (like clicking a button or a link or entering a value into a text field), the relevant code goes on to build a new interface (or modifies the existing one) by composing new HTML fragments and injecting them into the browser's view. This process goes on endlessly until the user quits the app.

This mechanism completely removes the logic of building the HTML response for a given request from the server. Server only has to provide a means to access the data & in cases where the data needs to be changed, validation of the inputs before changing it. The server also incorporates the business logic (such as cascading effects on related data) of what happens when data is thus modified.

An added benefit of using JavaScript frontend frameworks is that it allows clear separation between frontend & backend code bases. This has other couple of significant consequences.

Firstly, it allows development of these two to progress independently of each other as long as the interface remains the same.

Secondly, this model allows organizations to hire independent developers with specialized skills suiting the domain thereby achieving higher code quality and programmer productivity.

Lastly, pure JavaScript based frontends can be written in such a way that it can be converted into a mobile app with minimal effort. And such mobile apps can be deployed in both Android and iOS platforms, from a single code base with minimal additional effort to handle platform specific vagaries.

Conclusion

I hope this article gave you an insight into way the web has evolved to result in what we have today. This is by no means complete and overlooks a key trend -- mechanisms for live interaction via browser. By mechanism for live interaction, I'm referring to features such as live chat and streaming. Both these features require data to be pushed by the server to the client, after the client initiates a request expressing interest in the data. If time permits, we will explore this in a little bit more detail in one of the future articles.

Async Python Part 2 – A deeper look

2023-08-09T00:00:00+08:00

This article was originally published in Acadamia Sinica's LSL Blog.

Background

In the previous article we demonstrated with an example how asynchronous code can achieve massive scalability with the same hardware resources compared to a threaded model. In this example we would go a little bit deeper into async programming, exploring its implementation to further deepen our undertanding of the subject.

Concurrency vs Parallelism

Before we get into the nitty gritty details, it is useful to have some clarity on the concept of Concurrency vs Parallelism. In computing, concurrency is defined as the ability to execute more than one program or task simultaneously. Parallelism, on the other hand, is the ability to execute multiple tasks or programs at the same time. The difference is subtle and may not be obvious to the casual reader. It means that while concurrency doesn't require that the multiple tasks initiated be active at the same time, parallelism requires just that.

The following picture should help make things clearer.

As you can see, concurrent systems, while allowing multiple tasks to run simultaneously, at any point in time only one task has the CPU's attention. However, in a parallel system, multiple tasks run parallel to each other, usually on independent CPUs (or CPU cores, which are presented as independent CPUs to the OS).

Async Python programs are concurrent and not parallel. Even if you run the Python program on a machine with multiple CPUs, an async Python program runs on a single thread. Exception is when you explicitly spawn threads and schedule a function to run on that thread. But threading is a different subject and it's not the topic of this post.

Concurrency in Python

As with the previous article, where we demonstrated async benefits with an example, we'll use an example to illustrate concurrency in Python. However, we will not be using the Python provided async constructs. Rather, we will build our own concurrent runner to which various tasks can be queued and the runner's scheduler will dispatch them. But before we jump into that, let's establish some background information.

Consider the following Python code, consisting of two tasks.

def task1():
  pass

def task2():
  pass

If we were to run these, it would look like this(horizontal axis represents CPU execution time). That is, when task1 completes, task2 gets going. And if you have task3, task4, etc, they all follow the task prior to each.

task1()
       task2()
              task3()

Assume that we can implement the tasks' algorithms by dividing them into multiple subtasks:

# Task 1
def task1_part1():
  pass

def task1_part2():
  pass

def task1_part3():
  pass

def task1_part4():
  pass

# Task 2
def task2_part1():
  pass

def task2_part2():
  pass

def task2_part3():
  pass

def task2_part4():
  pass

def task2_part5():
  pass

In the concurent system, all we do is to queue up these tasks with a task scheduler and it would execute the individual sub tasks, interspersing each to maximise the processor utilization. The order in which the individual tasks are executed are not quite predictable and depends on the scheduler algorithm and other factors.

task1_part1()
             task2_part1()
                          task1_part2()task1_part3()
                                                    task2_part2()
                                                                 task1_part4()

As you can see, at any point in time, only one subtask has the Python interpreter thread's attention. When that subtask is completed, another subtask is scheduled to run. This subtask may be the part of the same or different parent task.

One key requirement for this to work is that the subtasks should be able to go through the start/stop/restart cycle while the execution state remains the same between the stop and restart steps. This is not obvious from our sample code above as it does not have any state shared between the different subtasks, but in a real world scenario this would not be the case.

Now we can write code to ensure the consistency of this state (by abstracting it as an object and get all the subtasks to update this object for any changes they make), but then it's additional housekeeping that we have to do. That too for every function to which we would like to apply this divide-and-conquer approach. A lot of work indeed.

What if this housekeeping is provided by the language framework itself?

Well that's what Python async provides you. As we saw in a previous article, it turns Python language provides just the right infrastructure for this -- the generators framework! To recap, generators are functions (function objects, rather) that can suspend their execution to provide a result to its caller and then resume from where it suspended when asked to do so. Sounds like our concurrent system's requirement, doesn't it?

This generator framework, together with the new keywords, async & await and a scheduler to manage various async tasks and schedule them based on some rules, provide the developers with a neat framework to build efficient & scalable algorithms that better make use of the processor resources.

Our own concurrency framework

As a last step to deepen our understanding of how generators and async programs are related, let us try and implement a simple concurrent framework ourselves. Note that is an oversimplified version and it's purpose is purely educational.

What we do in this example is to create a bunch of generators, and queue them up with our own async scheduler. The scheduler pops each queued up generator and runs it until it runs out of values to generate. It then moves on to the next queued up generator and so on. This is repeated until the queue is exhausted.

# async2_1.py
import random
from collections import deque
from time import sleep

def wait_for_io_result():
    sleep(random.random()/2)

def task1_part1():
    print("task1_part1.1")
    wait_for_io_result()
    yield
    print("task1_part1.2")
    wait_for_io_result()
    yield

def task1_part2():
    print("task1_part2.0")
    wait_for_io_result()
    yield
    print("task1_part2.1")
    wait_for_io_result()
    yield
    print("task1_part2.2")
    wait_for_io_result()
    yield

def task1_part3():
    print("task1_part3.1")
    wait_for_io_result()
    yield
    print("task1_part3.2")
    wait_for_io_result()
    yield
    print("task1_part3.3")
    wait_for_io_result()
    yield
    print("task1_part3.4")
    wait_for_io_result()
    yield
    print("task1_part3.5")
    wait_for_io_result()
    yield

def task1_part4():
  yield from task1_part1()

class MyConcurrentRunner:
    def __init__(self) -> None:
        self.tasks = deque([])

    def create_task(self, coro):
        self.tasks.append(coro)

    def run(self):
        while self.tasks:
            next_task = self.tasks.popleft()
            while True:
                try:
                    next(next_task)
                except StopIteration:
                    break


if __name__ == '__main__':
    runner = MyConcurrentRunner()
    runner.create_task(task1_part1())
    runner.create_task(task1_part2())
    runner.create_task(task1_part3())
    runner.create_task(task1_part4())
    runner.run()

What we have done is create a few generator functions, each of which yields a value after sleeping a random interval. The sleep is not relevant and is there to simulate a time consuming IO intensive background operation. Also, the sleep slows down the print messages, which helps us track and visualize the program's run progress much better.

Then, we created a class MyConcurrentRunner, which allows generators to be queued to it. Thereafter when its run() method is invoked, we execute the queued generators in a round-robin fashion by calling the built in next() with the generator popped from the queue.

While this is greatly oversimplified, Python's async framework works in a similar manner. The MyConcurrentRunner here is the event loop in async Python and the generator functions that we created are the async functions that we write. While here we explicitly queue the generators to our MyConcurrentRunner instance, in async Python, this happens implicitly when we await on an an async routine by calling it using the await func() syntax.

Of course Python's event loop is way more complex. It allows other types of async routines such as timer routines to be queued up, which when they expire the eventloop scheduler calls. The queued up routines are also scheduled in a more first-to-finish-first-served manner than our fixed order of queuing up.

Also, Python's event loop provides I/O multiplexing using the system's standard selectors module. This is what allows the TCP echo server in our previous article's code to scale immensely.

A note about selectors. Selectors are an OS level concept and Python selectors are a wrapper around this OS API. If you want to learn more on selectors just type from the terminal:

$ man select
$ man poll

Man pages for both these functions describe them as synchronous I/O multiplexing. Essentially these functions allow sockets to be queued up with OS so that when there's any network activity around the socket (in/out traffic or connection), it's reflected in the corresponding file descriptor returned by the function. The functions also allow you to specify a timeout value (which can be set to 0), after the elapse of which the function returns irrespective of the queued up sockets' state.

Python selectors use this API and the async counterparts of socket API are implemented using the selectors library.

Proof is in the pudding

It might surprise you to learn that the concurrency example above is a lot closer to the async event loop implementation than you may think. Async functions are represented by a coroutine object, much like generators are represented by a generator object. And coroutine is a specialization of a generator object. The primary difference between the two is that coroutine can be invoked with arguments (multiple times) whereas generator only emits values until it runs out of values to emit.

If you're still not convinced that generators form the basis for async Python, perhaps this little experiment will finally provide you the proof.

Consider the code below.

# async2_2.py
import dis

def gtask2():
    yield 10

def gtask1():
    yield from gtask2()

async def atask2():
    pass

async def atask1():
    await atask2()

# Disassemble both the generator task and the async task
dis.dis(gtask1)
dis.dis(atask1)

Run the above program from the terminal. You'll get the following output:

$ python async2_2.py

gtask1 disassembly:-
  8           0 LOAD_GLOBAL              0 (gtask2)
              2 CALL_FUNCTION            0
              4 GET_YIELD_FROM_ITER
              6 LOAD_CONST               0 (None)
              8 YIELD_FROM
             10 POP_TOP
             12 LOAD_CONST               0 (None)
             14 RETURN_VALUE
atask1 disassembly:-
 14           0 LOAD_GLOBAL              0 (atask2)
              2 CALL_FUNCTION            0
              4 GET_AWAITABLE
              6 LOAD_CONST               0 (None)
              8 YIELD_FROM
             10 POP_TOP
             12 LOAD_CONST               0 (None)
             14 RETURN_VALUE

Look at the bytecode at line 8 of both the functions. YIELD_FROM!

The YIELD_FROM in atask1 corresponds to the await atask2() call and is essentially retrieving values emitted by the atask1 coroutine, which in turn is a specialized generator object.

Conclusion

I hope this discussion helped deepen & clarify your understanding of asynchronous programming in Python and convince you of its close relationship with generators. While the content here is mostly conceptual and meant to give you an overview of how an async program is managed by the runtime, it's model is surprisingly similar to how things work in the real async event loop.

Hariharan is a software developer turned entrepreneur running his own software business in Taiwan. He has over 30 years of hands-on development experience in domains ranging from device drivers to cloud based applications and still enjoys coding.

Async Python - Part 1

2023-06-01T00:00:00+08:00

This article was originally published in Acadamia Sinica's LSL Blog.

Introduction

In the previous article on Python Generators we saw how generator functions are essentially objects that represent the function, execution of which can be suspended and resumed. The resumption of the function happens when it has new values to emit to the caller, which it does via the yield statement. With this background knowledge, in this (and quite likely a few) article we will start examining asynchronous programming in python.

Asynchronous programming is not an easy concept to grasp. Quite frankly it can be frustrating at times and in my experience, it is one of those subjects that requires repeated attempts to get your thinking in tune with the way the paradigm works. This is because our brain is inherently trained to think sequentially connecting information that occurs in a time series to form the complete picture. Asynchronous programming does not work like that. Computing occurs asynchronously, emitting results as and when they are available in an unpredictable sequence.

There are many articles on the web that introduce async programming in Python and the ones I have seen mostly start with explaining the basic syntax keywords (async & await) tha enable async programming and then go deeper. While this works for most people, I feel this approach works best for those who have had some exposure to async programming concepts (Javascript, which is inherently an async language) prior.

My approach is going to be a little bit different. First I want to start with a sample code that shows the differences & benefits of async programming proving what it can result in compared with the more conventional paradigms of concurrency. Once we make the case for async, we can move on to explain a production scenario where this can be applied.

In a subsequent lesson we will link the async constructs with what we learned in the Python Generators article and see how the former was adapted to implement the latter. And if necessary we'll cover the syntax, but hopefully with a different viewpoint than the many articles already existing on the web.

Example

The code below implements a TCP/IP echo server that will echo back the text written to it, but after converting it to upper case. There are two implementations, one traditional thread based server and another an async programming based single-threaded server.

Then we implement a client that will exercise this echo server. Since the server implementation is opaque to the client, we can run the client on the two server implentations and observe & compare the results.

Thread based server

#!/usr/bin/env python
# threaded_server.py
import socketserver
import threading
import signal
import time

quit_event = threading.Event()


def quit_handler(*args):
    quit_event.set()


class MyTCPHandler(socketserver.BaseRequestHandler):

    def handle(self):
        # self.request is the TCP socket connected to the client
        self.data = self.request.recv(1024).strip()
        print('Connection from {}'.format(self.client_address[0]))
        print('Data received: {0}'.format(self.data))
        # DB request
        time.sleep(0.100)     # sleep 100 milliseconds, simulating IO
        # just send back the same data, but upper-cased
        print('Sending back: {0}'.format(self.data.upper()))
        self.request.sendall(self.data.upper())


class MyServer(threading.Thread):
    '''Start a TCP/IP server at <host>:<port>'''
    def __init__(self, host: str, port: int, *args, **kwargs) -> None:
        super().__init__(*args, **kwargs)
        self._tcp_server = socketserver.TCPServer((host, port), MyTCPHandler)

    def stop(self):
        self._tcp_server.shutdown()
        self.join()

    def run(self) -> None:
        self._tcp_server.serve_forever()


if __name__ == "__main__":
    server = MyServer('localhost', 9090)
    server.start()

    print("Press CTRL+C to stop..")
    signal.signal(signal.SIGTERM, quit_handler)     # Termination signal
    signal.signal(signal.SIGINT, quit_handler)      # CTRL+C
    quit_event.wait(None)

    print('Quitting..')
    server.stop()
    print('Done.')

The code above is adapted from the sample in the official doc here. Besides the signal handlers to instrument a graceful exit, one key change is that before returning the uppercase version of the written string to the client, there is a delay of 100 milliseconds. This is to simulate an IO operation. To illustrate with a real-world example, you can imagine this delay as the time required to execute a complex SQL statement or time required to run face detection algorithms in a remote server.

Async server

#!/usr/bin/env python
# async_server.py; use Python v3.7+
import asyncio


class EchoServerProtocol(asyncio.Protocol):
    def connection_made(self, transport):
        peername = transport.get_extra_info('peername')
        print('Connection from {}'.format(peername))
        self.transport = transport

    def data_received(self, data):
        message = data.decode()
        print('Data received: {!r}'.format(message), flush=True)

        task = asyncio.get_running_loop().create_task(self.simulate_network_io(message))  # noqa
        task.add_done_callback(self.handle_network_io_result)

    async def simulate_network_io(self, message: str):
        # DB request
        return await asyncio.sleep(0.1, result=message.upper())

    def handle_network_io_result(self, task):
        message = task.result()
        print('Sending back: {!r}'.format(message))
        self.transport.write(bytes(message, 'utf-8'))
        self.transport.close()


async def main():
    # Get a reference to the event loop as we plan to use low-level APIs.
    loop = asyncio.get_running_loop()

    server = await loop.create_server(
        lambda: EchoServerProtocol(),
        'localhost', 9090)

    async with server:
        await server.serve_forever()


if __name__ == "__main__":
    asyncio.run(main())

Like threaded_server.py, async_server.py is also based on the code from the official doc here. Notable change is the delay introduced simulating an IO bound operation (simulate_network_io method of EchoServerProtocol). The implemtation of the server looks very different from the threaded server, but this understandable as we're using async programming for it. Also there are couple of tricky async stuff going on there, but we don't have to cover that now.

Client

Now that we have two identical servers, one threaded and another async, we need to write our client. Since we're going to compare the behavior of these two servers, to make a fair comparison, we need to use the same client. Without going too much into the details, here is the code.

#!/usr/bin/env python
# client.py; use Python 3.9+
from typing import Tuple, Any
import asyncio
import sys


class EchoClientProtocol(asyncio.Protocol):
    def __init__(self, message, on_con_lost):
        self.message = message
        self.on_con_lost = on_con_lost

    def connection_made(self, transport):
        transport.write(self.message.encode())
        print('Data sent: {!r}'.format(self.message))

    def data_received(self, data):
        print('Data received: {!r}'.format(data.decode()))

    def connection_lost(self, exc):
        # print('The server closed the connection')
        self.on_con_lost.set_result(True)


async def client(address: str, port: int) -> Tuple[asyncio.Transport, asyncio.Future[Any]]:     # noqa
    # Get a reference to the event loop as we plan to use
    # low-level APIs.
    loop = asyncio.get_running_loop()

    on_con_lost = loop.create_future()

    MESSAGE = 'the quick brown fox jumps over the lazy dog'
    transport, protocol = await loop.create_connection(
        lambda: EchoClientProtocol(MESSAGE, on_con_lost),
        address, port)

    return (transport, on_con_lost)


async def run_clients(count: int, address: str, port: int):
    transports = []
    futures = []
    for i in range(0, count):
        transport, con_lost_future = await client(address, port)
        transports.append(transport)
        futures.append(con_lost_future)

    # Wait until all the client connections are closed
    # Then close all the transports created.
    try:
        await asyncio.gather(*futures)
    finally:
        # close all transports
        for t in transports:
            t.close()


if __name__ == "__main__":
    iter = 100
    if len(sys.argv) < 3:
        print("Usage: client <address> <port> [<iterations>]\n")
        sys.exit(1)
    else:
        address = sys.argv[1]
        port = sys.argv[2]
    if len(sys.argv) > 3:
        iter = int(sys.argv[3])
    print(f"Creating {iter} connections to echo server at {address}:{port}...")
    asyncio.run(run_clients(iter, address, port))

The client takes three arguments, two of which are required. These are address of the server and its port. The last argument is the number of requests to make to the server during the test, which defaults to 10000, if omitted.

Execution results

Assuming that you have the three files saved, you can run it like this.

From Terminal 1:

$ python threaded_server.py

From Terminal 2:

$ python ./client.py localhost 9090 10000

Note the time it takes to process 10000 requests. Now do the same with the async server.

From Terminal 1:

$ python async_server.py

And from Terminal 2:

$ python ./client.py localhost 9090 10000

On my MacBook Pro i7 4870HQ (4 cores/8 hyperthreads), I get the following results.

1000 iterations:

Server type	Time
Threaded server	1.891 seconds
Async server	1.812 seconds

10000 iterations:

Server type	Time
Threaded server	36.851 seconds
Async server	20.066 seconds

As you can see as the number of iterations is increases, async server is able to scale up much better than the threaded version.

NOTE

The timings above were generated by profiling the code with cProfile, Python's builtin profiler. The commandline for this looks like this:

$ python -m cProfile ./client.py localhost 9090 100000 | grep 'function calls'
103731 function calls (88004 primitive calls) in 20.066 seconds

Piping the output to grep is to filter out the noise of profiler's individual function call timing details, which is not relevant for our experiment.

Discussion

A server program typically is designed to serve multiple clients. Each client could make a different request with possibly different parameters. This requires that the server separate the serving context of each request so that it can isolate the client's request. One of the simplest ways of isolating execution context is to use processor threads which inherently provides this capability.

However, when serving the request involves operations that are IO intensive, the thread is essentially blocked, waiting for the IO operation to complete. When the thread is blocked waiting for an IO to complete, if other requests come in, server program would spawn another thread to serve the new request. (Of course, no server would allow infinite number of requests and often there would be an upper limit on the number of concurrent threads that it'll spawn. But let's leave that aside). And as more and more threads are created, the OS scheduler has to spend more time managing the context of the threads which itself starts to eat up more and more CPU cycles. That is scheduler (which is also a thread) starts hogging the CPU.

In the case of the async server, there's only one thread that's serving all the client requests that are coming in. This thread is the one with an active event loop. Don't get overawed by the terminology -- think of an event loop as a do-while-true loop that takes an input queue (conceptual queue) of tasks and executes them one by one. As an when any of these tasks have completed their previous operation, they yield a result back to their caller. When this happens the loop moves on to the next ask in the queue. And if there are no more tasks, it just waits until new tasks are added to queue or is asked to stop waiting, whichever happens first.

We can apply the above concept to explain our example above. As each request comes in, a request to the database server is made via asyncio.get_running_loop().create_task(self.simulate_network_io(message). As the API name suggests, this creates a new task and puts it in the queue, which executes it immediately. However the code for simulate_network_io, consists of a sleep. Event loop, while executing this, gains control back and moves on to the next task while the just executed simulate_network_io is within it's sleep interval. When the sleep interval is exhausted the task is once again given attention in the loop (look keeps track of all its scheduled tasks), and this time the loop checks & notices that we have added a completion routine to the task and therefore invokes it. While at it, it also notices that the sleep returned a result to the task, which is used to write the response back to the client. You can refer to documentation of sleep here for details.

The last two paragraphs is essentially the crux of asynchronous programming. You can visualize it as a conceptual queue of tasks with one person (the loop) assigned to take each one out of the top of it and attend to it. The tasks themselves tell the loop which of its operations could take a long time to complete. When the loop encounters these operations, it starts the operation and and puts the task back in the queue. This process repeats until the person is asked to stop serving the task queue.

Real-world analogy

We can use a real-world analogy to explain this. Imagine being at home and doing the following:

------------------- time ----------------------->
Start the washing machine with dirty clothes
    Put kettle on the stove for tea
        Put two slices of bread in the toaster
            Kettle siren goes off
            Make tea
                Toaster goes pop
                Spread peanut butter on the toast and have it with tea
                    Washing machine goes 'beep beep'
                        Hang the clothes and while sipping tea and having your toast.

This is essentially what happens in async. You're doing many things concurrently and seemingly parallelly.

Client

An observant reader might have noticed that the client code uses asyncio. Why? This implementation detail merits a brief explanation.

To stress the server we need to make as many simultaneous connections to the server as possible. Only this would simulate a real-life server where client connections come from different machines all connected to the network. One way to simulate this would be to spawn n number of threads. But on a personal computer, with limited CPU cores, spawning thousands of threads will place excessive load on the OS scheduler, which in turn would affect our threaded server (remember the server is also running on the same machine). To get around this, we employ async programming, making the client code the same for both tests. We don't have to go deeper than this at the moment. As we progress with our discussion, the reasons for this will become clearer.

Wrapping up

To facilitate the async features, what one needs is a function whose state can be suspended and resumed. That means functions whose state (code & data) can be frozen so that it can be resumed later. This is exactly what generator functions provide. The only difference being that in generator functions, control is transferred back to the function via a call to the next(fn) iterator function. In async, the async function suspends itself when another async function is called with (via await) and control is transferred back to it when the awaited function is complete. And if the await is for a network operation, the event loop thread can make itself available for next async routine (via await) while the IO is in progress. This capability is what allows our example code to schedule 1000s of async routines in one event loop.

Hope this gives you a gentle introduction to asynchronous programming and some of its benefits. Bear in mind that asyncio is not a solution to everything that is wrong with multithreaded approach. The benefit comes at a cost -- code complexity. Debugging async code is hard as you would have to primarily resort to log messages to identify and troubleshoot any issues. A threaded program, on the other hand, allows you to break its execution, freezing all threads and step through the code using a traditional debugger.

If you have questions or comments, feel free to post them to the FB page where a link to this article is posted. I'll try to check them daily and answer any questions you may have.

Python Generators - A Look Inside

2023-05-29T00:00:00+08:00

This article was originally published in Acadamia Sinica's LSL Blog.

Background

One of the most powerful features of Python is its generator functions. It also happens to be a feature that leads to confusion among its users; it can seem quite mysterious as to how generators do their magic. A decent understanding of how generators work inside the Python interpreter can be quite helpful in unraveling this mystery and should go some way in helping you write better code.

Generator Functions

Before we get into the details of this article, it’s a good idea to recap what generator functions are and the best way to do that is by an example.

def my_range(start, end):
  # A function that mimics python’s standard range()
  index = start
  while index < end:
    yield index
    index += 1


for i in my_range(0, 10):
  print(i)

Nothing fancy about the code. It's just a limited implementation of Python’s standard function range(). A code example that has been extensively used in many Python books & articles. The magic happens when the interpreter encounters the keyword yield. During bytecode generation, this keyword tells the interpreter that the enclosing function is a generator function and it goes on to treat the function a little differently from a regular function. And this different treatment is what makes generator functions achieve its magic and it's this that is the topic of this article.

Python Functions

Before we dig into the details of generators, it's necessary to have a basic understanding of how the Python interpreter executes the source code that we provide. Specifically its functions.

A function in Python's interpreter has two distinct phases -- initialization and evaluation.

Initialization

All functions in Python will have a corresponding frame object. Initialization is the phase where the interpreter creates this frame object. The frame object consists, among other things, of the following:

code object
stack
execution pointer
variables, local & global

You can imagine this as a C struct with a few appropriately named members for each of the above(and more). The inspect module documents this here.

When a function is initialized, a corresponding frame object is created. Arguments to the function are passed as frame.f_local[arg1], frame.f_locals[arg2], etc. As the function is executed, the instruction pointer to the function's bytecode is advanced via the frame.f_lasti member.

Evaluation

Evaluation is actually a fancy way of saying running the function. And running the function involves passing the frame object created in the initialization phase to a function in the interpreter named PyEval_EvalFrameDefault. If you look the linked source code, you can see that it's second parameter is PyFrameObject f*, the frame object that we're talking about.

The PyEval_ function takes the frame object passed to it and starts executing the function's bytecode pointed to by frame.f_lasti.

With this background we can go into generator functions and their functioning.

Generator Functions

Initialization

The initialization phase for generator functions is slightly different from a normal function. First of all, invoking a generator function does not result in its immediate execution. Rather a generator object is created. A frame object is still created, but it's stored as a member of the generator object. Specifically, its gi_frame member.

You can verify this by the following code (using the my_range() defined above) from the Python console:

>>> a = my_range(0, 5)
>>> type(a)
<class 'generator'>
>>> dir(a)
[..., 'close', 'gi_code', 'gi_frame', 'gi_running', 'gi_yieldfrom', 'send', 'throw']

Evaluation

Evaluation or running of a generator function occurs when the builtin method next() is called with the generator object as its argument.

So using our original my_range() generator function:

>>> a = my_range(0, 5)
>>> next(a)
0
>>> next(a)
1
>>> next(a)
2
>>> next(a)
3
>>> next(a)
4
>>> next(a)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

What's going on here is that every time we call next(a), it calls PyEval_EvalFrameDefault with the generator.gi_frame pointer. However, PyEval_ pauses its execution when it encounters a yield statement. Note that I used the verb pause rather than stop. PyEval_ returns placing the yielded value from the generator making it available for the callee. At this point it's important to note that the frame (generator.gi_frame), unlike regular functions, is not destroyed. Rather, it's preserved with its execution state. This allows Python to resume the frame when it's passed to PyEval via the subsequent call to next().

The above cycle repeats until the generator function decides that it has reached its end of the yield iteration logic and returns. When this happens, next() generates a StopIteration exception. At this point the gi_frame held by the generator object can safely be destroyed.

Generators in a for loop

In the real world we hardly use generator functions using the next() builtin. Instead we would iterate through the generator using a for loop. The for loop is actually a syntactic sugar on top of the next() and StopIteration exception, that presents a coding style similar to the traditional loop to the user. That is the for loop iterates through the generator's yielded values until next() throws a StopIteration exception.

Communicating with generators

Generator functions also support other features such as allowing the callee to communicate with the generator function by sending it values. These are supported by methods .send() & .throw() which you can see in the output of dir().

>>> import inspect
>>> inspect.getmembers(a)
[...., ('send', <built-in method send of generator object at 0x1053decf0>), ('throw', <built-in method throw of generator object at 0x1053decf0>)]

From an implementation point of view, send() pushes its argument to the top of the generator's stack and starts executing the generator's frame from the last paused state. This would cause a statement such as a = yield x to pop the argument to send() and place it as the value of the variable a. Similarly, throw() would transfer control back to the generator function's paused code, throwing the exception object passed as its argument.

Conclusion

I hope the above discussion gives you a better idea of how Python generators are implemented by the interpreter that leads to its magic of continuously yielding values without having to complete its processing and store the result in a temporary local variable.

You can perceive a generator function as a closure as it preserves the function's execution pointer and its state, allowing it to be resumed. Come to think of it, this behavior is quite similar to threads in a multi-threaded environment. That is a threading system that doesn't require any support from the host CPU or the operating system. Of course one critical difference is that in a true multi-threaded system, threads are pre-emptively scheduled in and out. That doesn't happen here. Instead the threads (read, generator functions) have to yield control back to the callee for the function context switch to occur. This essentially is the behavior of a cooperative multitasking system. I'm not sure if anyone can remember this term -- remember Windows 3.x?!

This framework of stopping & resuming functions is quite critical to implementing another powerful feature of the language. A feature that has become quite useful in achieving massive scalability in this network-centric world. I'm talking about asynchronous programming, provided by the Python package asyncio. I hope to cover that in a follow-up article, contents of which will be easier to understand if you have a good grasp of the concepts discussed here.

Enabling click on Material Badge component

2023-02-14T00:00:00+08:00

In Angular Material, badge elements are declared as a directive on the host element around which the badge is to be placed. Depending on the value of other related directives, the badge is placed before or after and above or below the element. But if you try to bind a click handler on the host element itself, you'll find that the click only works on the host and not on the badge element which is typically positioned outside the host element.

As it turns out, the badge implementation creates a child element to the host which is positioned accordingly with the content specified as value to the matBadge directive. This child element has the style pointer-events: none, which is why you don't get the click events. So to enable click events, all you have to do is to remove this style or change its value to inherit. You can use Javascript to do this.

So for the following HTML:

<div id="todo-actions" matBadge="!" matBadgeOverlap="false" matBadgeColor="warn">&nbsp;</div>

This code will do the trick:

    const todoBadge: HTMLElement = document.querySelector("#todo-actions .mat-badge-content");
    if (todoBadge) {
      todoBadge.style.pointerEvents = "inherit";
    }

Note that the .mat-badge-content is the inner element created by the matBadge directive. Using #todo-actions .mat-badge-content ensures that the CSS style is changed for the specific badge element.

Finally, if you're using Angular, put the JS code in your AfterViewInit handler so that the matBadge directive would have created its child element for the badge.

Only drawback with this approach is that it's not guaranteed to work if the matBadge implementation changes as that can happen during a major framework overhaul.

Paginated Responses in NgRx/Data

2020-12-22T00:00:00+08:00

This is a quick post on how to support paginated GET responses in NgRx/Data. Like many other posts, done primarily with the objective of recording my research output while sharing it with others who might need it. (Quick google searches did not yield much on this front).

Background

I have a Django backend with a REST API that generates paginated responses. Essentially the responses look like this:

response = {
    "count": 1,
    "next": null,
    "previous": null,
    "results": [
        {
            "id": 1,
            "firstName": "Peter",
            "lastName": "Parker"
        },
        {
            "id": 2,
            "firstName": "John",
            "lastName": "Smith"
        }
    ]
}

The key point here is the response.next field. It is set to the next page URL, if more data is available at the remote. Using this allows us to avoid have to keep track of a page variable (though page index can be derived from this string) and provide infinite scrolling experience in the front-end.

Out of the box NgRx/data does not support this kind of responses. However, it provides extensive customization facilities which can be used to to support responses such as above.

How to do it

The class within NgRx/data that issues the actual REST API requests and handles its response is Entity DataService, the default implemention of which is provided as DefaultDataService<T>. As will be obvious from its name, it's a generic class, a concrete version of which will be generated for each registered entity. These concrete classes are created by the factory class, DefaultDataServiceFactory.

So the solution is to first create a customized version of DefaultDataService<T>, and then provide a customized DefaultDataServiceFactory, which would create this custom DefaultDataService<T> instead of the default one. Then tell NgRx to use the custom class factory by registering it in the modules provide array.

Here's the full code:

/**
 * DataService class that handles Django REST Framework's paginated
 * responses.
 */
export class PaginatedDataService<T> extends DefaultDataService<T> {
  private nextPageUrl: string = '';
  private count = 0;

  constructor(
    entityName: string,
    http: HttpClient,
    httpUrlGenerator: HttpUrlGenerator,
    config?: DefaultDataServiceConfig
  ) {
    super(entityName, http, httpUrlGenerator, config);
    this.nextPageUrl = this.entitiesUrl;
    this.count = 0;
  }

  // Override to store nextUrl as well as map response.results.
  getAll(): Observable<T[]> {
    return this.execute('GET', this.entitiesUrl).pipe(
      tap(response => {
        this.nextPageUrl = response.next;
        this.count = response.count;
      }),
      map(response => response.results)
    );
  }

  // Override to store nextUrl as well as map response.results.
  getWithQuery(queryParams: QueryParams | string): Observable<T[]> {
    const qParams =
      typeof queryParams === 'string'
        ? { fromString: queryParams }
        : { fromObject: queryParams };
    const params = new HttpParams(qParams);
    return this.execute('GET', this.entitiesUrl, undefined, { params }).pipe(
      tap(response => {
        // response.next would include the queryparams.
        this.nextPageUrl = response.next;
        this.count = response.count;
      }),
      map(response => response.results)
    );
  }

  /**
   * Get next page of results. Or empty array if remote data is
   * exhausted.
   */
  getMore(): Observable<T[]> {
    if (!this.hasMore()) {
      // or throwError?
      return of([]);
    }

    return this.execute('GET', this.nextPageUrl).pipe(
      tap(response => {
        this.nextPageUrl = response.next;
        this.count = response.count;
      }),
      map(response => response.results)
    );
  }

  /**
   * Returns total number of objects
   */
  totalCount() {
    return this.count;
  }

  /**
   * Returns boolean indicating if there's more data at server.
   */
  hasMore(): boolean {
    return !!this.nextPageUrl;
  }
}

/**
 * Custom DataServiceFactory that creates PaginatedDataService<T>,
 * instead of the DefaultDataService<T>
 */
@Injectable()
export class CustomDataServiceFactory extends DefaultDataServiceFactory {
  constructor(
    http: HttpClient,
    httpUrlGenerator: HttpUrlGenerator,
    @Optional() config?: DefaultDataServiceConfig
  ) {
    super(http, httpUrlGenerator, config);
  }

  create<T>(entityName: string): EntityCollectionDataService<T> {
    return new PaginatedDataService<T>(
      entityName,
      this.http,
      this.httpUrlGenerator,
      this.config
    );
  }
}

@NgModule({
  imports: [
    StoreModule.forRoot({}, {}),
    EffectsModule.forRoot([]),
    EntityDataModule.forRoot(entityConfig),
    StoreDevtoolsModule.instrument({ maxAge: 25, logOnly: environment.production }),
  ],
  providers: [
    { provide: DefaultDataServiceFactory, useClass: CustomDataServiceFactory },
  ]
})
export class AppStoreModule {
  constructor() {}
}

Disclaimer

A word of caution. I only tested this briefly as I decided to not use NgRx/Data for my own needs at the moment. I had spun my own implementation of something similar earlier, which is simpler and importantly, quite straightforward to use without having to come to grips with all the NgRx terminology.

I spent couple of hours today morning looking to NgRx/Data to see if it can replace my own data service implementation. While it definitely can, for my immediate needs, it's a little too much learning, not to mention wading into uncharted waters, for me.

But this should provide adequate clues for someone who wants to seriously use NgRx/Data and is stuck at getting around the lack of pagination support in it.

Controlling Pelican Output Files

2020-10-22T00:00:00+08:00

By default Pelican generation process results in a few files. Besides the site's homepage, a page each showing the list of articles, list of tags, list of categories and a page with a list of authors are produced.

Some sites may want to suppress some of these files. For instance, for a company website, only the the site homepage & the article list (if the site incorporates a blog) may be necessary.

You can achieve this by the following setting in pelicanconf.py:

ARCHIVES_SAVE_AS = ''
AUTHOR_SAVE_AS = ''
AUTHORS_SAVE_AS = ''
CATEGORY_SAVE_AS = ''
CATEGORIES_SAVE_AS = ''
TAGS_SAVE_AS = ''
TAG_SAVE_AS = ''

Specifying empty string values as the output filenames for the relevant HTML page results in the generation of the file being skipped.

Search Engine Optimization

2020-10-22T00:00:00+08:00

These are some of the notes from my research on SEO techniques for a website. 'Notes', and not a definitive guide. This is based on the guidelines outlined in Google's own docs and therefore is primarily geared towards improving a site's ranking in Google search.

Make sure your site is registered with google crawler. Though Google crawler will eventually find your site, it's always better to register it formally so the process of indexing your site can begin immediately.

To verify search for "site:yoursite.com" and see if you get results that lists the links to permanent pages within your site as separate search results. If you do, site is registered. If not, goto Search Console and register your site. Registering a site requires making a TXT record on DNS with a unique string that the search console will provide. This process is to verify that you're indeed the owner of the domain that you're trying to register.
Provide a sitemap file with links that point to the important pages in your site.
Make sure your server handles If-Modified-Since request header and returns 304 response code if the site has not been modified since the date value provided to the If-Modified-Since header. To test this, make a regular request to your site:
```
$ curl -I https://www.yoursite.com
HTTP/1.1 200 OK
Server: nginx/1.19.0
Date: Thu, 22 Oct 2020 01:45:53 GMT
Content-Type: text/html
Content-Length: 19963
Last-Modified: Wed, 21 Oct 2020 08:12:22 GMT
Connection: keep-alive
Vary: Accept-Encoding
ETag: "5f8fed66-4dfb"
Accept-Ranges: bytes
```
The Last-Modified response header indicates that last time the site's files were modified. Now make another request supplying this time in the request header.
```
$ curl -I https://www.yoursite.com  -H "If-Modified-Since:Wed, 21 Oct 2020 08:12:22 GMT"
HTTP/1.1 304 Not Modified
Server: nginx/1.19.0
Date: Thu, 22 Oct 2020 01:47:27 GMT
Last-Modified: Wed, 21 Oct 2020 08:12:22 GMT
Connection: keep-alive
ETag: "5f8fed66-4dfb"
```
Server should now respond with status code 304 indicating that the site has not changed since last modification. When crawler sees this response, it'll stop crawling the site again thereby reducing needless bandwidth consumption -- both for the server as well as for the search engine crawler.
If your server has portions that are not to be index by the search engine and you can't control access to them via user authentication, use robots.txt file hints to inform the crawler to skip these pages. Typically this will not be required, but it's good to know. You can also robots.txt to stop the crawler from indexing your site's static assets such as images & scripts.
Avoid pages that require URL parameters. Instead pages should use simple, clear and well terminated URLs.

Localizing Pelican Sites

2020-10-20T00:00:00+08:00

Localization

Localization is provided via the i18n_subsites plugin.

As mentioned before, localization is provided via the plugin i18n_subsites. This plugin employs jinja2.ext.i18n to achieve this.

Localization works at two specific levels:

Theme templates. Templates often contain hardcoded text and this needs to be localized. This is espcially the case for our design as the homepage contains text that is not pulled from any of the site articles or pages.
Pages & articles. These are typically published in markdown and these need to be localized as well.

We'll examine the localization process for both below.

Theme templates

Theme files are localized using Babel, which is used to extract the localizable content wrapped in {% trans % & {% endtrans %} blocks within the templates. Babel includes a tool - pybabel that helps you do this.

There are two distinct use-case scenarios. Initial creation of the translation files and subsequently updating them for any changes that occur during routine maintenance.

Initial creation

Extract the localizable strings to a message template file. This will create a new messages.pot file replacing the existing template file and its contents.
```
$ cd theme
$ pybabel extract --mapping ../babel.cfg -o messages.pot .
```
Note that since we're localizing themes, we have to work from the theme subfolder.
Create the translations catalog based on a generated message template file for each language that you want to support (specified in pelicanconf.py). In our case, we only have one extra language to support - zh.
```
$ cd theme
$ pybabel init --input-file messages.pot --output-dir translations/ --locale zh
```
Compile the created .PO catalog files to its compiled format -- a .MO file. This command is to be issued from the project home folder.
```
$ cd <project-home>
$ pybabel compile --directory theme/translations/
```

This would compile all the .PO files in theme/translations/ into a binary format .MO file. With this done, when pelican content command is issued to generate the site, for each supported language, a replican of all the theme HTML files for the specified language would be generated and placed under the <lang-code> folder under ./output.

So since we only support zh as an extra language, a zh language version of our site would be generated under ./output/zh folder.

Updating for changes

There's one additonal step involved. As we maintain the site, new strings will be added and/or existing ones would be tweaked. In this scenario we need to update the message template file (use pybabel extract command above) and the translations for each supported language. For this use the pybabel update subcommand.

Regenerate the messages template .POT file to capture all the new strings. Do this:
```
$ cd theme
$ pybabel extract --mapping ../babel.cfg -o messages.pot .
```

Update the new & modified strings in the .PO file.

$ cd theme
$ pybabel update --input-file messages.pot --output-dir translations/ --locale zh

Translate the strings and compile the PO file to MO file.

$ cd <project-home>
$ pybabel compile --directory theme/translations/

The above command ensures that existing translations are preserved while new strings are added to the PO file. Any changes to an existing string, will be marked by commenting off the original translation and the string will be marked as Fuzzy. You can disable this with the --no-fuzzy-matching option.

Pages & articles

Localization of pages and articles works in a different way. For each article or page that you wish to localize, create a duplicate file. You may wish to use the language code as the filename suffix to identify the file as being the different language version of the same file.

Then in all the files, specify the same slug value but different value for lang. The same value connects the artifacts as different language versions of the same article or page. For example, if youave a page about.md, you may want to create about_zh.md for its Chinese version and specify the same value for the slug field in both thereby connecting the two pages as being the same but for two different languages.

Brief Overview of Pelican

2020-10-19T00:00:00+08:00

Recently I switched this website from Django to a static site. This article gives an overview of the core concepts of Pelican, covering areas that are not directly captured in the official docs. This is written primarily to solidify my own understanding and also act as a reference for myself when I need to maintain it after a prolonged gap.

Static site generators

Static site generators work like the app servers that eventually output the HTML code to the client browser, but do this during the build process which the user invokes to assemble the site from its various fragments. Once generated, the resultant HTML, scripts & image files can be uploaded to an HTTP server such as Apache or NGINX.

Content

Pelican broadly divides the site content as belonging to one of the two types:

Pages: These are content, types of which are not constantly created over time. Examples include about page, privacy policy page, contact page, etc. These types of content are created once and usually they remain intact except that details inside them are updated when required.
Articles: Articles are running content, new types of which are constantly published and are presented to the visitor in a chronological (usually reverse) order. Examples being blog articles, news items, etc.

Site content is typically written in markdown. They can also be written in other popular publishing formats such as reStructuredText or even HTML.

File Metadata

Whatever format is used, special meta information is placed in content that controls how it's rendered into the final HTML. Example for meta information include tags, categories, date, title, description, etc. This doc shows how to place metadata in content files.

Themes

The presentation of the content is controlled by themes. Themes are essentially templates (using Jinja) which contain special placeholders into which content is supplanted during the generate process resulting in the final HTML output. These primarily consist of the following:

A site index template. This becomes the site's homepage or index.html.
A template for rendering the pages
A template for rendering articles
A few other templates such as one showing list of tags, articles belonging to a specific tag, list of authors, etc.

Some other points:

Themes also include the required CSS & JS files as well as any images.
Depending on your required site design constraints, you can either design your own theme templates or use one of the many pre-designed themes.
What theme file stands for what type of purpose is based on its filename.
Since they use Jinja templating system, they use the {% block %}, {{ variable }}, {{ variable | filter }} type of syntact that should be natural to a Django developer.

You can learn more about themes in the official docs here.

Custom theme

The primary target of Pelican seems to be personal sites where authors can publish content such as articles by writing it in markdown and then update the site by running the pelican to generate it. For such a site, the homepage would typically show the latest articles as and when they are published. This is what most of the themes in the theme library seem to do (I haven't explored them all. So I could be wrong here).

If you're using Pelican to generate a company website, then a homepage showing latest blog articles may not be what you want. Many company sites still have a blog, but typically the blog would be hosted in a subdomain (such as blog.example.com) or as a subsite under the main site.

To customize the site's homepage, Pelican does provide a special content meta tag that would render that content's HTML output as the site's homepage (example.com/index.html). However, rendered content (articles or pages) tend to consist mostly of text and this may not be ideally suited for a company site's homepage, which typically has a drastically different design than the site's inner pages.

So the solution is to create a custom theme with a suitable index.html that reflects the homepage design that you desire. You can still incorporate pelican content elements inside (such as latest blog articles) it, if you wish to do so.

Configuration

Pelican configuration file, pelicanconf.py, controls what theme to use and other relevant hints to the site generator. A full documentation of Pelican's settings is here.

Note however that, for publishing the site to the web a different configuration file, publishconf.py is used. This file imports all settings from pelicanconf.py, but overrides some of them for production configuration (borrowing the term from Angular).

One such configuration that is typically different in development and production configurations is the site's URL. For local dev configuration this will be an empty string and for production this will be the full URL of the domain(SITEURL setting). This matters are the generated hyperlinks are prefixed with this setting's value in the templates and setting its value correctly will ensure that the embedded hyperlinks to navigate between different pages within the site work as expected. Make sure you prefix the URL with the correct scheme. So if your site runs on HTTPS it should be https://www.example.com.

Plugins

Pelican supports plugins which extend the feature set of the core system. For instance localization is provided via the plugin i8n_subsites. There's also a plugin to automatically generate sitemap file that you can upload to search engines.

While some plugins are independent packages that can be installed using pip, others require the relevant repository to be cloned locally and its path given to pelican via the pelican configuration file (see below).

For example the sitemap plugin is installed as an independent package via pip install pelican-sitemap command whereas the localization plugin i8n_subsites is installed by replicating the repo locally.

For plugins that require its repository to be locally maintained, a useful convention is to store them under a plugins folder beneath the project's root folder.

The location where plugins are searched for is specified via the PLUGIN_PATHS configuration as shown below. The plugins itself to be applied are specified via the PLUGINS setting.

PLUGIN_PATHS = ["./plugins"]
PLUGINS = ["i18n_subsites", "sitemap"]

Historically plugins are maintained in this repository, but there's a new initiative to maintain the plugins under their own github organization here. Not all plugins from the historical repository has been ported over to the new organization.

Conclusion

I hope this gives a brief overview of Pelican and its core concepts. The documentation is excellent though it falls short of an overview doc like this. Perhaps there are other resources out there that this replicates. But I couldn't find any and hence I decided to write this up.

Reflecting server errors in Angular form validation

2020-02-12T00:00:00+08:00

Here's some sample code that shows how to display server validation errors sent by a django-restframework REST server.

<form [formGroup]='form' (ngSubmit)="doChangePassword()">
    <ion-item>
        <ion-label>{{ 'CURRENT_PASSWORD' | translate }}</ion-label>
        <ion-input formControlName="old_password" type="password"></ion-input>
    </ion-item>
    <ion-item class="ion-text-wrap ion-no-lines" *ngIf="form.controls.old_password.errors?.required && form.controls.old_password.dirty && form.controls.old_password.touched">
        <ion-label color='danger' class="ion-no-margin" stacked>{{ 'FIELD_IS_REQUIRED' | translate }}</ion-label>
    </ion-item>
    <ion-item class="ion-no-lines" *ngFor="let errorMsg of form.controls.old_password.errors?.serverErrors">
        <ion-label color='danger' class="ion-no-margin ion-text-wrap" stacked>
            <small>{{ errorMsg }}</small>
        </ion-label>
    </ion-item>
    <ion-item>
        <ion-label>{{ 'NEW_PASSWORD' | translate }}</ion-label>
        <ion-input formControlName="new_password1" type="password"></ion-input>
    </ion-item>
    <ion-item class="ion-text-wrap ion-no-lines" *ngIf="form.controls.new_password1.errors?.required && form.controls.new_password1.dirty && form.controls.new_password1.touched">
        <ion-label color='danger' class="ion-no-margin" stacked><small>{{ 'FIELD_IS_REQUIRED' | translate }}</small></ion-label>
    </ion-item>
    <ion-item class="ion-no-lines" *ngFor="let errorMsg of form.controls.new_password1.errors?.serverErrors">
        <ion-label color='danger' class="ion-no-margin ion-text-wrap" stacked>
            <small>{{ errorMsg }}</small>
        </ion-label>
    </ion-item>
    <ion-item>
        <ion-label>{{ 'CONFIRM_NEW_PASSWORD' | translate }}</ion-label>
        <ion-input formControlName="new_password2" type="password"></ion-input>
    </ion-item>
    <ion-item class="ion-text-wrap ion-no-lines" *ngIf="form.controls.new_password2.errors?.required && form.controls.new_password2.dirty && form.controls.new_password2.touched">
        <ion-label color='danger' class="ion-no-margin" stacked><small>{{ 'FIELD_IS_REQUIRED' | translate }}</small></ion-label>
    </ion-item>
    <ion-item class="ion-text-wrap ion-no-lines" *ngIf="form.controls.new_password2.errors?.matchError && form.controls.new_password2.touched && form.controls.new_password2.dirty">
        <ion-label color='danger' class="ion-no-margin" stacked>
            <small>{{ 'PASSWORDS_MISMATCH' | translate }}</small>
            </ion-label>
    </ion-item>
    <ion-item class="ion-no-lines" *ngFor="let errorMsg of form.controls.new_password2.errors?.serverErrors">
        <ion-label color='danger' class="ion-no-margin ion-text-wrap" stacked>
            <small>{{ errorMsg }}</small>
        </ion-label>
    </ion-item>
    <ion-button type="submit" expand="block" [disabled]="!form.valid">
        {{ 'CHANGE_PASSWORD' | translate }}
    </ion-button>
</form>

Specifically, note how ngIf & ngFor tags are used to conditionally detect and display error messages set on individual controls. The attribute serverErrors on a control is set by the page handler TypeScript code via a snippet like this:

  // Form definition
  this.form = formBuilder.group({
    old_password: ['', Validators.required],
    new_password1: ['', Validators.compose([
      Validators.required,
    ])],
    new_password2: ['', Validators.required],
  }, {

  // form submit handler
  async doFormSubmit() {
    try {
      await this.service.post(this.form.value);
      let toast = await this.toastCtrl.create({
        message: 'Server updated',
        duration: 3000,
        position: 'top'
      });
      toast.present();
      this.router.navigateByUrl('/home');
    } catch (err) {
      // 'err' contains server errors
      this.handleServerErrors(err.error);      
    }
  }

  // For fields with specific error messages, set the relevant field
  // control's error.serverError attribute to the server returned error
  // message. This then can be directly displayed in the template.
  handleServerErrors(err) {
    // Each key of the err object will correspond to the the field
    // name which has an error. Since the field names in the form match
    // the REST serializer fields, we can use this to attach the server
    // supplied error messages to corresponding UI input field.
    for (const controlName in err) {
      if (this.form.contains(controlName)) {
        this.form.controls[controlName].setErrors({serverErrors: err[controlName]});
      } else {
        // Need to handle generic form errors, that do not apply to a specific
        // field, but to the whole form.
      }
    }
  }

How to programmatically emulate Apple special keys such as media navigation & power keys

2019-06-28T00:00:00+08:00

Media keys along with some other keys are not handled through the standard apple keyboard event subsystem. They are treated differently, and therefore emulating them from software requires special code (EventTap won't work!).

Here's a function that does this:

void HIDPostAuxKey(uint32_t key, bool down)
{
    @autoreleasepool {

        NSEvent* ev = [NSEvent otherEventWithType:NSEventTypeSystemDefined
                                location:NSZeroPoint
                           modifierFlags:(down ? 0xa00 : 0xb00)
                               timestamp:0
                            windowNumber:0
                                 context:nil
                                 subtype:8
                                   data1:(key << 16)| ((down ? 0xa : 0xb) << 8)
                                   data2:-1
                                   ];
        CGEventPost(kCGHIDEventTap, [ev CGEvent]);
    }
}

This Objective-C code is adapted from an SO post that shows how to do this from Python.

Values for the key parameter NX_* macros defined in IOKit/hidsystem/ev_keymap.h. Some of them look like this:

#define NX_KEYTYPE_SOUND_UP  0
#define NX_KEYTYPE_SOUND_DOWN  1
#define NX_KEYTYPE_BRIGHTNESS_UP 2
#define NX_KEYTYPE_BRIGHTNESS_DOWN 3
#define NX_KEYTYPE_CAPS_LOCK  4
#define NX_KEYTYPE_HELP   5
#define NX_POWER_KEY   6
#define NX_KEYTYPE_MUTE   7
#define NX_UP_ARROW_KEY   8
#define NX_DOWN_ARROW_KEY  9
#define NX_KEYTYPE_NUM_LOCK  10

#define NX_KEYTYPE_CONTRAST_UP  11
#define NX_KEYTYPE_CONTRAST_DOWN 12
#define NX_KEYTYPE_LAUNCH_PANEL  13
#define NX_KEYTYPE_EJECT  14
#define NX_KEYTYPE_VIDMIRROR  15

#define NX_KEYTYPE_PLAY   16
#define NX_KEYTYPE_NEXT   17
#define NX_KEYTYPE_PREVIOUS  18
#define NX_KEYTYPE_FAST   19
#define NX_KEYTYPE_REWIND  20

#define NX_KEYTYPE_ILLUMINATION_UP 21
#define NX_KEYTYPE_ILLUMINATION_DOWN 22
#define NX_KEYTYPE_ILLUMINATION_TOGGLE 23

#define NX_NUMSPECIALKEYS  24 /* Maximum number of special keys */

Converting a std::vector into an NSArray

2019-06-26T00:00:00+08:00

An excellent piece of code to do the above is here:

//clang++ -std=c++11 -stdlib=libc++ -framework Foundation nsarray.mm -o nsarray
/* Note:
 * - libstdc++ has been frozen by Apple at a pre-C++11 version, so you must opt
     for the newer, BSD-licensed libc++
 * - Apple clang 4.0 (based on LLVM 3.1svn) does not default to C++11 yet, so
     you must explicitly specify this language standard. */
/* @file nsarray.mm
 * @author Jeremy W. Sherman
 *
 * Demonstrates three different approaches to converting a std::vector
 * into an NSArray. */
#import 
#include 
#include 
#include

int
main(void)
{
 @autoreleasepool {
 /* initializer list */
 std::vector strings = {"a", "b", "c"};

 /* uniform initialization */
 //std::vector strings{"a", "b", "c"};

 /* Exploiting Clang's block->lambda bridging. */
 id nsstrings = [NSMutableArray new];
 std::for_each(strings.begin(), strings.end(), ^(std::string str) {
  id nsstr = [NSString stringWithUTF8String:str.c_str()];
  [nsstrings addObject:nsstr];
 });
 NSLog(@"nsstrings: %@", nsstrings);

 /* Using a lambda directly. */
 [nsstrings removeAllObjects];
 std::for_each(strings.begin(), strings.end(),
  [&nsstrings](std::string str) {
   id nsstr = [NSString stringWithUTF8String:str.c_str()];
   [nsstrings addObject:nsstr];
  });
 NSLog(@"nsstrings: %@", nsstrings);

 /* Now with a range-based for loop. */
 [nsstrings removeAllObjects];
 for (auto str : strings) {
  id nsstr = [NSString stringWithUTF8String:str.c_str()];
  [nsstrings addObject:nsstr];
 }
 NSLog(@"nsstrings: %@", nsstrings);
 return 0;
 }
}

Again, thanks to this gist.

Set/get volume in OS X

2019-06-25T00:00:00+08:00

How to get/set master volume in OS X. Many sample codes online use the deprecated API. This uses the currently 'legal' API.

#include <CoreAudio/CoreAudio.h>
#include <AudioToolbox/AudioToolbox.h>
#include <AudioToolbox/AudioServices.h>
#include <AppKit/AppKit.h>
#include <CoreGraphics/CGEvent.h>

static AudioDeviceID getDefaultOutputDeviceID()
{
    AudioDeviceID outputDeviceID = kAudioObjectUnknown;

    // get output device device
    OSStatus status = noErr;
    AudioObjectPropertyAddress propertyAOPA;
    propertyAOPA.mScope = kAudioObjectPropertyScopeGlobal;
    propertyAOPA.mElement = kAudioObjectPropertyElementMaster;
    propertyAOPA.mSelector = kAudioHardwarePropertyDefaultOutputDevice;

    if (!AudioObjectHasProperty(
                kAudioObjectSystemObject, 
                &propertyAOPA))
    {
        printf("Cannot find default output device!");
        return outputDeviceID;
    }

    status = AudioObjectGetPropertyData(
            kAudioObjectSystemObject, 
            &propertyAOPA, 
            0, 
            NULL, 
            (UInt32[]){sizeof(AudioDeviceID)}, 
            &outputDeviceID);

    if (status != 0)
    {
        printf("Cannot find default output device!");
    }
    return outputDeviceID;
}

float getVolume()
{
    Float32 outputVolume;

    OSStatus status = noErr;
    AudioObjectPropertyAddress propertyAOPA;
    propertyAOPA.mElement = kAudioObjectPropertyElementMaster;
    propertyAOPA.mSelector = kAudioHardwareServiceDeviceProperty_VirtualMasterVolume;
    propertyAOPA.mScope = kAudioDevicePropertyScopeOutput;

    AudioDeviceID outputDeviceID = getDefaultOutputDeviceID();

    if (outputDeviceID == kAudioObjectUnknown)
    {
        printf("Unknown device");
        return 0.0;
    }

    if (!AudioObjectHasProperty(outputDeviceID, &propertyAOPA))
    {
        printf("No volume returned for device 0x%0x", outputDeviceID);
        return 0.0;
    }

    status = AudioObjectGetPropertyData(
            outputDeviceID, 
            &propertyAOPA, 
            0, 
            NULL, 
            (UInt32[]){sizeof(Float32)}, 
            &outputVolume);

    if (status)
    {
        printf("No volume returned for device 0x%0x", outputDeviceID);
        return 0.0;
    }

    if (outputVolume < 0.0 || outputVolume > 1.0) return 0.0;

    return outputVolume;
}

void setVolume(float volume)
{
    AudioDeviceID outputDeviceID;
    AudioObjectPropertyAddress propertyAddress = { 
        kAudioHardwareServiceDeviceProperty_VirtualMasterVolume, 
        kAudioDevicePropertyScopeOutput,
        kAudioObjectPropertyElementMaster 
    };

    outputDeviceID = getDefaultOutputDeviceID();

    // Deprecated API. Switch to AudioObjectSetPropertyData
    /*
    AudioHardwareServiceSetPropertyData(outputDeviceID, 
                                        &propertyAddress, 
                                        0, 
                                        NULL, 
                                        sizeof(Float32),
                                        &volume);
    */
    AudioObjectSetPropertyData(
            outputDeviceID,
            &propertyAddress,
            0,
            NULL,
            sizeof(Float32),
            &volume);
}

Adapted from this SO post.

WiX Tips

2019-05-07T00:00:00+08:00

Some tips on using WiX to generate MSI installers:

The GUID used as Product Id has to be unique for each release of the package. That is version 1.0 & 1.1 should have different product ids. So it's best to use auto-generated GUIDs for this like this:
```
<product Id="*"... />
```
In other words do not hard code the product id.
MSI only recognizes three quadrant product version numbers.
Product upgrades essentially work by uninstalling the current version and then installing the new version.
From MSI v3.5 onward upgrades are supported through the
```
<MajorUpgrade.. />
```
tag.
MSI does not define special actions for uninstallation. In fact, uninstallation is also treated as a type of installation. So if you want to run a custom action during uninstall, you have to use Condition predicates that cause them to run only during uninstallation. For example the following custom action is only run during uninstall as REMOVE="ALL" evaluates to TRUE only when all components are removed, which is during uninstall.
```
<Custom Action="CA_QuitHelper" Before="InstallValidate">REMOVE="ALL"</Custom>
```
If you want an action to be run during uninstall as well as during upgrade, additional condition predicates are necessary. Specifically:
```
<Custom Action="CA_QuitHelper" Before="InstallValidate">(REMOVE="ALL") OR (UPGRADINGPRODUCTCODE) OR (WIX_UPGRADE_DETECTED)</Custom>
```
. UPGRADINGPRODUCTCODE and WIX_UPGRADE_DETECTED are set to TRUE when doing inline product upgrade.
Note the use of Before="InstallValidate" in the above tag. This ensures that the task is launched before installation (or uninstallation) starts.
Similarly, if you want a custom action to be executed only during installation using the condition predicated NOT INSTALLED. For eg:
```
<Custom Action="CA_LaunchTask" After="InstallFinalize">NOT INSTALLED</Custom>
```
.
Custom action type 18, which executes an installed program, requires ExeCommand attribute specifying the command arguments, even if the program requires no arguments.
```
<CustomAction Id="CA_LaunchTask" FileKey="TaskBinaryEXE" ExeCommand="" Return="asyncNoWait" Impersonate="yes" />
```
.
The file that is referened to in FileKey above has to be a one of the
```
<File Id="TaskBinaryEXE".. />
```
keys.

Setting up a Windows VM for Kernel Debugging

2019-03-12T00:00:00+08:00

If you search the web using the title of this post, you'll get many links to articles most of which explain how to set this up using a virtual COM port on the VM. Coming from driver background that goes back over 20 years, at first this seemed quite natural. But then I remembered that using WinDBG over COM port to be painfully slow. And recent adventures in OSX land using LLDB led me to wonder if WinDBG over network would to be possible?

Well it turns out that WinDBG over Ethernet is indeed possible and is documented by Microsoft here. However, they miss out on certain key steps, which won't be obvious for the casual reader unless one reads the entire driver debugging documentation from the beginning.

This post captures all the requisite steps as a quick reference guide. I used one of the freely available MSEdge on Windows 10 VM for VirtualBox. It's only valid for 90 days, but 90 days was more than enough for my little excursion into the driver land. If you want a more permanent solution, you can use a Windows setup with a perpetual license.

Setup a Windows VM. Take the virgin system snapshot.
Default VM will only have one Network Adapter, which will be configured to NAT. You may continue to use NAT or change it to bridged. For remote debugging, however, we won't be using this adapter. For this we create a dedicated adapter and configure it to use the Host-Only network. So essentially the VM should have two NICs, one of which has to be configured to use the Host-Only network. Set this up and boot the VM. Make sure that this NIC has a valid IP address assigned and that you can ping the VM from the host through this IP. By default this NIC is configured to use DHCP. You may wish to change this to use static IP so that you can consistently refer to the VM through this fixed IP. But this is optional.
Install the Virtual Guest Additions package. This allows enabling clipboard sharing & folder sharing between host & guest machines, something that'll come very handy. Configure the above two (as well as device sharing if you're developing a driver for a physical device such as USB device) and take another snapshot.
Setup a dedicated boot configuration for debugging drivers. Use the following commands from Administrator Command Prompt:
```
C:\Windows\System32> bcdedit /copy {current} /d "Debug"
```
This creates a copy of the current boot configuration named 'Debug'.

Set the newly created boot configuration, Debug, as the default.

C:\Windows\System32> bcdedit /default <{debug_configuration_guid}

Reboot into the 'Debug' boot configuration just created. We need to configure this configuration with additional boot settings. It's a lot easier if we reboot to this making it the current configuration so that we can omit specifying the configuration guid in every bcdedit command.

Configure the newly created boot configuration, Debug, which is now the current configuration:

C:\Windows\System32> bcdedit /set nointegritychecks on
C:\Windows\System32> bcdedit /set testsigning on

Open up registry editor and create a DWORD (32-bit) entry named DEFAULT under HKLM\SYSTEM\CCS\Control\Session Manager\Debug Print Filter, with the value 0xF. If the key 'Debug Print Filter' does not exist, create it. This is the default debug message filter mask for messages printed from the kernel. By default this mask is set to 0x3 (I think), which only displays messages at ERROR and WARNING levels. If you want INFO & DEBUG level messages to be displayed at the debugger, you need to set the relevant bits (2 & 3) on this mask.
Follow steps in Microsoft's own documentation here to configure the boot configuration dbgsettings & set the busparams to the VM's Host-Only NIC bus, driver and function number values (Steps 3, 4 & 5).
Take another snapshot so that you can revert to this image if you end up messing up your environment through repeated test driver deployments!
Don't forget to setup the symbols path for WinDBG!
Assuming you did all these, you should be able to proceed with driver development tasks

Links:

- [Setting up KDNET Network Debugging Manually](https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/setting-up-a-network-debugging-connection) - [Debugging Drivers - Step by Step](https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/debug-universal-drivers--kernel-mode-#section-1-connect-to-a-kernel-mode-windbg-session)

Javascript Promises

2019-03-05T00:00:00+08:00

While explaining Promises to a friend I remembered reading a blog post couple of years back that explained the concept very well and made it all clear to me. However, I never bookmarked and searching for it, I can't find the specific post. Then I thought why not recreate some of the samples that the blog author had used to explain the concepts.

Here's what I came up with:

// Function to convert setTimeout into a Promise based // abstraction function delaySeconds(seconds) { return new Promise((resolve, reject) => { setTimeout(resolve, seconds*1000); }); } /* * A simple function that converts clunky setTimeout * callback based interface to a promise based * abstraction. */ function delayAndReturn(seconds, retval) { return new Promise((resolve, reject) => { setTimeout(() => { resolve(retval); }, seconds*1000); }); } // example of how to use the above abstraction console.log("Calling delaySeconds with 10 seconds delay"); delaySeconds(10).then(() => { console.log("delaySeconds completed waiting 10 seconds"); }); // Function returns a Promise object that is rejected. function rejectPromise() { return new Promise((resolve, reject) => { reject(); }); } console.log("Calling delayAndReturn with 2 seconds timeout"); /* * This sample demonstrates: * - how to chain promises together * - returning values from an 'outer' Promise to an 'inner' * one * - return values can be any concrete datatype, including * another Promise instance. * - returning and handling errors/rejections */ delayAndReturn(2, 10).then(ret => { console.log("delayAndReturn - retval: ", ret); console.log("delayAndReturn, waiting 3 seconds"); return delaySeconds(3); }).then(() => { console.log("delayAndReturn -- second promise completion handler"); return 20; }).then((ret) => { console.log('delayAndReturn - final return value: ', ret, ', calling rejectPromise()..'); return rejectPromise(); }).then(() => { console.log('empty Promise completion handler'); console.log('First delaySeconds promise should complete in ~5 seconds'); }).catch(err => { console.log('delayAndReturn promise chain err: ', err); });

It's not nearly as complete as what's in the original post, but guess it ought to be enough to grasp the core concepts.

Setting up OS X for kernel debugging

2019-01-26T00:00:00+08:00

Here are the steps to prepare an OS X system for debugging kernel modules such as drivers. The instructions are deliberately brief to save the reader from tons of text.
If you search the web, you will come across may resources that point to using GDB to debug the kernel. These resources were written in the early days of OS X when versions 10.4 or 10.5 were prevalent. Since around v10.9 (or thereabouts), Mac moved to lldb as their debugger of choice and have continually improved it to enable easier setup & more efficient workflow.

This short post has the instructions to setting up lldb for debugging OS X kernel versions 10.11 and above.

My setup is using a Parallels VM as the target on the development system. But it should work on a separate hardware that is connected via network to the host.

Debuggee is the target system where drivers are deployed and tested. This also referred to as the target system. Debugger is the host system, typically running your development tools with the driver source code that connects to the target system through network. This is also referred to as the host.

Download the Kernel Debug Kit matching the target VM OS, including its build number (clicking on the OS About box's version number field would reveal the build number next to it) and install it (installs to /Library/Developer/KDKs). Copy kernel.development or kernel.debug to the target’s kernel file folder. (Defaults to /System/Library/Kernels)
Define the VM’s boot arguments as:
```
devices.mac.boot_args="debug=0x146 -v watchdog=0 pmuflags=1 kcsuffix=development"
```
[debug=0x146 ]is to enable NMI as well as wait for debugger to connect on crash.

-v enable verbose mode boot which shows Linux like text messages during boot up process.

[watchdog=0] disables macOS watchdog from firing when returning from the kernel debugger.

Kcsuffix specifies the kernel to load during bootup. [kcsuffix=development]instructs bootloaded to start kernel.development as the OS kernel.

If you're using a separate machine, use nvram command:

    $ sudo nvram boot-args="debug=0x146 -v watchdog=0 pmuflags=1 kcsuffix=development"

Ensure that your driver kext is loaded by attaching the associated USB device to the VM. Only when a driver kext is loaded that we can add its symbols in lldb.

Start lldb with kernel as the debug target

lldb /Library/Developer/KDKs/KDK_10.14.1_18B75.kdk/System/Library/Kernels/kernel.development

Issue an NMI on the target. On a VM this can be done through the dtrace utility:
```
$ sudo dtrace -w -n “BEGIN { breakpoint(); }”
```
If the OS picked up the boot-args that you set earlier, system will halt waiting for debugger connection. OTOH, if the boot-args were not properly picked by the kernel, your system will reset.
From the Host, connect to the waiting target (waiting for debugger as it’s using the development kernel):
```
(lldb) kdp-remote <target IP address>
```

Add the kext symbols to lldb

(lldb) target symbols add ~/Library/Developer/Xcode/DerivedData/mydriver-fyooammjpvckpndqflrqugeldfjm/Build/Products/Debug/mydriver.kext.dSYM

Set the necessary breakpoints and issue continue to run the halted target.
Wait for breakpoints to hit and do source level tracing.

Happy days!

How to view kernel logs in OS X

2019-01-19T00:00:00+08:00

How to view kernel logs is OS X

You can use the command line tool log for this.

This command accepts a predicate argument that can be used to , well build predicates, that when evaluates to true, the log message will be printed. It takes a command argument that decides how the logs are to be displayed. For instance

log stream

produces a streaming output of the log. That is new log messages are displayed as they come (asynchronously).

log show

displays messages from the log history.

log --process 0

displays messages from process with ID 0, which is kernel. This can be used to display all kext log messages.

log show --predicate 'processID == 0'

displays messages printed by processID 0, which is the kernel process. This is the same as the previous command.

To display messages from a driver, you may use the sender predicate filter argument. So if your driver is named mytestdriver, you may display messages from it using the command:

log stream --predicate 'sender == "mytestdriver"'

Note that we're only specifying the actual executable name (value of CFBundleExecutable in the drivers plist file) and not the bundle identifier.

Replacing stream with show will show past messages from the log.

If you want to filter messages that contain a specific string pattern, say "user request":

log stream --predicate 'sender == "mytestdriver" and message contains "user request"'

Note that you can also use the Applications -> Uitlities -> Console app to get similar results, but from a GUI.

Ionic 4 HTTP CORS Issue

2018-11-07T00:00:00+08:00

Ionic 4 is a major upgrade to previous version of Ionic. One of the primary differences between Ionic 4 and its previous incarnations is that Ionic 4 uses web components, which are custom HTML elements written using web standards (as opposed to using a slap-on framework such as Angular).

Ionic 4 also makes a fundamental change in how the created app is rendered (I use rendered as apps are essentially HTML which is rendered using the native WebView control of the device's UI framework) in a device. While earlier versions used to refer to the top level index.html using file:// URI scheme, Ionic 4 internally spins up an HTTP server on localhost (yes for each app) and then loads the index.html through a standard HTTP request such as http://localhost:port/index.html.

Problem

I'm not sure what necessitated this change, but I reckon it probably had something to do with native API restrictions, which in turn was prompted by need for improved security. This, however, introduces a new (and common) problem. If your app interfaces with a remote server using HTTP requests, and you use the Angular HttpClient for making the requests, the remote server would typically reject the request as it would detect the request as coming from a different domain (domain would localhost) than the domain where it's hosted. This is due to CORS rules as set in the server configuration, which by default is set to reject all cross-domain requests.

Solution

You really have two options to solve this:

If you own & manage the server, change its CORS setting to allow requests from any domain
If you don't have control over the server or are reluctant to change the server's CORS setting, you have to switch to Ionic's native HTTP API to make HTTP requests.

Second options seems simple enough, but then it has another drawback. Native APIs are typically only available on the device and the HTTP native API happens to be one of them. Therefore using it means you won't be able to develop in the much quicker browser environment from your computer. This is where [[ionic-native-http-connection-backend]](https://github.com/silkimen/cordova-plugin-advanced-http) is installed and available for use and if so uses it. If it's not available or can't be used owing to the app being run on the browser, it defaults to Angular's native backend. So you get the best of both worlds!

Installation Woes

When I tried to install [cordova-plugin-advanced-http] again and it succeeded.

Conclusion

I'm not sure how stable [ionic-native-http-connection-backend](https://github.com/sneas/ionic-native-http-connection-backend) is, but if the rather minuscule [open issue](https://github.com/sneas/ionic-native-http-connection-backend/issues) count is anything to go by, I'm willing to take my chances with it for now.

Notes of the day

2018-10-11T00:00:00+08:00

A recent project requirement from a customer for a cross-platform solution prompted me to look into viable frameworks for this. Based on my knowledge of languages (C++, Python & Javascript), I narrowed the potential solutions to wxWidgets & Git Electron. Since I have been spending some time with Python over the last two years and given its ease of use, I through I'll look into wxPython, the Python projection of wxWidgets framework.

Here are my notes from this research today morning:

wxPython provides a genuine means to build cross platform GUI apps (installed through PyPi [pip install wxpython]).
Python code can be compiled into distributable binaries using cx_Freeze, another cross-platform tool that can generate binaries for Windows, Mac & Linux. cx_Freeze can provide executable files as well as installable binaries such as MSI or DMG files.
cx_Freeze source repo has sample code for various deployment scenarios. One of them is wxPython based application.
Py2Exe is a popular tool for this, but it can only generate Windows executables. Moreover it does not work with Python 3.6 and supports only up to Python 3.4 (This is reportedly owing to python byte code compatibility issues). So if cross-platform binaries are required, cx_Freeze is probably the answer.

Git's Electron is also a viable option. Especially using Ionic 4, which natively supports desktop apps through its build scripts. However, Ionic 4 is still in beta and the requirement could potentially require interfacing with native components -- either through IPC or some other means. And for this Python is a much simpler technology. Hence my focus on wxPython.

The project involves emulating input devices on a PC and a Mac. The objective is to create a scriptable mechanism where desired input combinations can be emulated on the target computer and then verify that the input is generated using another software. Obviously a candidate for Windows APIs [SendInput](https://docs.microsoft.com/en-us/windows/desktop/api/winuser/nf-winuser-sendinput) & [SetWinEventHook](https://docs.microsoft.com/en-us/windows/desktop/api/winuser/nf-winuser-setwineventhook). The question is how does one go about implementing this on a Mac. Well, it happens that OS X has parallel APIs to the Windows APIS above. Some notes on this: - Event emulation in Windows is performed through the [SendInput](https://docs.microsoft.com/en-us/windows/desktop/api/winuser/nf-winuser-sendinput) API. Another API which is closely associated with this is [SetWinEventHook](https://docs.microsoft.com/en-us/windows/desktop/api/winuser/nf-winuser-setwineventhook), which allows an independent app to trap almost all GUI events, along with input events, in the systen. Mac equivalent of these two technologies is [Quartz Event Services](https://developer.apple.com/documentation/coregraphics/quartz_event_services). - Event hooks are referred as 'event taps' in Quartz.

Cordova build errors

2018-04-17T00:00:00+08:00

Had this unusual errors crop up when building a cordova app for android. This started appearing after I installed cordova-plugin-file-opener2.

ERROR: In <declare-styleable> FontFamilyFont, unable to find attribute android:fontVariationSettings
ERROR: In <declare-styleable> FontFamilyFont, unable to find attribute android:ttcIndex

Going through [platforms/android/project.properties] file, I realized that there was something quite odd about it.

cordova.system.library.1=com.android.support:support-v13:26.+
cordova.system.library.2=me.leolin:ShortcutBadger:1.1.17@aar
cordova.system.library.3=com.google.firebase:firebase-messaging:11.0.1
cordova.system.library.4=com.android.support:support-v4:27.1.0
cordova.system.library.5=com.android.support:support-v4:24.1.1+
cordova.system.library.6=com.android.support:support-v4:+

There were multiple versions of the same library being defined.

I removed the last three lines -- library.4, library.5 & library.6, ensuring that the the highest version of [com.android.support:support] and its contents. But I'm assuming it's being used by the build system to do something similar to linking in the traditional C build system).

Anyway voila the problems disappeared!

Downloading & opening a file in an Ionic app

2018-04-17T00:00:00+08:00

If you search the Internet for examples to achieve the above, you'll get a number of resources that suggest using the cordova-file-transfer plugin. However, with Ionic 3+ and its native API, you can do this directly using a mix of Angular and Ionic File API.

Essentially, you need to use Angular HttpClient to download a URL as a Blob using the [responseType] option and then save the resultant Blob to a file on the device using the File API. Finally, open the saved file using the Ionic native FileOpener API.

Here's the method in a provider that is used to download files from a website. Note that [this.http].

/**
  * Download a file from the given url.
  *
  * Params:
  *   url - the URL to download the file from
  * Returns:
  *   A promise which upon resolution returns a dictionary 
  *      {
  *        fileEntry: FileEntry,
  *        type: The content-type string for the file
  *      }
  */
  download(url): Promise<{ fileEntry: FileEntry, type: string }> {
    return new Promise((resolve, reject) => {

      // Get the file from the given URL argument. Note responseType is set to 'blob'.
      // It can also be set to 'arraybuffer', but response.body will be of different
      // type and needs to be handled slightly differently.
      this.http.get(url, {
        observe: 'response',
        responseType: 'blob'})
        .subscribe(response => {

          // Create /Download folder if it doesn't exist
          this.file.createDir(this.file.externalRootDirectory, 'Download', true)
            .then(de => {

              // Write the downloaded Blob as the file. Note that the file will
              // be overwritten if it already exists!
              this.file.writeFile(
                de.toURL(),
                fileName, response.body,
                { replace: true })
                .then(fe => {

                  // All went well, resolve the Promise.
                  return resolve({
                    fileEntry: fe,  // FileEntry instance
                    type: response.body.type  // Content-Type for the file
                  });

                }).catch(err => {

                  // writeFile failed 
                  reject(err);

                })
            }).catch(err => {

              // createDir failed 
              reject(err);

            })
        }, err => {

          // Download failed 
          reject(err);

        });
    });
  }

One of the key points here is that the [responseType: 'blob'] allows us to retrieve the file type as determined by the server which we can use as argument to the FileOpener API to open the file.

The argument [observe: 'response'] is not really necessary and can be left out. In this case, the subscribe handler will get the blob object as its argument instead of the higher level HTTP response object.

The call to [File.createDir()] ensures that the Download folder is created and available before the downloaded content is saved into it.

Finally the call to [File.writeFile()] saves the file and the function returns the FileEntry return value from this call.

This being a method in a service, it only downloads the file saving it to the device storage. How the saved file is to be processed is left to the page that initiated the download request to decide.

If the page wants to open the file, it can done like this:

this.fileOpener.open(attachment.fileEntry.nativeURL, attachment.type)
  .then(() => {
    // nothing to do actually
  }).catch(e => {
    // show alert message that file open failed.
  });

Important thing to note here is that we're supplying the [FileEntry.nativeURL] can be used as second argument to help the device determine the most appropriate application to open the file with.

If there are no applications associated with the given file type, [open() ]would result in an error, which should be caught and reflected to the user with an appropriate message.

Celery on Windows

2018-01-17T00:00:00+08:00

Running celery on Windows, shows the following error when tasks are queued to be executed:

ValueError: not enough values to unpack (expected 3, got 0)

The way to fix this is to add --pool=solo to the celery daemon command line. This makes celery run as a solo process and without concurrency, which for development purposes shouldn't be a big deal.

So the entire daemon command would look something like this:

python -m celery -A config worker --pool=solo -l info

Taken from this Github thread.

Steps to Install Redis server

2017-10-02T00:00:00+08:00

Few steps necessary to build and install Redis on a server. There are no pre-built packages available for you to apt-get install it. So you have to grab the source, compile and install the generated binaries. Fortunately it all goes pretty smooth.

$ wget http://download.redis.io/releases/redis-4.0.2.tar.gz
$ tar xzf redis-4.0.2.tar.gz
$ cd redis-4.0.2
$ make
$ make test

# install compiled binaries to /ur/lib/bin
$ sudo make install

# setup redis-server as daemon
$ cd utils
$ sudo ./install_server.sh

# setup redis-server to autostart
$ sudo update-rc.d redis_6379 defaults

That ought to do it!

How to Integrate Sphinx with a Django Project for Automatic Documentation Generation

2017-09-14T00:00:00+08:00

A few points on integrating Sphinx with a Django redistributable project for generating documentation embedded in the source files.

Project directory structure should look like this:

django-redistributable/
  demo/
    demo/
      settings.py
      urls.py
      wsgi.py
    templates/
    manage.py
  docs/
    conf.py
    index.rst
    Makefile
    make.bat
  redistributable/
    models.py
    views.py
    templates/
    static/
  tests/
  setup.py
  MANIFEST.in
  README.rst
  LICENSE
  requirements.txt

Install [sphinx]
Run [sphinx-quickstart] folder. Provide default answers for all prompts.
Edit [docs/conf.py] such that the redistributable package and the demo project's apps are importable from the docs folder. At the top of the conf.py file:
Setup sphinx such that it can safely import the Django source file referred to in any RST files for autodoc. For this you need to setup the DJANGO_SETTINGS_MODULE environment variable to demo project's settings. Also need to initialize Django's AppRegistry with a call to [django.setup()].
Embed the package version in the redistributable folder's __init__.py file as __version__ = "x.y" and then import this version into conf.py. This will make the package version and documentation version concurrent.
Build the documentation issuing [make html ] folder.

Volvo Door Mirror Mismatched Folding

2017-09-14T00:00:00+08:00

One of the problems that I encountered with the S80 is that when I unlocked the car and pressed the button to unfold the door mirrors, they would be out of sync with each other. That is one would open and at the same time the other would close. You can open the closed mirror manually, but when you press the button to fold then mirrors, one would close and the the other would remain open.

After some research online, I figured out the problem -- the mirror's folded/open position is stored in memory and the car uses this position to instruct the mirror motors on what action to perform (fold/unfold) when you press the button to unfold the mirrors again. Now if you're in a hurry and happen to switch off the vehicle before the mirror folding operation could be completed, storing of the mirror position in memory cannot be completed accurately.

The solution to the problem turns out to be very simple.

Turn off the car.
Locate the fuse box in the passenger compartment. Locate fuses for left and right door power windows and mirrors. For 2004 S80 these are fuses 35 & 36 respectively. Here is the list of fuses in this fuse box. It looks like this:
3. Remove the left door fuse (fuse number 35).
Manually push the left door mirror position to be in sync with the right door mirror position.
Start the car.
Turn off the car
Replace the fuse you removed in step 2.
Start the car again and now press the door mirror button. If it was the left mirror whose position was wrongly recorded in memory, everything should be fine.
If the mirrors are still out of sync with each other, repeat steps 2-7 with right door fuse (fuse number 36) and manually bringing the right door mirror to be in sync with the left door mirror position.

This technique should work for all Volvo cars with electric door mirrors. Of course you do need consult your vehicle's manual for the right fuse to remove.

Volvo Engine Service Required Message

2017-03-12T00:00:00+08:00

This is a message that my Volvo (S80 2004 2.0T) started showing yesterday. The message was accompanied with a yellow/orange warning light, (image on the right) in the car dash.

This freaked me out when it first appeared as I was worried that the car could shutdown anytime while on the move. But apparently the color of the light, yellow, is an indicator that the issue is not major and therefore such a shutdown is not an immediate possibility. On the other hand, if the color of the indicator is red, then it is indeed cause for concern and the best option is to stop the car and seek professional help immediately.

In the past whenever I had issues with my car, my first reaction is to reach out to my mechanic for help. But with the Volvo, what I have learned is that with the amount of electronics and computer control that is embedded into it, it's best to first to research online, narrow down the possible causes to a select few and then approach the mechanic appraising him of your research results. And sometimes you might even be able to fix the issue yourself (like I did with the out of sync door mirror folding issue). Perhaps it's because my mechanic is not very familiar with Volvo that this is necessary. He's more of a VW/Audi guy having worked in their official dealership for many years before venturing out on his own. Unfortunately right now I don't have the time to find a Volvo specialist and this guy's place is close to where I live. More importantly I've found him to be trustworthy.

Anyway with that approach in mind, I started my research yesterday and here are some of my findings. I thought it's best to record them here for my own reference, should this happen again.

Internal Error Codes

While messages on the console provide a driver friendly generic message, internally the computer associates error codes with each problem and stores it. As many things can go wrong with something like an engine, Volvo displays the generic message 'ENGINE SYSTEM SERVICE REQUIRED' in the messages console, but stores a more discrete error code internally. While the official OBDII reader can obviously read this code, there is an easier way.

According to this post, with the doors closed and engine running, holding down the READ button at the end of the left stalk (used for indicators) along with the trip meter button for 10 seconds should show this code on the messages console, along with a beep.

I searched the web for a list of possible error codes and their description, but unfortunately I have not been able to find one.

Possible Causes for the Message

So far my research yielded many threads where others have encountered a similar problem and going through these, I have narrowed down the list of problems to a few:

Cruise Control

Somehow the cruise control stops functioning along with this message and I have the same symptom. One owner fixed this by changing the brake diaphragm position sensor and it was fixed.
The post linked above also relates the problem to the cruise control. I remember my car's cruise control stopped working two weeks ago on a trip to Taoyuan. So to be investigated.

Here's another thread that describes this problem and how he fixed it.

And this thread explains how to fix the brake pedal position sensor, without buying a new one.

Random Misfire

One owner attributed the problem to this. And I have been experiencing this on and off, but only when I start the car.

This owner pulled all the coils and spark plugs and found two spark plugs were not screwed in tight, which could have meant some compression loss and possibly a random misfire. He cleaned all the plugs and coil connections and also removed all the crumbing plastic sleeve on the wiring to the coils and wrapped all those wires with electrical tape, though he did not see any other damage to the wires, then installed new plastic sleeve. This seems to have fixed the problem as the message did not appear again after resetting it using the same method to reset the service indicator message.

Oil Sticking to the Oil Level Sensor

An owner reports this message due the following issue as thus: "*I've found that if I open my oil cap and then start the car, the message goes away. Of course I didn't have any OBDI codes or lights on, just the engine system service required message. I've done this twice now and it worked. I use lucas oil stabilizer with full synthetic, and I think sometimes the oil sticks to the oil level sensor and gives a false reading that the oil level is too high. Could be wrong but it works for me.*"

Another owner explains this as thus:

"*Sounds related to the vacuum system and when you pull the oil cap off that makes a difference. I have a vacuum\\booster issue that disables cruise as well, haven't bothered to fix it.*

* *

*If you're using non-premium gas theres a chance that over time you will scorch your exhaust valves and seats as the motor adjusts ignition advance to cope with the faster flame propagation. Result is that your valves spend more time exposed to the flames from the exiting gas. Believe it or not, this is the most common issue with OBD cars that request higher octane fuels. Detonation and so on not really an issue unless you have a defective knock sensor. Scorched valve seats though are much more common.*

*I dont think your fuel choice is related to the code you saw though.*"

#### Mass Air Flow Sensor {#mass-air-flow-sensor} A post in another forum alludes this problems to faulty or improperly seated Manual Air Flow (MAF) sensor. Apparently the owner got the message when using the car after it rained non-stop for a week, thereby causing the MAF to report air flow issues. And at another time, the MAF was not properly seated after cleaning which also caused the message to appear. UPDATE: The problem was due to faulty brake pedal position sensor, which automatically turns off the cruise control system as a safety measure (makes sense). This can be replaced or even repaired. The carbon rods inside this sensor become dirty which affects their sensitivity. Taking it out and cleaning it and putting it back can solve the problem. This [thread](https://www.matthewsvolvosite.com/forums/viewtopic.php?t=20166) discusses this further.

Volvo Service Reminder Message

2017-03-12T00:00:00+08:00

This is a message that appears on my Volvo S80 after driving for a period of time. Interestingly, this message appears even though I have not driven the car for the recommended service interval - 7500 miles, or about 12000km.

Did some research and it turns out that this message is displayed as a function of hours the engine has been running, duration since last service and distance covered. Essentially, this light will come on at 7,500 mile (12,000 km) intervals, after 750 hours of driving or after 12 months, whichever occurs first, to remind the driver that the service interval has been exceeded.

The light will stay on for 2 minutes after start until reset.

Here's how to reset this message without using a professional interface equipment:

1. Put the key in and turn it to ignition position I 2. Press and hold in the reset button for the trip odometer 3. Turn the key to ignition position II. Note! Turn the key to position II within 2 seconds. 4. Hold the reset button for the trip meter pressed in until the original value has been reset. *Note: For all vehicles from the 2003 model year a general yellow lamp will light when the reset button must be released.* 5. Release the reset button for the trip odometer within 4 seconds. The instrument gives an audible signal when resetting has succeeded 6. If the trip meter is already reset the reset button should be held in for a minimum of 10 seconds and a maximum of 14 seconds. *If the reset button for the trip odometer is not released within 4 seconds, the service reminder indicator (SRI) is not reset.*

Django template tag to retrieve template filename

2017-02-09T00:00:00+08:00

One useful Django template tag is to retrieve the template filename from inside the template. Unfortunately, Django built in tags do not provide one for this. However, you can write a simple tag to accomplish this.

Here's are two tags, that return either the base filename or the full filename.

from django import template

register = template.Library()

@register.simple_tag(takes_context=True)
def basefilename(context):
    '''Returns filename minus the extension'''
    return context.template_name.split('.')[0]

@register.simple_tag(takes_context=True)
def fullfilename(context):
    '''Returns full filename with the extension'''
    return context.template_name

To be complete, tthere should be another tag that returns the filename with its entire path. But I haven't figured out how to do this -- at least from within the confines of the template tag framework.

Useful, but rarely used git commands

2016-10-17T00:00:00+08:00

Some useful, but rarely used and hence easy to forget git commands. Recording these here as a note to myself.

Purpose

Command

Delete a Remote Tag

git push --delete origin <tagname>

Filenames diff between two commits

git diff commit commit2 --stat

List all tags, sorted by name, in reverse

git tag -l --sort="-v:refname"

How to Create a Random String

2016-10-14T00:00:00+08:00

Random strings are used everywhere in computing -- mostly as keys for cryptographic signing sensitive data. So how does one go about creating one?

My first thinking was to use Python code and its random library. I came up with this

import string
import random

def gen_random_string(length=32, charset=string.printable):
    '''
    A function to generate a string of specified length composed of random
    characters. The characters with which the string is composed can be
    customized thru the second parameter.

    Parameters:
        length - int, length of the string. Defaults to 32.
        charset - array of characters from which the letters to compose
                 the string are randomly chosen. Defaults to string.printable

    Returns:
        string

    Raises:
       None
    '''
    num_chars = len(charset)
    random_string = ''
    random.seed()
    for i in range(0, length):
        random_string += charset[random.randint(0, num_chars-1)]
    return random_string

It does produce a random string of the required length.

However, I found out a better way:

$ openssl rand -base64 48

This generates a random string of length 48 characters.

Not only is the latter more succinct (though it requires OpenSSL to be installed), I would think the generated string it's more random than one based in python random library.

Script to Create Secure Django Site Deployment Environment

2016-10-13T00:00:00+08:00

My first production level shell script is out. Yes, after two decades spent in the depths of code, I have finally published a shell script that is worthy of being called 'code'.

It's something that started as a few lines hack to update a Linux box with a series of packages that needed to be installed to deploy a Django web app that eventually grew to 300+ lines of madness (I can't find anything else to describe shell scripting language) that I have used to create consistent configuration across a few servers.

Without much ado, here it is: https://github.com/harikvpy/deploy-django.git.

Token Based Authentication in a Cordova App

2016-09-19T00:00:00+08:00

Any web based service that provides a mobile client requires some form of authentication before the user can access their permitted resources. The simplest approach is to require the user to enter their username and password for every session (or after the session expires). Simple to implement, but not the most friendly approach. User’s are notorious for forgetting their login credentials more so now than ever as more and more services require login ids and their own passwords.

A better approach is to use token based authentication whereby user is issued a signed token that they can then use to access their data from the online service. This introduces its own security challenges but this can be mitigated quite nicely with proper server side measures such as serving your entire site in HTTPS only. Of course, token based approach requires that your back-end implements an API that is suitably protected and accessible only through authenticated clients. But that’s a given and I’m not going to delve into that in this post.

Tokens however have a validity period attached to them after which they expire and are no longer useful. One way to extend their validity is to refresh them using a separate request typically provided through a separate API endpoint. Once this is setup, you can create a background task in your mobile app that would periodically keep refreshing your token so that it stays valid for prolonged periods of time providing user with a seamless experience a la Facebook app.

This is where the problems start. Creating a background task is typically achieved through a service which is a special kind of task (in Android at least) that remains active throughout the power cycle of your device. This means even when your soft turn off your device, the service runs in the background periodically refreshing the access token. Drains the battery a little, but who cares as long as your app provides the necessary user experience?

With this approach in mind, I started implementing a background service in my Cordova app. Searching the web, I found a few Cordova plugins for this and out of these, I chose the most popular one. Now the author does warn that some app stores may not certify apps that use the plugin and that it is only for enterprise apps, meaning apps that are distributed outside of legitimate app stores and run within a controlled environment. I still went ahead and tried it. Unfortunately my experience has not been all too great. Try as hard as I may, the background task fails to run even in my Android emulator.

Then there’s another limitation, on Android, for the background task to run reliably, the platform requires you to create a notification to the user highlighting to them that the task that they just closed is indeed still active. Quite rightly so too as the user needs to be informed that there’s a process that is draining their precious power bank, no matter however minuscule that consumption may be.

This caused me to pause a moment and think — perhaps my solution to the problem requires rethinking. Is there a way by which we can eliminate the need for periodic background token refresh altogether?

This is the approach that I can think of right now:

Extend the token validity period to one that covers the duration of use of even the most sporadic of your users. So if the most sporadic user’s interval of access is 3 months, make the token validity period so a comfortable range such as 6 months.
Store the token in offline storage and then upon application startup, look for this token. If it’s present, take the user to the app page that you would direct user to upon successful login within the app.
Refresh the token whenever the app is in the foreground and when the preset token refresh interval has elapsed. This would require that token acquire/refresh times to be saved in offline store such as localStorage of IndexDB.
Some token standards such as JWT provide a separate configurable interval for the maximum duration of token refresh. Extend this to a very long period — using the above as an example — 2 years or even longer.
To mitigate user losing their device, provide a web page listing all the devices using their account and allow them to revoke any device from that list. Once revoked the accessing the service from that device would require user to entire the username and password again.

Come to think of it, this seems to be reason why services like Facebook and Google provide the device list and revoke features for their services.

Notes

This approach is only recommendable for such services where the server does not hold highly sensitive information such as credit card information, financial account details, etc. If your service keeps track of user’s diet calorie intake that they enter after every meal, I guess then it’s not too big a loss if it gets into the wrong hands (unless you’re a celebrity of course)
The solution employs a blend of client side and server side code for what is perhaps the best user experience. High usability owing to the seamless experience and zero power impact as there’s no background service is running.

Pros

High usability provided by the seamless experience
Zero power impact as there’s no background task constantly running and draining power, however little.

Cons

Slightly lower security as losing a device can allow a malicious user to access the user data until the user revokes the device.
Additional development effort as the solution requires server side infrastructure to be built. Server side work involves changes to backend database to store the device-to-token association as well as a front end where the user can list and revoke their devices.

Perhaps there are ready made solutions out there that provide this whole thing as a pluggable component, but if there is one, I’m yet to discover that. Until then I have no choice but to embark on building one myself.

How to setup PostgreSQL on Linux for Django

2016-03-01T00:00:00+08:00

First step is to install PostgreSQL on Ubuntu. In its default installation, Ubuntu does not included PostgreSQL. So uou have to install it separately.
```
sudo apt-get install postgresql
```
Now install the python module for accessing PostgreSQL -- psycopg2. You have two choices here -- either install this from the OS package manager or manually using pip. Which approach to choose depends on your python development environment.

i. If you're using the system python as development environment, you may install psycopg2 from the OS package manager. You can do this by

    $ sudo apt-get install python-psycopg2

ii. However, if your python development environment is based on a virtual environment, the above approach will not work. This is because the above installs psycopg2 on the system python dist-packages, which does not copied into the virtual environment when it's created. This is the behavior even if the package is installed before the virtual environment is created. So in this case you need to revert to installing psycopg using the pip from the virtual environment.

This is where it gets a bit interesting. Pip base installation of psycopg, essentially installs the source code and then uses the OS compiler and libraries to build the python extension modules. One of the essential libraries for this is the PostgreSQL client development libraries. So you need to install this first.

    $ sudo apt-get install libpq-dev

Buidling a python extension module also requires python development environment, which is not installed by default. You have to install this by:

    $ sudo apt-get install python-dev

Now install psycopg2 using pip.

    (virtenv) $ pip install psycopg2

This will download the C source for the python extension module and will trigger these modules to be built. You should a series of messages where gcc is invoked to compile the C source. If all goes well, psycopg2 should be installed to your virtual environment.

Verify that psycopg2 was properly compiled and installed by issuing the following from python shell:

$ python
Python 2.7.6 (default, Jun 22 2015, 17:58:13)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import psycopg2

The next step is to configure PostgreSQL and then your Django project.

When PostgreSQL is installed, it will create the user account 'postgres' in the OS that will be the administrator account for managing PostgreSQL. To administer PostgreSQL, you need to login as this user and then start the PostgreSQL console using 'psql'. But before we go into PgSQL administration tasks, we need to set a password for the user 'postgres'. Do this by switching to root and then using the passwd command
```
$ sudo su
# passwd postgres
Enter new UNIX password: <password>
Retype new UNIX password: <password>
```

Create the database that you want to use with your Django project from PostgreSQL shell.

$ su - postgres
Password: <password>
postgres@hostname:~$
postgres@hostname:~$ createdb <dbname>

Note how we changed the user context to postgres, thePostgreSQL administrator before we created the database.

Another point here is that createdb is actually an external Perl script that is installed when PostgreSQL is installed. This, along with a few others such as createuser, are convenience wrappers to ease PostgreSQL administration directly from the bash rather than having to launch the PostgreSQL shell and then issue relevant commands.

Next step is to create a role that will be used to access this database. A role is a PostgreSQL internal user that is independent of the operating system user. You can assign access privileges to roles to different database objects. Every connection to the database server is made using the name of some particular role, and this role determines the initial access privileges for commands issued in that connection. For example, creating new databases requires a different privilege from the login privilege.

Only roles that have the LOGIN attribute can be used as the initial role name for a database connection. Without going too deep into the internals, you can use the helper script, createuser, to create a role with login privilege.

    postgres@hostname:~$ createuser -P 
    Enter password for new role: <password>
    Enter it again: <password>

Grant the necessar privileges on the new database to the newly created user. You do this from psql prompt from PostgreSQL administrator.
```
postgres@lubuntu1:~$ psql
psql (9.3.11)
Type "help" for help.
postgres=# GRANT ALL ON DATABASE  to
```

Note that you don't have to grant all privileges. You may be selective.

Now configure Django to use the newly created PostgreSQL by defining this in the project settings.py:

DATABASES = {
    'default': {
         'ENGINE': 'django.db.backends.postgresql_psycopg2',
         'NAME': '<dbname>',
         'USER': '<dbuser>',
         'PASSWORD': '<dbuser password>',
         'HOST': 'localhost',
         'PORT': '',
   }
}

From your Django project, issue

\$ python ./manage.py migrate

If your installation was successfully completed, various tables used by the Django project should be created in your newly minted database.

Persistent 'undo' in ViM

2015-12-08T00:00:00+08:00

I love ViM. It's universality, simplicity and keyboard-only-controllability appeals to me tremendously. I hear Emacs is better, but I can't be bothered to learn it. ViM suits me just fine.

However, there's one ViM shortcoming that has annoyed me endlessly. It's inability to maintain edit history across buffer switches. To put it simply, it maintains the edit history that allows undo's and redo's only for the current buffer being worked on. The moment you switch to another buffer, the history is cleaned up and a blank one is given to the new file.

Well, it turns out that ViM has the capability to preserve edit history for each file across buffer switches. Like many other features of the editor, you just gotta turn it on from the rc file. Here's how you do it

set undofile
set undodir=$HOME/.vim/undo
set undolevels=1000
set undoreload=10000

With this in your rc file, ViM creates a temporary file for each file being edited in the folder pointed to by undodir. Make sure you create this folder before you start ViM. ViM won't create it for you, if it doesn't already exist.

Note that undoreload=10000. So if you plan to edit large files with ViM, beware!

Lastly, since the edit history is stored in a file, it'll be available across program sessions. Isn't it awesome?

Thanks to this wonderful answer on SO for the tip.

Practices to reduce memory leaks in C++

2015-12-04T00:00:00+08:00

One of the fundamental differences between C++ and the more modern and if I dare say popular programming languages such as Java and C#, is the lack of a garbage collector in C++. Often this is used as an example to portray how C++ has not kept up with the times and how code written with the language is a potential minefield for memory leaks.

While the debate about the merits of one language over the other continues (and in all possibility would continue for many more years), there are a few steps one can take, as a programmer, that can pretty much minimize, if not eliminate, the chances of a memory leak.

I would like to list some of the techniques that I have employed over the years and have worked for me pretty well.

Use RAII

Use objects to abstract concepts

Use std::auto_ptr

Turn on memory tagging

Use std::vector\<> to allocate memory

Declare base class destructor as virtual

[{#useraii}Use RAII]

Use RAII (Resoure Acquisition Is Initialization) programming paradigm: Perhaps the best technique that one can employ to minimize memory leaks.

RAII is a term coined by Stroustrup to refer to the technique where the behavior of the constructor and automatic destructor of objects is employed to manage resources. In C++, there are two ways to allocate objects -- from the stack and from the heap. Unike Java and C#, where all objects are created by allocating memory from the heap, objects in C++ declared with local scope (within a function or meothod) are allocated from the stack and when these objects go out of scope, as that happens when a function returns, they objects are automatically destroyed.

This behavior can be exploited to make local resource management very simple. For example:

void foo()
{
    FILE* fp = fopen(...);
    if (!fp)
        return;
    // ...
    // do some other operation
    bakeacake()
    // call bar() giving the file as argument
    bar(fp);
    fclose(fp); // close the file
}

In this example, consider what happens if either bakeacake() or bar() throws an exeption. Control would immediately be transferred to the caller of foo() (and if it doesn’t handle the exception, to a higher level caller in the call chain that hopefully traps the exception) leaving the file resource dangling.

Given the above scenario, you can always write this function as

void foo()
{
    FILE* fp = fopen(...);
    if (!fp)
        return;
    try {
        // ...
        // do some other operation
        bakeacake()
        // call bar() giving the file as argument
        bar(fp);
    } catch (...) {     // catch the error
        fclose(fp); // remember to close
        throw;
    }
    fclose(fp); // close the file
}

and solve the problem of potential resource leak. However, this has repetitive code, something that is firstly, hard to maintain (as any changes in resource management would require updating two different places in the code) and secondly requires the programmer to exercise discipline while writing code (to remember or find out the various control paths and all the resources allocated so that they all can be released).

Another way to write this function would be to use a local object (allocated on the stack of foo()) that provides the required file operations. Consider this example:

void foo()
{
    File f(“a.out”);
    // ...
    // do some other operation
    bakeacake()
    // call bar() giving the file as argument
    bar(f);
}

The above approach will only work if you have access to the code of bar() and you can make the necessary changes to it so that it can accept a reference to File object rather than a FILE*. On the downside, now you have to change the implementation of another function to fit your new programming style.

Now consider the scenario where such a File object does not exist (and for some reason it is difficult to build one yourself) or you’re not at liberty to modify bar() as that function is used in many other modules code for which you do not have access to, you may try this approach:

class FilePtr {
    FilePtr(FilePtr& const);
    FilePtr& operator=(FilePtr& const);
public:
    FilePtr(char *const name, char* const mode) : fp_(0)
    { fp_ = fopen(name, mode); }
    FilePtr(FILE* fp) : fp_(fp)
    {}
    ~FilePtr()
    { 
        if (fp_) fclose(fp_);
    }
    operator FILE*() // crucial to making this work!
    { 
        return fp_;
    }
protected:
    FILE* fp_;
};

In this example, we first create a small wrapper class to whom we delegate the responsibility of managing the FILE* resource. When foo() returns, since the FilePtr object is allocated from the stack, it would be automatically destroyed resulting in its destructor being called which would close the FILE* gracefully.

There are two key points here that merit highlighting.

bar() function does not have to be modified. This is because FilePtr declares a FILE* operator, which will be automatically invoked by the compiler, when it cannot find an implementation of bar() that takes FilePtr (or FilePtr&) as its argument.

FILE* automatically gets released when the function returns, either through an error and consequent exception or when bar() returns.

[{#useobjects}Use objects to abstract concepts]

Use objects to abstract concepts as much as possible rather than using procedural style code.

One of the strengths of C++ is also its weakness. C++ allows you to mix C coding style (where program logic is expressed purely as procedural functions) into C++ code. While this provides immense flexibility and backward compatibility, it also leads you to maintain the same coding style that you’re used to (assuming that you spent considerable time coding in C and then eventually migrated to C++) preventing you from exploiting the language to its fullest.

Coupled with the RAII explained before, this approach will yield excellent results. Following are a few examples of scnearios where this approach will payoff rich dividends:

1. Synchronisation locks

2. Files as in the previous example

3. Temporary buffers (we’ll examine this soon)

4. Any other form of OS resources that have local scope

A few examples:

void a::foo()
{
    Guard g(mutex_)
    // code below, until the function exits, is
    // guaranteed to be safe across all threads
    // that access the resources protected by
    // the mutex mutex_.
    // update shared resources
    status_ = 1;
}

Here the constructor of Guard class acquires the mutex (waiting for it indefinitely) and its destructor automatically releases the same mutex.

void b::OnPaint()
{
    CPaintDC dc(this);    // RAII
    dc.SetBkColor(RGB(127, 127, 127));
    CRect rc; dc.GetClientRect(&rc);
    dc.FillSolidRect(&rc, RGB(0, 0, 0));
}

This is a classic MFC code. In Windows, drawing to a window in response to a WM_PAINT message requires one to call the BeginPaint API and once the drawing has been completed call the EndPaint API. Both these calls are encapsulated in the constructor and destructor of CPaintDC class respectively insulating the programmer from the explicit call to these APIs.

[{#usestdptr}Use std::auto_ptr]

Despite its destructive copy semantics, auto_ptr is still a very useful template that can be effectively employed to automatically release dynamic objects.

An example scenario where auto_ptr can be effective is in UI widgets that implement optional behaviors. For instance, imaging a window widget class that allows the user to set a background image as an option. Until the user sets this option, the widget draws the background using a solid color.

class Window {
    ...
    std::auto_ptr<Background> background_;
    ...
public:
    ...
    void setBackground(const wchar_t* imagefilename) throw(std::exception)
    {
        background_.reset(new     
        Background(imagefilename));
    }
    void Paint()
    {
        ...
        if (background_.get()) {
            // paint the background
        }
    }
};

Objects of the Window class would, by default, have an empty embedded Background object. However, when a user sets the background, a Background object would be created which will be tested for NULLness in the window paint routine. If the member variable points to a valid object (and hence is is not NULL), the image is painted as the background of the window.

The key point here is that the programmer does not have to remember to explicitly do anything to manage the Background object. When the window is destroyed which will result in a consequent deletion of the corresponding Window object, the associated Background object’s destructor will automatically be called where all the resources used to paint the background image can be released.

[Caveat 1:] As mentioned in the beginning, auto_ptr has destructive copy semantics making any object containing a member variable of this type not safe for copy constructor or operator assignment. So if your object requires either of these two constructs, you should stay away from auto_ptr. Alternative is boost::shared_ptr\<>.

[Caveat 2:] For the above reason, C++0x deprecates auto_ptr and instead provides a new template std::unique_ptr.

[Caveat 3:] std::auto_ptr is also not capable of managing array types whereas std::unique_ptr does.

[{#turnonmemorytagging}Turn on memory tagging]

Turn on tagging of heap memory allocated blocks in debug builds and compare heap memory snapshots between program startup and termination to ensure that all memory blocks are being released.

In Windows one can use the _CrtMemCheckpoint() and _CrtMemDifference() CRT calls to check if a function leaks any memory. Visual C++ also provides other Heap State Reporting Functions that can be used to monitor memory leaks.

I believe in LINUX, more specifically gcc, provides mtrace() which logs all memory allocation requests to a logfile which can then be parsed my mtrace utility to report if there are any leaks.

[{#usestdvector}Use std::vector\<char> instead of malloc.]

std::vector\<> uses RAII to acquire its memory and its destructor releases the memory. This way you don’t have to remember to release the memory you allocated at all of your return points.

For example:

void foo()
{
    char* buf = new char[1024];
    if (!buf)
        return;

    // dummy frame buffer
    volatile ulong* fb = 0x00040000;
    memcpy((void*)buf, (const void*)fb, 1024);

    if (cond1) {
        dosomething(buf);
        delete[] buf;
        return;
    }
    if (cond2) {
        dosomethingelse(buf);
        delete[] buf;
        return;
    }
    // do some other processing
    doyetanotherthing(buf);
    delete[] buf;
}

You’ll notice that that three different locations from where the function returns to the caller and hence three different calls to release the memory allocated at the beginning of the function. As mentioned in the first point, this requires the programmer to exercise discipline to remember to release the memory from all paths of return. Secondly, if any of dosomething(), dosomethingelse() or doyetanotherthing() throws an exception, the memory would never be released.

Now let’s look at how the same function, but now using std::vector for its memory allocation.

void foo() throw(std::exception)
{
    std::vector<char> buf(1024);

    // dummy frame buffer
    volatile ulong* fb = 0x00040000;
    memcpy((void*)&buf[0], (const void*)fb, 1024);

    if (cond1) {
        dosomething(buf);
        return;
    }
    if (cond2) {
        dosomethingelse(buf);
        return;
    }

    doyetanotherthing(buf);
}

Admittedly this is not such a great example as the control paths can be combined to eventually lead to a single point of return where the delete can be placed. But I hope you get the idea.

[{#baseclassdtorvirtual}Declare base class destructors as virtual.]

An often repeated mistake especially as the project scope expands and the classes that were designed and implemented in earlier versions are reused by inheriting from them to alter their behavior.

When deriving from a class whose destructor is not declared as virtual, and then subsequently deleting the object using a pointer to the base class would only invoke the destructor of the base class and not the derived class. Thus, the derived class resources would not be cleaned up.

Django Localization Issues and Solutions

2015-11-10T00:00:00+08:00

Django has excellent support for building a multi language multi culture application. However, there are a few interesting bits about its implementation of the internationalization feature that can prove to be minor annoyances in truly localizing yoru web based application. This post aims to discusses these issues and highlights the solutions for them.

It matters where you run 'makemessages' command from

The core command that prepares the application for localization, makemessages can be launched from two locations and depending on where it's launched it behaves slightly differently. The command can be launched from project root folder or from an application folder.

If started from the project root, it attempts to extract all the localizable strings from all installed apps and put them in the *.po file. However, if started from an application subfolder, it only extracts the strings from that application's files.

The key issue here is that makemessages which accepts an app name as an optional argument to generate migrations for that specific app. If you want to extract messages for a specific app, issue the command from the application folder.

The locale folder is optional for an application

Conventional wisdom will tell you that each application stores the localizable artifacts in its own private folder. It turns out that this is optional. makemessages will look for a locale subfolder under the app folder and if it finds one, it would place the .po file underneath it in the corresponding locale-name subfolder (more about this below) . If the locale folder is not found, the localizable strings for the app are collected in a .po file placed in the first path specified in the LOCALE_PATHS setting.

Language and locale names for Chinese have changed in 1.8

Earlier versions of Django used to refer to Chinese(Traditional) and Chinese(Simplified) as zh-tw respectively. Apparently, the change was brought about by the pan-nation usage of the language (and hence the related locale) that goes beyond Taiwan and China to countries such as Hong Kong, Malaysia and Singapore.

Interesting bit here is that 'makemessages' would still accept the old language codes as its argument and would send the messages file to a locale subfolder with that name. However, when trying out some localized strings, they won't show up.

Language names and locale names are different

Language names specified in the LANGUAGES tuple in settings are typically all lowercase. English (United States) is specified as 'en-us

There's an exception to this as well. If the country/region name has more than 2 characters, then it is represented in its pronoun form.

Finally, locale names use underscore to separate language and country codes whereas the Django language specification system uses all lowercase. So the locale name for English (United States) specified as 'en-us.

Beware of 'fuzzy' in PO files

Often as you tweak your website, you will invariably change some text. The change could be as simple as fixing a typo or adding a missing comma. After you change this, you would also generate a PO file using the makemessages command. This is where things get a little, shall we say 'fuzzy' (pun intdented). When you regenerate the PO file, GNU gettext utility parses the source code and starts extracting each string from it while looking for a matching entry in the PO file. On finding an exact match, it doesn't do anything as there could already be a msgstr "" entry for that string which it shouldn't mess up.

However, if it doesn't find any strings in PO file which is an exact match, it doesn't just create a brand new entry for the new string. Rather it has a concept called a close match. That is, the new string has a corresponding string in the PO file that, though not an exact match, is very close to the original string. In this case, makemessages followed by the new string while commenting out the old closely matching string.

Such strings are not by default stored in the MO file when the messages are compiled and consequently the localized text for the newly modified string won't show up. The system expects you to review the slight changes and make a decision whether the localized text needs to be appropriately updated. As part of this review, you have to remove the #, fuzzy and the commented original string which marks this message and its equivalent localized text as properly reviewed.

This mechanism is documented in detail here.

Enable/Disable Synaptics Touchpad on Ubuntu

2014-11-21T00:00:00+08:00

A day working from home brought out another problem the new Thinkpad. I did not have a spare mouse lying around and had to fall back to the flawed Touchpad/Trackpoint mechanism on the new T440. My original idea was to disable Touchpad completely through the x.org conf files and rely on the Trackpoint. But it turns out that while Trackpoint navigation works fine, clicking only works for the left-click. For some strange reason I couldn't get right-click to work on the Touchpad upper button area.
So I could not afford to dispense with the Touchpad altogether. At least not yet. Enabling it back through the x.org conf files would require reboot. Whenever a mouse is not handy I should be able to go back to the Touchpad as the primary pointing device. A mechanism to turn it on/off from shell as and when needed had to be devised.

Well, it turned out that this is pretty easy. Ubuntu came with a tool, xinput, which allows you to do just this. Here's how it works:

$ xinput set-prop <device id> "Device Enabled" 0

Essentially the tool allows setting various properties of the input device identified by the unique id and 'Device Enabled' is the option to enable/disable the device. To find the device id:

$ xinput list

and look for the id against the device 'Synaptics TouchPad'. On my machine it was 13.

A small bit of shell script can help automate this:

#!/bin/bash
declare -i ID
ID=`xinput list | grep -Eo 'Synaptics TouchPad\s*id\=[0-9]{1,2}' | grep -Eo '[0-9]{1,2}'`
xinput set-prop $ID "Device Enabled" 0

You can repeat the above to create another script to enable the device back. Using these two scripts you can toggle the touchpad on and off as and when necessary.

I know ideally I should figure out a solution whereby when the mouse is plugged in the touchpad is automatically turned off. But until then, I think I can live with this.

Ubuntu 12.04 on Thinkpad T440

2014-11-19T00:00:00+08:00

Being a slightly dated OS release, mating Ubuntu 12.04 to Thinkpad T440 hardware throws up a few challenges. This post lists these and how those can be worked around.
Disclaimer: I don't claim to have solved all these issues by myself. Instead all of them were solved by other people. What I did was to was crawl around the web, collect the necessary information and try it out one by one. I'm aggregating them all here for my own record at the same time hoping that someone else who has to go through the same pains would benefit from it.

Wireless Setup

Default installation of 12.04 won't detect the WiFi hardware rendering it useless. However, Ubuntu has updated 12.04 kernel since its original release and the updated kernel includes support for the new WiFi hardware. So to get WiFi support working, you have to do this:

$ sudo apt-get update
$ sudo apt-get install linux-generic-lts-saucy

Once above is finished, reboot the system. Afterward, make sure that all the patches as recommended by the Update Manager are applied and you should get the WiFi working normally.

Enable HDD TRIM

Precise 12.04 does not enable the SSD TRIM feature by default. You have to set it up yourself. There are numerous threads online on how to do this. However very few tell you how to go about doing this for an encrypted partition (like the one I have). This thread covers the latter case very well. I have tested it and it works like a charm.

Only change that I did was to perform the trimming daily by scheduling the cron job appropriately.

The important thing to highlight here is to NOT add the 'discard' option in fstab. Adding this will degrade the HDD performance quite a bit as every erase request to the filesystem would involve the time consuming trimming operation on the relevant 1M block.

Setup Default Screen Brigthness

This was another annoyance that was driving me nuts. Upon every boot up the screen brightness get set to the full level. I just couldn't get it to boot to the last level when the machine was shutdown. Apparently this requires installation of an additional package -- laptop-mode-tools! Once this is done, you have to edit one of its config files to set the default display backlight levels.

This blog post covers this well.

Disable Touchpad

T440 is exquipped with a touchpad the size of an airport. Presumably for those who reach computing nirvana through pinching and pulling using multiple appendages simultaneously. Unfortunately I'm not one of them. And I would think neither are most of my fellow developer comrades. We tend to rely more on the inherently flawed QWERTY system to tell the machine what we want it to do. So I felt the best option was to disable the damn thing and rely on the trackpoint (or a decent rodent). Here's how to do it:

$ sudo apt-get update
$ sudo vim /usr/share/X11/xorg.conf.d/50-synaptics.conf

Find the following section in the file and update the two Option lines as per the setting below. This essentially converts the entire touchpad into a giant three button configuration -- left, middle and right.

# This option enables the bottom right corner to be a right button on
# non-synaptics clickpads.
# This option is only interpreted by clickpads.
Section "InputClass"
        Identifier "Default clickpad buttons"
        MatchDriver "synaptics"
        Option "SoftButtonAreas" "65% 0 0 4% 40% 65% 0 4%"
        Option "AreaBottomEdge" "1"
EndSection

Here's the official Ubuntu documentation on Synaptics client and the meaning of various configuration options.

One issue with the trackpoint is that scrolling using the middle button doesn't work. I haven't found a hack to fix this, but when I do, I'll update it here.

Adding timedelta to a datetime.time

2014-06-26T00:00:00+08:00

One of Python's strengths is its exhaustive library. And date and time management is an integral part of this, provided by the datetime.

Logic suggests that timedelta. That is code like this would produce an error:

>>> from datetime import datetime, date, time, timedelta
>>> dt1 = datetime.now()
>>> td1 = timedelta(hours=1)
>>> dt1 + td1
datetime.datetime(2014, 6, 26, 15, 38, 27, 149200) # OK
>>> dt1.time()+td1
Traceback (most recent call last):
  File "", line 1, in 
TypeError: unsupported operand type(s) for +: 'datetime.time' and 'datetime.timedelta'

So how do you add (or subtract) a timedelta(or date) object? It turns out that a slightly convoluted way exists. Given a time object t1 and timedelta object td1, the following code does the trick.

>>> (datetime.combine(datetime.now().date(), t1)+td1).time()
datetime.time(15, 40, 55, 672503)

There is probably a shorter and better way to accomplish, but at the moment this is what I could come up with.

Convert time string to datetime.time object

2014-06-23T00:00:00+08:00

I'm still finding my way around the rather vast Python standard library. And today, I discovered a quick way to convert a pure time string (of format HH:MM[:SS]) into a standard datetime.time object.

datetime.strptime('08:00', '%H:%M').time()

That's it! A single, rather short line of code does the job.

The trick here is that the classmethod datetime.strptime() creates a datetime object with default values for the missing parameters. So the above code creates a datetime object with date set to 1900/1/1 and time set to 8:00AM, with seconds and microseconds set to 0. And from this datetime object we can extract the time component by calling the time() method. Cool!

Why we choose to

2014-05-04T00:00:00+08:00

Well, the context might look very obvious. But if you think a little deeper, you'll realize that there are many themes in life where this can be applied.

[Making is fun
At least for most
Rearing is less fun
Except for most
Yet we want to
Yet we dare to
Out of need
Or wanton greed
Out of fear
Or is end near
For we have little to leave
When called to leave
So we choose to
So we don't have to not to.]

Bliss!

2014-05-02T00:00:00+08:00

Was out on labor day for a medical appointment and had a chance to wander around an underground mall by myself, for couple of hours. The key thing here is 'by myself' -- I haven't managed to do this for quite a while. At least not since the arrival of our daughter, which was 2 years ago!

Anyway having lunch at the mall, relishing the what was to be a short-lived experience of my single days, I was inspired to write a small poem, based on my observations around the mall during that hour. I call it as a poem, not knowing what qualifies as one. You be the judge.

It started with the first four lines, but slowly took the following form on the ride back home. Context is my decision to go for a KFC burger as lunch, giving in to the momentary temptation, that could be safely pursued in the absence of my better half.

[A fleeting bliss]
[Derived from junk]
[Makes me wanna hiss]
[For what's inside is gunk.]
[
]
[Hues of purple and pink]
[Intent is not to diss]
[But can't stomach the kink]
[
]
[Seemed to last longer than my junk.]
[
]
[Heaving his chest, looks like a trunk]
[With wife and kids]
[In Taiwan for funk]
[
]
[Would have to search through junk.]
[
]
[A train full of my ilk]
[With phones pasted to their eyes]
[Deep in their world sunk]
[
]
[Having failed to notice THIS hunk.]
[
]
[Confined to the junk]
[At least I had my bliss]
[Though only for a blink.]

Comment on The Hindu on MH370

2014-04-02T00:00:00+08:00

I sent another comment to The Hindu on MH370 disappearance and the recent developments around it which they published. Here's the screen-cap:

The 'correction' here refers to the announcement by the authorities that the last words spoken by the pilots (or whoever was in control of the plane) to the air traffic control is not the original, and now famous, "All right, good night", but "Good night Malaysian three seven zero".

Crazy idea

2014-03-21T00:00:00+08:00

Here's a crazy idea on how to increase the chances of locating the missing plane. This is based on the assumption that the flight had indeed crashed in the ocean as many experts seem to believe.

Create a compact sonar device specifically designed to locate plane's missing black box and/or other possible debris. I guess such a device is already available. And if not, we ought to be able to create one by customizing the firmware of an existing hardware. A few good programmers, within a couple of pizza and coke loaded nights, ought to be able to turn this around.
Make this device send a periodic signal to a satellite. If that requires too much power and could drain the on-board battery too fast, let it send signals to a flying receiver such as a AWACS plane. Alternatively, float a few balloons such as the ones Google experimented with in New Zealand for cheap Internet access equipped the appropriate receiver.
Identify a few migratory marine creatures (is that the correct way to refer to them?) such as whales, sharks, and other large fish that move around the Indian Ocean. Attach the device to them and set them free at various locations in the ocean. This might seem too sci-fi, but remember that this is already being done. Marine biology relies on tagging large fish with radio transmitters to study their migration pattern. Also remember National Geographic's critter cam?
Keep resources stand-by to start data-mining and writing analytics code and remove the noise from the data generated by the hardworking creatures.

If the creatures do find us the missing plane, we humans give them something back in return by promising not to hunt or eat them ever! Well, may be for a couple of years giving them enough time to recover from our predatory onslaught!

May be crazy, but given how the world's most advanced existing hardware has yielded nothing so far, I would say there is no harm in trying this out. Provided that there are marine creatures do cover such a large distance and depth and that they can be caught and fitted with the device with relative ease. And provided that we can create these devices small enough, yet with enough battery power to last a few weeks or even couple of months, so as to not to affect their movement.

Linux Command Output Redirection Rules

2013-12-24T00:00:00+08:00

Here's another thing that I learned today -- command output redirection syntax.

> redirects the program's output to the specified file which is specified afterwards. If > is preceded by ampersand, shell redirects all outputs (error and normal) to the file right of >. If you don't specify ampersand, then only normal output is redirected.

In other words:

command &>  file # redirect error and standard output to file
command >   file # redirect standard output to file
command 2>  file # redirect error output to file

This is how things work on Ubuntu and bash. Other shells could very well be different.

MySQL Post Install Setup

2013-12-24T00:00:00+08:00

After MySQL is installed, either as part of Ubuntu Server installation sequence or separately through an appropriate apt-get command, it needs to be properly initialized for use by a client.

One key concept of any standalone database engine is that they have their own user namespace and associated access rights. MySQL is no different and therefore this needs to be appropriately setup before any client can create/update/delete entries into any table.

Essentially this involves the following steps, all to be undertaken as the Administrator (or the root account, by default):

Create a user account with an associated password. Password is optional, but recommended. Note that only the Administrator can create accounts with the default access rights setup as part of MySQL installation. So you need to connect to MySQL as root.
```
$ mysql -uroot -p<password> # connect as root
...
mysql> CREATE USER '<user>'@'localhost' IDENTIFIED BY '<password>'; # inside MySQL console
```
The key point here is that when logging into MySQL as root, the -p option has to be explicitly specified for it to prompt for password. Without it, you will get an access denied error. Of course you can supply the password inline in the command line right after the -p option as well.

Another thing that I kept getting wrong is that there is no space between the -u option and the value supplied to it. This is a little different from a lot of tools that accept various command line parameters. The same goes for the -p option as well.

Create the database(s) that will be used by the client
```
mysql> CREATE DATABASE <dbname>;
```
Grant appropriate privileges (or all privileges, as here) to the new client user account on the database(s) just created
```
mysql> GRANT ALL on <dbname>.* TO '<user>'@'localhost';
```

Now you should be able to connect to MySQL as the newly created user and manipulate the just created database any way you need to.

Setting up Ubuntu for Python Development

2013-12-23T00:00:00+08:00

This is more for my own reference as I often forget the various steps that I followed to get an environment set up. This post lists the steps that I followed to setup an Ubuntu Server for Python/Django/MySQL environment. The steps are drawn from various resources on the web -- official documentation from Ubuntu, some blog posts and a few pointers on stackexchange/stackoverflow. All credits due to all the kind souls who shared knowledge.

The environment that I want is Ubuntu Server, Python, Django with MySQL support. MySQL support is through MySQLdb.

These are the steps I followed on Ubuntu Server 12.04 LTS with MySQL. All commands are run from the home folder unless otherwise specified.

Install pip, the tool for installing & managing Python packages
```
sudo apt-get install python-pip
```

Install virtualenv and virtualenvwrapper.

sudo pip install virtualenv
sudo pip install virtualenvwrapper

Create the virtual environment, in this case myenv
```
virtualenv myenv
```
Activate the newly created virtual environment, so that Django and other dependent components will be installed to this virtual environment and not to the system-wide python folders.
```
source myenv/bin/activate
```
This should cause the shell prompt to be prefixed with (myenv) as in:
```
(myenv)hari@devmachine:~$
```
Install Django in the newly created virtual environment
```
pip install django
```
Note that we don't prefix the command with sudo here as Django is being installed to the virtual environment in our home folder.
Now we need to install the MySQL ORM module for Python so that we can access MySQL DB from inside Django. Unfortunately, Django does not come pre-built with this module. Neither does the standard Python 2.7, that is installed with Ubuntu installation.
```
sudo apt-get install python-dev libmysqlclient-dev
sudo apt-get build-dep python-mysqldb
sudo pip install pip --upgrade
pip install MySQL-python
```
MySQL interface from Python is through MySQLdb and this package is provided in source form. So when you install it, the installation attempts to compile the source files. These sources include a C file which links to the MySQL client libraries. This is why you have to install the libmysqlclient-dev package in the beginning followed by the build-dep python-mysqldb which installs gcc (if it's not already installed). Finally, we make sure pip is of the latest version before instructing it to install MySQL-python
Verify that MySQLdb is indeed installed correctly and accessible from Python by launching Python interpreter and importing it. You should see something like:
```
Python 2.7.3 (default, Sep 26 2013, 20:08:41)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import MySQLdb
>>>
```
Note how importing the moduld does not result in errors indicating that the installation was successful.

VirtualBox - Sharing a host folder on Ubuntu guest

2013-12-23T00:00:00+08:00

I have been using VirtualBox to run Ubuntu Server on my Windows PC over the last couple of weeks. The more time I spend with it, the more I find the need to get myself acclimatized with the good old command line method to interact with your OS. Though I'm a pretty fast learner and have spent quite a bit of time on Linux and its variants in the past, the sheer number of commands that one has to learn to manage the large array of interfaces in present-day OS is astounding.

For example, I wanted to share a folder from my Windows host to the Ubuntu guest. How do I go about doing it? How do I mount a USB stick on guest OS?

Luckily, answers to most of these questions are only a few clicks away with the help of Google. However, since I expect to carry out some of these tasks quite often in the future, I'm going to try and record things that I discover and re-learn in various blog posts, so as to act as a reference for myself. This post talks about how to go about accessing a Host folder that is shared through VirtualBox in the Ubuntu guest through command line.

1. Install Guest Additions Package

For this, you have to first install the Guest Additions package that comes with VB. It's rather straightforward if you have the Ubuntu GUI installed -- just select the supplied .iso image as your CD drive content and voila, the autorun-like mechanism pops. But on a OS without the GUI, you need to mount it yourself. This is how:

mount -o loop /dev/sr0 /media/cdrom

Of course this will succeed only after the Guest Additions CD has been inserted into the guest's CD drive. /media/cdrom is typically created when Ubuntu is installed. If not use a different folder path (one that exists) or create one to your liking and use the path to it.

Once successfully mounted, run the *VBoxLinuxAdditions.run* from /*media/cdrom* (you need root privileges, so either use *sudo* or *su* as root). This could take a little while as Guest Additions module has to be built.

2. Mount the Shared Folder

Once this is done, issue the following command to mount the shared folder

sudo mount -t vboxsf temp /mnt/winshare/temp

*temp* here is the folder that's shared from Windows and */mnt/winshare/temp* is the folder in the Ubuntu guest where this folder is mapped to. If */mnt/winshare/temp* does not exist, create it first. And temp must have been shared from **VirtualBox → Settings → Shared Folders**.

Play youtube videos on Raspbian using omxplayer

2013-10-02T00:00:00+08:00

This is a follow up post to the original one made by Jojen on setting raspbian to use omxplayer to play videos embedded in webpages. This yields much smoother playback on raspberry PI as omxplayer is built to use the PI GPU rather than software emulation.

BTW, some blog posts might suggest that you can install gnash and its browser plugin, but since it's not hardware accelerated, I found it's performance to be simply non-acceptable. So if you stumble upon those posts, you might as well save yourself some time and not go down that path.

Back to Jojen's post. Everything in the post works as it says it would, including playing the sample video the link to which he has at the bottom of his post. The trouble starts when you try to play youtube videos. I found this was due to two reasons:

Incorrect permissions on the script omxplayer-youtube.sh
Midory user addon Javascript failing to detect video links since youtube as has changed the relevant div tag id

The problems can be fixed thus:

Change the permissions on omxplayer-youtuble-sh (located in /opt/media-berry/server) through the command: [chmod 755 omxplayer-youtuble.sh]
Update the replace_video_user.js (located in \~/.local/share/midori/scripts) such that ["#watch7-video"] in line 61(at the time of writing this post).

If it all worked, you should see the youtube video thumbnail being replaced with a rectangle with a play button in the middle.

There's a caveat. After playing the main video, if you try to visit one of the related clips through the links on the sidebar at the right, the video that you just played will be played. This probably has to do with Javascript variable lifetime in the user script. I haven’t had the time to look into it, but when and if I do I’ll post the updated script.

Debugging Django Template Tags

2013-09-21T00:00:00+08:00

Here's a cool bit of code snippet that I found from the web (original post here). I'm recording it here for my own easy reference without having to google it all the time. It describes a simple technique to debug Django templates using the python debugger - pdb.

@register.filter 
def pdb(element):
    import pdb; pdb.set_trace()
    return element

Now, inside a template you can do

{{ template_var|pdb }}

and enter a pdb session (given you're running the local development server) where you can inspect element to your heart's content.

It's a very nice way to see what's happened to your object when it arrives at the template. :::

MQC Client

2013-09-04T00:00:00+08:00

Beging a legacy component, Mercury Quality Center web plugin does not work very well with new versions of Internet Explorer. MQC Client is a little program that addresses this by hosting the MQC web plugin inside it making it work with the newer IE versions that come as standard with Windows 7.

I have not tested it on Windows 8, but it ought to work.

More about the program and how to use it is here.

Friendly(X) UI

2013-07-30T00:00:00+08:00

One would think that in this day and age of highly interactive user experiences, simple message boxes can't go too wrong as far as UX is concerned. At least that's what I thought until I came across this dialog:

Perhaps you don't notice what I'm grumbling about -- the link provided to get the latest virus definition files (erased for confidentiality reasons) is neither a hyperlink nor a selectable text! That's right, you can't even select that text and copy-paste into the browser leaving you with the only choice of typing the link manually, a letter at a time.

Well, at least they got the grammar right and managed to convey the message correctly. So it's not as bad as it could've been!

Setting up Shared Repository in Git

2013-07-30T00:00:00+08:00

This is not really a post and more of a web clipping for my own future reference.

This page contains useful instructions on how to setup a shared repository with Git. I use it to back up by work on to a thumbdrive.

You might wonder why am I not using one of the free services such as GitHub or BitBucket. Answer is simple -- the code is closed source.

From my research most of the online Git services only provide free hosting for open source code. May be there is a provider out there who does offer true free hosting for closed source, but I don't have the time to pour through their licence agreement and ensure that free is really free without any catches.

Hence the decision to store things locally. In any case Git makes it so easy to set things up.

Typesafe Logging

2013-06-19T00:00:00+08:00

Problem

One of the common sources of runtime errors in C++ comes from using the good old vararg function types -- printf() or its variants. The problem stems from type mismatch between the type specifier in the format string and the actual arguments supplied to match this format string. Though most of the present-day software do not employ the printf function for user interaction, a variant of it is quite heavily used -- sprintf() for outputting log messages.

Often log messages are written in response to an error condition and formal testing does not always simulate all the possible error conditions which means some code goes untested. Also, log messages typically have an associated logging level which is usually set to a medium value by default. Testers often do not set the logging level to the lowest possible level before embarking on the testing process meaning that messages at the lowest level are also not exercised during testing. Another real-world situation is when the released code is maintained by support engineers who adds one or more arguments to the log statement for their own debugging purposes, but fails to test it thoroughly.

This post discusses a simple technique of adapting the superior C++ streams infrastructure as an extended logging framework. The mechanism is built such that it can sit on top of an existing legacy logging framework so that introducing it would not require rampant system-wide code changes.

As the reader might be aware, C++ puts lot of emphasis on writing type-safe code and C++ streams provides just this. (Check out Stroustrup's tips on how to avoid unsafe code here and one of his tips is to avoid using vararg functions).

Design Goals and Constraints

To reiterate, this is a solution which has to fit into an existing logging framework. The framework can be thought of the providing the following features, which is typical of many legacy logging infrastructure.

Multilevel logging with a global system wide logging level
Each message has an attached logging level and is only output if the global logging level is equal or lower than this level.
Provides printf() like function to concatenate multiple argument values into a log statement.

Given this, the design goals and various constraints can be summarized as below:

Goals

Provide a type-safe logging framework by leveraging on the C++ streams framework
Maintain backward compatibility by retaining the old logging infrastructure which means that the log output should be the same for both types of logging.
All log messages should get output to the destination as soon as the statement is executed before the next line is is executed.
Maintain logging levels as in the legacy framework and integrate it as an implicit feature.

Constraints

Can only utilize on the STL libraries which is available on most platforms and compilers
Don't use C++11 language features as certain compilers do not support it yet.

Solution

To anyone familiar with the STL library and its streams facility, the excellent stringstream class can be applied effectively to compose and output the log message. This allows values of multiple variables/objects to be captured as a string through a single statement. A typical logging code using this (one without any explicit framework) could look like:

std::stringstream ss;
ss << "Error creating connection to the given address, error code: " 
    << ::GetLastError() << endl;
g_logger->log(LOG_LEVEL_ERROR, ss.str());

Where g_logger is a global instance of a class that provides the legacy logging framework through its vararg methods. While this would definitely work, it has couple of shortcomings:

Every log statement requires three lines of code. First one to declare a local variable, second to compose the log message and third to write it to the log destination.
There's a necessity to declare a local variable which means that two log statements in the same block of code have to be using two separate locals. Alternatively, each log statement has to be in its own block.

An alternative implementation could be to declare a globally accessible instance of a specialized STL stream (which could implement its own stream buffer so as to redirect output to the legacy module-wide logger instance), say g_ls, to which data can be written as below:

using namesapce std;
g_ls << "Error creating connection to the given address, error code: "
    << ::GetLastError() << endl;
g_ls << flush;

This has the drawback that the programmer needs to call std::flush() after every logging statement so as to achieve the stated goal of immediate output to the logger destination before the execution of the next line of code. Not to mention the additional work of specializing the STL stream with its own stream buffer. And as in the previous example, we've to rely on the programmer being disciplined enough to make a call to flush() repetitiously after composing every log message.

Both the possible approaches presented above have the major drawback that it requires conscious and disciplined action from the user to make the logging work as per the goals set forth earlier. And this involves repetitious coding practice that is best avoided if robust and resilient software is the desired result.

What we need is a mechanism which upon entering a statement provides a blank slate for users to accumulate their messages in and upon exit dumps the content to the log destination. Which means we need an entry and exit trigger where we can plug in the necessary code to interface it with the legacy logging framework. Naturally, the object constructor and destructor comes to mind. Upon construction we get an object which acts as a blank slate where different messges can be stored and the destructor can then dump the slate contents to the logging medium. This can be implemented as:

class AutoLogger {
    int loglevel_;
    std::stringstream ss_;

    AutoLogger(); // hide default constructor!
    AutoLogger(const AutoLogger&); // a good practice
public:
    AutoLogger(explicit int level) : loglevel_(level) {}
    ~AutoLogger() {
        g_logger->log(level_, ss_.str());
    }
    std::stringstream& getStream() { return ss_; }
};

Thereafter we can employ it as below:

AutoLogger al1(LOG_LEVEL_ERROR);
al1.getStream() 
    << "Error creating connection to the given address, error code: "
    << ::GetLastError() << endl;

The above code meets most of our goals set forth at the beginning, except the third one that log messages should be written as soon as the statement that composed it is executed. In the above code, the buffered message would only be written when the variable goes out of scope, which is when its destructor would get called. That means code like this fails to meet the requirement refreshed above:

AutoLogger al1(LOG_LEVEL_ERROR);
al1.getStream() 
    << "Error creating connection to the given address, error code: "
    << ::GetLastError() << endl;
// more code...
al1.getStream() 
    << "Error retrieving the secondary server address"
    << ::GetLastError() << endl;

So we need a mechanism to prevent users from instantiating AutoLogger locally and this can be easily accomplished by hiding its constructor. The amended code would like like:

class AutoLogger {
    int loglevel_;
    std::stringstream ss_;

    AutoLogger(); // hide default constructor!
    AutoLogger(const AutoLogger&); // a good practice
    AutoLogger(explicit int level) : loglevel_(level) {}
public:
    ~AutoLogger() {
        g_logger->log(level_, ss_.str());
    }
    std::stringstream& getStream() { return ss_; }
};

This however now presents another problem, if constructor is hidden how is the user going to be able to instantiate an object and use it? Well, we provide a friend function to do just this. Being a friend, the function can instantiate that class and return it. So the code becomes:

class AutoLogger {
    int loglevel_;
    std::stringstream ss_;

    AutoLogger(); // hide default constructor!
    AutoLogger(const AutoLogger&); // a good practice
    AutoLogger(explicit int level) : loglevel_(level) {}
    friend AutoLogger getAutoLogger(int level);
public:
    ~AutoLogger() {
        g_logger->log(level_, ss_.str());
    }
    std::stringstream& getStream() { return ss_; }
};
AutoLogger getAutoLogger(int level) {
    return AutoLogger(level);
}

Now to use the framework, user will have to write code like this:

getAutoLogger(LOG_LEVEL_ERROR).getStream()
    << "Error creating connection to the given address, error code: "
    << ::GetLastError() << endl;

[This truly meets all the requirements that we set forth at the beginning of our discussion. You get increased typesafety and consequently no more runtime errors, log messages are written at the completion of every statement and logging levels have to be attached to every log message.]

Note that the key trick here is hiding the constructor of the AutoLogger class. This in no way restricts its usage, only its instantiation is constrained. In fact this is a common technique adopted by the class factory pattern to force users to use the factory to instantiate classes.

Additional Benefits

Besides the runtime safety benefits extolled earlier, STL streams based logging provides yet another capability that can significantly improve programming productivity and code quality. Since we're using STL streams as the interface for logging, any C++ class can be easily extended to support dumping its state to the log medium by overriding the output operator '\<\<'.

The significance of this becomes relevant when we observe how various C++ classes in a software are typically developed. Usually they are written individually, often using a console program to perform unit testing. Thereafter, when a minimum amount of class facade is written and tested, the component is integrated with the larger project where the necessary client code is written. It's beneficial for such component classes to include a method to dump their state to the logger. If the streams based logger is employed throughout, the overridden streams output ('\<\<') operator can be used to dump the object's state in both the unit-test console program (through cout or cerr) as well as the larger project (presumably through a class scuh as AutolLogger above).

As an example let's consider the following Employee class:

// employee.hpp
class Employee {
    // data
    std::string firstname_;
    std::string lastname_;
    // needs to be declared as friend so that the function
    // can directly access the class private data members
    friend std::basic_ostream<char>& operator<<(
        std::basic_ostream<char>&, const Employee&);
public:
    Employee(std::string const& f, std::string const& l)
        : firstname_(f), lastname_(l)
    {}
};
// implementation
std::basic_ostream<char>& operator<<(std::basic_ostream<char>& os,
        const Employee& emp)
{
    os << "Employee: " 
        << emp.lastname_ << ", " << emp.firstname_;
    return os;
}

With this in place, now we can unit test programs as:

#include <iostream>
#include "employee.hpp"
int main(void)
{
    Employee e("John", "Smith");
    std::cout << e;
}

And in the production code (with the above logging framework integrated):

Employee e = getAuthenticatedUser();
getAutoLogger(LOG_LEVEL_INFO).getStream() 
<< "Initializing document rights for user: "
    << e << std::endl;

Conclusion

The above example illustrates one very important aspect of the C++ language. With only the STL and clever usage of the auto object scope rules, one can develop pretty sophisticated mechanisms that can yield very robust and high quality software.

The other important aspect illustrated by this solution is the variable scope rules and how it can be effectively used to completely eliminate the possibility of memory leaks. Local scoped objects and the auto destruction semantics is one of the most powerful features of the language and effective employment of this can eliminate the need for explicit memory management in the code.

Explicit Overrides

2013-02-26T00:00:00+08:00

::: {dir="ltr" style="text-align: left;" trbidi="on"} In the last two days, I have been twice bitten by the loose nature of legacy C++ language.

I had developed a base class that abstracts the core features of a module a year back. The class itself was a great success. Since developing it for a specific module, I successfully refactored another module to use this and was able to do so in a relatively short window of 1 week. This is the background.

Recently, I had an opportunity to re-use this class for another module and having been successful earlier twice, I guess I was a little callous in using it without referring to the accompanied documentation. The design requires that I override a bunch of methods and admittedly I was as little cocky -- if I had designed and implemented it earlier, it should be a snap for me to re-use it. So I wrote the code, compiled it, removed a few syntax errors and dropped it into test. All good, except that couple of overridden methods were not being invoked!

WTF? It also so happened that the callback to the overridden methods were triggered by an external event. Off I went tracing this path to see what's going on. Nothing wrong there either. WTF??

With nowhere else left to investigate, I went back to the base class design. And it was then that I realized the rather embarrassing mistake that I had made. I guess the code is perhaps worth a thousand words -- so here it goes.

Base class implementation:

class Base {
protected:
   virtual void onInitialize();
   virtual void onShutdown(bool fGraceful);
};

And here's the derived class:

class Derived : public Base {
protected:
   virtual void onInitialize();
   virtual void onShutdown();
};

Note the mistake? I had forgotten about the bool fGraceful parameter to onShutdown() and as a result it was defined as a new method in the derived class!

This is precisely the kind of problem that explicit overrides feature in C++11 allows you to evade. If you have a conforming compiler (I didn't), you can declare the derived class as

class Derived : public Base {
protected:
   virtual void onInitialize() override;
   virtual void onShutdown() override;
};

What the trailing override does is to inform the compiler that onShutown() is a method that is overridden from its namesake in the base class. And the compiler, seeing that the prototype for the method in derived class doesn't match what's in the base, would raise an error which would allow you to spot the problem and fix it, even when you're overconfident and cocky!

As for me, I don't have the luxury of using a C++11 compiler and have to live with the possibility that I will make these mistakes again! :::

Back to the roots

2013-02-06T00:00:00+08:00

For the past couple of weeks, I have been working on something that I thought I had left behind a long time ago, a very long time ago -- a windows device driver!

I had launched my career writing drivers, first for OS/2 and then eventually for Windows. When I say Windows, I'm referring to Windows NT 3.51. That's correct, NT 3.51! And here's a surprise, the first Windows NT driver that I worked on ran on PowerPC platform! IBM was readying their new state-of-the-art PC platform and to play it safe wanted to give the user the choice of platforms to run -- OS/2 and Windows NT.

Being a new platform, the toolsets were not quite as well developed as they were for the more contemporary and stable x86. WinDBG (a far cry from the WinDBG of these days) had issues syncing the source line information in the PDB with the actual source files. Quite often I had to resort to using the disassembly to isolate the root cause and fix it. Another challenge that I faced then was that the x86 driver had plenty of inline assembly which had to be ported to either C or PowerPC's RISC instruction set.

But what sets apart the driver development experience then and now is the advances made with the virtualization technology. Gone are the days of NULL modem cable and the rather long wait with every WinDBG command. Instead you work with multiple VMs with a virtual serial port mapped to a named pipe and debugging different versions of the driver simultaneously! The fact that VM states can be saved and cloned at will really makes driver development a breeze.

One thing however, has not changed much. Much of NT driver is still written C with very little C++. It's a shame that MS has not managed to update their compiler such that it can generate a driver safe PE image. Let's hope that this will change in the near future.

Anyway, it's hard not to feel like I'm going back to my roots.

Back to the blogging world...

2013-01-30T00:00:00+08:00

I'm back online after a hiatus of almost 12 months.

Having maintained a self hosted blog on a shared hosting service and having gone through the pains of keeping the Wordpress version current, I'm gonna give the Blogger service a try.

Since I use Gmail as my primary email service, Blogger and its tight integration with Google services ought to make this a pleasant experience.

Also, Blogger provides free mapping of custom domains to the hosted blog. That means I just need to pay for my domain name -- no more hosting charges! Not sure if theres' a way by which I can get my domain mail account to work for free as well. Let's see.

May I find the drive to persevere with this effort for the long term. Wish me luck!