Category Archives: Tolon’s Coding Lab

Full Width and Height Google Maps with Angular Material and Bower

After having some trouble getting a full width and height map working with Angular Material, I am sharing my solution to hopefully spare others the same pain.

Screen Shot 2016-01-31 at 21.02.38

The solution isn’t anything special, but I did find it tricky as I hadn’t used Angular Material before.

For this example, create a new angularjs scaffold as described briefly at https://github.com/yeoman/generator-angular, or in a more detailed tutorial at http://yeoman.io/codelab/index.html.

Once your initial scaffold is complete, add the google-maps and angular-material bower packages using:

bower install --save google-maps angular-material

Replace the scaffolded content with a modified ng-view div, like this:

...
    <!-- Add your site or application content here -->
    <div ng-view="" flex class="layout-fill"></div>

    <!-- Google Analytics: change UA-RAAAWR-! to be your site's ID -->
...

Replace the main.html view content with this:

<md-content layout="column" flex class="layout-fill" class="content-wrapper">
  <div flex class="layout-fill" id="map"></div>
  </div>
</md-content>

Add the loader script into your controller main.js:

angular.module('srcApp')
  .controller('MainCtrl', function () {

    GoogleMapsLoader.load(function (google) {
      var mapOptions = {
        zoom: 11,
        center: new google.maps.LatLng(54.5767, -1.2355)
      };

      _map = new google.maps.Map(document.getElementById('map'), mapOptions);
    });

  });

And that’s it! I’ve used this technique on my NE Travel Data website showing live CCTV camera images on top of a Google Maps traffic layer. If you have the time, check it out at the link below.

Tolon Traffic Cameras

The code for this example can be found on Github at https://github.com/TolonUK/angularjs-googlemaps.

Beware setlocale() in Visual C++ 2012 onwards!

My new article in ACCU’s CVu magazine describes how investigating a crash in a popular open source library led to the discovery of a change in behaviour in the C runtime library implementation that could have a significant detrimental impact on applications using C runtime locales across multiple threads that are compiled with Visual C++ 2012 & 2013 – http://accu.org/var/uploads/journals/CVu264.pdf#page=12

Transporting SDO_GEOMETRY objects across an Oracle DBLINK in PLSQL

At my work we have a system that generally speaking consists of two Oracle 10g databases, the ‘SpatialDB’ and the ‘AppDB’. The SpatialDB holds a multi-gigabyte set of static national road-network geometry data and the AppDB holds the dynamic runtime data. A database link is defined in the AppDB so that it can query the SpatialDB and invoke stored procedures.

Generally what happens is that an action somewhere in the system (user interaction, external system communications, etc.), will result in something happening in the database.

In this case a package method in the AppDB needed to obtain a geometry from the SpatialDB. Surprisingly, this turned out to be a quite a difficult thing to achieve. It seems that along with cursors, LOBs and objects in general, an SDO_GEOMETRY cannot be transported across a database link as an out parameter of a package method. The following is a summary of the methods I discovered for overcoming this.

Method #1 – Use the SQL Engine

Whilst PLSQL cannot cope with transporting objects across a database link, it turns out that the SQL engine copes just fine. Objects from the remote database can be selected into a local table and then that table can be queried to obtain the result.

INSERT INTO LOCAL_TABLE SELECT ORA_GEOMETRY FROM REMOTE_TABLE@SPATIALDB;
SELECT ORA_GEOMETRY FROM LOCAL_TABLE;

Method #2 – Transport via WKT

Oracle 10g contains methods for transforming a geometry into a format called ‘Well Known Text’, which is a string representation for geometries. Two methods are provided for these transformations, SDO_UTIL.TO_WKTGEOMETRY and SDO_UTIL.FROM_WKTGEOMETRY.

The technique here is to transform the geometry into a WKT string on the remote side, transfer the string across as a PLSQL out parameter, then transform the string back into a geometry object on the local side.

However, the following limitations must be noted.

  • Transfering the string as a VARCHAR2 type will limit its length to 32767 characters.
  • No 3D support.
  • The SRID (reference system) is not preserved.
  • Complex geometries are not supported.

Method #3 – Transport via GML

Having read that Oracle’s GML implementation gave better geometry support, I thought I would give it a try instead of WKT. I was a bit concerned about the amount of valuable VARCHAR2 space that would be wasted on XML tags, but I never got as far as testing whether it would be an issue.

So in a similar fashion to the WKT support, in the SDO_UTIL package you will find a TO_GMLGEOMETRY method to convert your geometry object into a GML string. Unfortunately for users of 10g, you will not find any corresponding method to convert GML into a geometry object. Disappointing to say the least.

Method #4 – Transport via Custom String

My final solution was to write my own conversion routine that simply took the geometry object and encoded its data members into a CSV string. This format does not attempt to interpret the geometry data, so all geometries and all dimensions are supported. SRID data is also preserved and because the transformation is doing less work that the WKT methods, it turns out to be faster too.

The tables below give the times recorded for encoding and decoding geometries of varying sizes and quantities. Times are in seconds.

2 Vertex Line Encoding
Time taken to encode a line with 2 vertices.

WKT GML Custom
1 Geometry 0.009328 0.002073 0.000057
100 Geometries 0.124836 0.154757 0.002367

2 Vertex Line Decoding
Time taken to decode a line with 2 vertices.

WKT GML Custom
1 Geometry 0.001482 0.009657 0.000616
100 Geometries 0.036376 0.081836 0.046199

100 Vertex Line Encoding
Time taken to encode a line with 100 vertices.

WKT GML Custom
1 Geometry 0.001141 0.003194 0.000627
100 Geometries 0.293802 0.273061 0.06347

100 Vertex Line Decoding
Time taken to decode a line with 2 vertices.

WKT GML Custom
1 Geometry 0.001889 0.003083 0.002381
100 Geometries 0.24252 0.251887 0.239928

Observations

The encoding of the sdo_geometry objects to a custom string format is clearly faster that encoding to WKT or GML using the stand methods in SDO_UTIL. This is especially true for simple geometries with few vertices.

The decoding of the custom string format back into sdo_geometry objects did not exhibit the same performance benefit, although the performance of decoding one simple 2-vertex geometry did show a marked improvement.

Conclusions

There are two main reasons I can see for the custom string encoding methods having a performance advantage.

  1. The custom string format is a closer representation of the binary format of the object with less translation required by the encoding logic.

  2. The custom string format is implemented in PL/SQL and not Java like the other methods. Although Java code should have a performance advantage, it seems likely that the cost of switching context between PL/SQL and Java is a significant overhead.

Code

I hope you find the code useful. If you find any bugs or make any improvements, I would appreciate the feedback.

code on github

Spec

CREATE OR REPLACE PACKAGE pg_user_geom_util AS
-- copyright (c) 2012, Nicander Ltd
	
	-- Translates a geometry object into a string representation 
	-- for storage and transport. See also StringToGeom.
	FUNCTION GeomToString(pGeom IN mdsys.sdo_geometry ) RETURN VARCHAR2 DETERMINISTIC;
	PROCEDURE Test_GeomToString;
	
	-- Translate a geometry string representation into a geometry object.
	-- See also GeomToString.
	FUNCTION StringToGeom(pString IN VARCHAR2) RETURN MDSYS.SDO_GEOMETRY DETERMINISTIC;
	PROCEDURE Test_StringToGeom;
	
	-- Tests the conversion between sdo_geometry objects and a string representation.
	-- Tests for pGeomCount geometries, each with pGeomVertexCount vertices.
	PROCEDURE PerfTest_GeomStrings(pGeomCount IN INTEGER, pGeomVertexCount IN INTEGER);
	
	-- Runs PerfTest_GeomStrings(INTEGER, INTEGER) for a selection of values.
	PROCEDURE PerfTest_GeomStrings;
	
END pg_user_geom_util;
/

show errors package pg_user_geom_util;

Body

CREATE OR REPLACE PACKAGE BODY pg_user_geom_util AS
-- copyright (c) 2012, Nicander Ltd

-- Gets the next token in a separated list. See StringToGeom for an example.
PROCEDURE get_token( iStart   IN NUMBER,
           sPattern in VARCHAR2,
           sBuffer  in VARCHAR2,
           sResult  OUT NOCOPY VARCHAR2,
           iNextPos OUT NOCOPY NUMBER ) IS
  nPos1 number;
  nPos2 number;
BEGIN
  nPos1 := Instr(sBuffer, sPattern, iStart);
  IF nPos1 = 0 then
  sResult := rtrim(ltrim(substr(sBuffer, iStart, LENGTH(sBuffer) - iStart)));
  ELSE
  sResult  := Rtrim(Ltrim(Substr(sBuffer, iStart, nPos1 - iStart)));
  iNextPos := nPos1 + 1;
  END IF;
END;

--------------------------------------------------------------------------------
  
FUNCTION NumberToString(pNumber IN NUMBER) RETURN VARCHAR2 IS
BEGIN
  IF (pNumber IS NULL) THEN
    RETURN 'NULL';
  ELSE
    RETURN TO_CHAR(pNumber);
  END IF;
EXCEPTION
  WHEN OTHERS THEN
    --DBMS_OUTPUT.PUT_LINE('Exception caught: ' || DBMS_UTILITY.FORMAT_ERROR_STACK() || ', ' || DBMS_UTILITY.FORMAT_ERROR_BACKTRACE());
    RETURN 'NULL';
END;
  
--------------------------------------------------------------------------------
  
FUNCTION StringToNumber(pString IN VARCHAR2) RETURN NUMBER IS
BEGIN
    RETURN TO_NUMBER(pString);
EXCEPTION
  WHEN OTHERS THEN
    --DBMS_OUTPUT.PUT_LINE('Exception caught: ' || DBMS_UTILITY.FORMAT_ERROR_STACK() || ', ' || DBMS_UTILITY.FORMAT_ERROR_BACKTRACE());
    RETURN NULL;
END;
  
--------------------------------------------------------------------------------
  
  -- String format is as follows:
  --
  -- <GTYPE>,<SRID>,<POINT>,<ELEM_INFO>,<ORDINATES>
  --
  -- Where:
  -- <GTYPE> := <OPTIONAL_NUMBER>
  -- <SRID> := <OPTIONAL_NUMBER>
  -- <POINT> := NULL | SDO_POINT_TYPE, OPTIONAL_NUMBER, OPTIONAL_NUMBER, OPTIONAL_NUMBER
  -- <ELEM_INFO> := NULL | SDO_ELEM_INFO_ARRAY, NUMBER, {OPTIONAL_NUMBER, ...}
  -- <ORDINATES> := NULL | SDO_ORDINATE_ARRAY, NUMBER, {OPTIONAL_NUMBER, ...}
  --
  -- <OPTIONAL_NUMBER> := NULL | NUMBER
  -- entries in curly braces are repeated from n times
FUNCTION GeomToString(pGeom IN mdsys.sdo_geometry ) RETURN VARCHAR2 DETERMINISTIC IS
  vString VARCHAR(32767);
  vCount NUMBER;
BEGIN
  -- GTYPE
  vString := NumberToString(pGeom.SDO_GTYPE);
  
  -- SRID
  vString := vString || ',' || NumberToString(pGeom.SDO_SRID);
  
  -- POINT
  IF (pGeom.SDO_POINT IS NULL) THEN
    vString := vString || ',NULL';
  ELSE
    vString := vString || ',SDO_POINT_TYPE,' || NumberToString(pGeom.SDO_POINT.X) || ',' || NumberToString(pGeom.SDO_POINT.Y) || ',' || NumberToString(pGeom.SDO_POINT.Z);
  END IF;
  
  -- ELEM_INFO
  IF (pGeom.SDO_ELEM_INFO IS NULL) THEN
    vString := vString || ',NULL';
  ELSE
    vCount := pGeom.SDO_ELEM_INFO.COUNT;
    vString := vString || ',SDO_ELEM_INFO_ARRAY,' || vCount;
    FOR i IN 1..vCount LOOP
      vString := vString || ',' || NumberToString(pGeom.SDO_ELEM_INFO(i));
    END LOOP;
  END IF;
  
  -- ORDINATES
  IF (pGeom.SDO_ORDINATES IS NULL) THEN
    vString := vString || ',NULL';
  ELSE
    vCount := pGeom.SDO_ORDINATES.COUNT;
    vString := vString || ',SDO_ORDINATE_ARRAY,' || vCount;
    FOR i IN 1..vCount LOOP
      vString := vString || ',' || NumberToString(pGeom.SDO_ORDINATES(i));
    END LOOP;
  END IF;
  
  RETURN vString;
END;
  
--------------------------------------------------------------------------------
  
PROCEDURE Test_GeomToString IS
  vGeom MDSYS.SDO_GEOMETRY;
  vString VARCHAR2(32767);
BEGIN
  -- TEST CASE 1 - POINT
  dbms_output.put_line('----- TEST CASE 1 -----');
  vGeom := mdsys.sdo_geometry( 2001, NULL, MDSYS.SDO_POINT_TYPE(12345, 67890, NULL), null, null );
  dbms_output.put_line('Src Geom: ' || SDO_UTIL.TO_WKTGEOMETRY(vGeom));
  vString := GeomToString(vGeom);
  IF (vString IS NULL) THEN
    vString := '<NULL>';
  END IF;
  dbms_output.put_line('Dest String: ' || vString);
  
  -- TEST CASE 2 - LINE
  dbms_output.put_line('----- TEST CASE 2 -----');
  vGeom := MDSYS.SDO_GEOMETRY(2002,NULL,NULL,MDSYS.SDO_ELEM_INFO_ARRAY(1,2,1),MDSYS.SDO_ORDINATE_ARRAY(652737.95,6589964.213,652741.222,6589983.148,652746.77,6590018.745,652752.029,6590050.2,652759.351,6590085.732));
  dbms_output.put_line('Src Geom: ' || SDO_UTIL.TO_WKTGEOMETRY(vGeom));
  vString := GeomToString(vGeom);
  IF (vString IS NULL) THEN
    vString := '<NULL>';
  END IF;
  dbms_output.put_line('Dest String: ' || vString);
END;
  
--------------------------------------------------------------------------------
  
FUNCTION StringToGeom(pString IN VARCHAR2) RETURN MDSYS.SDO_GEOMETRY DETERMINISTIC IS
  vGeom MDSYS.SDO_GEOMETRY;
  vPos NUMBER := 1;
  vNextPos NUMBER := 1;
  vToken VARCHAR2(32767);
  vSep CONSTANT VARCHAR2(2) := ',';
  vCount NUMBER;
BEGIN
  vGeom := MDSYS.SDO_GEOMETRY('POINT(0 0)');
  
  -- GTYPE
  get_token(vPos, vSep, pString, vToken, vNextPos); vPos := vNextPos;
  vGeom.SDO_GTYPE := StringToNumber(vToken);
  
  -- SRID
  get_token(vPos, vSep, pString, vToken, vNextPos); vPos := vNextPos;
  vGeom.SDO_SRID := StringToNumber(vToken);
  
  -- POINT
  get_token(vPos, vSep, pString, vToken, vNextPos); vPos := vNextPos;
  IF (UPPER(vToken) LIKE 'SDO_POINT_TYPE') THEN
    vGeom.SDO_POINT := MDSYS.SDO_POINT_TYPE(NULL, NULL, NULL);
    get_token(vPos, vSep, pString, vToken, vNextPos); vPos := vNextPos;
    vGeom.SDO_POINT.X := StringToNumber(vToken);
    get_token(vPos, vSep, pString, vToken, vNextPos); vPos := vNextPos;
    vGeom.SDO_POINT.Y := StringToNumber(vToken);
    get_token(vPos, vSep, pString, vToken, vNextPos); vPos := vNextPos;
    vGeom.SDO_POINT.Z := StringToNumber(vToken);
  ELSE
    vGeom.SDO_POINT := NULL;
  END IF;
  
  -- ELEM_INFO
  get_token(vPos, vSep, pString, vToken, vNextPos); vPos := vNextPos;
  IF (UPPER(vToken) LIKE 'SDO_ELEM_INFO_ARRAY') THEN
    vGeom.SDO_ELEM_INFO := MDSYS.SDO_ELEM_INFO_ARRAY();
    get_token(vPos, vSep, pString, vToken, vNextPos); vPos := vNextPos;
    vCount := StringToNumber(vToken);
    IF (vCount > 0) THEN
      vGeom.SDO_ELEM_INFO.EXTEND(vCount);
      FOR i IN 1..vCount LOOP
        get_token(vPos, vSep, pString, vToken, vNextPos); vPos := vNextPos;
        vGeom.SDO_ELEM_INFO(i) := StringToNumber(vToken);
      END LOOP;
    END IF;
  ELSE
    vGeom.SDO_ELEM_INFO := NULL;
  END IF;
  
  -- ORDINATES
  get_token(vPos, vSep, pString, vToken, vNextPos); vPos := vNextPos;
  IF (UPPER(vToken) LIKE 'SDO_ORDINATE_ARRAY') THEN
    vGeom.SDO_ORDINATES := MDSYS.SDO_ORDINATE_ARRAY();
    get_token(vPos, vSep, pString, vToken, vNextPos); vPos := vNextPos;
    vCount := StringToNumber(vToken);
    IF (vCount > 0) THEN
      vGeom.SDO_ORDINATES.EXTEND(vCount);
      FOR i IN 1..vCount LOOP
        get_token(vPos, vSep, pString, vToken, vNextPos); vPos := vNextPos;
        vGeom.SDO_ORDINATES(i) := StringToNumber(vToken);
      END LOOP;
    END IF;
  ELSE
    vGeom.SDO_ORDINATES := NULL;
  END IF;

  RETURN vGeom;
END;
  
--------------------------------------------------------------------------------
  
PROCEDURE Test_StringToGeom IS
  vString VARCHAR2(32767);
  vGeom MDSYS.SDO_GEOMETRY;
BEGIN
  -- TEST CASE 1 - POINT
  dbms_output.put_line('----- TEST CASE 1 -----');
  vString := '2001,NULL,SDO_POINT_TYPE,12345,67890,NULL,NULL,NULL';
  dbms_output.put_line('src string: ' || vString);
  vGeom := StringToGeom(vString);
  dbms_output.put_line('dest geom: ' || SDO_UTIL.TO_WKTGEOMETRY(vGeom));

  -- TEST CASE 2 - LINE
  dbms_output.put_line('----- TEST CASE 2 -----');
  vString := '2002,NULL,NULL,SDO_ELEM_INFO_ARRAY,3,1,2,1,SDO_ORDINATE_ARRAY,10,652737.95,6589964.213,652741.222,6589983.148,652746.77,6590018.745,652752.029,6590050.2,652759.351,6590085.732';
  dbms_output.put_line('src string: ' || vString);
  vGeom := StringToGeom(vString);
  dbms_output.put_line('dest geom: ' || SDO_UTIL.TO_WKTGEOMETRY(vGeom));
END;

--------------------------------------------------------------------------------

PROCEDURE PerfTest_GeomStrings(pGeomCount IN INTEGER, pGeomVertexCount IN INTEGER) IS
  TYPE geom_array IS VARRAY(1000) OF MDSYS.SDO_GEOMETRY;
  vGeoms geom_array := geom_array();
  vWKTGeoms geom_array := geom_array();
  vGMLGeoms geom_array := geom_array();
  vCustomGeoms geom_array := geom_array();
  TYPE string_array IS VARRAY(1000) OF VARCHAR2(32767);
  vWkts string_array := string_array();
  vGmls string_array := string_array();
  vCustoms string_array := string_array();
  vWktStart timestamp;
  vWktEncDelta interval day to second;
  vWktDecDelta interval day to second;
  vGmlStart timestamp;
  vGmlEncDelta interval day to second;
  vGmlDecDelta interval day to second;
  vCustomStart timestamp;
  vCustomEncDelta interval day to second;
  vCustomDecDelta interval day to second;
  vTempOrdinates MDSYS.SDO_ORDINATE_ARRAY;
BEGIN
  dbms_output.put_line('----- ' || pGeomCount || ' geometries with ' || pGeomVertexCount || ' vertices -----');

  vGeoms.extend(pGeomCount);
  vWKTGeoms.extend(pGeomCount);
  vGMLGeoms.extend(pGeomCount);
  vCustomGeoms.extend(pGeomCount);
  vWkts.extend(pGeomCount);
  vGmls.extend(pGeomCount);
  vCustoms.extend(pGeomCount);
  
  FOR i IN 1..pGeomCount LOOP
    vTempOrdinates := MDSYS.SDO_ORDINATE_ARRAY();
    vTempOrdinates.extend(pGeomVertexCount*2);
    FOR j IN 1..(pGeomVertexCount*2) LOOP
      vTempOrdinates(j) := i + j;
    END LOOP;
    vGeoms(i) := mdsys.sdo_geometry(2002, NULL, NULL, MDSYS.SDO_ELEM_INFO_ARRAY(1,2,1), vTempOrdinates);
  END LOOP;
  
  -- encode to WKT
  vWktStart := systimestamp;
  FOR i IN 1..pGeomCount LOOP
    vWkts(i) := SDO_UTIL.TO_WKTGEOMETRY(vGeoms(i));
  END LOOP;
  vWktEncDelta := systimestamp - vWktStart;
  
  -- encode to GML
  vGmlStart := systimestamp;
  FOR i IN 1..pGeomCount LOOP
    vGmls(i) := SDO_UTIL.TO_GMLGEOMETRY(vGeoms(i));
  END LOOP;
  vGmlEncDelta := systimestamp - vGmlStart;
  
  -- encode to custom
  vCustomStart := systimestamp;
  FOR i IN 1..pGeomCount LOOP
    vCustoms(i) := GeomToString(vGeoms(i));
  END LOOP;
  vCustomEncDelta := systimestamp - vCustomStart;
  
  -- decode from WKT
  vWktStart := systimestamp;
  FOR i IN 1..pGeomCount LOOP
    vWKTGeoms(i) := SDO_UTIL.FROM_WKTGEOMETRY(vWkts(i));
  END LOOP;
  vWktDecDelta := systimestamp - vWktStart;
  
  -- decode from GML
  vGmlStart := systimestamp;
  FOR i IN 1..pGeomCount LOOP
    vGMLGeoms(i) := SDO_UTIL.FROM_GMLGEOMETRY(vGMLs(i));
  END LOOP;
  vGmlDecDelta := systimestamp - vGmlStart;
  
  -- decode from custom
  vCustomStart := systimestamp;
  FOR i IN 1..pGeomCount LOOP
    vCustomGeoms(i) := StringToGeom(vCustoms(i));
  END LOOP;
  vCustomDecDelta := systimestamp - vCustomStart;
  
  dbms_output.put_line('Encoding to WKT took ' || extract(second from vWktEncDelta) || ' seconds');
  dbms_output.put_line('Encoding to GML took ' || extract(second from vGmlEncDelta) || ' seconds');
  dbms_output.put_line('Encoding to custom took ' || extract(second from vCustomEncDelta) || ' seconds');
  
  dbms_output.put_line('Decoding from WKT took ' || extract(second from vWktDecDelta) || ' seconds');
  dbms_output.put_line('Decoding from GML took ' || extract(second from vGmlDecDelta) || ' seconds');
  dbms_output.put_line('Decoding from custom took ' || extract(second from vCustomDecDelta) || ' seconds');
  
  dbms_output.put_line('----------------------------------------');
  dbms_output.put_line(' ');
END;

--------------------------------------------------------------------------------

PROCEDURE PerfTest_GeomStrings IS
BEGIN
  -- geometries with 2 vertices
  PerfTest_GeomStrings(1, 2);
  PerfTest_GeomStrings(10, 2);
  PerfTest_GeomStrings(100, 2);
  PerfTest_GeomStrings(1000, 2);
  
  -- geometries with 10 vertices
  PerfTest_GeomStrings(1, 10);
  PerfTest_GeomStrings(10, 10);
  PerfTest_GeomStrings(100, 10);
  PerfTest_GeomStrings(1000, 10);

  -- geometries with 100 vertices
  PerfTest_GeomStrings(1, 100);
  PerfTest_GeomStrings(10, 100);
  PerfTest_GeomStrings(100, 100);
  PerfTest_GeomStrings(1000, 100);

  -- geometries with 1000 vertices
  PerfTest_GeomStrings(1, 1000);
  PerfTest_GeomStrings(10, 1000);
  PerfTest_GeomStrings(100, 1000);
  PerfTest_GeomStrings(1000, 1000);

END;

END pg_user_geom_util;
/

show errors package body pg_user_geom_util;

Lightweight Cooperative Multitasking with Boost.Context

I’m writing this in response to Kenny Kerr’s article Lightweight Cooperative Multitasking with C++ in the August 2012 issue of MSDN Magazine, in which Kenny describes a method of implementing cooperative multitasking in C++.

What is Cooperative Multitasking?

This is a method of achieving concurrency in an application without using multiple threads. The basic idea is that you can have multiple tasks running on just a single thread. Now of course only one of these tasks can be running at any one time, so each task must perform some work and then explicitly yield to another task.

Ideally whatever method is used for beginning tasks and yielding should automatically provide a separate context for each task. That is, each task should have its own stack and register data.

Why Kenny’s Article is Cool

Kenny’s article is all about how you can implement your own cooperative multitasking in C++ without having to resort to a specific API calls or a separate library. Yes it’s based on a previously known technique by Tom Duff, but it demonstrates that an awful lot can be done with a relatively small amount of code. This is why I like C++; it gives you the power to do things like this.

As cool as it is, I don’t think I’ll be adding it to my production code any time soon. Besides the problems I’d have getting it past code review, I found a better solution…

Enter Boost.Context

At the end of the article, Kenny reveals his hopes (and dreams?) of a day when he can implement cooperative multitasking in C++ without having to resort to unmaintainable macro hacks.

As luck would have it, on the 20th August 2012 version 1.51 of the Boost C++ libraries was released which contained the new library Boost.Context by Oliver Kowalke. This is a fairly low-level library, but it does provide a basic cooperative multitasking library with a separate context for each task.

Average Task, Boost.Context Style

In his article, Kenny implements a basic application to take numeric input from the command line and produce some statistics. It’s not what I’d call a classic concurrency problem, but it serves the purpose of demonstrating the tasks yielding control only when appropriate. I’ve taken the code and re-implemented it using Boost.Context. Enjoy!

#include <boost\context\all.hpp>

namespace ctx = boost::ctx;

ctx::fcontext_t fc_avg, fc_input, fc_main;
bool b_quit = false;

// --- The Average Task ---
// Computes the average of the input data and 
// then yields execution back to the main task
//-------------------------
// struct average_args - data for the task
// average_yield() - wrapper for yielding execution back to the main task
// average_task() - the task logic
// ------------------------
struct average_args
{
	int * source;
	int sum;
	int count;
	int average;
	int task_;
};
void average_yield() { ctx::jump_fcontext(&fc_avg, &fc_main, 0); }
void average_task(intptr_t p)
{
	average_args* args = (average_args*)p;
	args->sum = 0;
	args->count = 0;
	args->average = 0;
	while (true)
	{
		args->sum += *args->source;
		++args->count;
		args->average = args->sum / args->count;
		average_yield();
	}

	printf("ERROR: should not reach the end of average function\n");
}

// --- The Input Task ---
// Reads a number as input from the console and 
// then yields execution back to the main task
// ----------------------
// struct input_args - data for the task
// input_yield() - wrapper for yielding execution back to the main task
// input_task() - the task logic
// ----------------------
struct input_args
{
	average_args* aa;
	int * target;
	int task_;
};
void input_yield() { ctx::jump_fcontext(&fc_input, &fc_main, 0); }
void input_task(intptr_t p)
{
	input_args* pia = (input_args*)p;

	while (true)
	{
		printf("number: ");
		if (!scanf_s("%d", pia->target))
		{
			b_quit = true;
			return;
		}
    
		input_yield();
	}

	printf("ERROR: should not reach the end of input function\n");
}

void main()
{
	int share = -1;
	average_args aa = {&share};
	input_args ia = {&aa, &share};
	ctx::stack_allocator alloc;

	// construct the input task
	fc_input.fc_stack.base = alloc.allocate(ctx::minimum_stacksize());
	ctx::make_fcontext(&fc_input, input_task);

	// construct the average task
	fc_avg.fc_stack.base = alloc.allocate(ctx::minimum_stacksize());
	ctx::make_fcontext(&fc_avg, average_task);

	while (!b_quit)
	{
		ctx::jump_fcontext( &fc_main, &fc_input, (intptr_t) &ia);
		ctx::jump_fcontext( &fc_main, &fc_avg, (intptr_t) &aa);
		printf("sum=%d count=%d average=%d\n", aa.sum, aa.count, aa.average);
	}
	
	printf("main: done\n");
}

Footnotes

The Boost.Context library has been described as a good candidate for providing low-level cooperative multitasking support to the developing Boost.Coroutine library. Coroutine will provide a more high-level, abstract way of applying these concurrency techniques in your code.

I’m looking forward to Kenny’s next article in which he has promised to show how the Visual C++ team is working on what must be some kind of cooperative multitasking that he’s called ‘resumable functions’.

Reading an In-Memory SQLite Database

I’m currently working on a utility to improve upon the basic Error Lookup utility found in Microsoft Visual Studio. I am trying to improve it by collating all the error codes and messages found in the Microsoft Platform SDK into a database, which can be accessed and queried in a similar way to the existing utility, but with a extra bells and whistles (think free-text search, partial code matching, etc.). To make distribution of the utility easier, I decided to try embedding my error database into the executable file, which was straightforward. Using the database however required a bit more work, which I am presenting here in the hope that others will find it useful.

Embedding a SQLite Database in a Windows EXE or DLL

The data I am working with is in a SQLite database, which is a database consisting of a single file. To embed this file in my Visual Studio project I did the following.

  1. Add a resource file to your project (only if you don’t have one already).
  2. Right-click on the resource file and choose ‘Add Resource’.
  3. Select ‘Import…’ and choose your database file.
  4. Enter a resource type of ’10’.

Using an Embedded SQLite Database

Now you have your data embedded in your application, first you need to obtain a pointer to the data bytes and next you need to get SQLite to read the in-memory data. Although this looks simple, it took a few nights to understand how the SQLite storage model worked in order to get it working in a truly read-only manner.

This is how I do it in my code:

void open_mem_db()
{
    char errcode = 0;

    void* pBuffer = NULL;
    HRSRC hr = ::FindResource(NULL, MAKEINTRESOURCE(IDR_ERRORCODEDB), RT_RCDATA);

    if (hr)
    {
        HGLOBAL hg = ::LoadResource(NULL, hr);

        if (hg)
        {
            DWORD dwSize = ::SizeofResource(NULL, hr);
            pBuffer = ::LockResource(hg);

            set_mem_db(pBuffer, dwSize);

            int nInitResult = readonlymemvfs_init();
            assert(nInitResult == SQLITE_OK);
            errcode = sqlite3_open_v2( "_", &s_database,
                SQLITE_OPEN_READONLY, READONLY_MEM_VFS_NAME );
        }
    }
}

Remember to change IDR_ERRORCODEDB to your resource identifier. This makes use of some extra code to allow SQLite to read data straight from memory, which can be downloaded from http://www.tolon.co.uk/download/readonlymemvfs-0.1.zip or from github. Use of the code is entirely at your own risk and no warranty or guarantee is provided.

I am planning to develop the code further to tidy it up and support encryption of the embedded SQLite database, so if you have any comments or any improvements to the code I will gladly accept them and try to incorporate them into the code.

Recent Lessons

I’ve recently picked up TolonSpellCheckLib and Tolon NoteKeeper again after a long time away from it. I’m doing a lot of PL/SQL and testing at work at the moment, so I need to quench my thirst for C++ with a bit of ‘extra-curricular’ development.

Since last building my code in 2009 I have moved to Windows 7 64-bit and a new hard drive, so I was anticipating a fair bit of messing around in order to get a usable source/build tree again. As it turns out it wasn’t too bad, but I thought I would record a few useful notes on lessons I have learned from this process.

  • Lesson 1 – Use Source Control
  • Lesson 2 – Automate Your Build Process
  • Lesson 3 – LINK : fatal error LNK1181: cannot open input file ‘link.obj’
  • Lesson 4 – Compiler Warning Level 4 (/W4) is Your Friend
    • Lesson 1 – Use Source Control

      I’ve been using Subversion for a while now and was mightily chuffed at the ease at which I was able to upgrade my old source trees to the new version of the svn client (1.6 to 1.7) in order to check whether I had any uncommitted changes lying about. Upgrading was a simple matter of running ‘svn upgrade’ at the root of the source tree (after taking a backup of course).

      Without source control I would have been left manually comparing two old source trees with very little information to track down a problem if I got it wrong.

      Lesson 2 – Automate Your Build Process

      When I started experimenting with the enchant spell-checking library (see part 1 and part 2 of this escapade), I quickly realised that building it and all of its dependencies was going to get very complicated. I created a batch file that allows me to build this from scratch. To help stabilise my build I store compressed source versions of the dependencies in my source control repository as compressed files (any changes I need to make to these dependencies are managed by applying patches as part of my build process).

      Having an automated build pays off almost immediately as it means you can build from scratch again and again knowing that the code is being compiled and linked in exactly the same way. When working with external libraries, this helps track down platform issues, build option issues and bugs.

      So having got the latest code out of the repository onto my laptop, I simply ran ‘fullbuild.bat debug’ and the enchant library was very close to being successfully built at the first attempt. The only problem I encountered was the following lesson…

      Lesson 3 – LINK : fatal error LNK1181: cannot open input file ‘link.obj’

      If you get the above error whilst building using a makefile under Windows 7, do not panic. It seems that the environment variable handling is slightly different under Windows 7 so that when you come to the linking part of your traditional makefile, instead of calling:

      LINK <link flags> <link objects>

      the following is called

      LINK LINK <link flags> <link objects>

      The solution was to rename the LINK variable in the makefile to _LINK, so LINK = link became _LINK = link and all $(LINK) references became $(_LINK). Thanks to NeilRashbrook on http://forums.mozillazine.org/viewtopic.php?t=470087.

      Lesson 4 – Compiler Warning Level 4 (/W4) is Your Friend

      There are some good warnings that only get output as this level in Visual C++ 2008, which makes it worth having switched on. My particular favourite is that it will tell you when you’ve accidentally performed an assignment instead of doing a comparison, i.e. if (x = 5) rather than if (x == 5).

      In my opinion it’s definitely worth switching this on and getting rid of all the spurious warnings in order to find these kinds of bugs. When I first compiled Tolon NoteKeeper with this option I got over 1200 warnings, most of which came from external libraries. For now I have assumed that a certain set of warnings in external libraries can be safely ignored, so I have selectively switched them off by using #pragma warning as shown below.


      #include "stdafx.h"
      #include "NoteKeeper.h"
      #include "NkpStructs.h"
      #include "NoteKeeperDoc.h"

      #pragma warning(push)
      #pragma warning(disable:4244) // warning C4244: 'argument' : conversion from X to Y, possible loss of data
      #pragma warning(disable:4996) //warning C4996: 'vsprintf': This function or variable may be unsafe. Consider using vsprintf_s instead. To disable deprecation, use _CRT_SECURE_NO_WARNINGS. See online help for details.
      #include "planydec.h"
      #include "planybmp.h"
      #include "plwinbmp.h"
      #include "pljpegenc.h"
      #include "plpngenc.h"
      #include "pltiffenc.h"
      #include "plbmpenc.h"
      #pragma warning(pop)

      #pragma warning(push)
      #pragma warning(disable:4100) // warning C4100: unreferenced formal parameter
      #pragma warning(disable:4189) // warning C4189: local variable is initialized but not referenced
      #pragma warning(disable:4244) // warning C4244: 'argument' : conversion from X to Y, possible loss of data
      #include "cryptopp/gzip.h"
      #pragma warning(pop)

      // etc...

      EDIT: Build Script Files

      You can now download some of the automated scripts I use for building my software. See http://www.tolon.co.uk/miscellaneous/lab/dependency-build-scripts/.

Integrating your application with Enchant – Part 2

Enchanté

Welcome to the second article exploring how to integrate the spell checking library Enchant into your Visual C++ applications.

I will be using this library with my freeware program NoteKeeper, which I am currently maintaining/developing in two versions of Visual C++.

  • Visual C++ 6 SP6 with STLPort 5.1.5, Multithreaded DLL
    (cl.exe v12.00.8804, link.exe v6.00.8447)
  • Visual C++ 9 SP1, Multithreaded DLL
    (cl.exe v15.00.30729.01, link.exe v9.00.30729.01)

Continue reading Integrating your application with Enchant – Part 2

Integrating your application with Enchant – Part 1

Enchanté
Welcome to the first in a series of articles exploring how to integrate the spell checking library Enchant into your Visual C++ applications.

I will be integrating Enchant with the next release of NoteKeeper, which I am currently maintaining in an old version of MSVC and also developing a new version from scratch. Thus I will be using the following configurations.

  • Visual C++ 6 SP6 with STLPort 5.1.5, Multithreaded DLL
    (cl.exe v12.00.8804, link.exe v6.00.8447)
  • Visual C++ 9 SP1, Multithreaded DLL
    (cl.exe v15.00.30729.01, link.exe v9.00.30729.01)

Continue reading Integrating your application with Enchant – Part 1