Using reCAPTCHA from C++

I wanted to add reCAPTCHA to some old C++ CGI code, but I couldn't find a reCAPTCHA library for C++. Neither could a few other people I saw looking. So I wrote the following code, which I am happy to share:

#include <string>
#include <iostream>
#include <sstream>
#include <fstream>
#include <iomanip>
#include <curl/curl.h>

static ostringstream curl_result;

/* curl write callback, to fill tidy's input buffer...  */
uint write_cb(char *in, uint size, uint nmemb, void *out)
{
  int r = size * nmemb;
  in[r] = '\0';
  curl_result << in;
  return(r);
}

static bool recaptcha_check_answer(const char *private_key, const char *remoteip, 
 const char *challenge, const char *response, std::string &error) {
 if (!remoteip || !*remoteip) {
  error = "For security reasons, you must pass the remote ip to reCAPTCHA";
  return false;
 }
 //discard spam submissions
 if (!challenge || !*challenge || !response || !*response) {
  error = "Captcha challenge or response missing";
  return false;
 }

 CURL *curl;
 CURLcode res;
 struct curl_httppost *formpost=NULL;
 struct curl_httppost *lastptr=NULL;

 if(curl_global_init(CURL_GLOBAL_SSL)) {
  error = "Error initialising curl library";
  return false;
 }
 curl = curl_easy_init();
 if(!curl) {
  error = "Error initialising curl library";
  return false;
 }
 curl_formadd(&formpost,
  &lastptr,
  CURLFORM_COPYNAME, "privatekey",
  CURLFORM_COPYCONTENTS, private_key,
  CURLFORM_END);
 curl_formadd(&formpost,
  &lastptr,
  CURLFORM_COPYNAME, "remoteip",
  CURLFORM_COPYCONTENTS, remoteip,
  CURLFORM_END);
 curl_formadd(&formpost,
  &lastptr,
  CURLFORM_COPYNAME, "challenge",
  CURLFORM_COPYCONTENTS, challenge,
  CURLFORM_END);
 curl_formadd(&formpost,
  &lastptr,
  CURLFORM_COPYNAME, "response",
  CURLFORM_COPYCONTENTS, response,
  CURLFORM_END);

 const char *surl = "https://api-verify.recaptcha.net/verify";
 curl_easy_setopt(curl, CURLOPT_URL, surl);
 curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 0L);
 curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 1L);
 curl_easy_setopt(curl, CURLOPT_HTTPPOST, formpost);
 #ifdef SKIP_HOSTNAME_VERFICATION
 /*
  * If the site you're connecting to uses a different host name that what
  * they have mentioned in their server certificate's commonName (or
  * subjectAltName) fields, libcurl will refuse to connect. You can skip
  * this check, but this will make the connection less secure.
  */
 curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 0L);
 #endif
 curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_cb);

 res = curl_easy_perform(curl);

 /* always cleanup */
 curl_easy_cleanup(curl);
 if(res && res != CURLE_RECV_ERROR)
  {
  std::ostringstream err;
  err << "Error " << res << " contacting captcha server";
  error = err.str();
  return false;
  }

 std::string resp = curl_result.str();
 std::string::size_type pos = resp.find("\n");
 std::string part1 = pos == std::string::npos ? resp : resp.substr(0, pos);
 if(part1 == "true")
  return true;
 error = pos == std::string::npos ? "Unknown captcha error" : "Captcha error:" + resp.substr(pos);
 return false;
}

This code relies on the curl library to handle the communication with the reCAPTCHA server.

Using the word breaker from Microsoft SQL Server 2005 in a stand-alone C# program

I have been writing a program which will query a full text search index on Microsoft SQL Server 2005. The data being indexed are large blobs of text (multiple lines).

I need to extract only those lines of text which contain the search words for display with the results.

Full Text Search uses a word breaker (IWordBreaker COM interface) and stemmer (IStemmer COM interface) to break the indexed text into words, and to enable matching of alternate word forms (e.g. "welcome" will also match "welcomes", "welcomed" and "welcoming").

I therefore need to use the Sql Server word breaker and stemmer to determine if each line of text from the blob matches the search string, as I believe the standard word breaker and stemmer in Windows search uses a different algorithm.

I thought if I copied the two dlls (infosoft and langwrbk) from a SQL Server installation to the machine on which I am running my program, and ran regsvr32 on them, they would be installed in the registry, and I would be able to use them. Unfortunately, this does not work, as regsvr32 does not add any information to the registry.

I therefore needed to load the COM components in the dlls "by hand". I started with John Jeffery's code to load COM components by hand. Once I had got over some 32-bit/64-bit problems (your DLL has to match the bit size of your calling code), it worked.

Here is the LinqPad script I used to test the two interfaces, in case someone else needs to do the same thing:

const string SqlServerDllFolder = @"C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\Binn";
static LibraryModule langwrbk = LibraryModule.LoadModule(Path.Combine(SqlServerDllFolder, "LangWrbk.dll"));
static LibraryModule infosoft = LibraryModule.LoadModule(Path.Combine(SqlServerDllFolder, "infosoft.dll"));
static Guid stemmer = new Guid("D99F7670-7F1A-11CE-BE57-00AA0051FE20");
static Guid breaker = new Guid("173C97E2-AEBE-437C-9445-01B237ABF2F6");

void Main()
{
 WordBreaker breaker = new WordBreaker();
 string search;
 while((search = Util.ReadLine("Query string:").Trim()) != "") {
  Console.WriteLine("Original text:" + search);
  foreach (string word in breaker.Search(search)) {
   Console.WriteLine(word);
  }
 }
}

// COM Interface to Microsoft word breaker and stemmer
[Flags]
public enum WORDREP_BREAK_TYPE {
 WORDREP_BREAK_EOW = 0,
 WORDREP_BREAK_EOS = 1,
 WORDREP_BREAK_EOP = 2,
 WORDREP_BREAK_EOC = 3
}

[ComImport]
[Guid("CC907054-C058-101A-B554-08002B33B0E6")]
[InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
public interface IWordSink {
 void PutWord([MarshalAs(UnmanagedType.U4)] int cwc,
  [MarshalAs(UnmanagedType.LPWStr)] string pwcInBuf,
  [MarshalAs(UnmanagedType.U4)] int cwcSrcLen,
  [MarshalAs(UnmanagedType.U4)] int cwcSrcPos);
 void PutAltWord([MarshalAs(UnmanagedType.U4)] int cwc,
  [MarshalAs(UnmanagedType.LPWStr)] string pwcInBuf,
  [MarshalAs(UnmanagedType.U4)] int cwcSrcLen,
  [MarshalAs(UnmanagedType.U4)] int cwcSrcPos);
 void StartAltPhrase();
 void EndAltPhrase();
 void PutBreak(WORDREP_BREAK_TYPE breakType);
}

[ComImport]
[Guid("CC906FF0-C058-101A-B554-08002B33B0E6")]
[InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
public interface IPhraseSink {
 void PutSmallPhrase([MarshalAs(UnmanagedType.LPWStr)] string pwcNoun,
  [MarshalAs(UnmanagedType.U4)] int cwcNoun,
  [MarshalAs(UnmanagedType.LPWStr)] string pwcModifier,
  [MarshalAs(UnmanagedType.U4)] int cwcModifier,
  [MarshalAs(UnmanagedType.U4)] int ulAttachmentType);
 void PutPhrase([MarshalAs(UnmanagedType.LPWStr)] string pwcPhrase,
  [MarshalAs(UnmanagedType.U4)] int cwcPhrase);
}
[ComImport]
[Guid("fe77c330-7f42-11ce-be57-00aa0051fe20")]
[InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
public interface IWordFormSink {
 void PutAltWord([MarshalAs(UnmanagedType.LPWStr)] string pwcInBuf, [MarshalAs(UnmanagedType.U4)] int cwc);
 void PutWord([MarshalAs(UnmanagedType.LPWStr)] string pwcInBuf, [MarshalAs(UnmanagedType.U4)] int cwc);
}

[StructLayout(LayoutKind.Sequential)]
public struct TEXT_SOURCE {
 [MarshalAs(UnmanagedType.FunctionPtr)]
 public delFillTextBuffer pfnFillTextBuffer;
 [MarshalAs(UnmanagedType.LPWStr)]
 public string awcBuffer;
 [MarshalAs(UnmanagedType.U4)]
 public int iEnd;
 [MarshalAs(UnmanagedType.U4)]
 public int iCur;
}

// used to fill the buffer for TEXT_SOURCE
public delegate uint delFillTextBuffer([MarshalAs(UnmanagedType.Struct)]
 ref TEXT_SOURCE pTextSource);

[ComImport]
[Guid("D53552C8-77E3-101A-B552-08002B33B0E6")]
[InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
public interface IWordBreaker {
 void Init([MarshalAs(UnmanagedType.Bool)] bool fQuery,
  [MarshalAs(UnmanagedType.U4)] int maxTokenSize,
  [MarshalAs(UnmanagedType.Bool)] out bool pfLicense);
 void BreakText([MarshalAs(UnmanagedType.Struct)] ref TEXT_SOURCE pTextSource,
  [MarshalAs(UnmanagedType.Interface)] IWordSink pWordSink,
  [MarshalAs(UnmanagedType.Interface)] IPhraseSink pPhraseSink);
 void GetLicenseToUse([MarshalAs(UnmanagedType.LPWStr)] out string ppwcsLicense);
}

[ComImport]
[Guid("EFBAF140-7F42-11CE-BE57-00AA0051FE20")]
[InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
public interface IStemmer {
 void Init([MarshalAs(UnmanagedType.U4)] int ulMaxTokenSize, [MarshalAs(UnmanagedType.Bool)] out bool pfLicense);
 void GenerateWordForms([MarshalAs(UnmanagedType.LPWStr)] string pwcInBuf, 
  [MarshalAs(UnmanagedType.U4)] int cwc, 
  [MarshalAs(UnmanagedType.Interface)] IWordFormSink pStemSink);
 void GetLicenseToUse([MarshalAs(UnmanagedType.LPWStr)] out string ppwcsLicense);
}

// Word breaker to break a line of text into words, stem them, 
// and return a HashSet of all the words and stems
public class WordBreaker : IWordSink, IWordFormSink {
 IWordBreaker wordBreaker;
 HashSet words; // Words to search for
 IStemmer istm;
 
 public WordBreaker() {
  wordBreaker = (IWordBreaker)ComHelper.CreateInstance(langwrbk, breaker);
  bool pfLicense = true;
  wordBreaker.Init(true, 1000, out pfLicense);
  istm = (IStemmer)ComHelper.CreateInstance(infosoft, stemmer);
  pfLicense = true;
  istm.Init(1000, out pfLicense);
 }
 
 public HashSet Search(string text) {
  words = new HashSet();
  TEXT_SOURCE pTextSource = new TEXT_SOURCE();
  pTextSource.pfnFillTextBuffer = new delFillTextBuffer(pfnFillTextBuffer);
  pTextSource.awcBuffer = text.ToLower();
  pTextSource.iCur = 0;
  pTextSource.iEnd = text.Length;
  wordBreaker.BreakText(ref pTextSource, (IWordSink)this, null);
  return words;
 }
 
 #region IWordFormSink Members
 public void PutAltWord(string pwcInBuf, int cwc) {
  words.Add(pwcInBuf.Substring(0, cwc));
 }

 public void PutWord(string pwcInBuf, int cwc) {
  words.Add(pwcInBuf.Substring(0, cwc));
 }
 #endregion

 #region IWordSink Members
 public void PutWord(int cwc, string pwcInBuf, int cwcSrcLen, int cwcSrcPos) {
  istm.GenerateWordForms(pwcInBuf.Substring(0, cwc), cwc, this);
 }
 
 public void PutAltWord(int cwc, string pwcInBuf, int cwcSrcLen, int cwcSrcPos) {
 }

 public void StartAltPhrase() {
 }
 
 public void EndAltPhrase() {
 }

 public void PutBreak(WORDREP_BREAK_TYPE breakType) {
 }
 #endregion
}

static uint pfnFillTextBuffer(ref TEXT_SOURCE pTextSource) {
 // return WBREAK_E_END_OF_TEXT
 return 0x80041780;
}

// Code from https://gist.github.com/1568627
// By John Jeffery
static class ComHelper
{
 private delegate int DllGetClassObject(ref Guid clsid, ref Guid iid, [Out, MarshalAs(UnmanagedType.Interface)] out IClassFactory classFactory);

 public static object CreateInstance(LibraryModule libraryModule, Guid clsid)
 {
  var classFactory = GetClassFactory(libraryModule, clsid);
  var iid = new Guid("00000000-0000-0000-C000-000000000046"); // IUnknown
  object obj;
  classFactory.CreateInstance(null, ref iid, out obj);
  return obj;
 }

 static IClassFactory GetClassFactory(LibraryModule libraryModule, Guid clsid)
 {
  IntPtr ptr = libraryModule.GetProcAddress("DllGetClassObject");
  var callback = (DllGetClassObject) Marshal.GetDelegateForFunctionPointer(ptr, typeof (DllGetClassObject));

  var classFactoryIid = new Guid("00000001-0000-0000-c000-000000000046");
  IClassFactory classFactory;
  var hresult = callback(ref clsid, ref classFactoryIid, out classFactory);

  if (hresult != 0)
  {
   throw new Win32Exception(hresult, "Cannot create class factory");
  }
  return classFactory;
 }
}


[Guid("00000001-0000-0000-c000-000000000046")]
[InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
[ComImport]
interface IClassFactory
{
 void CreateInstance([MarshalAs(UnmanagedType.IUnknown)] object pUnkOuter, ref Guid riid, [MarshalAs(UnmanagedType.IUnknown)] out object ppvObject);
 void LockServer(bool fLock);
}

class LibraryModule : IDisposable
{
 private readonly IntPtr _handle;
 private readonly string _filePath;

 private static class Win32
 {
  [DllImport("kernel32.dll", CharSet = CharSet.Ansi, SetLastError = true)]
  public static extern IntPtr GetProcAddress(IntPtr hModule, string lpProcName);

  [DllImport("kernel32.dll")]
  public static extern bool FreeLibrary(IntPtr hModule);

  [DllImport("kernel32.dll", SetLastError = true)]
  public static extern IntPtr LoadLibrary(string lpFileName);
 }


 public static LibraryModule LoadModule(string filePath)
 {
  var libraryModule = new LibraryModule(Win32.LoadLibrary(filePath), filePath);
  if (libraryModule._handle == IntPtr.Zero)
  {
   int error = Marshal.GetLastWin32Error();
   throw new Win32Exception(error, "Cannot load library: " + filePath);
  }

  return libraryModule;
 }

 private LibraryModule(IntPtr handle, string filePath)
 {
  _filePath = filePath;
  _handle = handle;
 }

 ~LibraryModule()
 {
  if (_handle != IntPtr.Zero)
  {
   Win32.FreeLibrary(_handle);
  }
 }

 public void Dispose()
 {
  if (_handle != IntPtr.Zero)
  {
   Win32.FreeLibrary(_handle);
  }
  GC.SuppressFinalize(this);
 }

 public IntPtr GetProcAddress(string name)
 {
  IntPtr ptr = Win32.GetProcAddress(_handle, "DllGetClassObject");
  if (ptr == IntPtr.Zero)
  {
   int error = Marshal.GetLastWin32Error();
   string message = string.Format("Cannot find proc {0} in {1}", name, _filePath);
   throw new Win32Exception(error, message);
  }
  return ptr;
 }

 public string FilePath
 {
  get { return _filePath; }
 }
}

Naerok 12" bandsaw manual

I recently inherited a workshop containing a Naerok 12" bandsaw, and went looking on the Internet for a manual. Although I found manuals for similar saws, I couldn't find one for this model.

However, when tidying the workshop, I found the real manual! In order to help anyone else with the same model of bandsaw, I have uploaded a scan of the manual to http://www.trumphurst.com/naerok.zip.

ACS AET62 Fingerprint Reader and C#

I'm currently trying to interface to an ACS AET62 Fingerprint Reader from C#. The reader comes with an SDK, along with what appear to be extensive example applications. However, the applications contain bugs which the documentation does not help to resolve. I thought I would record the bugs I have found, and how I have resolved them.

ABSEnroll has a ref int parameter which is returned as a pointer to an ABS_BIR structure. This is declared in the provided BSTypes.cs file as containing a header followed by 2560 bytes of data. Unfortunately, this is not correct - it contains a header followed by up to 2560 bytes of data. The actual length of the whole structure (including the header) is contained in the first uint Length member of the structure.

This kind of variable length structure is quite common in C++, but there is no built-in way to marshal it correctly in C#.

The code provided in the SDK just calls Marshal.PtrToStructure on it - this will run off the end of the allocated memory, and assign garbage to the last (should be unused) bytes of the ABS_BIR structure. That's sort of OK in a 32-bit operating system (it is unlikely to try to access memory that doesn't exist), but in a 64-bit one with properly protected memory, it may well throw an exception.

The correct way to marshal the structure is in two parts - first marshal the header part with Marshal.PtrToStructure, then allocate the data byte array to the length therein (first subtracting the length of the header itself), and use Marshal.Copy to copy the remainder.

ppEnrolledTemplate = new BSTypes.ABS_BIR();
BSTypes.ABS_BIR_HEADER hdr = (BSTypes.ABS_BIR_HEADER) Marshal.PtrToStructure(ptr, typeof(BSTypes.ABS_BIR_HEADER));
ppEnrolledTemplate.Header = hdr;
int hdrlen = Marshal.SizeOf(hdr);
int bodylen = (int)(hdr.Length - hdrlen);
ppEnrolledTemplate.Data = new byte[bodylen];
Marshal.Copy(new IntPtr(tset + hdrlen), ppEnrolledTemplate.Data, 0, bodylen);

ABSVerify takes a reference to an IntPtr which is represented in C++ as **ABS_BIR - it is supposed to be an array of pointers to ABS_BIR. The example code provided just passed one ABS_BIR - it used Marshal to allocate a chunk of global memory and marshal the structure into it. The resulting IntPtr was passed as the ref parameter. This is all very fine if you only want to verify a single fingerprint, but if you want to determine which of a list of fingerprints is being scanned (as I did), it is no help at all.

My knowledge of C++ finally allowed me to figure out what was needed. - first, allocate an array of IntPtr (one for each ABS_BIR fingerprint scan you wish to pass). Then allocate memory for each structure into its corresponding IntPtr, and use Marshal to copy the structure into the allocated memory. Finally, pass a reference to the 0'th element of the array.

In actual fact, there is no need to Marshal the ABS_BIR structure during Enroll or Verify - it can be treated as an opaque array of bytes (with the length of the whole thing in the first 4 bytes) - other than the length, there is no other interesting data in the ABS_BIR structure.

Also, during Verify, you can make a single call to allocate enough memory for all the structures you want to pass, and calculate the values of the IntPtrs to pack the data into the single block of memory.

I hope this helps anyone else struggling with this SDK.

In response to anoop's comment, here is the code I use to call verify, to see if a scan corresponds to one of the scans in the database

BSTypes.ABS_OPERATION op = operation(1);
op.Timeout = 60000;
int res;
int matching_slot = 0;
res = BSApi.ABSVerify(_conn, ref op, (ushort)_templateIndex.Length, ref _templates[0], 
  ref matching_slot, 0); // BSTypes.ABS_FLAG_NOTIFICATION | BSTypes.ABS_FLAG_AUTOREPEAT);

To create the _templates array, you do this

int size = Marshal.SizeOf(typeof(BSTypes.ABS_BIR));
IntPtr[] templates = new IntPtr[number_of_elements];
templates[0] = Marshal.AllocHGlobal(size * number_of_elements);
byte[] zeroes = new byte[size];
int offset = 0;
int ind = 0;
foreach (byte[] b in list_of_scan_records) {
 if(ind > 0) templates[ind] = new IntPtr((int)templates[0] + offset);
 offset += size;
 Marshal.Copy(zeroes, 0, templates[ind], zeroes.Length);
 Marshal.Copy(b, 0, templates[ind], b.Length);
 ind++;
}

number_of_elements is the number of scans in the database, and list_of_scan_records is a list containing a byte array for each record

VMWare Workstation on Windows

Having abandoned CentOs as the host for my VMWare VMs, I went back to running Windows XP as the host.

A bit of a pain (two sets of virus detectors, and all the other bloatedness of Windows). That would have been OK, except that most times I started the client, it decided it had only 1 monitor, and rearranged my desktop accordingly. So every morning, the first few minutes would be spent putting things back on the correct monitor. Would you believe that moving a quick launch toolbar from one monitor to another reorders all the icons?

At least my USB 3 drive worked (and aren't they _fast_ !). And my headset.

However, performance was not good, and too many things caused screen updating to stop working. And VM clients had a tendency to jump from one (ATI Hydravision) virtual desktop to another (on the server).

The NTFS filesystem going corrupt, so that one of the 2GB files that made up my hard disk suddenly thought it was 2TB was the last straw.

I finally gave in, and decided to "de-virtualise" my desktop. I took loads of backups, made a BartPE CD, booted it, and repartitioned my disk. Then I restored the backup of my VM client desktop, and rebooted.

Then I was stuck in a boot loop - as soon as the Windows logo appeared, the machine rebooted. Safe mode (even safe mode with command line) did not help. The only way out was a repair install with the latest CD I had (XP64 SP1), followed by a whole day downloading updates and service packs and copies of Internet Explorer.

So I can now advise that virtualisation is not ready to host your desktop on the one machine, and that changing the hardware of your Windows working environment is still a horrible nightmare.

CentOs and VMWare Workstation

I had other problems with my CentOs VMWare WOrkstation host environment. I have a Logitek USB headset, which I use with Skype in my VM client running Windows XP.

However, a while ago the microphone stopped working - all I got from it was a load crackling noise. I tried buying a new headset, but that was the same - it seems to be a reaction between CentOs and VMWare.

Yet another reason to abandon CentOs :-(

CentOs and USB 3.0

I run CentOs 5 as a host operating system for VMWare Workstation, and use a VM client as my main working environment.

I was finding backing up my environment to be a slow and painful process using USB2 disks, or the Gigabit network card.

I read that USB 3.0 was much faster, and that Linux was the first O/S to support it, so I bought a controller card and disk.

Then I found CentOs didn't support it :-(

I was encouraged by the nice people at el-repo to try their experimental kernel for CentOs, and, after a lot of messing about, I managed to get it booted, and accessing the USB 3 disk. However, I couldn't also use their fglrx ATI Radeon driver at the same time (it only works with the stock CentOs kernel). As dual monitor support is vital to me, I gave up, and finally abandoned CentOs.