Turns out I was wrong and right at the same time.

I thought that the problem Thomson Reuters had discovered (detailed in the last post) was that Hoard’s spinlock implementation for Sparc was somehow broken, possibly by a newer version of the Sparc architecture (e.g., by using a horrible relaxed memory model). See this info from Sun about such models, including TSO and (yikes) RMO.

Suffice it to say that relaxed memory ordering breaks things in addition to being absolutely awful to reason about. But luckily, apparently saner minds have prevailed and that memory ordering — while supported by the Sparc architecture — is never enabled. Phew.

Anyway, while chasing down the bug I discovered an “impossible” sequence of events (a race, but under the protection of a lock), and switching from spinlocks to POSIX locks (slower, but safe) solved the problem. Aha! Plainly, something wrong with the spinlock! But, it turns out, the spinlock code is perfectly fine. It’s practically identical to what Hans Boehm does in his atomics library.

Next time both Hans and I have done things the same way, I will assume that Hans probably got it right, so I’m OK. 🙂 The real source of the problem was elsewhere, but it points up the perils of supporting cross-platform software (and legacy systems).

First, a little background.

Hoard has an optimization specifically for single-threaded programs. Unless you spawn a thread, Hoard does not enable any locking code or atomic operations (locks become simple loads and stores). IIRC, this increases performance by several percentage points for some programs, so it’s nothing to sneeze at.

It’s simple enough: Hoard has a Boolean value called anyThreadCreated, which is initially false. The Hoard library intercepts the pthread_create call (on Windows, it does something else that has the same effect). Among other things, any call to create a thread immediately sets anyThreadCreated to true. The spinlock implementation then enables real locking.

As you can imagine, if somehow a program were to run multiple threads with locks disabled, that would be bad.

Enter Solaris threads.

It turns out that Solaris has not one but two thread APIs.  They were the predecessor to the now-familiar POSIX threads API. However, some code still uses this old, non-portable API. Notably, the code running at Thomson Reuters.

Yeah, I knew about Solaris threads, since I programmed with them “back in the day” (in the mid to late 90’s), but I overlooked them.

What was happening was that Hoard was not intercepting thr_create (Solaris threads). It thus assumed that no threads had been created, even though they had. So Thomson Reuters’ multithreaded code was running with the spinlocks disabled.

No wonder it crashed. It’s surprising it worked at all.

So, now Hoard now properly intercepts thr_create. Bug fixed. Life is good.

That said, I still think that exposing programmers to the vagaries of hardware memory models should be a felony offense.