Answer by Chris Hall for Understanding pthreads locks and condition variables

I have looked deeper into how glibc (v2.30 on Linux & x86_64, at least) implements pthread_mutex_lock() and _unlock().

It turns out that _lock() works something like this:

  if (atomic_cmp_xchg(mutex->lock, 0, 1))    return <OK> ;             // mutex->lock was 0, is now 1  while (1)    {      if (atomic_xchg(mutex->lock, 2) == 0)        return <OK> ;        // mutex->lock was 0, is now 2      ...do FUTEX_WAIT(2)... // suspend thread iff mutex->lock == 2...    } ;

And _unlock() works something like this:

  if (atomic_xchg(mutex->lock, 0) == 2)  // set mutex->lock == 0    ...do FUTEX_WAKE(1)...               // if may have waiter(s) start 1

Now:

• mutex->lock: 0 => unlocked, 1 => locked-but-no-waiters, 2 => locked-with-waiter(s)

  'locked-but-no-waiters' optimizes for the case where there is no lock contention and there is no need to do FUTEX_WAKE in _unlock().

• the _lock()/_unlock() functions are in the library -- they are not in the kernel.

  ...in particular, the ownership of the mutex is a matter for the library, not the kernel.

• FUTEX_WAIT(2) is a call to the kernel, which will place the thread on a pending queue associated with the mutex, unless mutex->lock != 2.

  The kernel checks for mutex->lock == 2 and adds the thread to the queue atomically. This deals with the case of _unlock() being called after the atomic_xchg(mutex->lock, 2).

• FUTEX_WAKE(1) is also a call to the kernel, and the futex man page tells us:

  FUTEX_WAKE (since Linux 2.6.0)

  This operation wakes at most 'val' of the waiters that are waiting ... No guarantee is provided about which waiters are awoken (e.g., a waiter with a higher scheduling priority is not guaranteed to be awoken in preference to a waiter with a lower priority).

  where 'val' in this case is 1.

  Although the documentation says "no guarantee about which waiters are awoken", the queue appears to be at least FIFO.

Note especially that:

1. _unlock() does not pass the mutex to the thread started by the FUTEX_WAKE.

2. once woken up, the thread will again try to obtain the lock...

   ...but may be beaten to it by any other running thread -- including the thread which just did the _unlock().

I believe this is why you have not seen the work being shared across the threads. There is so little work for each one to do, that a thread can unlock the mutex, do the work and be back to lock the mutex again before a thread woken up by the unlock can get going and succeed in locking the mutex.