Python Twisted's DeferredLock -

can provide example , explain when , how use twisted's deferredlock.

i have deferredqueue , think have race condition want prevent, i'm unsure how combine two.

use deferredlock when have critical section asynchronous , needs protected overlapping (one might "concurrent") execution.

here example of such asynchronous critical section:

class networkcounter(object):     def __init__(self):         self._count = 0      def next(self):         self._count += 1         recording = self._record(self._count)         def recorded(ignored):             return self._count         recording.addcallback(recorded)         return recording      def _record(self, value):         return http.get(             b"http://example.com/record-count?value=%d" % (value,)) 

see how 2 concurrent uses of next method produce "corrupt" results:

from __future__ import print_function  counter = networkcounter() d1 = counter.next() d2 = counter.next()  d1.addcallback(print, "d1") d2.addcallback(print, "d2") 

gives result:

2 d1 2 d2 

this because second call networkcounter.next begins before first call method has finished using _count attribute produce result. 2 operations share single attribute , produce incorrect output consequence.

using deferredlock instance solve problem preventing second operation beginning until first operation has completed. can use this:

class networkcounter(object):     def __init__(self):         self._count = 0         self._lock = deferredlock()      def next(self):         return self._lock.run(self._next)      def _next(self):         self._count += 1         recording = self._record(self._count)         def recorded(ignored):             return self._count         recording.addcallback(recorded)         return recording      def _record(self, value):         return http.get(             b"http://example.com/record-count?value=%d" % (value,)) 

first, notice networkcounter instance creates own deferredlock instance. each instance of deferredlock distinct , operates independently other instance. code participates in use of critical section needs use same deferredlock instance in order critical section protected. if 2 networkcounter instances somehow shared state need share deferredlock instance - not create own private instance.

next, see how deferredlock.run used call new _next method (into of application logic has been moved). networkcounter (nor application code using networkcounter) not call method contains critical section. deferredlock given responsibility doing this. how deferredlock can prevent critical section being run multiple operations @ "same" time. internally, deferredlock keep track of whether operation has started , not yet finished. can keep track of operation completion if operation's completion represented deferred though. if familiar deferreds, guessed (hypothetical) http client api in example, http.get, returning deferred fires when http request has completed. if not familiar them yet, should go read them now.

once deferred represents result of operation fires - in other words, once operation done, deferredlock consider critical section "out of use" , allow operation begin executing it. checking see if code has tried enter critical section while critical section in use , if run function operation.

third, notice in order serialize access critical section, deferredlock.run must return deferred. if critical section in use , deferredlock.run called cannot start operation. therefore, instead, creates , returns new deferred. when critical section goes out of use, next operation can start , when operation completes, deferred returned deferredlock.run call result. ends looking rather transparent users expecting deferred - means operation appears take little longer complete (though truth takes same amount of time complete has wait while before starts - effect on wall clock same though).

of course, can achieve concurrent-use safe networkcounter more not sharing state in first place:

class networkcounter(object):     def __init__(self):         self._count = 0      def next(self):         self._count += 1         result = self._count         recording = self._record(self._count)         def recorded(ignored):             return result         recording.addcallback(recorded)         return recording      def _record(self, value):         return http.get(             b"http://example.com/record-count?value=%d" % (value,)) 

this version moves state used networkcounter.next produce meaningful result caller out of instance dictionary (ie, no longer attribute of networkcounter instance) , call stack (ie, closed on variable associated actual frame implements method call). since each call creates new frame , new closure, concurrent calls independent , no locking of sort required.

finally, notice though modified version of networkcounter.next still uses self._count is shared amongst calls next on single networkcounter instance can't cause problems implementation when used concurrently. in cooperative multitasking system such 1 used twisted, there never context switches in middle of functions or operations. there cannot context switch 1 operation in between self._count += 1 , result = self._count lines. execute atomically , don't need locks around them avoid re-entrancy or concurrency induced corruption.

these last 2 points - avoiding concurrency bugs avoiding shared state , atomicity of code inside function - combined means deferredlock isn't often particularly useful. single data point, in 75 kloc in current work project (heavily twisted based), there no uses of deferredlock.