Transducers are fundamental
Transducers are being introduced in the next version of clojure. There are a few posts about them already, but I found the notion of transducers more elegant and natural than I have seen described. They simplify a lot of disparate clojure concepts and make it easier to write more performant code without any loss in generality. They are quite beautiful, actually, and I believe the concept is elemental, like function composition.
From the announcement: a transducer is a function that takes one reducing function and returns another. That's very general and totally relies on understanding a reducing function. A reducing function is a function that has the structure of what you'd pass to reduce. It is a function which accepts an element of input and a previous reduction, and returns a new reduction.
Making it more concrete by way of example, let's start with a simple reducing function: This reducing function reduces a sequences of numbers into a map containing the count and sum of the numbers.
Now that we have a reducing function, let's transduce it, using only what we know so far. Again, per the definition, this means we'll need to return a function that's a reducing function (one that accepts the previous reduction and a new element, and returns the next reduction). Let's try to make a transducer that will cause our mean-reducer to increment all of the inputs.
We've just implemented a crude transducer, and all it can do is increment.
One of the parts of the announcement of transducers was that clojure's core functions (map, filter, take, etc) that normally operate on sequences will gain a new arity that returns a transducer when called with a single argument that's a function. The new code for the 1-arity map looks much like what we already wrote (ignore the other arities for now).
Now we can see in fact how crude our original transducer is, it can be constructed trivially with map.
Neat, but if you stop and think for a minute, the above returns the same result as the non-transducers form: Here, map is operating on a sequence like it did before. Now we know what transducers are, but haven't quite had the epiphany yet. So why is the transducer form different and more importantly, better?
This is the right question to ask! And will lead to understanding the elegance of transducers. Don't proceed until you convince yourself that the previous two examples generate the same result.
Decoupling From Sequences
When using the thrush, or ->>, form above, map accepts an input sequence one by one, and runs inc over each of the elements, generating a new sequence, which is then reduced by reduce. This creation of logic via composition of sequence-iterating functions complects the created logic with the machinery of sequences. This also means that unnecessary intermediate sequences are created just to execute your logic.
If we could separate our logic from sequences, we could write more performant code by avoiding incidental sequences. We could also use the same logic across things that aren't necessarily sequences (eg: clojure.core/reducers, channels).
Transducers let us do that by composing functions "inside" the traversal of the input, instead of composing sequence-traversing functions "outside" on the sequences themselves.
They are similar to traditional function composition (eg: comp) in this way, but a comped function iterated over a sequence can only know about a single element at a time, and doesn't have state. Transducers can encapsulate state (eg: the take transducer uses an atom to count elements), and can control generation of output (eg: the filter transducer can decide whether or not to pass a token to the passed-in reducing function, whereas a composed function that's mapped needs to generate one output per input).
We've seen how you can create a transducer, and the advantages they have, but how do you use them? Above we used our crude transducer and our (map inc) transducer with reduce, but because they no longer require sequences, they can be used in other ways. Here are some examples.
Finally, transducers can of course be composed, allowing the building up of logic without requiring them to be used with sequences.
Now go read the release announcement.