In this episode, we talk to Paul deGrandis about Vase.
The RPM Challenge
Vase
Pedestal
The High Velocity Edge by Steve Spear
Datalog for program analysis
Mike Nygard
Pedestal documentation
Stilobate
Cycling Grand Tours
The Global Cycling Network
The show is available on iTunes! You can also subscribe to the podcast using our podcast feed.
You can send feedback about the show to podcast@cognitect.com, or leave a comment here on the blog. Thanks for listening!
EPISODE COVER ART
PRODUCER
Our theme music for this episode is Thumbs Up (for Rock N' Roll) by killthenoise with Feed Me which was used under a Creative Commons License.
In this episode, we talk with Paul deGrandis about Vase.
RUSS: Hello, and welcome to Episode 118 of The Cognicast, a podcast by Cognitect, Inc. about software and the people who create it.
Today I’m thrilled to introduce the second of our new trio of hosts, Stuart Sierra. Many folks from the Clojure world will know Stuart from his code contributions, particularly the Component library. Stuart is also a frequent conference speaker and the author of not one, but two books on Clojure. Those of us who work with him know Stuart as that lovely, caring guy who keeps pretending to be a curmudgeon.
This week Stuart will be talking to another Cognitect, Paul deGrandis. Paul is a lead developer, instructor, and architect at Cognitect and, most importantly, at least for the purposes of this episode, Paul is also the genius lurking behind the Vase project. But more on that when we get to the show.
But before we do, we have a few announcements. There’s going to be a Clojure Bridge in Toronto on February 24th and 25th of 2017, so go on over to ClojureBridge.org to find out more.
Clojure/west is happening in Portland, Oregon on March 30th and 31st for the conference with training available on March 27th and 28th. The training will include an intro to Clojure and clojure.spec. Have a look at 2017.clojurewest.org.
Finally, EuroClojure will be happening in Berlin on July 20th and 21st. The call for papers as well as opportunity grant applications will open on Monday, March 13th. Go to 2017.euroclojure.org for more information.
If you have a Clojure related event you’d like us to mention, please drop us a line at podcast@cognitect.com.
That’s about it, so on to Stuart and Paul in Episode 118 of The Cognicast.
[Music: "Thumbs Up (for Rock N' Roll)" by Kill the Noise and Feed Me]
STUART: Let’s see. Are you ready?
PAUL: Yep.
STUART: All right. Here we go. Hello, everyone. Today is Friday, January 27th, 2017, and this is The Cognicast. I’m Stuart Sierra, and I’m happy to welcome my fellow Cognitect Paul deGrandis back to the show. Thanks for joining us.
PAUL: Thank you so much for having me.
STUART: You are welcome. I’m happy to have you here. As we mentioned to you earlier, we like to start off the show by asking a question of all of our guests, and that is to describe some experience of art. It could be anything at all, just something that you found interesting.
PAUL: Yeah. This question always gives me trouble. I always have a hard time finding something in my brain, but I think I have a good one. In the month of February there’s a contest called the RPM.
STUART: Okay.
PAUL: The goal is to write and record an entire album of music in the month of February.
STUART: Ah, like National Novel Writing Month in November.
PAUL: Exactly. Yeah. Exactly, and so they always choose cold months or dreary months where creativity or motivation is usually lacking.
STUART: Right.
PAUL: The experience of trying to write and record and then master songs all in a month is daunting, but the result is you give up things that aren’t important. You very quickly find things that you can shed.
STUART: Right.
PAUL: I find a lot of the recordings that come out of RPM are somehow more honest or more authentic.
STUART: Sure, sure.
PAUL: The feeling, the essence of the songs really come through.
STUART: Nice.
PAUL: It was started here in New Hampshire, but now it’s an international thing, and there’s these great parties. That’s just around the corner, so I’m really looking forward to that.
STUART: Cool. Have you participated in this as a writer or a participant?
PAUL: I have many years now in a row, and I will be this year as well.
STUART: Excellent. Maybe we’ll borrow some of your music and play it on the show sometime.
PAUL: Embarrassingly so.
[Laughter]
STUART: Excellent. All right, well, that sounds like fun and definitely an opportunity for all kinds of desperate, creative measures.
PAUL: Yes. Yes.
STUART: The topic we brought you here to talk about today or the most interesting one I think we are scheduled to talk about today, I’m not going to say the name of it before you do. I will let you establish the precedent for how we’re going to pronounce the name of this project.
PAUL: Sure. I feel like I want people to feel free to pronounce it however they want. I say the project’s name is Vase [vās], but oftentimes people will pronounce that thing as Vase [väz], and that’s okay too.
STUART: Yeah.
PAUL: I’m alright with Vase [väz].
STUART: Yeah. It seems to be one of those words that people grew up pronouncing it one way and are shocked to discover that someone might pronounce it any other way: vāz, vās, väz. Whatever.
PAUL: Yes, exactly.
STUART: We’ll go with vāz for this – or was it vās? Sorry, vās for this episode. Tell us about Vase. What is it?
PAUL: Sure. Vase is a project that’s over two years in the making now. It is a library to help you write data driven microservices on top of Pedestal. In 2014, at Clojure/conj, I had given a talk where Vase was introduced, but not available. Not available as open source at the time. I said tongue in cheek, you know, Vase is a small container that sits on top of Pedestal and is very limited in what it can hold.
STUART: Right.
PAUL: That’s very true today. The data descriptions are very limited in what you can say, but they’re fully extensible, so you could always make them say more. That is the general what, you know. The why is more interesting, I think.
STUART: Okay. Well, we’ll get into that in just a second. You said it’s a container for API services. Where does that fit? If I’m building an application, what piece is Vase providing?
PAUL: Sure. When writing a service or an API oftentimes you’re just trying to expose data over HTTP so other clients can use that data.
STUART: Right.
PAUL: Maybe it’s a smart, single page application, or maybe it’s a mobile application. All of that data plumbing chews up a lot of time, but it doesn’t deliver a lot of value to your customers. Vase is there to automate all of the mundane data plumbing that would otherwise be chewing up your time, so you can focus more on things that face the customers or things that deliver real value for the business.
Where it fits is instead of writing all of this Clojure code in Pedestal, instead of writing all of this database code in Clojure, Vase does all that data plumbing for you. It takes Datomic schema, it takes some route descriptions, it takes some clojure.spec specs, and it will automatically do all of the data validation, all of the data marshalling, and all of the data shaping for you - all of the really mundane, boring, data plumbing stuff that we just have to do in order to get to value.
STUART: Got it. Basically I have a database, and I have some data that describes what kind of data is in there: the schema, the specs, and so on. I give that to Vase and, boom, I have a Web API for my rich client app or whatever.
PAUL: Correct. Yep, and it follows a lot of same conventions, and so HTTP status codes are enforced for you and all these other things that are just annoying to deal with. They’re all taken care of.
STUART: Let’s go on to that topic we skipped over a second ago. Why? Why Vase?
PAUL: Yeah, so–
STUART: Aside from we needed something to sit on top of Pedestal because we needed a pedestal.
[Laughter]
PAUL: Yes, exactly. Yeah, besides the pun–
STUART: Which is a great reason. Let’s face it.
PAUL: Yes. Yes, we love our puns at Cognitect, for sure. There’s this whole mindset about delivering value to customers and taking actions that deliver value. I think it’s best said in a book called The High-Velocity Edge by an author whose last name is Spear. He says, “Today sustainable competitive advantage has to be won by creating internal capacity to improve and innovate fast and without letup.” That one quote from the book perfectly describes every company that I ever sit down with through the consulting business of Cognitect.
STUART: Mm-hmm.
PAUL: They know they have to move forward. They know they have to move forward faster. They know they’re sitting on a bunch of data that has a lot of value in it.
STUART: Yeah.
PAUL: But they can’t seem to get that value out, right? They can’t seem to go fast enough. Their tools don’t let them go fast enough.
STUART: Mm-hmm.
PAUL: And so the real why of Vase is: Let’s get all that value out of the data as fast as possible. Let’s automate all of that mundane plumbing so you can focus just on delivering value. And let’s do it in a tool that allows you to evolve and move forward fast.
Vase is about doing all of those things. It’s about: I show up and instead of spending weeks or months writing a service and doing all of this plumbing, I spend minutes and I have my service, I have my data model, and I have my validations all set to go. As I move forward through time, as my business evolves, as my value propositions evolve, the services and the data model can evolve very freely. Each individual team within an organization could evolve a Vase service independently without even bothering each other. I mean it’s very easy to evolve these things however you need to, and so that’s the real why of Vase - to get to value faster.
STUART: Got it. Got it. It sounds like one of those cases where we’re trying to unlock the database or un-silo the database and connect it with other applications and other sources of data.
PAUL: Very much the case. In larger organizations, and even in small organizations, technology departments get commercial software off the shelf or start with a framework, and that slowly silos their technology and slowly silos their data.
STUART: Right.
PAUL: Then they hit a point where they can’t even move forward any more. They’ve siloed it so much that they can’t move forward any more.
STUART: Sure. Sure.
PAUL: Vase is about blasting through all that and getting past all of that.
STUART: It’s a Vase used as a missile. Is that the analogy here?
PAUL: Yeah.
[Laughter]
STUART: It’s a large stone pot traveling at high velocity. Well, cool. That sounds great. What would I need if I were going to adopt it? You said it’s built on top of Pedestal, so I assume it requires Pedestal as the minimum foundation.
PAUL: Yep. Pedestal is required to have the HTTP endpoints. It is very specific to how Pedestal works, so Vase can’t go without Pedestal.
STUART: Okay.
PAUL: The data model currently is tied to Datomic, but there’s nothing stopping you from extending Vase to support a data model that was not Datomic specific. But Vase out of the box right now, if you’re on Clojure, you need to be on Clojure 1.9.
STUART: Mm-hmm.
PAUL: You’re on the latest Pedestal and you’re using Datomic.
STUART: Cool. Well, that certainly describes a lot of projects I work on, so that would be great.
PAUL: Yep.
STUART: Let’s get into some of the details of how this works. I’ve only looked at it briefly. I’ve looked at some of the example apps and some of the code that’s currently in development, I think.
PAUL: Yep.
STUART: I notice in the examples there are fairly detailed descriptions of routes and schema and queries and so forth in EDN files, in EDN data files, but surprisingly not a lot of code.
PAUL: Correct. Yeah, all the code is generated automatically by the library itself. Vase, the library, has a bunch of Pedestal interceptors.
STUART: Mm-hmm.
PAUL: Then those descriptors have what we call action literals in them. Actual literals will expand either into more EDN or into interceptors. And so by the end of reading that EDN file, you have absolutely everything you need in order to interact with the database or to interact with Pedestal, so no code at all.
I mean that’s the beauty of Vase. You show up. You know, I do a lein-new Vase my service. I have a data description file, and that’s the end of my service. As soon as I turn on Pedestal or do a lein-run or something like that, I can keep editing that data file. As soon as I hit “save,” all of those changes are live on the service.
STUART: Oh, cool, so it’s dynamic as well. It’ll automatically reflect changes to those files as you’re working on them.
PAUL: Yep, and dev mode. Dev mode is, as soon as you save that file, you see the changes and then in prod mode it gets locked down.
STUART: Got it. Got it. I think I saw – correct me if I’m wrong here, but it seemed – the documentation seemed to suggest that you could also create these services dynamically. You could actually post them to the server and it would create that service based on the description you sent. Is that right?
PAUL: In the original version of Vase that was default out of the box.
STUART: Okay.
PAUL: People could come along and say, “I want to expand the service. I want to expand the service,” and so on.
STUART: Yeah.
PAUL: In the version of Vase that is released as open source, that is no longer default out of the box.
STUART: Okay.
PAUL: But that code is one function, and it’s a very easy function. All the parts of Vase are functions or protocols, so it’s really easy to sort of take control of the whole situation and make the app behave the way you want it to behave. No more of posting services and having them magically show up, but–
STUART: Okay. I could see that might be a security risk in some situations.
[Laughter]
PAUL: Yes. Yeah, for sure.
STUART: Cool, so these descriptor files, if I want to write one of them, walk me through what it entails. What do I have to provide to Vase to get this API service up and running?
PAUL: There are three major parts to a Vase description file. Like you said, it’s just EDN. It’s just data. And the three sessions all have three very specific namespaced keys, and that’s how Vase actually operates. It’s going to look for these three keys.
STUART: Okay.
PAUL: The very first part are called norms. Vase norms are just the description of your schema. You could write standard Datomic schema chunks, but Vase also supplies a reader literal for writing shorthand chunks of Datomic schema, which saves a lot of space and makes it a little cleaner to read and a little cleaner to write. That’s the first chunk.
The next chunk is called specs, vase.specs. These are clojure.specs, just like standard clojure.spec library in Clojure 1.9.
STUART: Mm-hmm.
PAUL: You write those however you want. Literally however you write them in Clojure is how you write them here in the Vase descriptor.
STUART: Got it.
PAUL: Those are optional. I mean if you want to have validation, you can write those. If you don’t want to have validation, you certainly don’t need to write those.
Then the next section is called Vase API. That describes a bunch of namespaced APIs. There are two things in that sentence, and we’re going to pull them apart right now.
STUART: Okay.
PAUL: One is that a single Vase descriptor can have multiple APIs described in it. It’s totally legit.
STUART: Okay.
PAUL: However you want to distinguish those APIs, whether you want version names or you want not version names or whatever, you basically tagged those APIs using a namespaced keyword. Then those APIs, they describe routes.
STUART: Okay.
PAUL: The routes are described how you would expect. There’s some URL, then there’s some HTTP method, and then there’s an action literal. Is this doing a query? Is this doing a transaction? Is this just responding with static data? Is this doing a redirect? And so on.
STUART: Okay. An API is a collection of routes. Is that right?
PAUL: Correct. Yep.
STUART: Okay. I might have more than one API.
PAUL: Correct. Yep. You might have – not that this is a good practice, but maybe you have version 1 of your API and version 2 of your API, and so on. Or maybe you have the main API or the core API. Just like you name files in Clojure–
STUART: Sure.
PAUL: –or name files in any programming language in order to organize them, you name and tag APIs the same way.
STUART: I see, so the route itself is not necessarily the top most thing in the hierarchy. There is the API as a grouping outside of that.
PAUL: Correct. Yep, and so that tag for that API will be the first thing, right? When you start a service, it gets inserted at an endpoint in your Pedestal application like /API. Then /your tag name.
STUART: Okay.
PAUL: Then /your route name, right?
STUART: Yeah.
PAUL: So everything gets totally name spaced as an API.
STUART: Great. We definitely like our name spaces in the Clojure world.
PAUL: Yes.
STUART: That’s a good thing to see.
PAUL: Yes, for sure.
STUART: There’s a question I had related to this because this is a topic I’ve encountered a number of times recently. That is, when you are trying to design some kind of higher level abstraction that is model or domain specific, there is often a question that comes up of do I design this as a pure data structure and make a DSL out of Clojure data or EDN data, or do I write a collection of functions and/or macros that a user or an application can call to assemble whatever the abstraction is? Of course these have tradeoffs either way, so I’m curious what led you to choose the data structure route for Vase.
PAUL: I think I’ve been on this plotline for a long time.
STUART: Yeah. Exactly. We all have.
PAUL: And so this is the natural progression of that plotline. That is that if Vase is a data structure, if the description of Vase is a data structure, then anything can generate it and anything can consume it, and you can store it wherever you want.
STUART: Got it.
PAUL: I can also pull it apart, and I can query it either with Datomic or CoreLogic or whatever I want. And so having Vase as data was really important for a few reasons: for the construction and for the generation, but also for the parsing and sort of for a hidden benefit. There are tools in Vase that allow you to query your own descriptor and enforce properties on top of it.
STUART: Oh, cool.
PAUL: If you had some business rule that you wanted to ensure was always followed, you can certainly do that in Vase. You can’t do that in the DSL side or in the purely code side without doing program analysis.
STUART: Right.
PAUL: That’s a huge headache, right?
STUART: Sure. Sure.
PAUL: Yeah, avoiding that was important from the very get go, from when we first imagined Vase as a solution.
STUART: Got it. Yeah, and that’s something that I think we’ve all – a conclusion that a lot of us have come to is that data has the advantage that it’s data. You can do stuff with it besides whatever you did with it the first time. You can parse it. You can analyze it. You can generate it. Things that might be much harder or even impossible if all you have is functions or macros being called.
PAUL: Yeah, 100%, and you know there’s a lot of really great research out there about using Datalog to do program analysis.
STUART: Mm-hmm.
PAUL: You read these research papers, and you’re like – you read about all of the pain they have to go through in order to get some AST representation that fits into some sort of Datalog set of facts.
STUART: Right. Right.
PAUL: The moment that you get rid of code as the primary source and you say, “I’m going to express this in a data structure,” you’ve solved all of those problems right away.
STUART: Right. Right. Well, great. Yeah, I wouldn’t want to try to query the full possible range of what Clojure code can be in any language, really.
PAUL: For sure.
STUART: I think that would be far too difficult to be practical.
PAUL: There’s also a really funny sort of benefit to Vase being pure EDN and using reader literals. Many years ago now at some conference I gave a talk, and I said one great trick about Clojure is abusing the reader.
STUART: Yes.
PAUL: I feel like people don’t abuse the reader enough. I feel like every single time I go out to a Clojure function somebody asks me, “What do you mean by abuse the reader?” It’s like Vase is the perfect case study about this is what I’m talking about, you know, like the reader is there for your utility.
STUART: Right.
PAUL: Any way that you can make it do the job for you, you should be doing that.
STUART: That’s interesting. I did notice the reader literals, and you mentioned them earlier. If you’re not – anyone who might not have seen these before, these are the symbols in Clojure that start with a hash sign and then a symbol and then usually some other Clojure data structure, and that actually gets turned into basically a function call that is evaluated inside the reader. While your data is being read, you can invoke a piece of code that will transform it into some other piece of data, and that is what the reader actually hands back to your program.
PAUL: Yeah. It’s great because it’s not happening at compile time. The reader in Clojure is the same reader in ClojureScript, and I’ll let everybody’s imagination think about what you could do with Vase descriptors between the reader in both locations. Yeah, it’s great that it happens at read time, right?
STUART: Yeah.
PAUL: This is a pure data thing.
STUART: And so what kinds of things do the reader literals get used for in Vase? Are they like constructors for particular kinds of things?
PAUL: Yep. They are constructors, so in some cases they get turned into just hash maps. The schema chunks, that reader literal just gets turned into a vector of a bunch of hash maps so that it can be consumed by Datomic.
STUART: Got it.
PAUL: But the other reader literals all get turned into a record type, and that record type then fulfills or satisfies a number of protocols throughout Pedestal so that they become, you know, first class citizens with everything in Pedestal.
STUART: I see. They’re a combination of making the syntax more convenient to write and also, I guess, if they’re creating records, that’s something that isn’t always obvious how to express in EDN.
PAUL: Right. The choice for choosing records to back the reader literals was also so that we could control the printer as well, right?
STUART: Aah.
PAUL: If you wanted to turn around and print a bunch of Vase services back out, they would print the same way, right?
STUART: Oh, nice, so it all round trips.
PAUL: It round trips perfectly through the printer and the reader, yeah.
STUART: Cool. Yeah, that’s something that is great when you’re working with pure Clojure data structures, all the built in ones, but it can sometimes get a little complicated when you start adding your own types and your own records.
PAUL: That’s the rationale. When you generate that record, or when Vase generates the record from the reader literal, it’s doing all of the code gen that it needs to do by satisfying all these protocols in Pedestal. That’s where the code gen piece comes in.
STUART: I have my application. I’ve got my descriptor files for Vase. I assume these are basically combining the things you would use to describe a particular type of thing, not a type in the programming language sense, but a type in the domain space that you’re working with.
PAUL: Yep.
STUART: I’ve defined these. I run the app, and I’ve got a Web service. What form does that Web service take? I’ve seen in a couple of demos it runs a Swagger based UI tool. Is that right?
PAUL: Yep, so it can do a number of things, and it will do more things in the future. Out of the box it doesn’t do Swagger. It doesn’t have a Swagger UI. It has sort of a really hand rolled, very simplistic API browser endpoint.
STUART: Okay.
PAUL: If you hit just /API or whatever, wherever your Vase service is rooted at in your tree, you’ll see a list of all the routes and the methods and the things that you can pass to it, so all the information you would get from a Swagger UI, but not presented in any sort of pretty or interactive way.
STUART: Got it.
PAUL: Then after that all the endpoints, just JSON, yep, you would just interact with them however you would interact with any other JSON based Web service whether you want to cURL it or you want to call it from another Web service or whatever.
STUART: Okay. It does use JSON as the lingua franca of the Internet, which I suppose it is by now.
PAUL: Yeah, and perfect term because the original version of Vase used EDN and then another version of it used transit JSON. You can make it use all of those. I mean it’s very easy to change what the wire format is, what the data format is. It’s not locked down by any stretch of the imagination. But the default one is JSON specifically because it’s everywhere.
STUART: Right.
PAUL: This is a bridge into all other tools. It shouldn’t be an island. Vase shouldn’t create a bunch of islands for you. It should create a bunch of services that have very natural bridges between them and all of the other tools you have.
STUART: Yeah. When it comes to accessibility and compatibility, it’s hard to beat JSON these days.
PAUL: Yes. Very true.
STUART: Well, cool. You’ve mentioned that this will be open source. What does that mean? Do you know when that’s going to happen?
PAUL: I think by the time the listeners hear this we will have made the announcement and the repo will be opened. It’ll be open sourced under the EPL, and it will follow the same contributing guideline as Pedestal.
STUART: That is?
PAUL: That is you need to sign the Cognitect contributors agreement, which is all electronic now. It’s very easy to fill out.
STUART: Mm-hmm.
PAUL: Post an issue for any issues you have or feature requests so that the community at large can discuss its merits and its design, and then pull requests thereafter.
STUART: Cool. Will you be looking for people to add or extend features in Vase once it’s released?
PAUL: I think that what I’m expecting to see – whether this will happen or not, what I’m expecting to see is that many people will write add-on libraries, not that Vase itself will take on more core action literals.
STUART: Right.
PAUL: I think the action literals that ship with Vase, that’s what you get. You will not get any more unless somebody makes a very compelling argument for more.
STUART: Yeah.
PAUL: But I would see that people would write, like, a Vase-blahblahblah. That would add more action literals for you.
STUART: I see. They could add a different database backend, for example.
PAUL: Yes.
STUART: Cool. Sounds great.
PAUL: Yep, or new endpoint actions are very common. Yeah, and that’s the great thing. I mean Vase was designed very specifically to make it extensible at a lot of different spots so that code contributions wouldn’t have to always be changing the core. Vase was designed to be very, very small and very, very simple and straightforward. Then all of the pieces that you would want to change, those are all open for extension.
STUART: Great. Those are the flowers in the vase, if we could torture that analogy a bit more.
[Laughter]
PAUL: Yes, exactly.
STUART: This all sounds fantastic, and it’s built on top of Pedestal, of course, so I think we will switch gears slightly here, and I’ll ask you what’s been going on with Pedestal since I think you are still leading the development of that.
PAUL: Yep, still leading the maintenance and the development there.
STUART: The last time you were on the show it was just after or just before releasing version 0.3.
PAUL: Yes.
STUART: I think a couple of weeks ago version 0.5.2 just came out, so clearly stuff has been happening. Tell us about it.
PAUL: Yeah. Thinking about Pedestal 0.3 is like, you know, thinking about a childhood toy.
[Laughter]
PAUL: It seems so far ago and such a different–
STUART: It does seem long ago.
PAUL: Yeah. A lot has happened in Pedestal. From 0.3 to 0.5 there was a progression in moving and refactoring the sort of surface level of Pedestal.
STUART: Mm-hmm.
PAUL: The core pieces of pedestal, the interceptor chain and things like that, were very well engineered to begin with. Some changes to them have happened. But for the most part the very, very core pieces have stayed the same, but the surface area of Pedestal was huge. It was really hard to get your hands around. It was really hard to get your head around.
STUART: Mm-hmm.
PAUL: The real goal moving from 0.3 to 0.5 was to reduce that surface area to a set of very small, concrete APIs and those APIs would all be protocols.
STUART: Aah.
PAUL: I wanted all of Pedestal to be programmed against protocols, so you could extend it however you needed to.
STUART: Nice.
PAUL: In that process, once things were put behind protocols, it became really easy to start breaking Pedestal up into other modules. Like Pedestal Log is a separate module now. Pedestal Route is a separate module. Pedestal Interceptor, the interception chain by itself, is a separate module.
Previously, because all these things were tied together, people would often compare Pedestal to Ring. But for me Pedestal really isn’t a HTTP library. It’s not a Web library. Pedestal is a toolset, is a set of libraries for writing backend applications.
STUART: Okay.
PAUL: And so it could do lots of things. The interceptor chain has nothing about HTTP in it. It is not HTTP specific. It’s just about combining both chain of responsibility pattern and the interceptor pattern together to write concurrent and high performance applications.
STUART: Yeah. That’s an interesting thing. I think one of the most interesting changes that’s happened in Pedestal recently was that refactoring that turned the interceptor pattern, framework, into its own standalone library, so you can now use that as a tool to build an application even if what you’re doing has nothing to do with Web apps or HTTP.
PAUL: Exactly. I’ve been sort of giving meet-up talks and talks here at Cognitect about how I really see Pedestal and teaching people to think about Pedestal from the core abstractions first.
STUART: Mm-hmm.
PAUL: Backwards, you know, so let’s start at interceptors. Okay, let’s connect interceptors into a chain. Okay, now let’s process a chain with a container around it and so on. I think that’s the way you should think about it.
I’m seeing the community do that as well. There are messages on the mailing list coming through about people just using the chain. I just want to use the chain to process this piece of data or to connect it.
STUART: Yeah.
PAUL: Internally at Cognitect we’re doing that too. We have Pedestal Kafka for using the Pedestal interceptor chain with Kafka. We’re seeing more and more of that, and that really excites me.
STUART: Describe for someone who – assume we haven’t used interceptors or worked with them. Is there a short answer to: What do interceptors give me? What capability do they provide?
PAUL: Sure. Interceptors provide the capability to separate out your processing into individual steps. In the case of Pedestal, they’re described. They can be described as data, which is great.
STUART: Mm-hmm.
PAUL: Then assemble those, compose those steps together however you want and execute them as a chain, at which point any interceptor could go asynchronous or not, but all processing, logging, and debugging, all of that nonsense still happens as if nothing was asynchronous at all, nothing was concurrent at all. And so really it’s the interceptor pattern and the interceptor chain are designed to simplify concurrent programming, in general.
STUART: Okay. I have some big, complicated, procedure with many steps, and some of those might be asynchronous or depend on asynchronous IO. I can compose them all together using interceptors and not have to deal with the differences between what’s happening asynchronously and what’s happening synchronously on a thread and so on.
PAUL: Correct. This pattern is used a ton.
STUART: Yeah.
PAUL: In computing, right? The servlet model is the exact same model. The way messaging middleware work is the exact same model, right? Very robust applications that handle a lot of traffic, that handle a lot of processing are engineered this way specifically because this pattern is so well suited for that kind of use case.
STUART: Got it. Got it.
PAUL: That was the major change.
STUART: Yeah.
PAUL: I mean I’ll go through a couple of other really quick changes.
STUART: Sure. Sure.
PAUL: The logging API is separate, and it also includes metrics, so Pedestal by default publishes metrics to JMX, but you can redirect that to AWS CloudWatch or StatsD or whatever you want. There’s runtime metrics that happen. Pedestal’s router got even faster.
STUART: Mm-hmm.
PAUL: For a small to medium sized application, it will route in 60 nanoseconds or so.
STUART: Wow.
PAUL: It can pick routing decisions quite fast, and it routes actually in effectively constant time, so log32(N) time.
STUART: Got it.
PAUL: Just a lot of other smaller optimizations like that, and all those small optimizations add up across the entire picture, right?
STUART: Sure.
PAUL: Making the interceptor chain faster, making routing faster, making logging faster, pushing things out to protocols helped a lot to speed up some calls in some cases.
STUART: Mm-hmm.
PAUL: Small gains across the whole thing.
STUART: Where do you see – as it continues to evolve, where would you hope to see Pedestal and maybe Vase as well fit in to the broader Clojure ecosystem? Obviously there’s a fairly robust ecosystem of libraries in Clojure these days and including a lot of things related to Web or backend server development. What role would you like Pedestal and Vase to fill there?
PAUL: That is a fantastic question. It’s one that I think people don’t actually ask a lot, but it is really important to me, so thank you for asking.
STUART: You’re welcome. Happy to help.
PAUL: I think that I want to see more people treat Vase as a tool to simplify robust, backend, concurrent development regardless of the library they choose for the IO or for the container, right?
STUART: Mm-hmm.
PAUL: I could imagine an Aleph application and a chunk of that application’s processing is done on an interceptor chain.
STUART: Interesting.
PAUL: That’s a really straightforward idea. But going forward, I like this idea that people internally in Cognitect and externally in the community are using just the interceptor chain or just the interceptor chain and the logging to extend Pedestal’s capabilities into other systems, to tie that interceptor chain into other systems like Pedestal, Kafka, and so on.
STUART: Nice.
PAUL: My real focus will be of course we’re going to keep continuing and evolving the HTTP specific stuff, but I really want to start growing the catalog for Pedestal of, you know, here is Pedestal for all of these other protocols. Here is Pedestal for 9P even.
STUART: Yeah.
PAUL: Or here’s Pedestal for Kafka, or here’s Pedestal for, you know, whatever.
STUART: Yeah.
PAUL: Whatever you’re doing. And so you could imagine a large collection of libraries that are just additions to the interceptor chain.
STUART: I see.
PAUL: Whether they’re HTTP specific or not. For Vase specifically I hope that, in sort of the same regard, we see people extend Vase’s reader literals, the action literals to extend it into new systems. I think the core part of Vase probably needs to be tightened up a little bit.
STUART: Mm-hmm.
PAUL: The current version of Vase, as people are sort of experiencing it now, is where Pedestal probably was at 0.3.0.
STUART: Okay.
PAUL: It probably needs to be tightened up a little bit. Probably the API could be a little cleaner. We certainly took all of the lessons we learned from Pedestal when we were cleaning up the API for Vase, but I’m sure there’s more work to be done. But then I would be really excited to see people extend Vase through the action literals to push Vase into new systems.
STUART: Basically you plan to take over all data processing everywhere in Clojure. Is that right?
PAUL: Yeah.
[Laughter]
PAUL: Pretty much. I want to have people – I’m on this real big mission to have people stop wasting their time, you know?
STUART: Right.
PAUL: I don’t want people to keep building tools or to keep wasting their time. I want people to solve real problems and to deliver real value. Writing ETL jobs and writing all these data processing pipelines and the infrastructure for doing that.
STUART: Yeah.
PAUL: That’s not delivering value. That’s just the annoyance of having to show up at work.
STUART: Right, just getting system A to talk to system B and moving the data from place-to-place.
PAUL: Yeah. I just want to get rid of all that. I want to show up and say, “You have a real hard problem.”
STUART: Yeah.
PAUL: “That real hard problem can only be solved by a distributed system.” I want to have tools ready to make that as straightforward as possible so I can get to a value today instead of a value in six months.
STUART: Well, this all sounds super. I hope that vision comes to fruition, and I hope I can play a part in it.
PAUL: I welcome your company, Stuart.
STUART: Thank you.
[Laughter]
STUART: I was. I have been involved off and on in Pedestal. Not recently, but I was there way back at the beginning before even the 0.3 release, I think.
PAUL: Yeah. There are definitely. I have definitely touched lines of code where I get blame and your name shows up.
STUART: There are still things to blame on me in Pedestal. I’m sure of it. Well, great. Before I jump topics here, I’ll ask is there anything else that you wanted to tell people about Vase or Pedestal or any of those things.
PAUL: I think by the time the listeners hear this, the Vase documentation will start to be coming together, but won’t be complete. But I do want to say a big thanks to Mike Nygaard for helping pull together all of the Pedestal documentation and the new Pedestal site. It’s been really great for a lot of new users and for existing users. And a wider thanks to the community for putting the effort in to extend those guides and to fill them out with detail. The documentation story around Pedestal is much better than it has been in the past. Certainly it’s thanks to everybody else and not me at all.
STUART: Great. Great. We should have mentioned there, there is a dedicated site for it now.
PAUL: Correct, Pedestal.io.
STUART: People can go there to find out everything they ever wanted to know about Pedestal, including whatever pun we’ll be working on for the next thing.
[Laughter]
PAUL: Yeah. I’m starting to – we’re going to be grasping at straws, but maybe that’s the next thing like the straw that you put into a vase. I don’t know.
STUART: Yes. You put a straw in the vase, and there’s a flower on the straw in the vase on the pedestal. I don’t know.
I remember. I don’t know if this is still in the code, but at one point there was a thing somewhere in the Pedestal code named stylobate.
PAUL: The stylobate is still there, yeah.
STUART: It’s still there. I remember when I first saw that, I had to go look it up on Wikipedia to figure out what it was. Now I’ve forgotten what it was. Do you remember?
PAUL: Everybody does. Yeah, everybody. For me, even working in the code, I feel like a whole year went by where I kept going, like, what is a stylobate?
[Laughter]
PAUL: A stylobate is the foundation, the very bottom piece of a pedestal, that really wide, broad piece of a pedestal.
STUART: Okay.
PAUL: That is the stylobate.
STUART: All right, well, I’ll try to remember that for a little while, but I’ll probably forget it again in a month, the next time I see it.
PAUL: Yeah, not worth putting in the mind palace.
STUART: Yeah. Yeah, not essential information. It’s down in the internals of Pedestal, anyway. It’s not something you necessarily need to know about.
PAUL: Nobody would ever see it. Yeah.
STUART: Well, cool. We have, I think, a little more time here, so if you’re willing, I wanted to ask you about another thing unrelated to Clojure code. At least I think it’s unrelated. That is competitive cycling, which I’ve heard is something that you’ve been involved in.
PAUL: Yeah. I have ridden a bike since I was very little, maybe five years old or so, four. I don’t know when I first started riding a bike, but very little, and have ridden a bike my entire life.
STUART: You’re doing it right now, in fact.
PAUL: I’m sitting down looking at two bicycles. There are two bicycles sitting in my office right now, actually. But last year I had started really focusing on what does it take to actually be good at cycling. If you were going to do cycling for long distances, high elevation gains, long trips, or competitively very fast, what does it take to do that?
I set off trying to figure this out, teach myself these things, and get these skills. This is my second year now, and I actually train with professionals and ex-professionals, so people who have ridden in the professional peloton in the grand tours like Tour de France and the Giro d’Italia. I ride with those guys pretty regularly now.
STUART: Wow. I imagine that’s pretty tough keeping up with them. From what little I know of the sport, I know it can be some of those long tours, like the Tour de France, can be absolutely grueling.
PAUL: Quite demanding. Yeah, so for a grand tour like that. There’s a lot of different kinds of races, and maybe we’ll get into that. A grand tour, there are three grand tours: one that happens in France, one that happens in Italy, and one that happens in Spain. They’re all 21 days.
STUART: Okay.
PAUL: Every day is like 100 to 120 miles. Probably, well, I mean there are some flat stages, but a lot of the stages are called mountain stages, and so they’ll go over the Alps.
STUART: Wow.
PAUL: They might go over the Alps and then back over the Alps again, so sometimes you go back and forth over them. Yeah, so imagine you’re riding your bike 21 days in a row, 100 miles or more every single day, over a mountain chain every single day.
STUART: I don’t want to. I don’t want to imagine that. I’m sure it is fun for those who can do it.
PAUL: Yeah. Cycling is such a funny sport. It’s just a great sport to watch. I think a lot of people think, when they think of cycling or competitive cycling, it’s just a bunch of people racing on a bicycle from the start to the finish line.
STUART: Mm-hmm.
PAUL: But in a race there’s lots of stages. There’s sprint stages where you can get points, and there’s mountain stages where you can get points. You’re just trying, as a team. It’s a team sport, so there’s anywhere from three to seven of you. You’re trying as a team to accumulate as many points. The finish line, that’s only one stage, right?
STUART: Mm-hmm.
PAUL: The finish line is actually not that important.
STUART: Okay.
PAUL: You just have to finish in order to get your points.
STUART: Got it.
PAUL: That’s the only reason to finish. Yeah, the teams are really fluid. You might be biking, and you might make an attack. You might try to go faster than everybody else in the group. Other people from other teams might follow you, and now suddenly you’ve sort of formed yourself a mean team just out of all these individuals that decided to do this. You have to work together regardless of what real team you are on in order to make those things work out for you.
STUART: Interesting.
PAUL: Yeah, there’s all these tactics. It’s just a resource management game, right?
STUART: Yeah. Yeah.
PAUL: You can only ride a bicycle for five hours. After that you’re out of energy completely.
STUART: Uh-huh.
PAUL: You’ve got five hours worth of energy. Your job is to either make the other team expend more energy and make sure that you don’t expend more energy than the rest of the other teams. It’s like all these tactics. It’s just an energy management game, a resource management game.
STUART: Fascinating, like those Eurogames where you have to turn your technology resources into food resources and things like that.
PAUL: It’s exactly like that. Imagine a seven-way Eurogame where alliances get randomly created and torn apart on a whim and you’re just exchanging one resource for another. That’s exactly what competitive cycling is like.
STUART: Cool. It sounds like it should be a board game.
PAUL: You know I would not be surprised if there’s some really old French board game that is this game.
STUART: That’d probably be the only way in which I could actually participate in this.
PAUL: [Laughter]
STUART: But I was fascinated by this. This came up in one of our chat rooms earlier this week. I just enjoyed your description of cycling as being like chess at 20 to 50 miles per hour.
PAUL: It’s exactly like that. Yeah. You’re making these moves. You’re trying to shut down parts of the “board.”
STUART: Mm-hmm.
PAUL: You’re purposefully trying to block parts of the road so that other teams can’t use them.
STUART: Yeah. Yeah.
PAUL: You never go slower than 20 miles an hour.
STUART: Wow.
PAUL: You will go up to 50, 55, and you have to eat.
STUART: Yeah.
PAUL: Every race, in the middle of the race, has a feed stop, but you don’t ever stop moving, right? You have to grab food while you’re moving past, and you’re eating sandwiches in the middle of the race at 20 miles an hour. It’s really hilarious. Yeah, it’s just a really funny story. Chess at 20 to 50 miles an hour.
STUART: While eating sandwiches.
PAUL: While eating sandwiches.
STUART: [Laughter] Yeah, that sounds rather terrifying, frankly.
[Laughter]
STUART: To be in the middle of that. I think another topic that came up that I found kind of fascinating was the amount of energy that a competitive cyclist can put out in the course of a race.
PAUL: Yes. Yeah. It is mind boggling compared to other humans, and then completely not mind boggling when you compare it to electronics we use every day.
STUART: Right. Right.
PAUL: The average rider, when they get on a bike, probably rides a bike at 110-ish watts or so, 115 watts.
STUART: Okay.
PAUL: That’s to go, like, 12 to 15 miles per hour.
STUART: Okay.
PAUL: And a professional sprinter in the Pro Peloton will put out 1,500 watts.
STUART: Wow.
PAUL: That’s to go like 40 miles an hour.
STUART: They could power a hairdryer with the bicycle.
[Laughter]
PAUL: Yes. Yeah, and so that was the joke that we all started making. It’s like that’s basically like a hairdryer.
STUART: Yeah.
PAUL: You know 900 watts. The joke is like, oh, you got a hairdryer.
STUART: Okay, so none of us are going to be relying on bicycles to power our computers and electronics any time soon.
PAUL: No, it is the worst - the worst.
[Laughter]
STUART: You can actually measure these things, right? You have little gadgets on bikes nowadays that can tell you how much power you’re producing. Is that right?
PAUL: Yep. Yep. Power meters in all different forms, so there are power meters on trainers. There are power meters on the bike. You’re measuring your power and your power output, again because it’s a resource management thing.
STUART: Right. Right.
PAUL: I happen to know how much power I can put out for an hour. Then after that I’m out of energy. I’ve depleted my glycogen stores, right? I can’t – I can’t put out any more power than that.
STUART: It’s a Eurogame with your own biological resources that we’re talking about here.
PAUL: Yeah. Yeah, and you know that your body can only process 100 calories an hour, right?
STUART: Uh-huh.
PAUL: You’re limited to how many resources you can take on to try to replenish yourself. Yeah, it’s a complete resource management game.
STUART: Fascinating, and I’m sure there are algorithms for optimizing or finding optimal paths in that space.
PAUL: Yeah, there are, and people spend a lot – I mean like the professional teams spend a lot of time and money trying to figure out what’s the most aero position, what’s the most effective use of food at what time, at what speed, at what cadence and so on - all that stuff. But at the end of the day it’s humans on bicycles.
STUART: Right.
PAUL: Human problems happen. Crashes happen.
STUART: Yeah.
PAUL: You’re psyche happens, and you’re amped up, and your heart rate is going faster than it normally would when you’re on the trainer.
STUART: Yeah. Yeah.
PAUL: Yeah, so there’s all these different caveats. Science will be damned from time-to-time.
STUART: Yes. Science will only get you so far.
PAUL: Yeah.
STUART: Well, that is fascinating. Are there places where people can learn about some of this stuff or maybe see, watch races that would explain some of these tactics as they happen?
PAUL: Yeah. I think if this sounds cool to you or you want to get a taste of it, there is a YouTube channel called The Global Cycling Network.
STUART: Mm-hmm.
PAUL: All great videos, how-tos, introductories. They cover races and the broadcasters are very, very friendly and very, very welcoming. That’s actually how I got into it. I just started watching those YouTube videos, then slowly met people local to me, and then started training more and more, so I think that’s a great place to start.
STUART: Cool. You heard it here. If you want to learn to play chess at 30 miles an hour while eating a sandwich, go to YouTube.
PAUL: Exactly.
STUART: Well, I think we are coming close to the end of the show here. As you know, there is – well, first I have to ask, is there anything else that you wanted to talk about or let people know about while you’re here?
PAUL: No, this has been wonderful for me. Thank you very much.
STUART: You are welcome. But before we let you go, we have one more question that we ask all of our guests. That is, do you have any advice for people listening or for us?
PAUL: I think my only bit of advice would be: Be kind. I think if everybody just thought, “Today I’m going to be kind,” the world would be a better place.
STUART: It would indeed. That does sound nice. All right, well, I think that wraps it up. Thank you, Paul, for taking the time to talk with us.
PAUL: Thanks so much, Stuart.
STUART: Thank you to everyone for listening. This has been The Cognicast.
[Music: "Thumbs Up (for Rock N' Roll)" by Kill the Noise and Feed Me]
RUSS: You have been listening to The Cognicast. The Cognicast is a production of Cognitect, Inc. Cognitect are the makers of Datomic, and we provide consulting services around it, Clojure, and a host of other technologies to businesses ranging from the smallest startups to the Fortune 50. You can find us on the Web at cognitect.com and on Twitter at @Cognitect. You can subscribe to The Cognicast, listen to past episodes, and view cover art, show notes, and episode transcripts at our home on the Web, cognitect.com/cognicast. You can contact the show by tweeting @Cognicast or by emailing us at podcast@cognitect.com.
Our guest today was Paul deGrandis, on Twitter and most of the other things at @ohpauleez. He’s so clever. Our host in his inaugural outing was Stuart Sierra who is @StuartSierra on Twitter. Stuart is also clever, even if his Twitter handle isn’t.
Episode cover art is by Michael Parenteau. Audio production is by Russ Olsen and Daemian Mack. The Cognicast is produced by Kim Foster. Our theme music is Thumbs Up (for Rock N' Roll) by Kill the Noise with Feed Me. I'm Russ Olsen. Thanks for listening.
