Archive Tags Feed

Alex Sheluchin's Blog

Thoughts, expressions, reviews and ideas

PhuzQL: A Fuzzy GraphQL Explorer with Babashka, Pathom, and FZF (PoC)

I've recently been exploring new ways to make use of Pathom's indexes. The result is a very basic proof of concept implementation of an interactive GraphQL explorer. I'm going with the working title PhuzQL. This article explains the idea and implementation components.

You can find the code in my PhuzQL repo.

PhuzQL: A GraphQL Fuzzy Finder

First, a quick demo gif... PhuzQL POC Demo

Components

Pathom3

If you haven't heard of Pathom before, I covered a brief introduction in the previous post on Pathom3 Instrumentation. In short, Pathom is a Clojure library for navigating a graph of related attributes.

The point of leverage in this PoC is Pathom's index-io index, which tells us which attributes are directly reachable from the attributes you already have. In the demo above, we can only request specific attributes about all films once we've included the :swapi.FilmsConnection/films node in our query.

Further, the entire Pathom environment is created by consuming the SWAPI GraphQL index. This is done with pathom3-graphql which uses dynamic resolvers to translate the GraphQL index to a Pathom graph that can easily be queried using EQL.

See the docs on Pathom3 GraphQL Integration for more details.

Babashka & fzf

Babashka is a scripting environment for Clojure. It has a very fast startup time which is important here to make the interactive experience smoother. PhuzQL uses it to invoke fzf, the interactive fuzzy-finder used to filter and display query results. In the PoC implementation, I'm using the fzf.clj library which makes it easier to use fzf from Clojure. The preview window invokes an external Babashka script each time each time the attribute selection changes.

Potential Improvements

This project is very much in the experimental state. Some improvements that could be made:

It has been fun piecing this together so far. I'm not entirely certain I'll be iterating on it more in the near future, but I'll take some hammock time to think about applications that could be interesting. In general, I think putting these components together shortens the feedback loop in working with GraphQL APIs. It can be used for API exploration, data analysis and IDE integration.

Published: 2025-02-21

Tagged: interactive clojure explorer graphql fzf proof-of-concept babashka api pathom

Pathom3 Instrumentation

In this article I will explain how to get performance insights into your Pathom3 resolvers by using Tufte. My aim is to show a very basic example of how it can be done, without doing a deep dive on any of the topics.

Pathom

If you are unfamiliar with Pathom, its docs define it as "a Clojure/script library to model attribute relationships". In essence, Pathom allows you to create graph of related keywords and query it using the EDN Query Language (EQL). It supports read and write operations using resolvers and mutations. The "magic" of it is that it produces an interface which abstracts away function calling by handling all the graph traversal internally when responding to EQL requests. What does that mean? A short example should suffice:

;; create a few resolvers to model related attributes
(pco/defresolver all-items
  "Takes no input and outputs `:all-items` with their `:id`."
  []
  {::pco/output [{:all-items [:id]}]}
  {:all-items
   [{:id 1}
    {:id 2}
    {:id 3}]})

(pco/defresolver fetch-v
  "Takes an `:id` and outputs its `:v`."
  [{:keys [id]}]
  (Thread/sleep 300)
  {:v (* 10 id)})

;; query the graph for some data
(p.eql/process
 (pci/register [all-items fetch-v])
 ;; ask for the `:v` attribute of `:all-items`
 [{:all-items [:v]}])
; => {:all-items [{:v 10} {:v 20} {:v 30}]}

Source: Pathom3 docs on Batch Resolvers.

As you can see, once the graph is established, you only need to tell Pathom what you want, not how to get it. As long as there is enough data to satisfy the input requirements of some initial resolver, its output can be used as input to whatever other resolver(s) need to be used in order to satisfy the entire request. Pathom will continue traversing the graph using whatever data it has at each point in order to get all the requested attributes. An elaborate chain of function calls is reduced to a single EQL expression.

While this does offer developers a great deal of power, one trade-off is that it becomes a little bit harder to understand exactly what your program is doing when you send your query to the Pathom parser. The above example creates a very simple graph without much mystery, but real applications often include a large number of resolvers, often with multiple paths for getting certain attributes.

Tufte

Tufte is useful for understanding what happens when you send a query to your Pathom parser. From the Tufte example in its repo's README, the basic usage is like this:

(tufte/profile ; Profile any `p` forms called during body execution
  {} ; Profiling options; we'll use the defaults for now
  (dotimes [_ 5]
    (tufte/p :get-x (get-x))
    (tufte/p :get-y (get-y))))

In plain English, we need to use p to wrap individual expressions and profile to wrap a set of p expressions to profile them together.

Profiling Pathom Queries

To put it together, we need to understand one last piece: Pathom Plugins. Plugins allow developers to extend Pathom's functionality by wrapping specific parts of its internal execution process with arbitrary extension code. The various places you can add wrapping are identified by keywords. In our case, we want to wrap individual resolver calls with p and the entire process (which may call many resolvers) with profile. The keywords for these extension points are:

NOTE: this article is specifically for Pathom's EQL interface.

With this knowledge, we can create some extension functions and register the plugin:

(defn tufte-resolver-wrapper
  "Wrap a Pathom3 resolver call in `tufte/p`."
  [resolver]
  (fn [env input]
    (let [resolver-name (-> (get-in env [::pcp/node ::pco/op-name])
                            (name)
                            (keyword))
          identifier (str "resolver: " resolver-name)]
      (tufte/p identifier (resolver env input)))))

(defn tufte-process-wrapper
  "Wrap a Pathom3 process in `tufte/profile`."
  [process-ast]
  (fn [env ast] (tufte/profile {} (process-ast env ast))))

(p.plugin/defplugin tufte-profile-plugin
  {::p.plugin/id `tufte-profile-plugin
   ::pcr/wrap-resolve tufte-resolver-wrapper
   ::p.eql/wrap-process-ast tufte-process-wrapper})

The last step is to include this plugin in Pathom's environment when processing a query:

;; Add handler to print results to *out*
(tufte/add-basic-println-handler! {})

(p.eql/process
;; Only the first form is new, everything else is as before.
 (-> (p.plugin/register tufte-profile-plugin)
     (pci/register [all-items fetch-v]))
 [{:all-items [:v]}])
 ; => {:all-items [{:v 10} {:v 20} {:v 30}]}

Tufte Results - no batch

If you follow along with the Batch Resolvers docs linked above, you can see how to optimize such a situation to avoid the N+1 query and the extra 600ms of processing time it causes. Let's replace the fetch-v resolver with its batch version and profile it again:

(pco/defresolver batch-fetch-v
  "Takes a _batch_ of `:id`s and outputs their `:v`."
  [items]
  {::pco/input  [:id]
   ::pco/output [:v]
   ::pco/batch? true}
  (Thread/sleep 300)
  (mapv #(hash-map :v (* 10 (:id %))) items))

(p.eql/process
  (-> (p.plugin/register tufte-profile-plugin)
      (pci/register [all-items #_fetch-v batch-fetch-v]))
  [{:all-items [:v]}])
; => {:all-items [{:v 10} {:v 20} {:v 30}]}

Tufte Results - batch

Comparing results, we can see the processing time saved by the batch version, exactly how much time was spent in each resolver and which resolvers were called. Again, this is a very simplified example. In a real-world scenario your may end up calling a large number of resolvers to produce the result, so having Tufte's stats at hand can be very useful.

Pathom Viz

As a final note, I want to point out that Pathom has its own tool for gaining such insights. It's called Pathom Viz and provides an excellent visual interface that shows everything you get from the above and more. It's a great tool and I use it often. Using Tufte as I've outlined above is an alternative lightweight approach that I've found useful.

Wrapping Up

In this article I covered a basic introduction to Pathom, its extension points and how to integrate it with Tufte in order to get performance and execution insights. Nothing groundbreaking here, but I did a quick search and didn't find any similar content, so hopefully this helps someone in the future.

You can find the complete working example code in my fnguy-examples repo.

Published: 2025-02-14

Tagged: clojure instrumentation tufte monitoring pathom performance

SCXML-Inspired State Charts In Clojure(script)

Note: This post is the second in my asked-clojure series.

Simple Statecharts Diagrams

In this article I attempt to distill some of the core concepts in the Statecharts library that has been gaining interest lately. I will do so by covering a little bit of history, explaining key ideas and illustrating them through short examples:

Getting started with the coin flip example, I ran into a few things I wasn't certain of. I wanted a minimal implementation that I could add to in order to make it a gradual learning progression. I had a coin with two states: heads and tails, and I wanted to add a flip transition between them that would randomly transition to one of the two states. My implementation seemed too complex for such a simple idea, so I turned to asking for input in a Slack thread. I received some great advice about simplifying the transition, a pragmatic naming convention for element ids and some rationale for why my original approach isn't the right way to go about it.

A final note: while the library is by Fulcrologic and includes Fulcro integration helpers, you can also use it without Fulcro. It's a fairly small library with very few dependencies. That's what I'll be doing here, and in subsequent posts, I plan to extend the examples below by adding Fulcro data management and UI rendering.

A Short History

Note: while this section does offer some valuable context about how statecharts fit within the broader scope of state machines, feel free to skip ahead to the examples if you just want to see some code.

The history of state machines dates back to the early-mid 19th century, pioneered by notables such as Claude Shannon, Edward F. Moore and John von Neumann, among others. A reasonable place for us to start is with the concept of finite-state machines (FSM), which are the basis for Statecharts.

A state machine is an abstract description of a system with a finite number of possible states, only one of which represents the system at any given time. This is the high-level definition as established through the research on FSMs.

Citizen Quartz Multi-Alarm III

In 1987, David Harel published Statecharts: A Visual Formalism for Complex Systems, a paper which extended the work on FSMs in several key areas:

Furthermore, as the title indicates, Harel presented a new method for visualizing stateful systems. The paper is a fun read, using the Citizen Quartz Multi-Alarm III watch (pictured above) as the basis for demonstrating how to use Statecharts to describe complex systems. I recommend reading it!

In 2002, a working draft was published by the World Wide Web Consortium (W3C) on a new standard called Call Control eXtensible Markup Language (CCXML). This is an XML-based language designed for call control in telephony applications. It incorporated many of the concepts from Harel's paper and combined them with XML encoding.

Later on, in 2005, the first draft of State Chart extensible Markup Language (SCXML) was published by the W3C. This is a standard inspired by both CCXML and Harel's Statecharts with the goal of providing a "generic state-machine based execution environment". Since its proposal, it has graduated to become a W3C Recommendation, indicating it has passed several rounds of review and should be considered "production ready". An interesting turn of events is that the next version of CCXML will likely be based on SCXML.

Lastly, the Statecharts library is based on SCXML structure and semantics, but deviates from the standard slightly when it comes to "executable content". In short - and covered in more detail below - Statecharts leans into making the ideas and constructs of SCXML more idiomatic in Clojure(script), while still making an effort to maintain compatibility with the standard. This approach allows Statecharts to make the most of the work that has gone into the standard and associated tooling.

Core Concepts

State

From Wikipedia:

A state is a description of the status of a system that is waiting to execute a transition.

Compound State

A state containing one or more sub-states.

Atomic State

A state without sub-states.

Transition

A change from one state to a target state. These are often the result of events, but the SCXML standard does allow eventless transitions as well.

Configuration

The set of states (including sub-states) that are currently active.

Executable Content

Hooks that allow the state machine to modify its data model and interact with external entities.

In the Statecharts library, these are represented by a function of two arguments: [env data]. You'll see these in :cond values and other places like script elements.

Examples

The examples below are available in full in my fnguy-examples repo. You will need to refer there to see all the scaffolding functions I use to run these statecharts and interact with them. That code is largely based on the examples in the Statecharts docs and is not optimized for production use.

Coin Flip

This was my starting point. I decided to start with this simple idea of creating a statechart that represents a coin flip and build up from there. The basis is a coin that can be flipped any number of times and would land on either heads or tails:

First let's define the chart:

(def coin-flips
  (let [heads :id/heads
        tails :id/tails
        pick-rand #(rand-nth [heads tails])]
    (statechart {}
      (state {:id heads}
        (transition {:event :flip
                     :target tails
                     ;; BUG! `pick-rand` will be called in each visited transition
                     :cond (fn [_ _] (= tails (pick-rand)))})
        (transition {:event :flip
                     :target heads
                     :cond (fn [_ _] (= heads (pick-rand)))}))
      (state {:id tails}
        (transition {:event :flip
                     :target heads 
                     :cond (fn [_ _] (= heads (pick-rand)))})
        (transition {:event :flip
                     :target tails
                     :cond (fn [_ _] (= tails (pick-rand)))})))))

With the chart defined, we now move on to interacting with it by first starting it and then sending a single :flip event:

(fnsc/start-new-sc! 1 ::coin-flips coin-flips)
; Event loop started
; Processing event on statechart => :com.fnguy.coin-flip/coin-flips
; target(s) => [:id/heads]
; Enter :id/heads
; after enter states:  => #{:id/heads}  ➊
; conflicting? => #{}
(fnsc/send-event! 1 :flip)
; Processing event on statechart => :com.fnguy.coin-flip/coin-flips
; external-event =>
; {:type :com.fulcrologic.statecharts/chart,  ❷
;  :name :flip,
;  :target 1,
;  :data {},
;  :com.fulcrologic.statecharts/event-name :flip}
; Running expression #function[com.fnguy.coin-flip/fn--36551/fn--36554]  ❸
; (sp/run-expression! execution-model env expr) => false
; (boolean (run-expression! env cond)) => false
; evaluating condition #function[com.fnguy.coin-flip/fn--36551/fn--36556]
; Running expression #function[com.fnguy.coin-flip/fn--36551/fn--36556]
; (sp/run-expression! execution-model env expr) => false  ❹
; (boolean (run-expression! env cond)) => false
; conflicting? => #{}
; enabled transitions => #{}
; entry set => [#{} #{} {}]
; after enter states:  => #{:id/heads}
; conflicting? => #{}  ❺

The above shows a basic REPL interaction where I start the statechart and then send it a single :flip event. After each interaction, I've also included some of the log statements that were emitted using the default logging config with a bit of manual edits to suit the blog format. A few things to notice:

I won't be repeating the entire log output for each interaction, but it's helpful to be aware of what's in there (subject to change) as it's useful for troubleshooting.

While this chart seems like it works, someone in the Slack thread helpfully pointed out a small logical bug: pick-rand can potentially be executed multiple times. If the first transition's :cond evaluates to a falsey value, the next sibling transition will call pick-rand again and might also eval to false, causing neither transition to execute. That's not what we want!

You might be thinking that moving the pick-rand evaluation somewhere to a higher scope and allowing both sibling transitions to look at that value is the fix, but that's not exactly right. Each transition element should be viewed as an independent node in the graph. Also, the chart, once initiated, is just a nested data structure. Adding in dynamic computations is possible, but there are specific ways to do that. You can't just add in a let form wherever, since it would only be evaluated once. We could technically solve this by putting the flip result data in the data model, but I wanted to keep this example minimal and adding a data model struck me as too elaborate.

The alternative implementation suggested to me is even simpler than my original:

(def simpler-coin-flip
  (statechart {}
    (state {:id :state/coin}
      (transition {:id :transition/coin->heads
                   :target :state/heads
                   :event :event/flip
                   :cond (fn [& _] (rand-nth [true false]))})
      (transition {:id :transition/coin->tails
                   :target :state/tails
                   :event :event/flip})
      (state {:id :state/heads})
      (state {:id :state/tails})))))

Improvements over the original:

The key thing to understand here is that when you have multiple transitions within a state, they are visited in document order. If a transition is enabled, subsequent siblings will not be. In other words, if :transition/coin->heads's :cond returns true, the transition moves to its :target state and :transition/coin->tails will be ignored. If, on the other hand, the predicate returned false, the next transition (to tails) would happen unconditionally. Now we have just a single evaluation of the predicate, so the bug noted above is fixed while also simplifying the code.

I've also made some slight adjustments to the code by giving namespaced :id values to all of the elements. This was suggested to me by Slack user @Michael W (see their Statecharts-based chatbot repos) and greatly helps to make the logging easier to understand. One additional logging-related improvement you could make is to convert the anonymous :cond fn to a named function using defn. You can see the difference in these two before/after log statements:

evaluating condition #function[com.fnguy.coin-flip/fn--40594]
evaluating condition #function[com.fnguy.coin-flip/random-flip]

For such a simple chart, this 👆 small optimization may not be worth it, but as complexity grows, it can be very helpful for observability.

Lastly, in my examples repo I also included another version of the coin flip statechart which remembers the total number of :heads and :tails results. It's a small change so I won't include it here. Try it yourself and refer to the repo if you need a hint.

Rock Paper Scissors

This example covers a game of rock paper scissors, which offers slightly more complicated mechanics than a coin flip does:

Here is the complete statechart:

(def rps
  (statechart {}
    (data-model {:expr {:state/player1 {:player/score 0}
                        :state/player2 {:player/score 0}}})
    (state {:id :state/game
            :initial :state/r1}
      (state {:id :state/r1}
        (transition {:id :transition/r1->r2
                     :event :event/throw
                     :target :state/r2
                     :cond not-draw?})
        (on-exit {} (update-winner-score)))
      (state {:id :state/r2}
        (transition {:id :transition/r2->tie-breaker
                     :event :event/throw
                     :target :state/tie-breaker
                     :cond ->tie-breaker?})
        (transition {:id :transition/r2->game-over
                     :event :event/throw
                     :target :state/game-over
                     :cond ->game-over?})
        (on-exit {} (update-winner-score)))
      (state {:id :state/tie-breaker}
        (transition {:id :transition/tie-breaker->game-over
                     :event :event/throw
                     :target :state/game-over
                     :cond not-draw?})
        (on-exit {} (update-winner-score)))
      (state {:id :state/game-over}
        (on-entry {}
          (log {:level :info
                :expr log-game-winner}))))))

Things to notice:

  1. there are no states to represent the players
  2. a data-model element keeps track of player scores
  3. namespaced keywords (:transition, :event, :state) for improved logging
  4. on-entry, on-exit and log elements
  5. functions are defined separately

When I started writing this chart, having player states seemed like an obvious requirement. My thinking was that each player would have at least their score and would wait for some event to play each round, producing an outcome for each :event/throw. This would require keeping two parallel compound states: rounds and players, thereby allowing rounds and players to be simultaneously active and reacting to events by transitioning their internal states or at least updating their state-local data models. When I tried this approach, I ran into problems with transition conflicts and had to change some of the transitions to :type :internal to get closer to a working model. Something felt off - the solution I was headed toward was once again becoming too complicated.

It took some time to arrive at the above definition. Of course, there are many ways one could model this simple game. I will explain the choices I made.

Data Model Implementatation

In my trials, I used Fulcro's FlatWorkingMemoryDataModel. This particular implementation "puts all data into a single scope (a map which itself can be a nested data structure)", meaning there is just one big map that keeps track of the data model of the statechart and all of its sub-states. The docs also mention an alternative implementation, the WorkingMemoryDataModel, which "scopes data to the state in which it is declared". The source code suggests to use it with caution:

WARNING: This model is not recommended for many use-cases. The contextual paths turn out to be rather difficult to reason about. The flat data model is recommended. (Source.)

I stuck with the recommended flat model that puts all the data into a single top-level map. With that decided, the first part of my reasoning for having player states - that each player would "hold" their score - was no longer valid.

Events and Transitions

The second part of my reasoning was that each player state would be responsible for determining its own throw outcome. Intuitively, this made sense to me (see state). A player would be modeled as a system waiting to execute a transition: from an idle state where their hand is not in play, to a thrown state resulting in one of the three RPS hand gestures. The round winner would then be determined, player scores incremented accordingly and the game would transition to the next round.

The trouble here is that using parallel states brings on additional complexity regarding transition conflicts and the value of an idle player state transitioning to a thrown state and then back again is not clear.

My solution resolved on getting rid of the players state and the parallelism with the rounds state such a design would necessitate. Instead, the :round/winner is decided as an integral part of the throwing event (:event/throw):

(defn random-throw [] (rand-nth [:rock :paper :scissors]))

(defn throw->winner []
  (let [p1-throw (random-throw)
        p2-throw (random-throw)
        winner (pick-winner p1-throw p2-throw)]
    winner))

(comment
  (fnsc/send-event! ::rps :event/throw
                    ;; Send as part of the event's data
                    {:round/winner (throw->winner)}))

Now let's see how this applies to the initial game state, :state/r1:

(defn not-draw? [_env data] (not (data->round-draw? data)))

(defn update-winner-score []
  (script-fn [_env data]
    (let [winner-id (data->round-winner data)
          new-score (data->new-winner-score data)]
      [(ops/assign [winner-id :player/score] new-score)])))

;; One part of the chart, not a top-level definition
(state {:id :state/r1}
  (transition {:id :transition/r1->r2
               :event :event/throw
               :target :state/r2
               :cond not-draw?})
  (on-exit {} (update-winner-score)))

This highlights some important design decisions when creating statecharts:

As mentioned, of course there are other ways a game of rock paper scissors could be designed as a statechart. The above explains the process I went through and the lessons learned along the way. Design is rarely an aspect with clear answers, just a set of trade-offs that need to be considered to arrive at a good fit for the problem at hand.

Wrapping Up

This article covered statecharts history as well as a few of the core concepts, demonstrated through simple examples. I must stress that the concepts covered are by no means comprehensive. I excluded important aspects of the SCXML standard, such as history, invocations, event queue processing and many others. The Statecharts library has direct analogs for many of these, and likewise has a few of its own concerns that I've not written about here such as different data model implementations, invocation processors, testing and Fulcro integration. My objective here was to explain raw statecharts in the simplest form rather than provide a comprehensive explanation of the all the facets. Refer to the source material to learn more.

The examples above cover states, transitions, guards, events and how to leverage a statechart's data model. Everything was done strictly in the REPL with plain statecharts with no additional required dependencies.

I initially planned to cover a third example by modeling a game of poker as a statechart. This post is already quite lengthy, so I decided to put the poker example off to a future article in which I'll focus more on the Fulcro integration helpers to build a simple webapp to serve as the UI for the game's statechart.

Links

Published: 2025-01-18

Tagged: cljc clojure scxml statecharts fulcro harel state-machines asked-clojure

Archive