Clojure Cassandra Driver Performance -

i'm testing performance of alia in clojure on 6 node cassandra cluster. when multi-threading can 400 writes/sec. firebrand cassandra driver , manually handling threading in java able 5000 writes/sec 96 threads.

am doing wrong in utilization of agents here? cpu usage ~25% on machine running on seems low.

update: @ author of alia's suggestion, utilizing prepared statements instead of raw statements realized gains of 2500/sec in synchronous, single-threaded fashion. still need test multi-threading clojure , separately utilizing async function built alia/the underlying java driver see faster.

update 2: seeing similar results mpenet below additionally utilizing async functionality built driver.

(ns alia-perf-test.core   (:gen-class)   (:require [qbits.alia :as alia]             [qbits.hayt :as hayt]))  (defn exec-query [session query]   (alia/execute session (hayt/->raw query)))  (defmacro time-query   [expr]   `(let [start# (. system (nanotime))          ret# ~expr]      (/ (double (- (. system (nanotime)) start#)) 1000000.0)))  (defn write-entity   [total-time session entity]   (let [query (hayt/->raw (hayt/insert :entities (hayt/values entity) (hayt/using :timestamp 1234)))         query-time (time-query (alia/execute session query))]       (+ total-time query-time)))  (defn generate-entity []   {:id (str (java.util.uuid/randomuuid)) :num 0})  (defn write-something   [write-agent session]   (send-off write-agent write-entity         session          (generate-entity)))  (defn -main [& args]   (let [cluster (alia/cluster ["server1"                                "server2"                                 "server3"                                 "server4"                                 "server5"                                 "server6"]                               :pooling-options {:core-connections-per-host [:local 16 :remote 16]                                                 :max-connections-per-host  [:local 1000 :remote 1000]                                                 :max-simultaneous-requests-per-connection [:local 32 :remote 32]                                                 :min-simultaneous-requests-per-connection [:local 16 :remote 16]})         session (alia/connect cluster)]     (alia/set-consistency! :any)     (exec-query session (hayt/create-keyspace :aliaperftest                                               (hayt/with {:replication                                                           {:class "networktopologystrategy"                                                            :dc1 3 :dc2 3}})))     (exec-query session (hayt/use-keyspace :aliaperftest))     (exec-query session (hayt/create-table :entities                                            (hayt/column-definitions {:id :varchar                                                                      :num :int                                                                      :primary-key [:id]})))     (let [num-entities 10000           write-agent (agent 0)]       (dotimes [n num-entities]         (write-something write-agent session))       (await write-agent)       (println "wrote" num-entities "entities in" @write-agent "ms -"            (* (/ num-entities @write-agent) 1000.0) "ops/sec"))      (exec-query session (hayt/drop-table :entities))     (exec-query session (hayt/drop-keyspace :aliaperftest))     (alia/shutdown session)     (alia/shutdown cluster)     (shutdown-agents))) 

update: have got 100k requests in ~8 seconds in asynchronous mode (using alia's success handler, single node), using atom receive results , waiting response arrive.

it might possible squeeze more performance out of using custom executor depending on setup , batching, didn't go far. see: https://github.com/mpenet/alia/blob/master/docs/guide.md#executors