netmq - ZeroMq Sockets Design -

netmq - ZeroMq Sockets Design -

i need send large volumes of short messages single process different agents.

approximately 10-15 2-3 kb messages/sec 10 agents.

some of agents run on same machine, while others on separate machines in same vlan.

the requirement minimum latency between generation of message , start of processing within main service. maximum of 50ms @ peaks acceptable ( though not desired ).

i planning utilize netmq transport.

q1: best configuration of socket types , protocols?

q2: better utilize dealer/router, push/pull or maybe push/router setup?

q3: should use ipc sockets agents on same machine or tcp fast plenty both types of agents?

q4: need create tcp , ipc sockets within same context or create separate context?

on subject: messaging, transport, latency

zeromq scaleable formal communication patterns framework. gives set of smart objects , hides lot of internal gritty-nitties designers ( beneficial ).

zeromq designed messaging

transport-agnostic, means, bulk of use-cases not take care, on particular transport-class ( { inproc: | ipc: | tcp: | pgm: | epgm: } ) delivery takes place

low latency optimised ( "batteries included" principle )

the key thing, seem have missed is, main difference comes fact, application shall build on zeromq-archetypes' behaviour, presented using use-case metaphoric names ( push/pull ), understanding of of import design.

latency sky-rocket, if poor design decision makes software utilize "wrong" ( not selected ) zeromq-archetype, behavioural-model in turn introduce unevitable need handle / wait / block work-flow model, due mis-understood internal behaviour of req/rep illustration , alike.

this way, lowest achievable latency results smart design, not (*)mq-library.

having craftmanship in near real-time scheme design, zeromq fit it's best of genuine services in code, not expect poor scheme architecture magic speeds , latencies close none linking in few of zeromq-socket methods.

on .context( niothreads )

a practice isolate design features, able validate respective impact on final criteria. should minimum overall end-to-end latency case, there may step increment number of io-threads, instantiated , used localhost application via .context() method.

10 x 15 x 3kb per sec not big issue raspberry pi processor.

the more of import is, whether sole step bring important improvement on other design-compromises and/or other io-workload related activities.

test it.

however test "in-vivo", not "in-vitro" under set of artificially academic circumstances. gain without losing, "surrounding" architecture implemented , measure in context of designed modus operandi.

where go next?

good point start spend week of hands-on testing pieter hintjen's book "code connected, volume 1" (aspdf). there have discussed both fabulous indepth knowledge-pool , code-examples, help move forward.

after moderator's kind intervention delete unfair insultations, allow merit-based facts focus:

a4: no, book ephasises, possible, zmq.context() singleton , in cases, architecture/design/tests prove beneficial, zmq.context() instance may have more 1 i/o-thread handle specific traffic-patterns on low-level ( without other assistance / command user-code side ). in sense, makes no sense until implemented , pre-tested solution went live, speculate on such potential benefit. 1 time solution operational, possible increment parameter , experimentally measure it's possible influence on overall processing performance.

a to-always-remind practice take indeed due care enforcing gracefull terminations of socket(s) & context life-cycles, avoid dirty leaks. such practice more recommended.

a3: yes, may utilize tcp-transport-class 'em too. on non-windows localhosts ipc-option becomes feasible. speed questions shall rather measured, not opinion-based. if code struggle gets shaving downwards additional [us]-s , [ns]-s ( zmq.stopwatch goes downwards 3 tens of nanoseconds in it's resolution ), may, based on code-design, expect slight benefit avoiding tcp-protocol wire-line framing assembly/re-assebly/decode. mutual question @ additional costs may become available measurable / observable speed-up.

a2: broad answer. without single word on processing architecture / taskunit's workflow, there no serious way reply q2.

a1: missing criteria optimisation goal "the best". having no criteria / metrics, generic rule valid -- best 1 such one, meets spec & works fine , stable within given time & budget.

zeromq netmq