20
Creating a neo4j cluster with Docker
It has been a while since I posted anything not related to Oracle Databases, so I thought was a good idea to change the "trend" a bit.
If you follow Tech news, you may have heard recent news about Neo4j recent developments
I didn't know anything about it, so I decided to give it a try.
If you just want to try it out, you have the option to create a Free Aura account and try it directly in their cloud
I really recommend to check the Documentation and some of the presentation available to see what a Graph Database is really about.
For this post, I will focus in more the Administration point of view so I thought it would be more interesting for me to deploy small cluster locally so I can see more about the actual system operation mode.
Since you can use their Docker official images directly, there is very littel friction if all you want is to have a quick deployment to test it out in your local system.
I don't have much knowledge about "resources requirements" but I was just able to create a small Casual Cluster with 3 Core nodes and 1 Read-only Replica in my laptop without any noticeable overhead.
This is the docker-compose file I'm using for this example where I just added local Volumes for each container to their default example as well as set hostnames for each container.
The reason behind that is to make easier to check some of the information generated for each container
16:21:22 |=| penguin in ~/neo4j ○
→ cat docker-compose.yml
version: '3.8'
x-shared:
&common
NEO4J_AUTH: neo4j/Welcome1
NEO4J_ACCEPT_LICENSE_AGREEMENT: "yes"
NEO4J_causal__clustering_initial__discovery__members: core1:5000,core2:5000,core3:5000
NEO4J_dbms_memory_pagecache_size: "100M"
NEO4J_dbms_memory_heap_initial__size: "100M"
x-shared-core:
&common-core
<<: *common
NEO4J_dbms_mode: CORE
NEO4J_causal__clustering_minimum__core__cluster__size__at__formation: 3
networks:
lan:
services:
core1:
image: neo4j:4.3-enterprise
hostname: core1
networks:
- lan
ports:
- "7474:7474"
- "7687:7687"
environment:
<<: *common-core
NEO4J_causal__clustering_discovery__advertised__address: core1:5000
NEO4J_causal__clustering_transaction__advertised__address: core1:6000
NEO4J_causal__clustering_raft__advertised__address: core1:7000
volumes:
- /home/solifugo/neo4j/core1/data:/data
- /home/solifugo/neo4j/core1/logs:/logs
- /home/solifugo/neo4j/core1/import:/var/lib/neo4j/import
- /home/solifugo/neo4j/core1/plugins:/plugins
core2:
image: neo4j:4.3-enterprise
hostname: core2
networks:
- lan
ports:
- "7475:7474"
- "7688:7687"
environment:
<<: *common-core
NEO4J_causal__clustering_discovery__advertised__address: core2:5000
NEO4J_causal__clustering_transaction__advertised__address: core2:6000
NEO4J_causal__clustering_raft__advertised__address: core2:7000
volumes:
- /home/solifugo/neo4j/core2/data:/data
- /home/solifugo/neo4j/core2/logs:/logs
- /home/solifugo/neo4j/core2/import:/var/lib/neo4j/import
- /home/solifugo/neo4j/core2/plugins:/plugins
core3:
image: neo4j:4.3-enterprise
hostname: core3
networks:
- lan
ports:
- "7476:7474"
- "7689:7687"
environment:
<<: *common-core
NEO4J_causal__clustering_discovery__advertised__address: core3:5000
NEO4J_causal__clustering_transaction__advertised__address: core3:6000
NEO4J_causal__clustering_raft__advertised__address: core3:7000
volumes:
- /home/solifugo/neo4j/core3/data:/data
- /home/solifugo/neo4j/core3/logs:/logs
- /home/solifugo/neo4j/core3/import:/var/lib/neo4j/import
- /home/solifugo/neo4j/core3/plugins:/plugins
readreplica1:
image: neo4j:4.3-enterprise
hostname: readreplica1
networks:
- lan
ports:
- "7477:7474"
- "7690:7687"
environment:
<<: *common
NEO4J_dbms_mode: READ_REPLICA
NEO4J_causal__clustering_discovery__advertised__address: readreplica1:5000
NEO4J_causal__clustering_transaction__advertised__address: readreplica1:6000
NEO4J_causal__clustering_raft__advertised__address: readreplica1:7000
volumes:
- /home/solifugo/neo4j/replica1/data:/data
- /home/solifugo/neo4j/replica1/logs:/logs
- /home/solifugo/neo4j/replica1/import:/var/lib/neo4j/import
- /home/solifugo/neo4j/replica1/plugins:/plugins
16:21:26 |=| penguin in ~/neo4j ○Once you we have the Docker Compose file, we just need to start it up (remember to use the Detach option)
16:24:47 |=| penguin in ~/neo4j ○
→ sudo docker-compose up -d
Starting neo4j_core1_1 ... done
Starting neo4j_readreplica1_1 ... done
Starting neo4j_core2_1 ... done
Starting neo4j_core3_1 ... done
16:25:00 |=| penguin in ~/neo4j ○
→ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
41b29cd57174 neo4j:4.3-enterprise "/sbin/tini -g -- /d…" 22 minutes ago Up 25 seconds 7473/tcp, 0.0.0.0:7475->7474/tcp, :::7475->7474/tcp, 0.0.0.0:7688->7687/tcp, :::7688->7687/tcp neo4j_core2_1
b861f241ffe7 neo4j:4.3-enterprise "/sbin/tini -g -- /d…" 3 hours ago Up 25 seconds 7473/tcp, 0.0.0.0:7477->7474/tcp, :::7477->7474/tcp, 0.0.0.0:7690->7687/tcp, :::7690->7687/tcp neo4j_readreplica1_1
e2f39748efad neo4j:4.3-enterprise "/sbin/tini -g -- /d…" 3 hours ago Up 25 seconds 0.0.0.0:7474->7474/tcp, :::7474->7474/tcp, 7473/tcp, 0.0.0.0:7687->7687/tcp, :::7687->7687/tcp neo4j_core1_1
961f5ed4ab31 neo4j:4.3-enterprise "/sbin/tini -g -- /d…" 3 hours ago Up 25 seconds 7473/tcp, 0.0.0.0:7476->7474/tcp, :::7476->7474/tcp, 0.0.0.0:7689->7687/tcp, :::7689->7687/tcp neo4j_core3_1
16:25:25 |=| penguin in ~/neo4j ○
→Having the system up, you can just access to the different Databases using Neo4j using your browser
You can check some of the system information directly in the Neo4j Browser
I really recommend to explore the included Example guides to see how the system works and what make this such a different Database
We already saw Neo4j Browser which is the Developer focused tool to allow you to visualize the query results.
But you also have the command line tool called Cypher Shell which allow more automatization/scripting for Administrators.
Since we already have it on each of the deployed container, we can just access to any of them (be aware of the different roles of each node) to do use the cypher-shell
12:39:24 |=| penguin in ~/neo4j ○
→ sudo docker exec -it neo4j_core1_1 bash
root@core1:/var/lib/neo4j# cypher-shell -u neo4j -p Welcome1
Failed to connect to neo4j://localhost:7687, fallback to bolt://localhost:7687
Connected to Neo4j using Bolt protocol version 4.2 at bolt://localhost:7687 as user neo4j.
Type :help for a list of available commands or :exit to exit the shell.
Note that Cypher queries must end with a semicolon.
neo4j@neo4j>You can do queries inside the different databases and see the Text results
neo4j@system> :use neo4j
neo4j@neo4j> MATCH (Movie) WHERE Movie.title IS NOT NULL RETURN Movie.title LIMIT 5;
+--------------------------+
| Movie.title |
+--------------------------+
| "The Matrix" |
| "The Matrix Reloaded" |
| "The Matrix Revolutions" |
| "The Devil's Advocate" |
| "A Few Good Men" |
+--------------------------+
5 rows
ready to start consuming query after 97 ms, results consumed after another 2 ms
neo4j@neo4j>To check if there are some sessions
neo4j@neo4j> CALL dbms.listConnections() YIELD connectionId, connectTime, connector, username, userAgent, clientAddress;
+------------------------------------------------------------------------------------------------------------------------+
| connectionId | connectTime | connector | username | userAgent | clientAddress |
+------------------------------------------------------------------------------------------------------------------------+
| "bolt-2" | "2021-06-25T17:28:01.594Z" | "bolt" | "neo4j" | "neo4j-browser/v4.3.0" | "100.115.92.25:42372" |
| "bolt-210" | "2021-06-25T18:04:33.586Z" | "bolt" | "neo4j" | "neo4j-cypher-shell/v4.3.1" | "127.0.0.1:60896" |
| "bolt-10" | "2021-06-25T17:28:07.431Z" | "bolt" | "neo4j" | "neo4j-browser/v4.3.0" | "100.115.92.25:42390" |
| "bolt-14" | "2021-06-25T17:28:07.504Z" | "bolt" | "neo4j" | "neo4j-browser/v4.3.0" | "100.115.92.25:42398" |
| "bolt-208" | "2021-06-25T18:04:33.405Z" | "bolt" | "neo4j" | "neo4j-cypher-shell/v4.3.1" | "127.0.0.1:60892" |
| "bolt-13" | "2021-06-25T17:28:07.489Z" | "bolt" | "neo4j" | "neo4j-browser/v4.3.0" | "100.115.92.25:42396" |
| "bolt-209" | "2021-06-25T18:04:33.548Z" | "bolt" | "neo4j" | "neo4j-cypher-shell/v4.3.1" | "127.0.0.1:60894" |
| "bolt-12" | "2021-06-25T17:28:07.472Z" | "bolt" | "neo4j" | "neo4j-browser/v4.3.0" | "100.115.92.25:42394" |
| "bolt-15" | "2021-06-25T17:28:07.665Z" | "bolt" | "neo4j" | "neo4j-browser/v4.3.0" | "100.115.92.25:42400" |
+------------------------------------------------------------------------------------------------------------------------+
9 rows
ready to start consuming query after 69 ms, results consumed after another 7 ms
neo4j@neo4j>You can also see things like Databases available:
neo4j@neo4j> show databases;
+----------------------------------------------------------------------------------------------------------+
| name | address | role | requestedStatus | currentStatus | error | default | home |
+----------------------------------------------------------------------------------------------------------+
| "neo4j" | "localhost:7687" | "follower" | "online" | "online" | "" | TRUE | TRUE |
| "neo4j" | "localhost:7687" | "leader" | "online" | "online" | "" | TRUE | TRUE |
| "neo4j" | "localhost:7687" | "follower" | "online" | "online" | "" | TRUE | TRUE |
| "neo4j" | "localhost:7687" | "read_replica" | "online" | "online" | "" | TRUE | TRUE |
| "system" | "localhost:7687" | "leader" | "online" | "online" | "" | FALSE | FALSE |
| "system" | "localhost:7687" | "follower" | "online" | "online" | "" | FALSE | FALSE |
| "system" | "localhost:7687" | "follower" | "online" | "online" | "" | FALSE | FALSE |
| "system" | "localhost:7687" | "read_replica" | "online" | "online" | "" | FALSE | FALSE |
| "test" | "localhost:7687" | "follower" | "online" | "online" | "" | FALSE | FALSE |
| "test" | "localhost:7687" | "follower" | "online" | "online" | "" | FALSE | FALSE |
| "test" | "localhost:7687" | "leader" | "online" | "online" | "" | FALSE | FALSE |
| "test" | "localhost:7687" | "read_replica" | "online" | "online" | "" | FALSE | FALSE |
+----------------------------------------------------------------------------------------------------------+
12 rows
ready to start consuming query after 71 ms, results consumed after another 4 ms
neo4j@neo4j>Nodes and their role in the cluster:
neo4j@neo4j> CALL dbms.cluster.overview();
+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| id | addresses | databases | groups |
+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| "85256f9d-8dd9-439a-b26a-e0e082d56c96" | ["bolt://localhost:7687", "http://localhost:7474"] | {neo4j: "FOLLOWER", system: "FOLLOWER"} | [] |
| "d351aabc-4af2-4c8f-9861-9d94ec9c96f9" | ["bolt://localhost:7687", "http://localhost:7474"] | {neo4j: "READ_REPLICA", system: "READ_REPLICA"} | [] |
| "d4907a20-d8be-4b73-bae2-eebe49c8de34" | ["bolt://localhost:7687", "http://localhost:7474"] | {neo4j: "LEADER", system: "FOLLOWER"} | [] |
| "e76056db-e196-491f-8f6e-a5a60a0ee2a8" | ["bolt://localhost:7687", "http://localhost:7474"] | {neo4j: "FOLLOWER", system: "LEADER"} | [] |
+--------------------------------------------------------------------------------------------------------------------------------------------------------+
4 rows
ready to start consuming query after 16 ms, results consumed after another 10 ms
neo4j@neo4j>Here you can also see how you can build commands and make the call directly from the host instead of connect to each node to check the current role of each one for a database
18:46:46 |=| penguin in ~/neo4j/core1/logs ○
→ sudo docker exec neo4j_core1_1 bash -c "echo 'CALL dbms.cluster.role(\"neo4j\")' | cypher-shell -u neo4j -p Welcome1"
Failed to connect to neo4j://localhost:7687, fallback to bolt://localhost:7687
role
"FOLLOWER"
18:47:13 |=| penguin in ~/neo4j/core1/logs ○
→ sudo docker exec neo4j_core3_1 bash -c "echo 'CALL dbms.cluster.role(\"neo4j\")' | cypher-shell -u neo4j -p Welcome1"
role
"LEADER"
18:48:51 |=| penguin in ~/neo4j/core1/logs ○
→ sudo docker exec neo4j_readreplica1_1 bash -c "echo 'CALL dbms.cluster.role(\"neo4j\")' | cypher-shell -u neo4j -p Welcome1"
Failed to connect to neo4j://localhost:7687, fallback to bolt://localhost:7687
role
"READ_REPLICA"Or check the sessions running on each cluster node using a simple bash for loop
19:11:49 |=| penguin in ~/neo4j/core1 ○
→ for i in `sudo docker ps --format "{{.Names}}"` ; do echo "## Node $i ##" ; sudo docker exec $i bash -c "echo 'CALL dbms.listConnections() YIELD connectionId, connectTime, connector, username, userAgent, clientAddress' | cypher-shell -u neo4j -p Welcome1" |grep -v Failed ; echo "" ; done
## Node neo4j_readreplica1_1 ##
connectionId, connectTime, connector, username, userAgent, clientAddress
"bolt-10", "2021-06-25T18:11:55.725Z", "bolt", "neo4j", "neo4j-cypher-shell/v4.3.1", "127.0.0.1:60928"
"bolt-9", "2021-06-25T18:11:55.572Z", "bolt", "neo4j", "neo4j-cypher-shell/v4.3.1", "127.0.0.1:60926"
## Node neo4j_core1_1 ##
connectionId, connectTime, connector, username, userAgent, clientAddress
"bolt-10", "2021-06-25T17:28:07.431Z", "bolt", "neo4j", "neo4j-browser/v4.3.0", "100.115.92.25:42390"
"bolt-14", "2021-06-25T17:28:07.504Z", "bolt", "neo4j", "neo4j-browser/v4.3.0", "100.115.92.25:42398"
"bolt-208", "2021-06-25T18:04:33.405Z", "bolt", "neo4j", "neo4j-cypher-shell/v4.3.1", "127.0.0.1:60892"
"bolt-13", "2021-06-25T17:28:07.489Z", "bolt", "neo4j", "neo4j-browser/v4.3.0", "100.115.92.25:42396"
"bolt-209", "2021-06-25T18:04:33.548Z", "bolt", "neo4j", "neo4j-cypher-shell/v4.3.1", "127.0.0.1:60894"
"bolt-12", "2021-06-25T17:28:07.472Z", "bolt", "neo4j", "neo4j-browser/v4.3.0", "100.115.92.25:42394"
"bolt-15", "2021-06-25T17:28:07.665Z", "bolt", "neo4j", "neo4j-browser/v4.3.0", "100.115.92.25:42400"
"bolt-2", "2021-06-25T17:28:01.594Z", "bolt", "neo4j", "neo4j-browser/v4.3.0", "100.115.92.25:42372"
"bolt-210", "2021-06-25T18:04:33.586Z", "bolt", "neo4j", "neo4j-cypher-shell/v4.3.1", "127.0.0.1:60896"
"bolt-237", "2021-06-25T18:11:57.219Z", "bolt", "neo4j", "neo4j-cypher-shell/v4.3.1", "127.0.0.1:60934"
"bolt-235", "2021-06-25T18:11:57.064Z", "bolt", "neo4j", "neo4j-cypher-shell/v4.3.1", "127.0.0.1:60930"
"bolt-236", "2021-06-25T18:11:57.191Z", "bolt", "neo4j", "neo4j-cypher-shell/v4.3.1", "127.0.0.1:60932"
## Node neo4j_core3_1 ##
connectionId, connectTime, connector, username, userAgent, clientAddress
"bolt-7", "2021-06-25T18:11:58.602Z", "bolt", "neo4j", "neo4j-cypher-shell/v4.3.1", "127.0.0.1:60938"
"bolt-6", "2021-06-25T18:11:58.452Z", "bolt", "neo4j", "neo4j-cypher-shell/v4.3.1", "127.0.0.1:60936"
## Node neo4j_core2_1 ##
connectionId, connectTime, connector, username, userAgent, clientAddress
"bolt-10", "2021-06-25T18:11:59.842Z", "bolt", "neo4j", "neo4j-cypher-shell/v4.3.1", "127.0.0.1:60940"
"bolt-12", "2021-06-25T18:12:00.041Z", "bolt", "neo4j", "neo4j-cypher-shell/v4.3.1", "127.0.0.1:60944"
"bolt-11", "2021-06-25T18:11:59.995Z", "bolt", "neo4j", "neo4j-cypher-shell/v4.3.1", "127.0.0.1:60942"Once of the advantages of adding independent local Volumes to each container, is to be able to check directly some of the logs and files generated by each core directly form the host.
For example, we can see the debug.log file to see the core start process and the Instance ID of each node:
18:26:58 |=| penguin in ~/neo4j/core1/logs ○
→ tail -f debug.log
2021-06-25 17:26:46.972+0000 INFO [c.n.s.e.EnterpriseNeoWebServer] ======== Neo4j 4.3.1 ========
2021-06-25 17:26:47.552+0000 INFO [c.n.c.c.s.m.ClusterStateMigrator] Cluster state version found on disk is: ClusterStateVersion{major=1, minor=2}
2021-06-25 17:26:47.569+0000 INFO [c.n.c.c.s.m.ClusterStateMigrator] Cluster state version expected by this Neo4j process is: ClusterStateVersion{major=1, minor=2}
2021-06-25 17:26:47.582+0000 INFO [c.n.c.i.CoreIdentityModule] Found ServerId on disk: ServerId{79dee5e4} (79dee5e4-d3c2-46ae-af93-36d4780189b1)
2021-06-25 17:26:47.588+0000 INFO [c.n.c.i.CoreIdentityModule] This instance is ServerId{79dee5e4} (79dee5e4-d3c2-46ae-af93-36d4780189b1)
2021-06-25 17:26:47.925+0000 INFO [o.n.b.BoltServer] Bolt server loaded
2021-06-25 17:26:53.879+0000 WARN [a.e.DummyClassForStringSources] Using serializer [com.neo4j.causalclustering.discovery.akka.marshal.UniqueAddressSerializer] for message [akka.cluster.UniqueAddress]. Note that this serializer is not implemented by Akka. It's not recommended to replace serializers for messages provided by Akka.
2021-06-25 17:26:55.478+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=187, gcTime=112, gcCount=1}
2021-06-25 17:26:56.404+0000 INFO [c.n.c.n.Server] [raft-server] bound to '0.0.0.0:7000' with transport 'EpollServerSocketChannel'
2021-06-25 17:26:56.422+0000 INFO [c.n.c.n.Server] [catchup-server] bound to '0.0.0.0:6000' with transport 'EpollServerSocketChannel'
2021-06-25 17:26:56.429+0000 INFO [c.n.c.n.Server] [backup-server] bound to '127.0.0.1:6362' with transport 'EpollServerSocketChannel'
2021-06-25 17:26:56.434+0000 INFO [c.n.d.StartupSystemGraphDbmsOperator] Starting up 'system' database
2021-06-25 17:26:56.583+0000 INFO [c.n.d.ClusteredDbmsReconciler] Database 'system' is requested to transition from INITIAL{db=system/00000000} to STARTED{db=system/00000000}
2021-06-25 17:26:56.625+0000 INFO [c.n.c.c.CoreDatabaseManager] Creating 'DatabaseId{00000000[system]}'.
2021-06-25 17:26:57.523+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=332, gcTime=390, gcCount=1}
2021-06-25 17:26:58.386+0000 INFO [o.n.g.f.EditionLocksFactories] [system/00000000] Locking implementation 'forseti' selected.Another interesting log is the query.log, where you can see the different queries done on each cluster member
18:34:02 |=| penguin in ~/neo4j/core1/logs ○
→ tail -f query.log
2021-06-25 17:32:58.647+0000 INFO Query started: id:85 - 0 ms: 0 B - bolt-session bolt neo4j-browser/v4.3.0 client/100.115.92.25:42390 server/172.18.0.4:7687> system - neo4j - SHOW DATABASES - {} - runtime=null - {type: 'system', app: 'neo4j-browser_v4.3.0'}
2021-06-25 17:32:58.654+0000 INFO id:85 - 7 ms: -1 B - bolt-session bolt neo4j-browser/v4.3.0 client/100.115.92.25:42390 server/172.18.0.4:7687> system - neo4j - SHOW DATABASES - {} - runtime=system - {type: 'system', app: 'neo4j-browser_v4.3.0'}
2021-06-25 17:32:58.743+0000 INFO Query started: id:86 - 1 ms: 0 B - bolt-session bolt neo4j-browser/v4.3.0 client/100.115.92.25:42390 server/172.18.0.4:7687> system - neo4j - CALL dbms.clientConfig() - {} - runtime=null - {type: 'system', app: 'neo4j-browser_v4.3.0'}
2021-06-25 17:32:58.748+0000 INFO id:86 - 6 ms: -1 B - bolt-session bolt neo4j-browser/v4.3.0 client/100.115.92.25:42390 server/172.18.0.4:7687> system - neo4j - CALL dbms.clientConfig() - {} - runtime=system - {type: 'system', app: 'neo4j-browser_v4.3.0'}
2021-06-25 17:32:58.755+0000 INFO Query started: id:87 - 1 ms: 0 B - bolt-session bolt neo4j-browser/v4.3.0 client/100.115.92.25:42400 server/172.18.0.4:7687> system - neo4j - CALL dbms.showCurrentUser() - {} - runtime=null - {type: 'system', app: 'neo4j-browser_v4.3.0'}
2021-06-25 17:32:58.762+0000 INFO id:87 - 7 ms: -1 B - bolt-session bolt neo4j-browser/v4.3.0 client/100.115.92.25:42400 server/172.18.0.4:7687> system - neo4j - CALL dbms.showCurrentUser() - {} - runtime=system - {type: 'system', app: 'neo4j-browser_v4.3.0'}
2021-06-25 17:33:26.888+0000 INFO Query started: id:88 - 38 ms: 0 B - bolt-session bolt neo4j-cypher-shell/v4.3.1 client/127.0.0.1:60778server/127.0.0.1:7687> neo4j - neo4j - MATCH (Movie) WHERE Movie.title IS NOT NULL RETURN Movie.title LIMIT 5; - {} - runtime=null - {}
2021-06-25 17:33:26.997+0000 INFO id:88 - 147 ms: 152 B - bolt-session bolt neo4j-cypher-shell/v4.3.1 client/127.0.0.1:60778 server/127.0.0.1:7687> neo4j - neo4j - MATCH (Movie) WHERE Movie.title IS NOT NULL RETURN Movie.title LIMIT 5; - {} - runtime=pipelined - {}
2021-06-25 17:33:41.038+0000 INFO Query started: id:89 - 26 ms: 0 B - bolt-session bolt neo4j-cypher-shell/v4.3.1 client/127.0.0.1:60778server/127.0.0.1:7687> system - neo4j -
show databases; - {} - runtime=null - {}
2021-06-25 17:33:41.053+0000 INFO id:89 - 43 ms: -1 B - bolt-session bolt neo4j-cypher-shell/v4.3.1 client/127.0.0.1:60778 server/127.0.0.1:7687> system - neo4j -
show databases; - {} - runtime=system - {}In conclusion, really interesting database specially for people like me so used to a completely different beast like it is Oracle Databases.
At administration level, you will always find similar solution specially if you are running a cluster with different instances so you will need to have similar processes and structures to make sure everything is on sync all the time.
I'm actually thinking about doing some tests on exactly that and see how the system keeps everything on sync while losing a node or how the system handle "rollbacks". Let's see if I can get some time to work on that soon :)
20