Tag: network management

Proof of concept for network monitoring with graph database

~ dnzydn ~ Leave a comment

Proof of concept for network monitoring with graph database

This project is about to understand modelling network as we draw it and how to virtualise. For the beginning our core network, which is based on IGP/MPLS, was the cover. Later on we tried to see how we can virtualise the network and can we use the data to monitor the network like changing node colours or status as they go up and down.

All the logs or alarms will be mirrored to this platform and will set the node’s status which will give log reduction for the operator in the first place. After understanding the problem operator can dig into logs for further analysis.

I decided to use BGP-LS for monitoring IGP network which will immediately reflect the up/down igp node and prefixes. An open source java based github project telefonicaid/netphony-network-protocols is used with adding support for graph database connection, processing BGP-LS UNREACH NLRI. Its modified to add new node and alter node status.

https://github.com/denizaydin/netphony-network-protocols

Modelling the Network

We used a simple approach for starting. One layer show current IGP network, logical network. The other shows how they connect each other interface by interface, the physical network.

For IGP layer every igp node connects via each other by means of links. Multiple connections between nodes are represented as a single link. There BGP-LS is best suits for this. As a result if two node have a link between them this means that they have a connection. One of the problem is that if node has no link to another node, reboot or etc you can not get node nlri unreach message. So while implementing BGP-LS speaker if we get unreach prefix nlri, we run a database query for searching the node has another active link or not. If not this means that the node is down. You may model you network also by using directions of links representing downstream or upstream which can be used for additional purposes.

For the physical network each connection between nodes are represented as they we draw it.

Routers contains interfaces which may contain logical interfaces (subint or vlan) and then which are connected via circuits to each others.

Topology Layers
Topology Layer for Network Monitoring

 

Basically logical layer show the IGP/MPLS/IP backbone, the other shows the physical network. More layers can be added. For example another layer below the physical layer may show the fibers etc that the circuits or the connection between interface, or another layer above the logical layer that shows the services build on the routers. Also YANG model can be used.

More information about the model :

Simple Graph Database Model for Network Management

 

Initial, Proof of Concept Model
Initial, Proof of Concept Model

The database

Graph database best suits for this project as it stores the data as you draw:). We used neo4j.

Also we use directed graph database as in some layers direction of the edges are  imported. Also we labeled edges.

For pysical layer;

Routers (vertex) CONTAINS (edge) interfaces (vertex) which may CONTAINS (edge) intefaces (vertex) are CONNECTED (edge) to the circuits (vertex). Direction of the edge’s shows the relation between node/vertex.

For logical layer;

Routers (vertex) LINK (edge) Routers (vertex) Or

devices (vertex) UPSTREAM (edge) Routers (vertex)

For the logical network direction of the edge is shows the downstream or upstream relation. By this way you can easily query a nodes upstream neighbour.

GUI

For the virtualisation of the graph database, I have used Cisco NEXT UI. It gives great abilities like map utility, grouping of vertex and colouring and scaling functions, auto draw capabilities.

I have used the PHP scripting for web services and serve data to the GUI and make some final modification for the GUI. These are;

Grouping vertex’es and set status of them : Group interfaces under router and set router status according to the interface status. If an interface is down set router status to warn which will be represented as orange on the GUI. If you expand the router router will shown as green and interface will be shown and the colour will be red. Another grouping function is group router under POP and POP’s under city. More can be done. Also edge’s between same nodes or nodeSet(node groups like city ore POP) are automatically grouped by Cisco NEXT UI. Some additional changes are made to Cisco NEXT UI for representing edge group colour/status according to edge group. If all the links between two cities are down the edge group colour of the edge set will be red, if one is down it will be shown as orange. These grouping function for the node,pop and city should be done at while data loading for the GUI. I have wrote a php script that will load node with the pop name as an attribute than later on ad nodeSet according the pop’s and cities. So changing the nodeSet status (pop,city or for the router model on physical layer logical interfaces are grouped under corresponding physical interface and physical interfaces grouped under routers)

Clearing status of node or edge from the GUI.

All of the programming and and design are made for just learning and see how its gonna work. But even if in this simple form it’s being used for our operation department.

Also I have encountered a problem to show the node’s positions according to geo locations as Cisco Next UI lacks of local map support. I have added Turkey Map support but not able to find a proper SVG map and geo projection method. So I have add another properties to the node like GUI coordinates that can be adjusted from GUI but will not set the actual locations.

For the future development, design should be separated into micro services like db management, monitoring management & collector, GUI for DB management and another for monitoring. REST/etc should be used for service messaging and YANG models should be used for design and modelling and messaging.

Future model for to be devepoled
Future model for to be devepoled

You may see some sample codes from :

https://codepen.io/deniza/pen/bqVVvJ

https://codepen.io/deniza/pen/dzKRRj

 

PHP codes to load from db,

for logical layer ;

loadTopology.php

<?php
function accessProtected($obj, $prop) {
$reflection = new ReflectionClass($obj);
$property = $reflection->getProperty($prop);
$property->setAccessible(true);
return $property->getValue($obj);
}

require_once ‘vendor/autoload.php’;

use Neoxygen\NeoClient\ClientBuilder;

$client = ClientBuilder::create()
->setAutoFormatResponse(true)
->addConnection(‘default’, ‘http’, ‘10.2.24.157’, 7474,true, ‘neo4j’,’neo4j’)
->build();
// temporary used arrays
$data[‘nodes’]=array();
$data[‘links’]=array();
$data[‘nodeSet’]=array();
$data[‘connection’]=array();
#popid nodeset this will be added to the data nodeSet in the end!
$pop[‘nodeSet’]=array();
$city[‘nodeSet’]=array();
$link[‘links’]=array();

$popID=1;
$error[‘nodes’]=array();
// arrays used for creating NEXTUI CTM MODEL. ‘nodes’,’links’ and ‘nodeSet’ array are used for this purpose.
$return[‘nodes’]=array();
$return[‘links’]=array();
$return[‘nodeSet’]=array();

# FIX ME: nodesetID, this must be unique between ID’s returned from database, currently picking a large enouh value to start with. DO NOT FORGET TO INCREASE nodesetID EACH TIME A NODESET IS CREATED
$nodesetID=666000;

// IGP/MPLS router link model
$q=”MATCH (sourcerouter:router)-[link:LINK]->(targetrouter:router) RETURN ID(sourcerouter) AS sourcerouterID,sourcerouter,ID(link) AS linkID,link,ID(targetrouter) AS targetrouterID,targetrouter”;

$response = $client->sendCypherQuery($q);
$result = $client->getRows();
$labels = $client->getLabels();

for($i=0; $i<count($result[‘sourcerouterID’]);$i++){
/* return format
sourcerouterID sourcerouter linkID link targetrouterID targetrouter
*/
#print_r(“processing row number:”.$i.” sourcerouterID:”.$result[‘sourcerouterID’][$i].” linkID:”.$result[‘linkID’][$i].” targetrouterID:”.$result[‘targetrouterID’][$i].”\n”);

if (!isset($result[‘sourcerouter’][$i][‘name’])) {
$result[‘sourcerouter’][$i][‘name’]=”null”;
}
if (!isset($result[‘targetrouter’][$i][‘name’])) {
$result[‘targetrouter’][$i][‘name’]=”null”;
}

#print_r(“processing row number:”.$i.” sourcerouter:”.$result[‘sourcerouter’][$i][‘name’].” targetrouter:”.$result[‘targetrouter’][$i][‘name’].”\n”);

# latitude and longitude settin. GUI values may be different than original values. It’s hard to project actual values on map.

if (isSet($result[‘sourcerouter’][$i][‘guilatitude’]) && isSet($result[‘sourcerouter’][$i][‘guilongitude’]) ) {
$result[‘sourcerouter’][$i][‘latitude’]=$result[‘sourcerouter’][$i][‘guilatitude’];
$result[‘sourcerouter’][$i][‘longitude’]=$result[‘sourcerouter’][$i][‘guilongitude’];
} else if ( !isSet($result[‘sourcerouter’][$i][‘dblatitude’]) || !isSet($result[‘sourcerouter’][$i][‘dblongitude’])) {
$result[‘sourcerouter’][$i][‘latitude’]=”42.16898908798602″;
$result[‘sourcerouter’][$i][‘longitude’]=”31.45923313006913″;
} else if ( $result[‘sourcerouter’][$i][‘dblatitude’] == “0” && $result[‘sourcerouter’][$i][‘dblongitude’] == “0” ) {
$result[‘sourcerouter’][$i][‘latitude’]=”42.16898908798602″;
$result[‘sourcerouter’][$i][‘longitude’]=”31.45923313006913″;
} else {
$result[‘sourcerouter’][$i][‘latitude’]=$result[‘sourcerouter’][$i][‘dblatitude’];
$result[‘sourcerouter’][$i][‘longitude’]=$result[‘sourcerouter’][$i][‘dblongitude’];
}

if (isSet($result[‘targetrouter’][$i][‘guilatitude’]) && isSet($result[‘targetrouter’][$i][‘guilongitude’]) ) {
$result[‘targetrouter’][$i][‘latitude’]=$result[‘targetrouter’][$i][‘guilatitude’];
$result[‘targetrouter’][$i][‘longitude’]=$result[‘targetrouter’][$i][‘guilongitude’];
} else if ( !isSet($result[‘targetrouter’][$i][‘dblatitude’]) || !isSet($result[‘targetrouter’][$i][‘dblongitude’])) {
$result[‘targetrouter’][$i][‘latitude’]=”42.16898908798602″;
$result[‘targetrouter’][$i][‘longitude’]=”31.45923313006913″;
} else if ( $result[‘targetrouter’][$i][‘dblatitude’] == “0” && $result[‘targetrouter’][$i][‘dblongitude’] == “0” ) {
$result[‘targetrouter’][$i][‘latitude’]=”42.16898908798602″;
$result[‘targetrouter’][$i][‘longitude’]=”31.45923313006913″;
} else {
$result[‘targetrouter’][$i][‘latitude’]=$result[‘targetrouter’][$i][‘dblatitude’];
$result[‘targetrouter’][$i][‘longitude’]=$result[‘targetrouter’][$i][‘dblongitude’];
}
#check popid and popname of source and target router
if (!isset($result[‘sourcerouter’][$i][‘popid’])) {
$result[‘sourcerouter’][$i][‘popid’]=9999999999;
$result[‘sourcerouter’][$i][‘popname’]=”null”;
}
if (!isset($result[‘targetrouter’][$i][‘popid’])) {
$result[‘targetrouter’][$i][‘popid’]=9999999999;
$result[‘targetrouter’][$i][‘popname’]=”null”;
}

#binding source router to target router, creating link
$keys = array_keys($result[‘link’][$i]);
$link[‘links’][$result[‘linkID’][$i]][‘id’]=$result[‘linkID’][$i];
$link[‘links’][$result[‘linkID’][$i]][‘model’]=’igp’;
$link[‘links’][$result[‘linkID’][$i]][‘source’]=$result[‘sourcerouterID’][$i];
$link[‘links’][$result[‘linkID’][$i]][‘sourcerouter’]=$result[‘sourcerouter’][$i][‘name’];

$link[‘links’][$result[‘linkID’][$i]][‘target’]=$result[‘targetrouterID’][$i];
$link[‘links’][$result[‘linkID’][$i]][‘targetrouter’]=$result[‘targetrouter’][$i][‘name’];
for($k = 0; $k < count($result[‘link’][$i]); $k++) {
$link[‘links’][$result[‘linkID’][$i]][$keys[$k]] = $result[‘link’][$i][$keys[$k]];
}

if(!(array_key_exists($result[‘sourcerouterID’][$i],$data[‘nodes’]))) {
$keys = array_keys($result[‘sourcerouter’][$i]);
$data[‘nodes’][$result[‘sourcerouterID’][$i]][‘id’]=$result[‘sourcerouterID’][$i];
for($k = 0; $k < count($result[‘sourcerouter’][$i]); $k++) {
$data[‘nodes’][$result[‘sourcerouterID’][$i]][$keys[$k]] = $result[‘sourcerouter’][$i][$keys[$k]];
}
#print_r(“creating sourcerouter with id:”.$data[‘nodes’][$result[‘sourcerouterID’][$i]][‘id’].” name:”.$data[‘nodes’][$result[‘sourcerouterID’][$i]][‘name’].”\n”);
}

if ($result[‘link’][$i][‘status’]>=12){
if ($result[‘sourcerouter’][$i][‘status’] == 0){
$data[‘nodes’][$result[‘sourcerouterID’][$i]][‘status’]=”4″;
$data[‘nodes’][$result[‘sourcerouterID’][$i]][‘message’]=”check links!”;
$data[‘nodes’][$result[‘sourcerouterID’][$i]][‘statusupdater’]=”data proceses script”;
#print_r(“links status:”.$result[‘link’][$i][‘status’].” sourcerouter status:”.$result[‘sourcerouter’][$i][‘status’].”\n”);
}
}

$popID=$result[‘sourcerouter’][$i][‘popid’];
$popname=preg_replace(‘/^N1-/’, ”, $result[‘sourcerouter’][$i][‘popname’]);
$cityname=”null”;
if (preg_match(‘/-/’,$popname)) {
list($cityname, $popname) = split(‘[-]’, $popname,2);
if ($cityname == ’34’ || $cityname == ‘234’ ) {
$cityname = 34;
}
}
if(!(array_key_exists($popID,$pop[‘nodeSet’]))) {
$popID=$result[‘sourcerouter’][$i][‘popid’];
$pop[‘nodeSet’][$popID][‘id’]=$result[‘sourcerouter’][$i][‘popid’];
$pop[‘nodeSet’][$popID][‘name’]=$result[‘sourcerouter’][$i][‘popname’];
$pop[‘nodeSet’][$popID][‘model’]=’pop’;
$pop[‘nodeSet’][$popID][‘root’]=$result[‘sourcerouterID’][$i]; //root is current routernodeset
$pop[‘nodeSet’][$popID][‘longitude’]=$result[‘sourcerouter’][$i][‘longitude’];
$pop[‘nodeSet’][$popID][‘latitude’]=$result[‘sourcerouter’][$i][‘latitude’];
$pop[‘nodeSet’][$popID][‘nodes’]=array($result[‘sourcerouterID’][$i]);
$pop[‘nodeSet’][$popID][‘status’]=$data[‘nodes’][$result[‘sourcerouterID’][$i]][‘status’];
#print_r(“creating pop with id:”.$pop[‘nodeSet’][$popID][‘id’].” name:”.$pop[‘nodeSet’][$popID][‘name’].”\n”);
}

if ($pop[‘nodeSet’][$popID][‘status’]<$data[‘nodes’][$result[‘sourcerouterID’][$i]][‘status’]){
#print_r(“changing status of pop id:”.$popID.” name:”.$pop[‘nodeSet’][$popID][‘name’].” from:”.$pop[‘nodeSet’][$popID][‘status’].” to:”.$data[‘nodes’][$result[‘sourcerouterID’][$i]][‘status’].”\n”);
$pop[‘nodeSet’][$popID][‘status’]=$data[‘nodes’][$result[‘sourcerouterID’][$i]][‘status’];
}
if (!(in_array($data[‘nodes’][$result[‘sourcerouterID’][$i]][‘id’],$pop[‘nodeSet’][$popID][‘nodes’]))) {
array_push($pop[‘nodeSet’][$popID][‘nodes’],$data[‘nodes’][$result[‘sourcerouterID’][$i]][‘id’]);
}

if(!(array_key_exists($cityname,$city[‘nodeSet’]))) {
$city[‘nodeSet’][$cityname][‘name’]=$cityname;
$city[‘nodeSet’][$cityname][‘model’]=’city’;
$city[‘nodeSet’][$cityname][‘root’]=$popID; //root is current pop
$city[‘nodeSet’][$cityname][‘longitude’]=$pop[‘nodeSet’][$popID][‘longitude’];
$city[‘nodeSet’][$cityname][‘latitude’]=$pop[‘nodeSet’][$popID][‘latitude’];
$city[‘nodeSet’][$cityname][‘nodes’]=array($pop[‘nodeSet’][$popID][‘id’]);
$city[‘nodeSet’][$cityname][‘status’]=$pop[‘nodeSet’][$popID][‘status’];
#print_r(“creating city with name:”.$cityname.”\n”);
}

if ($city[‘nodeSet’][$cityname][‘status’]<$pop[‘nodeSet’][$popID][‘status’]){
#print_r(“changing status of city name:”.$cityname.” from:”.$city[‘nodeSet’][$cityname][‘status’].” to:”.$city[‘nodeSet’][$cityname][‘status’].”\n”);
$city[‘nodeSet’][$cityname][‘status’]=$pop[‘nodeSet’][$popID][‘status’];
}
if (!(in_array($pop[‘nodeSet’][$popID][‘id’],$city[‘nodeSet’][$cityname][‘nodes’]))) {
array_push($city[‘nodeSet’][$cityname][‘nodes’],$pop[‘nodeSet’][$popID][‘id’]);
}

if(!(array_key_exists($result[‘targetrouterID’][$i],$data[‘nodes’]))) {
$keys = array_keys($result[‘targetrouter’][$i]);
$data[‘nodes’][$result[‘targetrouterID’][$i]][‘id’]=$result[‘targetrouterID’][$i];
for($k = 0; $k < count($result[‘targetrouter’][$i]); $k++) {
$data[‘nodes’][$result[‘targetrouterID’][$i]][$keys[$k]] = $result[‘targetrouter’][$i][$keys[$k]];
}
#print_r(“creating targetrouter with id:”.$data[‘nodes’][$result[‘targetrouterID’][$i]][‘id’].” name:”.$data[‘nodes’][$result[‘targetrouterID’][$i]][‘name’].”\n”);
}
if ($result[‘link’][$i][‘status’]>=12){
if ($result[‘targetrouter’][$i][‘status’] == 0){
$data[‘nodes’][$result[‘targetrouterID’][$i]][‘status’]=”4″;
$data[‘nodes’][$result[‘targetrouterID’][$i]][‘message’]=”check links!”;
$data[‘nodes’][$result[‘targetrouterID’][$i]][‘statusupdater’]=”data proceses script”;
#print_r(“links status:”.$result[‘link’][$i][‘status’].” targetrouter status:”.$result[‘targetrouter’][$i][‘status’].”\n”);
}
}
$popID=$result[‘targetrouter’][$i][‘popid’];
$popname=preg_replace(‘/^N1-/’, ”, $result[‘targetrouter’][$i][‘popname’]);
$cityname=”null”;
if (preg_match(‘/-/’,$popname)) {
list($cityname, $popname) = split(‘[-]’, $popname,2);
if ($cityname == ’34’ || $cityname == ‘234’ ) {
$cityname = 34;
}
}
if(!(array_key_exists($popID,$pop[‘nodeSet’]))) {
$popID=$result[‘targetrouter’][$i][‘popid’];
$pop[‘nodeSet’][$popID][‘id’]=$result[‘targetrouter’][$i][‘popid’];
$pop[‘nodeSet’][$popID][‘name’]=$result[‘targetrouter’][$i][‘popname’];
$pop[‘nodeSet’][$popID][‘model’]=’pop’;
$pop[‘nodeSet’][$popID][‘root’]=$result[‘targetrouterID’][$i]; //root is current routernodeset
$pop[‘nodeSet’][$popID][‘longitude’]=$result[‘targetrouter’][$i][‘longitude’];
$pop[‘nodeSet’][$popID][‘latitude’]=$result[‘targetrouter’][$i][‘latitude’];
$pop[‘nodeSet’][$popID][‘nodes’]=array($result[‘targetrouterID’][$i]);
$pop[‘nodeSet’][$popID][‘status’]=$data[‘nodes’][$result[‘targetrouterID’][$i]][‘status’];
#print_r(“creating pop with id:”.$pop[‘nodeSet’][$popID][‘id’].” name:”.$pop[‘nodeSet’][$popID][‘name’].”\n”);
}

if ($pop[‘nodeSet’][$popID][‘status’]<$data[‘nodes’][$result[‘targetrouterID’][$i]][‘status’]){
#print_r(“changing status of pop id:”.$popID.” name:”.$pop[‘nodeSet’][$popID][‘name’].” from:”.$pop[‘nodeSet’][$popID][‘status’].” to:”.$data[‘nodes’][$result[‘targetrouterID’][$i]][‘status’].”\n”);
$pop[‘nodeSet’][$popID][‘status’]=$data[‘nodes’][$result[‘targetrouterID’][$i]][‘status’];
}
if (!(in_array($data[‘nodes’][$result[‘targetrouterID’][$i]][‘id’],$pop[‘nodeSet’][$popID][‘nodes’]))) {
array_push($pop[‘nodeSet’][$popID][‘nodes’],$data[‘nodes’][$result[‘targetrouterID’][$i]][‘id’]);
}

if(!(array_key_exists($cityname,$city[‘nodeSet’]))) {
$city[‘nodeSet’][$cityname][‘name’]=$cityname;
$city[‘nodeSet’][$cityname][‘model’]=’city’;
$city[‘nodeSet’][$cityname][‘root’]=$popID; //root is current pop
$city[‘nodeSet’][$cityname][‘longitude’]=$pop[‘nodeSet’][$popID][‘longitude’];
$city[‘nodeSet’][$cityname][‘latitude’]=$pop[‘nodeSet’][$popID][‘latitude’];
$city[‘nodeSet’][$cityname][‘nodes’]=array($pop[‘nodeSet’][$popID][‘id’]);
$city[‘nodeSet’][$cityname][‘status’]=$pop[‘nodeSet’][$popID][‘status’];
#print_r(“creating city with name:”.$cityname.”\n”);
}

if ($city[‘nodeSet’][$cityname][‘status’]<$pop[‘nodeSet’][$popID][‘status’]){
#print_r(“changing status of city name:”.$cityname.” from:”.$city[‘nodeSet’][$cityname][‘status’].” to:”.$city[‘nodeSet’][$cityname][‘status’].”\n”);
$city[‘nodeSet’][$cityname][‘status’]=$pop[‘nodeSet’][$popID][‘status’];
}
if (!(in_array($pop[‘nodeSet’][$popID][‘id’],$city[‘nodeSet’][$cityname][‘nodes’]))) {
array_push($city[‘nodeSet’][$cityname][‘nodes’],$pop[‘nodeSet’][$popID][‘id’]);
}

}
// MATCH the igprouters which do not have any link
$q=”MATCH (sourcerouter:router) WHERE not ((sourcerouter)-[:LINK]->(:router)) AND NOT exists(sourcerouter.ip) RETURN sourcerouter,ID(sourcerouter) AS sourcerouterID”;

$response = $client->sendCypherQuery($q);
$result = $client->getRows();
$labels = $client->getLabels();

for($i=0; $i<count($result[‘sourcerouterID’]);$i++){
/* return format
sourcerouter sourcerouterID
*/
#print_r(“processing row number:”.$i.” sourcerouterID:”.$result[‘sourcerouterID’][$i].”\n”);

if (!isset($result[‘sourcerouter’][$i][‘name’])) {
$result[‘sourcerouter’][$i][‘name’]=”null”;
}
# latitude and longitude settin. GUI values may be different than original values. It’s hard to project actual values on map.

if (isSet($result[‘sourcerouter’][$i][‘guilatitude’]) && isSet($result[‘sourcerouter’][$i][‘guilongitude’]) ) {
$result[‘sourcerouter’][$i][‘latitude’]=$result[‘sourcerouter’][$i][‘guilatitude’];
$result[‘sourcerouter’][$i][‘longitude’]=$result[‘sourcerouter’][$i][‘guilongitude’];
} else if ( !isSet($result[‘sourcerouter’][$i][‘dblatitude’]) || !isSet($result[‘sourcerouter’][$i][‘dblongitude’])) {
$result[‘sourcerouter’][$i][‘latitude’]=”42.16898908798602″;
$result[‘sourcerouter’][$i][‘longitude’]=”31.45923313006913″;
} else if ( $result[‘sourcerouter’][$i][‘dblatitude’] == “0” && $result[‘sourcerouter’][$i][‘dblongitude’] == “0” ) {
$result[‘sourcerouter’][$i][‘latitude’]=”42.16898908798602″;
$result[‘sourcerouter’][$i][‘longitude’]=”31.45923313006913″;
} else {
$result[‘sourcerouter’][$i][‘latitude’]=$result[‘sourcerouter’][$i][‘dblatitude’];
$result[‘sourcerouter’][$i][‘longitude’]=$result[‘sourcerouter’][$i][‘dblongitude’];
}
#check popid and popname of source and target router
if (!isset($result[‘sourcerouter’][$i][‘popid’])) {
$result[‘sourcerouter’][$i][‘popid’]=9999999999;
$result[‘sourcerouter’][$i][‘popname’]=”null”;
}

if(!(array_key_exists($result[‘sourcerouterID’][$i],$data[‘nodes’]))) {
$keys = array_keys($result[‘sourcerouter’][$i]);
$data[‘nodes’][$result[‘sourcerouterID’][$i]][‘id’]=$result[‘sourcerouterID’][$i];
for($k = 0; $k < count($result[‘sourcerouter’][$i]); $k++) {
$data[‘nodes’][$result[‘sourcerouterID’][$i]][$keys[$k]] = $result[‘sourcerouter’][$i][$keys[$k]];
}
#print_r(“creating sourcerouter with id:”.$data[‘nodes’][$result[‘sourcerouterID’][$i]][‘id’].” name:”.$data[‘nodes’][$result[‘sourcerouterID’][$i]][‘name’].”\n”);
}

$popID=$result[‘sourcerouter’][$i][‘popid’];
$popname=preg_replace(‘/^N1-/’, ”, $result[‘sourcerouter’][$i][‘popname’]);
$cityname=”null”;
if (preg_match(‘/-/’,$popname)) {
list($cityname, $popname) = split(‘[-]’, $popname,2);
if ($cityname == ’34’ || $cityname == ‘234’ ) {
$cityname = 34;
}
}
if(!(array_key_exists($popID,$pop[‘nodeSet’]))) {
$popID=$result[‘sourcerouter’][$i][‘popid’];
$pop[‘nodeSet’][$popID][‘id’]=$result[‘sourcerouter’][$i][‘popid’];
$pop[‘nodeSet’][$popID][‘name’]=$result[‘sourcerouter’][$i][‘popname’];
$pop[‘nodeSet’][$popID][‘model’]=’pop’;
$pop[‘nodeSet’][$popID][‘root’]=$result[‘sourcerouterID’][$i]; //root is current routernodeset
$pop[‘nodeSet’][$popID][‘longitude’]=$result[‘sourcerouter’][$i][‘longitude’];
$pop[‘nodeSet’][$popID][‘latitude’]=$result[‘sourcerouter’][$i][‘latitude’];
$pop[‘nodeSet’][$popID][‘nodes’]=array($result[‘sourcerouterID’][$i]);
$pop[‘nodeSet’][$popID][‘status’]=$data[‘nodes’][$result[‘sourcerouterID’][$i]][‘status’];
#print_r(“creating pop with id:”.$pop[‘nodeSet’][$popID][‘id’].” name:”.$pop[‘nodeSet’][$popID][‘name’].”\n”);
}

if ($pop[‘nodeSet’][$popID][‘status’]<$data[‘nodes’][$result[‘sourcerouterID’][$i]][‘status’]){
#print_r(“changing status of pop id:”.$popID.” name:”.$pop[‘nodeSet’][$popID][‘name’].” from:”.$pop[‘nodeSet’][$popID][‘status’].” to:”.$data[‘nodes’][$result[‘sourcerouterID’][$i]][‘status’].”\n”);
$pop[‘nodeSet’][$popID][‘status’]=$data[‘nodes’][$result[‘sourcerouterID’][$i]][‘status’];
}
if (!(in_array($data[‘nodes’][$result[‘sourcerouterID’][$i]][‘id’],$pop[‘nodeSet’][$popID][‘nodes’]))) {
array_push($pop[‘nodeSet’][$popID][‘nodes’],$data[‘nodes’][$result[‘sourcerouterID’][$i]][‘id’]);
}

if(!(array_key_exists($cityname,$city[‘nodeSet’]))) {
$city[‘nodeSet’][$cityname][‘name’]=$cityname;
$city[‘nodeSet’][$cityname][‘model’]=’city’;
$city[‘nodeSet’][$cityname][‘root’]=$popID; //root is current pop
$city[‘nodeSet’][$cityname][‘longitude’]=$pop[‘nodeSet’][$popID][‘longitude’];
$city[‘nodeSet’][$cityname][‘latitude’]=$pop[‘nodeSet’][$popID][‘latitude’];
$city[‘nodeSet’][$cityname][‘nodes’]=array($pop[‘nodeSet’][$popID][‘id’]);
$city[‘nodeSet’][$cityname][‘status’]=$pop[‘nodeSet’][$popID][‘status’];
#print_r(“creating city with name:”.$cityname.”\n”);
}

if ($city[‘nodeSet’][$cityname][‘status’]<$pop[‘nodeSet’][$popID][‘status’]){
#print_r(“changing status of city name:”.$cityname.” from:”.$city[‘nodeSet’][$cityname][‘status’].” to:”.$city[‘nodeSet’][$cityname][‘status’].”\n”);
$city[‘nodeSet’][$cityname][‘status’]=$pop[‘nodeSet’][$popID][‘status’];
}
if (!(in_array($pop[‘nodeSet’][$popID][‘id’],$city[‘nodeSet’][$cityname][‘nodes’]))) {
array_push($city[‘nodeSet’][$cityname][‘nodes’],$pop[‘nodeSet’][$popID][‘id’]);
}
}

/*
foreach($nodes as $id => $id_value) {
#print_r( “Key=” . $id . “, Value=” . $nodes[$id][‘name’] . “\n”);
}
*/

foreach($data[‘nodes’] as $x => $x_value) {
//echo “Key=” . $x . “\n”;
$temp_node=array();
foreach($data[‘nodes’][$x] as $y => $y_value) {
//echo “Key=” . $y . “, Value=” . $y_value. “\n”;
$temp_node[“$y”]=$y_value;
}
array_push($return[‘nodes’],$temp_node);
}

foreach($link[‘links’] as $x => $x_value) {
//echo “Key=” . $x . “\n”;
$temp_link=array();
foreach($link[‘links’][$x] as $y => $y_value) {
//echo “Key=” . $y . “, Value=” . $y_value. “\n”;
$temp_link[“$y”]=$y_value;
}
array_push($return[‘links’],$temp_link);
}

foreach($data[‘nodeSet’] as $x => $x_value) {
//echo “Key=” . $x . “\n”;
$temp_nodeSet=array();
foreach($data[‘nodeSet’][$x] as $y => $y_value) {
//echo “Key=” . $y . “, Value=” . $y_value. “\n”;
$temp_nodeSet[“$y”]=$y_value;
}
array_push($return[‘nodeSet’],$temp_nodeSet);

}

foreach($pop[‘nodeSet’] as $x => $x_value) {
//echo “Key=” . $x . “\n”;
$temp_nodeSet=array();
foreach($pop[‘nodeSet’][$x] as $y => $y_value) {
//echo “Key=” . $y . “, Value=” . $y_value. “\n”;
$temp_nodeSet[“$y”]=$y_value;
}
array_push($return[‘nodeSet’],$temp_nodeSet);

}
//No need for city group, already viweing one city
foreach($city[‘nodeSet’] as $x => $x_value) {
//echo “Key=” . $x . “\n”;
$temp_nodeSet=array();
$temp_nodeSet[‘id’]=$nodesetID;
$nodesetID++;
foreach($city[‘nodeSet’][$x] as $y => $y_value) {
//echo “Key=” . $y . “, Value=” . $y_value. “\n”;
$temp_nodeSet[“$y”]=$y_value;
}
array_push($return[‘nodeSet’],$temp_nodeSet);

}
//$topology_data=json_encode($return);
if (!$error[‘nodes’]) {
echo json_encode($return);
} else {
echo json_encode($error);
}
?>

<?php

function accessProtected($obj, $prop) {
$reflection = new ReflectionClass($obj);
$property = $reflection->getProperty($prop);
$property->setAccessible(true);
return $property->getValue($obj);
}

require_once ‘vendor/autoload.php’;

use Neoxygen\NeoClient\ClientBuilder;
$client = ClientBuilder::create()
->setAutoFormatResponse(true)
->addConnection(‘default’, ‘http’, ‘10.2.24.157’, 7474,true, ‘neo4j’,’pet09tr&’)
->build();
$id=$_POST[‘id’];
$latitude=$_POST[‘latitude’];
$longitude=$_POST[‘longitude’];
$return=”error, no return from db or unable to connect to db”;
if ( isSet($_POST[‘id’]) && isSet($_POST[‘longitude’]) && isSet($_POST[‘latitude’])) {
$id=$_POST[‘id’];
$guilatitude=$_POST[‘latitude’];
$guilongitude=$_POST[‘longitude’];
$client->sendCypherQuery(“MATCH (n) WHERE ID(n)=$id SET n.guilatitude=$guilatitude,n.guilongitude=$guilongitude RETURN n”);
$result=$client->getResult();
if ($result->getNodesCount() == 1 ) {
$return=”updated”;
} else {
$return=”failed, returned number of node count:”.$result->getNodesCount();
}
} else {
$return=”not enough parameters for update”;
}
echo json_encode($return);
?>

 

Simple Graph Database Model for Network Management

~ dnzydn ~ Leave a comment

Simple Graph Database Model for Network Management

This is a very simple approach to modelling network via graph database.

I have tried to model network as we draw on whiteboard while working on project for a simple network monitoring tool to detect failure or alarms from the devices.

First step is to model the device with its connected interfaces which will be enough for the beginning (device-[contains]-interface).  Each device will be connected to another device via its’s interfaces. I model the interface as a vertex/node. Logical interfaces will be the leaf of the physical interfaces. Thus we can set the status of each interface individually without touching the root interface. If there is an alarm on the root interface it will be very easy to set all leaf interface status.

 

Level 2 network data model for Graph Database

 

Why I don’t directly connect the device interfaces to the remote device interface?  Generally devices are connected via another physical/logical layer. Devices can be directly connected (without any physical devices) which can be local cabling that can be a fiber, a copper or they can be connected via another provider or another logical layer by means of vlan’s or etc. For all them, this connection represented as a node/vertex named Circuit. With this approach, we are able to add another dependency for the connection between devices to a another layer.

With this information you can get the effected circuits after a breakout in the lower layer like fiber cut or upper provider failure. (Return circuit depends on fiber? which will be an easy query for graph database). Also you can add more dependency at lower layers like this fiber depends on that route or fiber group.

Next step is modelling the logical connections between devices, our backbone. I modelled IGP connections as LINK and IP connections for devices that do not have IGP configured as UPSTREAM (access devices like DSLAM). BGP-LS node and link notifications reflected to the database to set node or relation status (with reach and unreach NLRI). If an IGP node is down this means all the devices connected this node (connected to backbone via single logical ip link) are down. First you get the nodes that have an upstream relation with the node that is subject to BGP-LS message. If the current node is down this means all the nodes that behind it is down. (of down set all nodes-[upstream]->downnode status to down). If a link is down, set link status to down, check if nodes related both with the link has an another link thats status is up (return node-[:LINK {status:up}]-node. no link returns means that node’s unreachable. Set its status and all nodes which has upstream like connected  behind it is down.  If its an up message you set the status of all nodes behind it as warn (actually those node may be down and may need further lookup). Also messages coming from devices or any monitoring tool like syslog, snmp,  cacti graph alarms etc. reflected to the database.

Above that you can add multiple layers like services that’s depend on a device. Or you can more node types like BGP sessions or L2 tunnels or customers that connected or depend on a device.

Grouping of devices may also be modelled via separate nodes like CITY and node should have BELONGS to relation to that group. Also nested groups can be used like; device BELONGS to a pop, pop BELONGS a CITY or you may add group membership as an attribute to nodes. Sure this will be needed for virtualisation of network. Other wise IT’s very hard to browse hundreds of nodes via GUI.

TOOLS :

Drawing Tool : https://www.draw.io