Importing topology data into the system

Topology data can be imported using the /data/import/graph endpoint of the REST API. The topology data to be imported is described using a JSON structure.

When you import topology data, it is added to a partition that is named according to the specified source and source prefix.

Related topics

Endpoints in the REST API

Import modes - full and delta

There are two import modes, full and delta.

Full mode

For every full mode import, you must provide an item for each node and relationship in the items section of the JSON structure. Any existing nodes and relationships in the partition that are not specified are deleted.

Delta mode

For every delta mode import, you only need to specify changed nodes and relationships. To delete nodes and relationships using delta mode, you must specify the nodes and relationships to be deleted by using the destroyed or destroyed_rels sections of the JSON structure.

Request details

The following table provides key information on the JSON structure.

JSON	Key	Required/ Optional	Description
`{` `"source": "Source name", "source_prefix": "Source prefix",` "uuid": "1234", "complete": true, "mode": "delta",	source	Required	Display name of the source. Case sensitive. A partition is created for every source, with the partition name: `source_prefix + _ + source.` Non-alphanumeric characters in partition name are replaced with underscores.
	source_prefix	Required	Unique prefix for a source provider which is prepended to an item's node kind and used as the node kind in the datastore. For example, SN_Host. You can use the same prefix for different imports.
	uuid	Optional	The request can specify the uuid, but it must be globally unique. If the request does not specify a uuid, the system generates one for it. The API call returns the uuid, so that it can be used in subsequent batches. The batches are linked by uuid.
	complete	Optional	If it is not specified, it defaults to true. It marks the import as completed. When importing multiple batches for one logical import (transaction), set complete to false on all batches before the last, and then set complete to true on the last batch.
	mode	Optional	If mode is not specified, it defaults to `full` mode. The supported modes are: `delta` and `full` Full mode: An item for every node and relationship must be provided in each import, that is, any that are not specified are deleted. Delta mode: Only changed nodes and relationships are specified; deletions must be explicitly provided in "destroyed".
`"items": [` { "kind": "kind_name_1", "key": "1", "data": { "attr_one": "value1", "attr_two": "2" ... }, "void" : ["data_attr_name1", "data_attr_name2"] }, { "kind": "kind_name_2", "key": "2", "data": { "attr_one": "value1", "attr_two": "2", "attr_three": "value3" } }, { "kind": "Relationship_name", "role1": { "role": "role_one", "kind": "kind_name", "key": "11" }, "role2": { "role": "role_two", "kind": "kind_name", "key": "22" }, "data": { "rel_attr_name": "value4" } }, ], }	items	Required	List of node nodes and relationships between nodes. May be empty, to handle the case that a sender that sends batches may only notice that there is no more data to send having sent a batch.
	kind	Required	The node kind, for example, Host, Port, Network, relationship, and so on.
	key	Required for node, optional for relationships.	The identifier of the node. It must be unique among nodes with same kind. If the key does not exist, the system creates a new node. If the key does exist, the system updates the matching node.
	data	Optional	List of key/value pair attributes of the node. For example: `{"endpoint": "value1", "count": 100}` Every key in the data must be a valid identifier . A string is considered a valid identifier if it only contains alphanumeric letters (a-z, A-Z) and (0-9), or underscores (_). A valid identifier cannot start with a number, or contain any spaces. It must not start with double underscore (__).
	void	Optional	List of attribute names to remove. If the node already exists and provided attribute names exist then remove them from the node, the difference between setting a key as empty or null in data is that the key is kept in the datastore.
	role1 and role2	Required	Properties for a relationship. Relationships are optional. The roles "role1" and "role2" define the nodes in the relationship.
	kind	Required	The node kind in the relationship.
	role	Required	The name of the role. Valid characters are a-z, A-Z, 0-9 and underscore (_)
	key	Required	The identifier of the node in the relationship, that is, the keys of nodes created earlier in the batch.
`"destroyed": [` `{` `"kind": "kind_name_3",` `"key": "3"` `}` `],`	destroyed	Optional	Only used in delta mode. Contains nodes that will be deleted from datastore.
	kind	Required	The node kind.
	key	Required	The node key.
`"destroyed_rels": [` `{` `"kind": "rel_name_1",` `"key": "10"` `}` `],` }	destroyed_rels	Optional	Used in delta mode. Contains relationships that will be deleted from datastore. Only relationships which have a key (provided on creation) attribute can be destroyed.
	kind	Required	The node kind.
	key	Required	The node key.

Response uuid

If a uuid is not provided in the request then the system generates one. The response returns the generated or provided uuid to the caller. The caller can use the uuid from the response for subsequent batches.

Permissions

You need the following permissions to use the /data/import/graph endpoint of the REST API:

api/access
api/datastore/import
model/datastore/partition/*/write (this permission is required for creating partitions)

Viewing results of an import

When you have imported data, you can see information in the data import. To do this:

From the main menu, select Explore > Data.
In the Miscellaneous section, click Import Records.
From the list, select the imported record you want to see. The record displays numbers of items in the following states:

- Imported
- Created
- Updated
- Deleted
- Failed

Example topology imports

The following examples show JSON source for a series of topology imports:

To import topology data in “full” mode

{
            "source": "CompanyName",  # Payload defines two nodes to be imported
            "source_prefix": "CN",
            "items": [
                {
                    "kind": "Broker",
                    "key": "dc0cc9d7-2745-4b16-b97a-8945d2ce6435",
                    "data": {
                        "name": "NP_pargeter",
                        "broker_type": "kafka",
                        "brokers": "server:9092",
                        "topic": "events"
                    }
                },
                {
                    "kind": "Host",
                    "key": "8643bebf-85ab-4622-a56b-0158ec1f154a",
                    "data": {
                        "name": "Win_225_DD",
                        "status": "OK",
                        "host_type": "Windows",
                        "num_of_cpus": "4"
                    }
                }
            ]
        }

To import nodes and relationship in “full” mode

{
            "source": "CompanyName",   # Import two nodes and single relationship
            "source_prefix": "CN",
            "items": [
                {
                    "kind": "Broker",
                    "key": "dc0cc9d7-2745-4b16-b97a-8945d2ce6435",
                    "data": {
                        "name": "NP_pargeter",
                        "broker_type": "kafka",
                        "brokers": "server:9092",
                        "topic": "events"
                    }
                },
                {
                    "kind": "Host",
                    "key": "8643bebf-85ab-4622-a56b-0158ec1f154a",
                    "data": {
                        "name": "Win_25",
                        "status": "OK",
                        "host_type": "Windows",
                        "num_of_cpus": "4"
                    }
                },
                {
                    "kind": "HostBrokerRelationship",
                    "key": "a7262740-8381-48a2-b4fc-597afae08d04",
                    "role1": {
                        "role": "role1",
                        "kind": "Host",
                        "key": "8643bebf-85ab-4622-a56b-0158ec1f154a"
                    },
                    "role2": {
                        "role": "role2",
                        "kind": "Broker",
                        "key": "dc0cc9d7-2745-4b16-b97a-8945d2ce6435"
                    },
                    "data": {
                        "name": "NP_materialize",
                        "max_msg_size": "1MB"
                    }
                }
            ]
        }

To import nodes and delete node attribute in “full” mode

{
            "source": "CompanyName",   # To import two nodes and delete an attribute of one node
            "source_prefix": "CN",
            "items": [
                {
                    "kind": "Broker",
                    "key": "dc0cc9d7-2745-4b16-b97a-8945d2ce6435",
                    "data": {
                        "name": "NP_pargeter",
                        "broker_type": "kafka",
                        "brokers": "server:9092",
                        "topic": "events"
                    }
                },
                {
                    "kind": "Host",
                    "key": "8643bebf-85ab-4622-a56b-0158ec1f154a",
                    "data": {
                        "name": "Win_225_DD",
                        "host_type": "Windows",
                        "num_of_cpus": "4"
                    }, "void": ["status"]   # Deleting Status attribute
                }
            ]
        }

To import two nodes and a single relationship in delta mode

{
            "source": "CompanyName", 
            "source_prefix": "CN",
            "mode": "delta",
            "items": [
			
			{
                    "kind": "New_Host",			#Added Node
                    "key": "8643bebf-86ab-4622-a56b-0158ec1f154a",
                    "data": {
                        "name": "Win_22dd",
                        "status": "OK",
                        "host_type": "Windows",
                        "num_of_cpus": "4"
                    }
                },
				{
                    "kind": "New_Host2",			#Added Node2
                    "key": "8643bebf-86ab-4622-a56b-0158ec1f154b",
                    "data": {
                        "name": "Win_33dd",
                        "status": "OK",
                        "host_type": "Windows",
                        "num_of_cpus": "4"
                    }
                },
				
				{
                    "kind": "NewHostBrokerRelationship",		#Added Relationship
                    "key": "a7262740-8381-48a2-b4fc-597afae08d04",
                    "role1": {
                        "role": "role1",
                        "kind": "New_Host",
                        "key": "8643bebf-86ab-4622-a56b-0158ec1f154a"
                    },
                    "role2": {
                        "role": "role2",
                        "kind": "New_Host2",
                        "key": "8643bebf-86ab-4622-a56b-0158ec1f154b"
                    },
                    "data": {
                        "name": "NP_materialize",
                        "max_msg_size": "1MB"
                    }
                }
                
            ]
        }

To import a single node and delete a node and relationship in delta mode

{
            "source": "CompanyName",
            "source_prefix": "CN",
            "mode": "delta",
			"destroyed": [
			{
			"kind": "New_Host2",							#Destroy Node
			"key": "8643bebf-86ab-4622-a56b-0158ec1f154b"
			}  
			],
			"destroyed_rels": [				#Destroy Relationsships
				{
			"kind": "HostBrokerRelationship",
			"key": "a7262740-8381-48a2-b4fc-597afae08d04"
				}  
			],
            "items": [
			{
                    "kind": "New_Host3",			#Added Node3
                    "key": "8643bebf-86ab-4622-a56b-0158ec1f154b",
                    "data": {
                        "name": "Win_55_a",
                        "status": "OK",
                        "host_type": "Windows",
                        "num_of_cpus": "4"
                    }
                }
               
            ]
        }

Import topology data in two batches (full mode)

# First batch
{
    "source": "CompanyName",
    "source_prefix": "CN",
    "uuid": "a797d9b0-789c-4259-910b-95762c2b447c",
    "complete": false,	
    "mode": "full",
    "items": [
        {
            "kind": "Broker",
            "key": "dc0cc9d7-2745-4b16-b97a-8945d2ce6435",
            "data": {
                "name": "NP_pargeter",
                "broker_type": "kafka",
                "brokers": "server:9092",
                "topic": "events"
            }
        }
    ]

    # Last batch
    "source": "CompanyName",
    "source_prefix": "CN",
    "uuid": "a797d9b0-789c-4259-910b-95762c2b447c",
    "complete": true,
    "mode": "full",
    "items": [
        {
            "kind": "Host",
            "key": "8643bebf-85ab-4622-a56b-0158ec1f154a",
            "data": {
                "name": "NP_adscription",
                "host_name": "some_host",
                "status": "OK",
                "host_type": "Windows",
                "num_of_cpus": "4"
            }
        },
        {
            "kind": "HostBrokerRelationship",
            "key": "a7262740-8381-48a2-b4fc-597afae08d04",
            "role1": {
                "role": "role1",
                "kind": "Host",
                "key": "8643bebf-85ab-4622-a56b-0158ec1f154a"
            },
            "role2": {
                "role": "role2",
                "kind": "Broker",
                "key": "dc0cc9d7-2745-4b16-b97a-8945d2ce6435"
            },
            "data": {
                "name": "NP_materialize",
                "max_msg_size": "1MB"
            }
        }
    ]
}