Migrating databases
The database toolkit is a command line tool created to leverage specific operations applied to the BMC Client Management database.
As a database administrator, use the database toolkit with a dedicated command-line tool to manage database operations. For instance, the toolkit’s copy command facilitates migrating a source database to a destination, such as transferring a Client Management database from an Oracle to a PostgreSQL instance.
The tool offers two different commands:
- copy: to copy the content of an existing database into another database, see To copy a database.
- update: to update the content of specific file path rows in a database, see To update database values.
The copy command connects to the two different databases, collects the required information, performs a comparison to determine the operations to be executed on the destination database, including the following:
- The destination database cleanup.
- The destination database initialization using the BMC Client Management SQL scripts.
- The destination database update to create dynamic objects, i.e. objects not created by the SQL scripts.
- The destination database is populated with rows value copy.
Click here to see the list of command help outputs.
To copy a database
The copy database command is designed to assist customers in migrating to a different database system. For example, customers using an Oracle database may want to switch to PostgreSQL. This command facilitates the transfer, allowing the environment to operate with the new database system.
The command first connects to the source database to gather relevant information and then connects to the destination database to collect data there as well. Using the gathered information, the command calculates and executes the necessary operations. The collected data include:
- The default tablespace for the database.
The tablespaces in the database.
- The tables.
- The columns for the collected tables, along with their data type.
- The primary key constraints.
- The foreign key constraints.
- The indexes.
- The functions.
- The views.
The collected data is categorized into two datasets:
- Static objects: Static objects are those created by the default BMC Client Management database creation script.
- Dynamic objects: The system generates dynamic objects throughout the product lifecycle.
After having connected the two databases and collected information for each, the command first verifies the source database includes all the static objects. The static objects, created by the default SQL script, must be present in the source database. The command then checks if the destination database needs initialization. Initialization is required if any static object is missing or incompatible with the expected format.
For example, a table might require a text-based column, but the column may be defined as numeric instead.
If all static objects are missing, a simple initialization is enough. However, if any object is incompatible, a cleanup of the destination database is needed first. This cleanup removes all existing static objects to allow the initialization script to run properly.
For example, if the Devices table needs to be created but already exists, it will be removed during the cleanup.
Once the destination database is initialized (with or without a preliminary cleanup), the command identifies the dynamic objects to create, including tables, columns, constraints, and indexes. After the final objects are created in the destination database, the two databases will be compatible, allowing the rows to be copied from the source to the destination. This final step is the destination database population, where all rows for both static and dynamic tables are copied from the source to the destination.
By default, the command does nothing unless a specific command line switch is provided. To cleanup, initialize, update, and populate the destination database, use the -clear, -init, -update, and -populate command line parameters. If these switches are not supplied, the command notifies you that operations need to be executed and specifies which command line parameter is required to proceed. Some operations may consume time and displays the progress as percentages.
Additionally, the row population is incremental, so you can stop the database copy during this step. The next execution of the command will resume from the table that was being copied when the process was stopped.
Tablespace
The database copy command may need special configuration regarding the tablespaces. By default, the default SQL script used to initialize the destination database will use standard tablespace names having the BCM_ prefix.
There are different use cases:
- The destination database can keep these default tablespace names.
- The destination database must use different tablespace names.
- The source database is using the default tablespace names.
- The source database is using different tablespace names.
All these situations can be handled using tablespaces settings. The command line parameter -output-tablespaces-path can be used to write the default settings on disk. Once modified, the command line parameter -input-tablespaces-path can be used to instruct the command on how to handle the tablespaces during the database copy.
The tablespace settings are defined in a JSON file format, which consists of two sections:
Section | Details |
---|---|
Custom | |
Rename |
By using these custom and rename sections, all the different use cases can be handled.
Filters
The database copy operation is driven by filters, specifically during the update operation. While creating objects from the default SQL script is straightforward, the update operation is responsible for creating the dynamic objects. Filters are used to make decisions during this operation and follow an event-driven model. The command triggers events during execution whenever specific conditions are met, and the events defined in the filters are used to resolve each situation.
For example, when dynamic objects are collected from the source database, the filters will determine whether these objects need to be created and copied to the destination database or if they can be ignored. If they need to be processed, the actual object names are retrieved from the filters. This is necessary because different database systems have different naming conventions. By default, SQL Server keeps the names as they are, while Oracle converts them to uppercase and PostgreSQL to lowercase. When copying objects between database systems, it is essential to retrieve their original names, and the filters help facilitate this process.
The filters support the following event types:
Event type | Description |
---|---|
MISSING_TABLE | |
MISSING_COLUMN | |
MISSING_PKEY | |
MISSING_FKEY | |
MISSING_INDEXES | |
COPY_TABLE | |
COPY_COLUMN |
To update database values
The update values command is designed to help customers update predefined column rows in their BMC Client Management database, specifically those containing file paths. While the copy database command helps customers migrate to a different database system, it can also be used to change the operating system running the BMC Client Management Master. In this case, file paths stored in the database may need to be updated. Additionally, it’s possible to move the Master from folder A to folder B without changing the underlying operating system. In this scenario, the file paths stored in the database must also be updated to reflect the change.
The command connects to a database and collects a list of tables and columns. Modifications are then applied based on the provided filter settings. The command includes a default filter, but it is also possible to export this default configuration and import it back after making modifications.
Command | Description |
---|---|
UPDATE_COLUMN |
Command help output parameters
Parameters | |
---|---|
Command | Description |
copy database | This command copies a source database content into a destination database. The source and destination databases can be SQL Server, Oracle or PostgreSQL. The command prepares the destination database structure to ensure that all the rows can be copied for each table. Upon execution, the destination database includes all the source database content. |
update values | This command updates the text columns for rows with values that match a specified prefix, replacing them with modified values based on a new prefix. It is commonly used to change the prefix of identified columns, such as file paths |
-log STRING | This command defines the log file path. When configured, the command writes the log on the terminal and in the specified log file. When not configured, it will only write the log on the terminal |
-debug | This command enables the debug log messages. This mode also generates messages with an extended format, including the date and time for each record and an identification class for each logged line. |
-shorten-values | This command copies each text column using a predefined buffer size. For example, If the text column's maximum length is 2000 characters or less, that length is used as the buffer size. If it's greater than 2000 characters, the limit is set to 2000 to avoid high memory usage. The command will fail if a source row requires more characters. Use this parameter to shorten the source value and proceed. |
-transaction-size INTEGER | This command defines how many rows per transaction can be inserted while populating destination tables. Caution because depending on the defined parameter value, the command may be slow or use a lot of memory. Default size is 10.000 rows per transaction. |
-sql-files-path STRING [mandatory] | This command defines the SQL scripts path. These scripts are required to create the default objects in the dst database. Depending on the database type, a different set of SQL scripts will be used. |
-output-filters-path STRING | This command writes the default event filters to the defined file path. The content can then be updated and provided as input-filters-path to a next call. |
-input-filters-path STRING | This command reads the event filters from the provided file path. The default event filters can be obtained using -output-filters-path. |
-output-tablespaces-path STRING | This command writes the default tablespace mappings to the defined file path. These mappings define the list of custom tablespaces if the default BCM ones shall not be used, and the list of renaming rules to be applied if the source tablespaces name shall be updated. |
-input-tablespaces-path STRING | This command reads the default tablespace mappings from the provided file path. The default mappings can be obtained using -output-tablespaces-path. |
Source Database Parameters | |
-db-type STRING [mandatory] | This command defines the type for the src database. Accepted strings are odbcdirect or odbc for SQL Server, postgres for PostgreSQL, oracle for Oracle. |
-db-host STRING | This command defines the hostname or IP address for the src database. This value is optional for SQL Server and mandatory otherwise. When set with a valid SQL Server connection string, the whole parameters are merged to build the final connection string. |
-db-port INTEGER | This command defines the TCP port number to connect to the src database. This value is required for PostgreSQL only. |
-db-name STRING [mandatory] | This command defines the name for the src database. |
-db-user STRING [mandatory] | This command defines the user to connect to the src database. |
-db-pass STRING [mandatory] | This command defines the password to connect to the src database. Cautions as values supplied on the command line. |
-db-cstr STRING | This command defines a connection string to connect to the src database. This value can be merged with other database command line parameters to build the final connection string on SQL Server. Use the connection string value to configure specific settings, such as driver and encryption. |
Destination Database Parameters | |
-dst-db-type STRING [mandatory] | This command defines the type for the dst database. Accepted strings are odbcdirect or odbc for SQL Server, postgres for PostgreSQL, oracle for Oracle. |
-dst-db-host STRING | This command defines the hostname or IP address for the dst database. This value is optional for SQL Server and mandatory otherwise. When set with a valid SQL Server connection string, the whole parameters are merged to build the final connection string. |
-dst-db-port INTEGER | This command defines the TCP port number to connect to the dst database. This value is required for PostgreSQL only. |
-dst-db-name STRING [mandatory] | This command defines the name for the dst database. |
-dst-db-user STRING [mandatory] | This command defines the user to connect to the dst database |
-dst-db-pass STRING [mandatory] | This command defines the password to connect to the dst database. Caution as values supplied on the command line are visible when enumerating system tasks via dedicated tools. |
-dst-db-cstr STRING | This command defines a connection string to connect to the dst database. This value can be merged with other database command line parameters to build the final connection string on SQL Server. Use the connection string value to configure specific settings, such as driver and encryption. |
Operation Parameters | |
-clear | This command authorizes the command to perform clear operations on the dst database such as tables, functions or primary key constraints deletion. This is only required when the dst database must be emptied before the scripts can be executed. |
-init | This command authorizes the command to initialize the dst database with the scripts dedicated to objects creation. This initialization operation cannot be executed unless the dst database was first cleared, if required. |
-update | This command authorizes the command to update the dst database with dynamic objects identified in the src database. Dynamic objects are elements collected from the src database but not defined in the SQL scripts. The database can only be updated after a successful initialization. |
-populate | This command authorizes the command to copy the src database rows in the dst tables when required. Only tables having a different row count will be copied from a src table into its equivalent dst table. The copy operation can only be executed if the src and dst tables include compatible columns, which is verified by the update operation. |