Configuring the dataset and sync interval

Use the Dataset console to configure a dataset and set the sync interval.

To configure the dataset and sync interval

1. On the Helix Data Connector UI, select Dataset console.
2. Add a dataset.
   1. Click Add Dataset.
   2. Enter a name for the dataset.
   3. Select the sync interval from the values provided in the Interval list. Example- 5 h
      
   4. Click Save.
      The dataset is saved in the draft state. 
      To make this dataset active, you must associate at least one form with this dataset.
      You can select from a list of all the BMC Helix Innovation Suite forms by clicking Data form selection or BMC category groupings.
      You cannot assign the same form to multiple datasets. Attempting to do so will display a warning.
      It is not mandatory to perform both Steps 3 and 4. You can perform either Step 3  or Step 4 as per your requirements.
      
3. Select forms from Data form selection.
   1. Click Data form selection.
   2. Select the forms you want to associate with the dataset and click Select. 
      The selected forms are displayed on the Dataset UI.
4. Select forms from BMC category groupings.
   1. Click BMC category groupings.
   2. On the Select category screen, select the category of forms and click Next.
   3. On the Data select forms screen select the forms you want to associate with the dataset and click Select. 
      The selected forms are displayed on the Dataset UI.
5. Enable syncing of attachments
   1. By default, syncing of attachments is disabled for the forms. 
   2. To allow syncing attachments, select the form, click Sync Attachment and select Yes from the list as shown in the following image:
      
6. Make the dataset active.
   1. On the Dataset screen, click Edit.
   2. Select the Status as Active and click Save.
      The dataset is added to the Dataset Console as shown in the following image:
      
      The form jobs in each dataset are run as per the scheduled interval.  You can view the dataset job details in the Job Console.

Best practices for syncing attachments

Including attachments in your dataset sync can significantly increase sync completion time and, depending on your destination type (for example, Snowflake), might lead to higher storage requirements and compute costs. Include attachments only if you plan to decompress, deserialize, and use their content.

Additionally, note the following:

• Attachment Size Limit— Snowflake allows a maximum attachment size of 8MB. Text data in columns exceeding 16MB will be truncated for Snowflake as a destination.
  Refer to Snowflake documentation for details.
• Compression Format— Data in the attachment column is stored in a compressed format. Use the decompression helper classes available in the BMC Java API to retrieve the original attachment. For more information, see the Knowledge article, Java code to decompress/extract Remedy attachments without AR Server.

Best practices for optimizing sync performance

Consider the following best practices to ensure efficient data synchronization:

• Impact of adding more forms and attachments
  The sync time depends on the number and size of forms, the type of data, the size of data, the resources given to the target database, the warehouse configuration(Snowflake), and network latency. Including attachments significantly increases sync size and completion time. Syncing large datasets at very short intervals can cause delays. If a sync takes longer than the scheduled interval, BMC Helix Data Connector skips the next scheduled sync.
  For example, If sync timings are scheduled for 12 A.M., 1 A.M., and 2 A.M., but the 12 A.M. sync exceeds an hour, the system skips the 1 A.M. sync and starts the next sync at 2 A.M.
• Setting sync intervals based on the use case
  • Improve sync performance by categorizing data forms based on the use case.
    For example, for Incident data, such as SLA, escalations, status syncing to the destination might be required more frequently. Hence add these forms to a dataset with a shorter sync interval configured.
    Change data might be queried or analyzed on a daily basis. Hence, a longer interval should be considered to alleviate the load for datasets in the shorter intervals. 
    Lastly, CMDB and assignments might be only required for weekly or monthly reporting. In this case, consider a reasonable daily sync to alleviate the load for data sets set to a shorter interval.
  • Instead of placing all data in a single dataset, create multiple datasets based on your use case and reporting requirements. Consider offloading some forms to alleviate loading times for data required in shorter syncs and to optimize the sync job completion time.
• Dataset limitations for data forms
  A data form belongs to only one dataset at a time. To add a form to another dataset, first remove it from the existing dataset.
  • The Dataset column shows if a form is already assigned to a dataset.
  • Adding a form to a new dataset triggers a full sync, not an incremental sync.

Related topic

• Syncing data from BMC Helix SaaS to a destination database