TrackerInitialJob performance observation

A performance observation was conducted on the tracker initial load job when fetching data from TeamForge and loading them to datamart.

For this observation, the tracker initial load job was run on the following key configuration parameters of the environment (configuration/resource allocation/capacity of the test server):
  • 64-bit CentOS 6.6
  • 8 CPU machine type with 8 GB RAM and 8 GB swap space
  • TeamForge application, ETL service and database server are all installed on the same box
  • The ETL service is configured to use a maximum heap size of 1524 MB
  • Postgres database
  • Maximum connections of database configuration server - 400
  • Database client connection pool configuration set to 30
It took, approximately, 15 hours and 30 minutes for the tracker initial load job to process the following number of records:
Note: All the relevant processes (for example, building dimensions, populating details around facts and hierarchies (team, artifacts and planning folder) were run in a sequence and not in parallel.
  • Projects: 570
  • Trackers:1692
  • Planning folders: 4731
  • Artifacts: 184992
  • Customers: 1692
  • Statuses: 1692
  • Releases: 4409
  • Categories: 1692
  • Teams: 45
  • Users: 1556
  • Flex fields: 3772
  • Flex field values: 15427
  • Audit change rows: 3516016
  • Audit entry rows: 3110684
  • Total number of fields: 27950
  • Total number of field values: 62580
  • Number of artifact transaction fact rows generated: 769212
  • Number of artifact daily snapshot fact rows generated: 536672
Dimension/fact name ETL Start Date ETL End Date Record Count Duration
flex_field_dimension 8/11/2015 5:16:03 8/11/2015 14:15:11 4511 9 hrs (approx)
flex_field_value_ dimension 17881
pf_bridge 15621
artifact_dependency_bridge 32515
artifact_transaction_fact
Batch 1 8/11/201514:15:11 8/11/2015 15:02:04 122350 47 minutes
Batch 2 8/11/2015 15:02:04 8/11/2015 16:40:42 130570 1 hr 38 minutes
Batch 3 8/11/2015 16:40:43 8/11/2015 17:42:59 107961 1 hr 02 minutes
Batch 4 8/11/2015 17:42:59 8/11/2015 18:21:24 104260 39 minutes
Batch 5 8/11/2015 18:21:24 8/11/2015 19:03:47 107449 42 minutes
Batch 6 8/11/2015 19:03:47 8/11/2015 20:07:46 104023 1 hr 3 minutes
Batch 7 8/11/2015 20:07:46 8/11/2015 20:41:37 92599 34 minutes
    Total number of artifact_transaction_fact rows 769212  
artifact_daily_snapshot_fact
Batch 1 8/11/2015 20:41:38 8/11/2015 20:42:33 70275 11 minutes
Batch 2 8/11/2015 20:42:33 8/11/2015 20:44:02 87406
Batch 3 8/11/2015 20:44:03 8/11/2015 20:45:32 78238
Batch 4 8/11/2015 20:45:37 8/11/2015 20:47:09 72840
Batch 5 8/11/2015 20:47:19 8/11/2015 20:48:36 72919
Batch 6 8/11/2015 20:48:36 8/11/2015 20:50:33 79406
Batch 7 8/11/2015 20:50:34 8/11/2015 20:52:24 75588
    Total number of artifact_daily_snapshot fact rows 536672  
      Overall time 15 hrs 30 minutes (approx)