TrackerInitialJob performance observation - parallel process

Using parallel processing, it was established that the time taken to process the data was considerably less when compared to the sequential flow.

For this observation, the tracker initial load job was run on the following key configuration parameters of the environment (configuration/resource allocation/capacity of the test server):
  • 64-bit CentOS 6.6
  • 8 CPU machine
  • Overall 8 GB RAM and 8 GB swap space
  • TeamForge application, ETL service and database server are all installed on the same box
  • The ETL service is configured to use a maximum heap size of 2048 MB
  • Postgres database
  • Maximum connections of database configuration server - 150
  • Database client connection pool configuration set to 40
It took, approximately, 8 hours 40 minutes for the tracker initial load job to process the following number of records:
  • Projects: 570
  • Trackers:1692
  • Planning folders: 4731
  • Artifacts: 184992
  • Customers: 1692
  • Statuses: 1692
  • Releases: 4409
  • Categories: 1692
  • Teams: 45
  • Users: 1556
  • Flex fields: 3772
  • Flex field values: 15427
  • Audit change rows: 3516016
  • Audit entry rows: 3110684
  • Total number of fields: 27950
  • Total number of field values: 62580
  • Number of artifact transaction fact rows generated: 769212
  • Number of artifact daily snapshot fact rows generated: 536672
Dimension/fact name ETL Start Date ETL End Date Record Count Duration Remarks
flex_field_dimension 27/08/15 05:59 27/08/15 06:23 4511 24 mins This includes all the dimensions.
flex_field_value_ dimension 17881
pf_bridge 27/08/15 06:23 27/08/15 09:00 15621   Running as three threads in pareallel.
artifact_dependency_bridge 27/08/15 06:23 27/08/15 14:43 32515  
artifact_transaction_fact
Batch 1 27/08/15 06:23 27/08/15 07:15 122350 52 miutes
Batch 2 27/08/15 07:15 27/08/15 09:10 130570 1 hr 55 minutes
Batch 3 27/08/15 09:10 27/08/15 10:16 107961 1 hr 7 minutes
Batch 4 27/08/15 10:16 27/08/15 10:56 104260 40 minutes
Batch 5 27/08/15 10:56 27/08/15 11:42 107449 46 minutes
Batch 6 27/08/15 11:42 27/08/15 12:51 104023 1 hr 9 minutes
Batch 7 27/08/15 12:51 27/08/15 13:28 92599 37 minutes
Total number of artifact_transaction_fact rows 769212  
artifact_daily_snapshot
Batch 1 27/08/15 13:28 27/08/15 13:29 70275 12 minutes
Batch 2 27/08/15 13:29 27/08/15 13:31 87406
Batch 3 27/08/15 13:31 27/08/15 13:32 78238
Batch 4 27/08/15 13:32 27/08/15 13:34 72840
atch 5 27/08/15 13:34 27/08/15 13:36 72919
Batch 6 27/08/15 13:36 27/08/15 13:38 79406
Batch 7 27/08/15 13:38 27/08/15 13:40 75588
Total number of artifact_daily_snapshot fact rows 536672  
Overall time 8 hrs 40 minutes (approx)