Released on: 4 July , 2017
Based on vanilla Gerrit version: v2.13.9.
Highlights
- Fixed the issue in which the RabbitMQ connections and threads are kept permanently in
memory even if the project was not active.
Cause of the issue:
This issue is caused by the default configuration of RabbitMQ client which opens the
dedicated thread pool (that spawns up to 2 * number of CPU threads by default) per
connection. The issue manifests with large number of threads being spawned over time up
to,
- (number of repositories + number of projects) for threads named AMQP Connection
AA.BB.CC.DD.YYYY.
- (number of repositories + number of projects) * (2 * number of CPUs) for threads
named pool-XXX-thread-Y (especially when the machine is heavily loaded and
multiple changes are being reviewed and pushed to multiple projects).
Solution:
- Shared thread pool used by all RabbitMQ connections: It can spawn up to
(factor * number of CPUs) threads where factor can be modified by setting
teamforge.notificationMaxRabbitMQExecutorsFactor in
gerrit.config (default value is 2) file. Threads are removed
from thread pool after being idle for more than
teamforge.notificationMaxRabbitMqThreadIdleTime seconds in
gerrit.config (by default 3*60s) file. This addresses the
issue with the pool-XXX-thread-Y threads explosion.
- Connection Expiration: Connection to get closed after being idle for more
than teamforge.notificationMaxRabbitMQConnectionIdleTime seconds in
gerrit.config (default is 5*60s) file. This addresses the issue
with the AMQP Connection AA.BB.CC.DD:YYYY threads explosion as they are not
being quickly (marked as "not used" after
teamforge.notificationMaxRabbitMQConnectionIdleTime idle time and
guaranteed to be collected within 2 *
teamforge.notificationMaxRabbitMQConnectionIdleTime idle time)
collected.
- You can now search for the changes using Change-Id triplet (Repository~Ref~Change-Id).
When the first 5 characters of the Repository name (Example: IdeaField) matches the
Change-Id pattern, the changes searched for will be shown. This fix is sent back to open
source Gerrit.
Bug fixes
For complete details of the bug fixes, see v2.13.9.