The PGSQL_MAINTENANCE_WORK_MEM variable specifies the maximum amount of memory to be used in maintenance operations such as VACUUM.
See What are the right PostgreSQL settings for my site? for values recommended by CollabNet.