Cups performance with 1200 queues

[bse]

> I work for a major retailer and we use cups to print massive graphic jobs to 1200 stores.  We have a custom app that generates the files in either Postscript or PDF format.  We use Cups v1.24 with the lpd backend.  Our production cups environment handles as much as 10,000 jobs a day load balanced between two 8 processor servers with 16GB of RAM each.

Consider a newer version of Cups.
Consider a version with some small patches.
Really!

Just some comments:
At my university we are using a slightly modified cups 1.3.9 + pykota to take care of the printing. 400 printers, 20000 users, up to 10000 jobs each day, 40000 pages printed each day. The clients are Windows XP, Windows Vista, MacOSX, Linux. All printers (except a few) uses PostScript. The printserver is a old Dell 2950, 2 dual core 2GHz CPU (4 cores in total), 16 GB memory.

At the start in 2006 we designed the system so we should be able to insert more hardware in case it became neccesary. The design allowed the incoming side (samba, IPP, lpd), the processing and output side (cupsd, pykota), the database for pykota (postgres), the ripping engines to run on separate servers.

We were seriously discussing using more hardware last summer/autumn when the server was constantly running with cupsd pinned at 100%, a load average of more than 3 with peaks in the order of 8 to 10.

When we renewed the system, using the then latest versions of Cups, Samba, PyKota, RedHat Linux, Kerberos and so on, there was a (very) small improvement, and we decided to keep the system on one box, and try to improve the performance.

The unmodified cupsd was running at 100%, becoming a bottleneck, and the clients could get hangs, timeouts and other problems. A rewrite of cupsd to reduce the number of syscalls, use nonblocking writes, more fair behaviour when selecting next socket and some other small stuff have resulted in a CPU-usage of less than 30% for cupsd.

Current (8.45 in the morning) load for the server, including cupsd, 400 samba processes, pykota for all printers, 330 jobs during the last 30 minutes, is 1.5

cupsd have since restart at 06.05 in the morning used 5 minutes of CPU time. That is 5 minutes out of 160 minutes = 3% If we adjust for the fact that the users came in at 7.30, it is still 5 minutes / 75 minutes = 7% CPU utilisation.

The load increases later in the day when the amount of printjobs increases, but that is mainly due to more samba and python processes.

The reduction in CPU use for cupsd is dramatic, and makes Cups scale much better. All other processes as samba, pykota, ripping, are spread out on all processors. We can see an increase in load when the server needs to rip from PostScript to bitmap, but the ripping does not affect cupsd.

In conclusion:
With a few small patches to cupsd
* the systems run with much lower CPU usage,
* the clients got much better response, especially when the system is loaded
* scaling is much better. A single cupsd on a 'normal' machine could probably handle 100000 jobs a day

>
> My questions are these:
>
> 1) Are there any docs with recommended settings for cupsd.conf running in an enterprise?
> 2) I've increased "BrowseTimeout 90" and "BrowseInterval 900" which initially reduced my cupsd from running at 100% down to a reasonable level.  Now we again have the same issue with the cupsd hitting 99-100% utilization.  The job queue can get as many as 400 jobs pending.
> 3) The only processing we do is take the PDF jobs and convert them to PCL using the foomatic "hp-color_laserjet_4600_series-hpijs.ppd" driver or one very similar from foomatic.  At what point does cups actually hit the processor hard?  Does cups render the job completely and then add to the queue or does it simply line the jobs up in the queue and do the rendering of all jobs simultaneously in the queue while they're in a pending state?
> 4) We use the lpd backend if that matters.
>
> Here's my cupsd.conf minus the policies:
> MaxLogSize 2000000000
> MaxJobs 10000
> JobRetryLimit 1000
> LogLevel info
> SystemGroup sys root cups
> Listen 80
> Listen 631
> Listen /var/run/cups/cups.sock
> Browsing On
> BrowseOrder allow,deny
> BrowseAllow @LOCAL
> BrowseInterval 90
> BrowseTimeout 900
> DefaultAuthType Basic
> == policies removed ==
>
>
> I have to use stable code delivered with Oracle Enterprise Linux 5.  So I can't compile the latest version.  I can't image anyone using cups as heavily as we are would like to share my experiences with everyone.
>
> Jeff Borders
>