Large cups systems?

[Seth Galitzer]

Kurt Pfeifle wrote:
> Seth Galitzer wrote:
> 
>> My users print from your typical host of desktop office productivity
>> applications: office suite, acrobat, various web browsers & email
>> clients, pre-generated PS files, nothing out of the ordinary.  Most
>> small print jobs seems to complete fine, but anything over a few-hundred
>> KB (that I've noticed) seems to stall out the queue to where I have to
>> dump it and restart it.
> 
> You're restarting *what*? The queue, or cupsd? (Usually the
> "cupsenable printername" should work...)

I've been trying restarting the queue first, and if that is
unsuccessful, then restarting cupsd.  I'm sorry to say I haven't
familiarized myself with the cli tools yet.  Usually I prefer them, but
I've been doing most of this with the GUI.
> 
> Also, what is the connection/backend you're using for talking to your
> print devices?
> 
> Given your complaints, I assume lpd://.... And I also assume that
> these are not real network printers, but cheaply turned into net-
> work printers by putting a "printbox" in front of them, and hooking
> the actual print device onto their parallel port.

I'm using socket:// for all but one of the printers.  All but two of my
printers are HP using internal EIO or MIO JetDirect cards.  Of these,
most are LaserJets, but two are InkJets.  Of the two that are not HP,
one is an Epson Stylus Color 3000 with a Type B Ethernet I/F card, and
the other is a Savin copier/printer with its own internal print server.
 The Savin is the only one I'm using lpd:// on.  Frankly, I don't have a
reason for that other than the cups server I replaced used it there.
that server predates me so I don't know the reasoning for using that
backend.

> 
> (Apologies if I'm wrong on that.)
> 
> These cheap print boxes do very often go on strike if you send too
> large jobs towards them. The CUPS lpd backend will notice that, and
> will in turn disable said queue. (And you can change this automatic
> disabling by setting the "ErrorPolicy" to a different one; easiest
> via the web interface....)

I added the ErrorPolicy setting to my conf file.  Thanks for the tip.

> 
> ---
> 
> And are you sure you've got enough of empty disk space left on the
> disk/partition where CUPS spooling happens (usually /var/spool/cups) ??
The partition is small but I have a few GB free on there.  I'll look at
growing this partition and give myself a little more headroom there.
> 
> Are you sure you've enough memory for CUPS to process print jobs?
I have 2GB RAM allocated for this VM.  Watching top, it doesn't look
like I'm touching more than a few-hundred MB so far.

> 
> What does 'grep -E "RIPCache|RIP_MAX_CACHE" /var/log/cups/error_log*'
> return?

D [15/Oct/2007:16:08:32 -0500] [Job 2018] envp[19]="RIP_MAX_CACHE=8m"

(repeated a couple of times with different job numbers)

> 
> You may be able to get some better performance for larger print jobs
> if you set "RIPCache 100m" in your cupsd.conf (instead of the default
> value of 8m).

What would be an appropriate setting?  Is 100m reasonable for typical
jobs, ie jobs that typically are less than 10MB?  I'll have to grep my
logs and see what kind of file size I've been processing.

> 
> 
>> I have to restart the daemon nearly daily because I get several queues
>> stuck in this bad state to where a restart of the queue itself is not
>> sufficient to get jobs flowing again.
> 
> 
>> I'm looking through the error_log and am not convinced I see much
>> helpful in there.  It's pretty verbose.  
> 
> Seems you have debug or debug2 LogLevel set.
> 
> Each line starts with a letter (I, E, D, or d, mostly). E indicate real
> errors. I orginate in LogLevel info. D are from LogLevel debug. d are
> from LogLevel debug2.

Looks like I have LogLevel debug.  Thanks for the info on the flags.
That'll be useful for parsing logs from now on.

> 
>> When I tail -f it, it updates
>> several times per second.  If there's anything I should be looking for,
>> please let me know, 
> 
> Whenever you have one of these hanging jobs, identify its CUPS job ID.
> Then grep for (and/or extract) all the lines with that ID. Identify
> which the originating user/host/application/printfile was. Which file
> format. Which print options went with it. Which filter and backend
> processes where started for that job. Which process IDs did these
> filters and backends have. Are some of these still appearing in the
> "ps aux" listing? Zombies?
> 
> If a job hangs with a given CUPS job ID of NNMM, you can still grab
> the input file(s) from the spool directory. They are named "d00NNMM-001",
> "d00NNMM-002" etc.
> 
> The respective job control file is "c00NNMM", and you can extract the
> IPP job attributs for the job with the "testipp" utility shipping with
> the CUPS sources.
> 
> You can grep the error_log for "argv[" and "envp[" to see which argu-
> ments and env variables your cupsd saw for each job.
> 
> You can hire someone to analyze your log files, even remotely by ssh-ing
> in....

Nah, I just need to learn how to read them.  With the info you provided,
that will help immensely.  I suppose this is documented somewhere and I
just missed it...

> 
>> otherwise, I'd be pasting a huge dump which probably
>> wouldn't be productive. 
> 
> You could upload it somewhere, and post an URL.
> 
>> The following seem fairly common (and look like
>> actual errors):
>> "cupsdAuthorize: No authentication data provided" (tasks seem to keep
>> going after this one)
>> "cupsdUpdateCUPSBrowse: Refused 172 bytes from foo" (where foo is the IP
>> of the server it's running on)
>>
>> Like I said, if you can give me an idea of what to look for, I'll do it.
> 
> How many clients? How many printers? How many jobs during a typical
> week? How many jobs during the peak hour of the week? How many pages
> per job on avarage?

One cups server, 13 printers, 200+ users.  Looking at my logs from last
week, an average of about 700-750 jobs for an average of about 1000
pages per day during the week.

> 
> 
>> I'll paste my sanitized cupsd.conf here.
>>
>> #begin cupsd.conf
>> LogLevel debug
>> SystemGroup admins
>> Listen foo:631		#foo is the server IP
>> Listen localhost:631
>> Listen /var/run/cups/cups.sock
>> Browsing On
>> BrowseOrder allow,deny
>> BrowseAddress bar.255	#bar is my subnet
>> BrowseInterval 15
>> BrowseTimeout 30
> 
> Aren't the defaults (60/300) good enough for you? This setting
> certainly increases load and strain on your system and on your
> network... (doubling the frequency of browse broadcasts for
> [how many?] printers, and reducing the BrowseTimeouts to a tenth
> of the default value).

As I mentioned to Bernd and earlier here, I took these from another
server conf.  I pulled them out and am now using defaults.

> 
>> DefaultAuthType Basic
>> DefaultEncryption Required
>> <Location />
>>   Order allow,deny
>>   Allow from bar.*
>> </Location>
>> <Location /admin>
>>   Order allow,deny
>>   Allow from bar.*
>>   Require user @SYSTEM
>> </Location>
>> <Location /admin/conf>
>>   Order allow,deny
>>   Allow from bar.*
>>   Require user @SYSTEM
>> </Location>
>> <Policy default>
>>   <Limit Send-Document Send-URI Hold-Job Release-Job Restart-Job
>> Purge-Jobs Set-Job-Attributes Create-Job-Subscription Renew-Subscription
>> Cancel-Subscription Get-Notifications Reprocess-Job Cancel-Current-Job
>> Suspend-Current-Job Resume-Job CUPS-Move-Job>
>>     Require user @OWNER @SYSTEM @faculty_users @staff_users @admins
>>     Order deny,allow
>>   </Limit>
>>   <Limit Pause-Printer Resume-Printer Set-Printer-Attributes
>> Enable-Printer Disable-Printer Pause-Printer-After-Current-Job
>> Hold-New-Jobs Release-Held-New-Jobs Deactivate-Printer Activate-Printer
>> Restart-Printer Shutdown-Printer Startup-Printer Promote-Job
>> Schedule-Job-After CUPS-Add-Printer CUPS-Delete-Printer CUPS-Add-Class
>> CUPS-Delete-Class CUPS-Accept-Jobs CUPS-Reject-Jobs CUPS-Set-Default>
>>     AuthType Basic
>>     Require user @SYSTEM @admins
>>     Order deny,allow
>>   </Limit>
>>   <Limit Cancel-Job CUPS-Authenticate-Job>
>>     Require user @OWNER @SYSTEM @admins
>>     Order deny,allow
>>   </Limit>
>>   <Limit All>
>>     Order deny,allow
>>   </Limit>
>> </Policy>
>> #end cupsd.conf
>>
>> And here's my sanitized samba.conf, as well:
>> #begin smb.conf
>> [global]
>>    panic action = /usr/share/samba/panic-action %d
>>    workgroup = myworkgroup
>>    netbios name = muserver
>>    netbios aliases = cups
>>    server string = print server
>>    printcap name = cups
>>    load printers = yes
>>    printing = cups
>>    printer admin = @admins
>>    invalid users = root
>>    log file = /var/log/samba/log.%m
>>    max log size = 1000
>>    syslog = 0
>>    security = ADS
>>    realm = MY.REALM
>>    password server = *
>>    encrypt passwords = true
>>    passdb backend = smbpasswd
>>    socket options = TCP_NODELAY
>>    preferred master = no
>>    announce version = 2.0
>>    wins support = no
>>    wins server = my.wins.server
>>    dns proxy = no
>>    preserve case = yes
>>    short preserve case = yes
>>    obey pam restrictions = yes
>> [printers]
>>    comment = All Printers
>>    path = /var/spool/samba
>>    browseable = no
>>    guest ok = no
>>    writable = no
>>    printable = yes
>> [print$]
>>    path = /var/lib/samba/printers
>>    browseable = yes
>>    read only = yes
>>    write list = @admins
>>    guest ok = yes
>> [drivers]
>>    path = /var/lib/samba/printers/drivers
>>    browseable = yes
>>    read only = yes
>>    write list = @admins
>>    guest ok = yes
>> # end smb.conf
> 
> While this may be your explicitely stored smb.conf, you'll see *all* the
> parameters used by Samba when running the testparm command with the
> -v option. (Samba uses some params silently).
> 
> Try
> 
>   testparm -vs | grep -E "lp|print|driver|cups"
> 
> to discover some of the values used by Samba for printing without you
> having specified them in your smb.conf.

For completeness:

# testparm -vs | grep -E "lp|print|driver|cups"
Load smb config files from /etc/samba/smb.conf
WARNING: The "printer admin" option is deprecated
Processing section "[printers]"
Processing section "[print$]"
Processing section "[drivers]"
Loaded services file OK.
Server role: ROLE_DOMAIN_MEMBER
        netbios aliases = cups
        server string = print server
        lpq cache time = 30
        load printers = Yes
        printcap cache time = 750
        printcap name = cups
        cups server =
        iprint server =
        addprinter command =
        deleteprinter command =
        show add printer wizard = Yes
        os2 driver map =
        printer admin = @admins
        min print space = 0
        max reported print jobs = 0
        max print jobs = 1000
        printable = No
        printing = cups
        cups options =
        print command =
        lpq command = %p
        lprm command =
        lppause command =
        lpresume command =
        printer name =
        use client driver = No
        force printername = No
[printers]
        printable = Yes
[print$]
        path = /var/lib/samba/printers
[drivers]
        path = /var/lib/samba/printers/drivers


> 
>> I think you'll agree, they're both pretty vanilla. 
> 
> I told you about the browse values above. Not vanilla. And you're saying
> you're using a VM. Is it VMWare? Which version? And you do indeed have
> the bandwidth required? The disk space?

I didn't realize those Browse values weren't standard.  They are
corrected now.  Yes, this is a VMWare VM.  The host is also Gentoo
Linux, running VMWare Server 1.0.3.44356.  This guest VM has a 10GB
(3.8GB free) virtual disk and 2GB RAM.  I'm using the e1000 NIC driver,
so it's at least simulating a GB copper connection (physical network is
GBe), but I haven't tested it to see what it is actually using.  The
printer network is on a separate segment from the rest of my network,
and on a separate NIC from the VM and host machine.

> 
>> I think I said in my
>> other response that I'm not trying to do anything outrageous here, just
>> basic network printing for Windows and Mac OSX clients via samba, and
>> Linux clients via remote cups.
>>
>> Any suggestions would be greatly appreciated.  I'll be happy to post
>> further data.  Just let me know what would be helpful.
>>
>> Thanks.
>> Seth
> 

Here's the latest version of the top part of my cupsd.conf:
LogLevel debug
SystemGroup admins
Listen foo:631
Listen localhost:631
Listen /var/run/cups/cups.sock
Browsing On
BrowseOrder allow,deny
BrowseAddress bar.255
BrowseAllow @LOCAL
DefaultAuthType Basic
DefaultEncryption Required
ErrorPolicy retry-job

Thanks for all your help.  I'll monitor it overnight and see if the
changed settings are helpful.

Seth