Re: IPSec SPD limit?

From: Ju Ichi (freebsd-security@ichi.net)
Date: 08/26/02


From: Ju Ichi <freebsd-security@ichi.net>
To: "Sam Leffler (at Usenix)" <sam@usenix.org>, <freebsd-security@FreeBSD.ORG>
Date: Mon, 26 Aug 2002 12:59:15 -0400

On Friday 23 August 2002 7:49 pm, Sam Leffler (at Usenix) wrote:
> > We are trying to setup a large IPSec SPD (in excess of 1000 SAs) on the
> > following hardware/software config:
> >
> > Compaq DL360 with dual 1.4GHz processsors
> > 2GB RAM
> > 4GB swap space
> >
> > 4.6.1-RELEASE-p11
> > racoon-20020507a
> >
> > We get a "send: No buffer space available" when trying to read in the
> > /etc/ipsec.conf file if it has more than about 1000 entries. Also, if we
> do
> > a setkey -DP after trying to read in /etc/ipsec.conf we get
> > "recv: Resource temporarily unavailable" after it lists some of the SAs.
> >
> > Several kernel tweaks have been tried. For example, we have tried setting
> > MAXUSERS from 0 to 1024 on bit boundaries (0, 128, 256, 512, and 1024).
> > FWIW, setting it to 1024 seems to be evil. ;-) We have also tried
> various
> > settings in the kernel config file on NMBCLUSTERS, NMBUFS, NBUF, MAXDSIZ,
> > MAXSSIZ, DFLDSIZ, and MAXFILES. In addition, we have tweaked
> > kern.ipc.somaxconn, net.inet.tcp.sendspace, net.inet.tcp.recvspace,
> > net.inet.udp.recvspace, and net.inet.udp.maxdgram after reading some
> > performance tuning web pages. I can provide additional details as needed,
> > but didn't want to make this initial request too long.
> >
> > Does anyone know of any limits on the number of entries the SPD can hold
> and
> > if so how to make the limits higher?
> >
>
> setkey -DP returns the SA's via a PF_KEY socket. You're hitting the upper
> bound on the amount of data that can be stored in a socket of this type as
> all the data is returned en masse (i.e. the process isn't given an
> opportunity to read data). PF_KEY sockets inherit the send+recv space
> reserves of raw sockets:
>
> ebb# gdb -k /kernel /dev/mem
> ...
> (kgdb) p raw_sendspace
> $1 = 8192
> (kgdb) p raw_recvspace
> $2 = 8192
>
> You can either work around this by upping these values or patch the PF_KEY
> code to set the reserves on the socket explicitly (and provide sysctl's a la
> udp+tcp to control the upper bounds).
>
> Sam

Thanks! I changed /usr/src/sys/net/raw_cb.h as follows:

        < #define RAWSNDQ 8192
        < #define RAWRCVQ 8192
        ---
> #define RAWSNDQ 65535
> #define RAWRCVQ 65535

So, now we have:

        (kgdb) p raw_sendspace
        $1 = 65535
        (kgdb) p raw_recvspace
        $2 = 65535

We are able to get the policy loaded by using "setkey -c" with sleep
statements as Nate suggested, but still are getting "recv: Resource
temporarily unavailable" when doing a setkey -DP. Anymore ideas on other
values to up?

Also, Nate, do you know of a way to dump the poicy with setkey so it all
shows? In other words, using setkey -c we can slow down the rate of putting
entries in, but there doesn't seem to be a way to slow down the rate at which
the policy is dumped.

Thanks,
Ju

To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-security" in the body of the message