Solaris CIFS Client

Internals

Gordon Ross
Project Lead
www.opensolaris.org/os/project/smbfs
May 2008
1

Solaris CIFS Client Components

nsmb netsmb driver
smbfs Loadable file system module
libsmbfs Used by CIFS client commands
smbutil(1) List shares, lookup names, etc.
mount_smbfs(1M) Mount a CIFS share
pam_smbfs_login(5) Save p/w hash at login
mdb(1)/kmdb(1) modules for nsmb and smbfs

Solaris CIFS Client Architecture

User Apps

mount_smbfs

smbutil

Data
libsmbfs

VFS layer

SMBFS

CIFS Server
NSMB

In-Kernel Data Structures

VFS

smbfs

vfs_t

smbmntinfo_t

netsmb smb_share_t (TID)

vnode_t

smbnode_t (FID)

smb_vc_t (UID)

network
4

Internals of nsmb Data Structures

nsmb data structures are a hierarchy,
the top of which is smb_vclist
A virtual circuit (VC) in nsmb is an object that
holds all state associated with a connection to
one server with one set of credentials (smb_vc_t)
Below VCs are share connections (smb_share_t)
Below shares are file handles, but those are not
managed by the nsmb driver (see smbfs)
Message data carried in STREAMS mblk_t chains.
Need ability to split and concatenate messages

Internals of nsmb MT Locking

Lock order is top-to-bottom in the hierarchy
see comments on locking in smb_conn.h
With each VC is a list of current requests,
and this uses a readers/writer lock
A blocking semaphore is used to serialize
sending of network data
In most cases we don't need to hold multiple
locks simultaneously
The code has many locking assertions; use a
debug build to test changes

Internals of nsmb Entry Points

ioctl calls from libsmbfs smb_dev.c:
nsmb_ioctl(), smb_usr_X()

Connection management smb_conn.c:

smb_sm_X(), smb_vc_X(), smb_share_X()

Request handling smb_rq.c:

smb_rq_init(), smb_rq_simple()
smb_rq_enqueue(), smb_rq_reply(),
smb_rq_done()

Packet building/parsing subr_mchain.c:

mb_init(), mb_put_X(), mb_get_X()
where X is uint8, uint16le, etc.

Internals of nsmb Build/Parse Request

Create requests

smb_rq_init() or smb_rq_alloc()

Mark start/end of word or byte sections

smb_rq_wstart(), smb_rq_wend(),
smb_rq_bstart(), smb_rq_bend()

Build body of each part by calling

mb_put_X(), where X is uint8, uint16le, etc.

Dispatch the request

smb_rq_simple() see next slide
Parse results with by calling
mb_get_X() functions
8

Internals of nsmb Dispatch a Request

1.Put the request on the list typical call stack:
smb_rq_simple / smb_rq_simple_timed /
smb_iod_addrq, or smb_iod_multirq

2.Send pending requests (if necessary)

smb_iod_sendall usually via _addrq

3.Wait for reply typical call stack:

smb_rq_simple / smb_rq_simple_timed /
smb_rq_reply / smb_iod_waitrq

In step 3, the I/O Daemon (IOD) does its work...

trans requests may send/wait multiple times,
and those stay in the list until fully complete.
9

Internals of nsmb I/O Daemon

The I/O Daemon (IOD) is a reader thread

started for each active VC, running this loop:
1. Wait for a packet and read it.
2. Parse the SMB header for the multiplex ID (MID),
which identifies the request this response belongs to.
3. Search the list of current requests for the one with the
matching XID. If found, link the response onto the list
of responses and signal the condition variable, which
continues the thread waiting for that response.

Responses with no matching request happen

after requesting thread wait timeout.

10

Request List Management

All requests go on the request list
Only the first (M) of them are sent, where
M is the negotiated max. multiplex value
Every thread adding or removing requests
checks whether send work is needed.
Normally, the thread adding a request
will do its own send.
Avoid sends from the IOD thread due to
deadlock concerns (send could block).
11

Internals of smbfs Data Structures

smbfs.h: smbmntinfo_t
FS-specific, per-mount
smbfs_node.h: smbnode_t
FS-specific, per-vnode
Node identity is: Path Name!
No NFS-style file handle
SMB file ID (FID) is held in smbnode_t,
currently shared (carefully) by user open
A lot like NFS, i.e. locking, zone support
12

Internals of smbfs Code Organization

Lots of files due to differing licenses,
tracking copies of originals
> smbfs_vfsops.c: VFS operations vector
> smbfs_vnops.c: Vnode operations vector
> smbfs_node.c: Node identity and attributes
> smbfs_subr2.c: Node hash (from NFS)
> smbfs_subr.c: Other miscellaneous helpers
> smbfs_rwlock.c: Reentrant read-write lock
> smbfs_io.c: Input/output shim layer
> smbfs_client.c: Zone code (from NFS)
> smbfs_smb.c: message building & parsing
13

Internals of smbfs Surprises

Single credentials per mount is unusual
All access uses the credentials of
the user who made the mount (!)
Owner, group, and modes are fake,
specified by client mount options
Server does most access checks,
client does unix mode checks
Some UNIX file names are illegal in CIFS
Other CIFS differences...
> No link, symlink, delete open file, ...
14

Internals of libsmbfs Data Structures

smb_ctx_t Handle per server/share
> Open FD into the netsmb driver
> Copy of ioctl data going to or from driver
> CIFS server address, etc.
> Authentication information

BSD-style mbuf simulation

> Used for building/parsing RAP messages, etc.
> Intentionally similar to in-kernel,

but not identical

15

Internals of libsmbfs Entry Points

Server or share context ctx.c:
Create a "handle"
> smb_ctx_readrc() defaults, SMF, .nsmbrc
> smb_ctx_resolve() Connect, authenticate
> smb_ctx_lookup() Does tree connnect (!)
Remote administration protocol (RAP)
> smb_ctx_init()

> rap.c: smb_rap_X()

List shares using RAP: netshareenum.c

Build & parse network messages:
> mbuf.c: mb_get_X(), mb_put_X()

where X is uint8, uint16le, ...

16

Internals of libsmbfs Authentication

smb_ctx_resolve() drives authentication,

but actions differ significantly by

authentication type:

> NTLM uses simple password hash, computed

in the driver during ioctl SMBIOC_NEGOTIATE
> With NTLM, the driver may be asked to get the

password from the keychain instead of using

one passed in See: smb_dev.h: SMBVOPT_*
> Kerberos, used with Active Directory (AD),
requires SPNEGO protocol (see: libsmbfs)
In this case, the driver treats the auth. data as
opaque and copies to or from the libsmbfs
See struct members: ioc_intok, ioc_outtok
17

Internals of smbfs Commands

cmd/fs.d/smbclnt/smbutil/view.c
> smbutil view Nice, simple example
> Uses the RAP functions in libsmbfs

cmd/fs.d/smbclnt/mount/mount.c
> Mount is almost as simple

Both basically do
>
>
>
>

smb_ctx_init()
smb_ctx_readrc()
smb_ctx_resolve()
smb_ctx_lookup()
18

mdb modules usage

Examine nsmb data structures:
(List all VCs, shares)
ADDR ::nsmb_rqlist (All requests on a VC)
::nsmb_pwtree
(List all p/w hashes)

> ::nsmb_vc -rv

>
>

Examine smbfs data structures:

> ::smbfs_vfs
> ::smbnode

(List all smbfs mounts)

(List all smbfs nodes)

Example outputs (next slide)

19

mdb modules sample output

$ mdb -k 0
> ::nsmb_vc -rv
// smb_vc_t uid server user
d71331c0 0 CHELUB cifs_user
d824f558 PUBLIC
> ::smbnode
// smbnode vnode rpath
d84d6b48 d85e5540 \file
e5056e20 dd88ac00 \

20

Handy dtrace Probes

Debug messages use dtrace probes
To see debug messages, use:
#!/usr/sbin/dtrace -s
sdt:nsmb::debugmsg2
{
printf("\n\t %s: %s", stringof(arg0), stringof(arg1));
}
sdt:nsmb::debugmsg3
{
printf("\n\t %s: %s", stringof(arg0), stringof(arg1));
tracemem(arg2, 16);
}

More dtrace scripts on our wiki

21

More Information
CIFS Client Project (smbfs) on OpenSolaris:
http://opensolaris/org/os/project/smbfs

External Wiki: (search for CIFS)

http://www.genunix.org/wiki

Mailing lists:
cifs-discuss@opensolaris.org

22