[Open-FCoE] [fcoemon PATCH v2 00/11] rfcoemon restructuring

Multanen, Eric W eric.w.multanen at intel.com
Tue Dec 15 18:11:38 UTC 2009

>-----Original Message-----
>From: Shyam_Iyer at Dell.com [mailto:Shyam_Iyer at Dell.com]
>Sent: Tuesday, December 15, 2009 9:35 AM
>To: Multanen, Eric W; devel at open-fcoe.org
>Cc: open-iscsi at googlegroups.com
>Subject: RE: [Open-FCoE] [fcoemon PATCH v2 00/11] rfcoemon restructuring
>> -----Original Message-----
>> From: devel-bounces at open-fcoe.org [mailto:devel-bounces at open-fcoe.org]
>> On Behalf Of Eric Multanen
>> Sent: Tuesday, December 15, 2009 10:31 AM
>> To: devel at open-fcoe.org
>> Subject: [Open-FCoE] [fcoemon PATCH v2 00/11] rfcoemon restructuring
>> v2 - restore code in the service script which destroys FCoE interfaces
>> for
>> which DCB is not required when the service is stopped.
>> This patch set provides a fairly significant redesign of fcoemon.
>> fcoemon
>> was suffering from a number of issues, including being too reactive to
>> events.  This resulted in unstable behavior.  This note describes the
>> high
>> level functions of the redesigned fcoemon.  The following patches
>> provide the
>> implementation.
>> fcoemon will now have the following high level structure:
>> 1.  Read in FCoE interface configuration files.  This information is
>> used
>>     to create a list of FCoE interfaces.  As link and dcbd events
>>     occur and dcbd queries are resolved, the FCoE interface list will
>> manage
>>     the creation and destruction of FCoE interfaces.  Each FCoE
>> interface
>>     element in the list keeps track of its network interface (from the
>>     config file and could be a VLAN), its real network interface
>>     from link events during runtime), and its next action (create,
>> destroy,
>>     etc.).
>> 2.  As link events are received from the system, fcoemon determines
>> which ones
>>     are relevent for the FCoE interfaces.  Relevent link events are
>> those which
>>     occur on interfaces which are configured to be FCoE interfaces
>>     the FCoE interface list above), or on the underlying real network
>> interface,
>>     if the FCoE interface is configured on a VLAN.  A list of relevant
>> real
>>     network interfaces is maintained.  The list is used to track the
>>     operstate of the real network interfaces and, if DCB is required,
>>     the status of dcbd queries for the real network interface.
>> 3.  The core loop of fcoemon:
>> 	- listens for link events from rtnetlink and updates the state
>> the
>> 	  real network interface list. Link down events will terminate
>> any
>> 	  pending dcbd query sequences and a subsequent link up will
>> start the
>> 	  sequence over from the beginning.  Delete link events (VLAN
>> deleted,
>> 	  driver unloaded) will mark the FCoE interface for destruction.
>> 	- listens for dcbd events and response messages.  If the
>> 	  matches the current dcbd query state, then move to the next
>> state.
>> 	  If a dcbd event indicates a change in a DCB feature of
>> 	  then start the dcbd query sequence over.
>> 	- manages a timeout for maintaining a connection to dcbd.  If
>> dcbd
>> 	  service stops, this timeout will clean up as needed and re-
>> establish
>> 	  the dcbd connection once dcbd starts up again.
>> 	- at the end of each event/timeout cycle, after all link and
>> 	  and network interface events have been handled, perform any
>> 	  pending next actions.
>> 	  For network interface elements, that could mean sending the
>> next
>> 	  dcbd query, or, if the dcbd queries are complete, analyze the
>> 	  results and determine if the FCoE interface should be
>> 	  will now handle all FCoE interfaces whether or not DCB is
>> required.
>> 	  For FCoE interfaces, this means creating or destroying the
>> 	  interface.
>> 4.  All FCoE interfaces, whether DCB is required or not, will be
>> managed by
>>     fcoemon now.  Previously, the init.d start script would handle
>> configured
>>     FCoE interfaces which were configured without DCB required.
>> dcbd query sequence
>> ===================
>> When DCB is required for an FCoE interface, the real network device
>> performs a
>> series of dcbd queries to obtain the current DCB configuration.  The
>> sequence
>> is as follows:
>> 1.  DCB state - is DCB enabled on the interface?
>> 2.  PFC configuration (Priority Flow Control)
>> 3.  FCoE configuration
>> 4.  PFC operational state
>> 5.  FCoE operational state
>> If an error occurs at any step, or if all the steps are completed and
>> the
>> dcbd state does not match the conditions to either create or destroy
>> the
>> FCoE interface (see next sections), then a retry timer is started.
>> Once the
>> timer expires, the dcbd query sequence is reinitiated from the
>> beginning.  The
>> timer begins at 0.2 seconds and increases by multiples of that amount
>> up to
>> ten retries.  Once the max retries has been reached, without a
>> successful
>> completion of the dcbd query sequence, then the FCoE interface is
>> marked for
>> destruction.
>> The function of this retry mechanism is to protect against destroying
>> the
>> FCoE interface due to intermittent problems.  Such as when links go
>> down
>> momentarily due to configuration changes.  When this occurs, dcbd may
>> take
>> a couple seconds to resynchronize with the peer device.
>> Required conditions to create an FCoE interface
>> ===============================================
>> 1.  Link is up on the network interfaces required for the FCoE
>> interface, and
>> 2a. DCB is not required.
>> 	OR
>> 2b. DCB is required
>> 	AND DCB is enabled
>> 	AND PFC is enabled
>> 	AND App:FCoE is enabled
>> 	AND PFC and App:FCoE are operational
>> 	AND PFC and App:FCoE operational values match
>> Required conditions to destroy an FCoE interface
>> ================================================
>> 1.  The network interface required for the FCoE interface is removed
>> 	Such as driver is unloaded or VLAN interface is destroyed.
>> 	This is detected by DELLINK rtnetlink events.
>> 2.  The network interfaces required for the FCoE interface are up,
>> 	AND DCB is required,
>> 	AND on root network interface
>> 		DCB is disabled
>> 			OR
>> 		App:FCoE is disabled
>> 			OR
>> 		PFC and App:FCoE are operational
>> 		AND PFC and App:FCoE operational values mismatch
>> 3.  When DCB is required and the conditions to create the FCoE
>> interface are
>>     not satisfied after going through the maximum number of retry
>> sequences,
>>     as described in the "dcbd query sequence" section, then the FCoE
>> interface
>>     will be destroyed.
>> ---
>> Eric Multanen (11):
>>       Update version string of fcoemon.
>>       Remove FCoE interface management from service start script
>>       Add dcbd query timeout and retry mechanism
>>       Update the fcoemon state machine to be insensitive to
>> intermittent dcb changes
>>       Update the FCoE and network interface lists.
>>       Add separate arguments to specify FCoE and network interface
>>       Fix the print_errors() routine
>>       Remove extraneous dcb members from the network interface
>> structure.
>>       Fixup dcbd connection timeout code
>>       Remove unused dcbd routine.
>>       Obtain the real device of a VLAN interface using an ioctl.
>>  etc/initd/initd.fedora |   66 --
>>  etc/initd/initd.suse   |   37 -
>>  fcoemon.c              | 1504
>> --------
>>  fcoemon.h              |   67 +-
>>  fcoeplumb.in           |  121 ++--
>>  5 files changed, 789 insertions(+), 1006 deletions(-)
>> --
>> Signature: Eric Multanen <eric.w.multanen at intel.com>
>> _______________________________________________
>> devel mailing list
>> devel at open-fcoe.org
>> http://www.open-fcoe.org/mailman/listinfo/devel
>Shouldn't the requirement to administer DCB  be FCoE independent ? After
>all DCB is required for other protocols like iscsi as well.
>How about using a library interface to manage DCB over a network
>interface so that both open-iscsi and open-fcoe could administer DCB?

dcbd is an independent process and can be administered by client programs (e.g. dcbtool).
fcoemon is also a dcbd client, but currently only queries DCB configuration and status.  It
does not actively administer the DCB configuration.  open-iscsi could become a client of dcbd
as well if there is a need.

More information about the devel mailing list