Your IP : 172.28.240.42

Current Path : /usr/lib/python2.7/dist-packages/landscape/broker/
Upload File :
Current File : //usr/lib/python2.7/dist-packages/landscape/broker/store.pyc


Tc@sdZddlZddlZddlZddlZddlmZddlmZddl	m
Z
ddlmZm
Z
dZdZd	efd
YZdZdS(sMessage storage.

The sequencing system we use in the message store may be quite confusing
if you haven't looked at it in the last 10 minutes. For that reason, let's
review the mechanics here.

Our goal is to implement a reasonably robust system for delivering messages
from us to our peer. The system should be smart enough to recover if the peer
happens to lose messages that we have already sent, provided that these
messages are not too old (we'll see below what 'too old' means).

Messages added to the store are identified by increasing natural numbers, the
first message added is identified by 0, the second by 1, and so on. We call
"sequence" the number identifying the next message that we want to send. For
example, if the store has been added ten messages (that we represent with
uppercase letters) and we want start sending the first of them, our store
would like like::

    sequence: 0
    messages: A, B, C, D, E, F, G, H, I, J
              ^

The "^" marker is what we call "pending offset" and is the displacement of the
message we want to send next from the first message we have in the store.

Let's say we now send to our peer a batch of 3 sequential messages. In the
payload we include the body of the messages being sent and the sequence, which
identifies the first message of the batch. In this case the payload would look
like (pseudo-code)::

    (sequence: 0, messages: A, B, C)

If everything works fine on the other end, our peer replies with a payload that
would like::

    (next-expected-sequence: 4)

meaning that the peer has received all the three messages that we sent, and so
the next message it expects to receive is the one identified by the number 4.
At this point we update both our pending offset and our sequence values, and
the store now looks like::

    sequence: 4
    messages: A, B, C, D, E, F, G, H, I, J
                       ^

Great, now let's pretend that we send another batch, this time with five
messages::

    (sequence: 4, messages: D, E, F, G, H)

Our peer receives them fine responding with a payload looking like::

    (next-expected-sequence: 9)

meaning that it received all the eight messages we sent so far and it's waiting
for the ninth. This is the second successful batch that we send in a row, so we
can be reasonably confident that at least the messages in the first batch are
not really needed anymore. We delete them and we update our sequence and
pending offset accordingly::

    sequence: 9
    messages: D, E, F, G, H, I, J
                             ^

Note that we still want to keep around the messages we sent in the very last
batch, just in case. Indeed we now try to send a third batch with the last two
messages that we have, but our peer surprisingly replies us with this payload::

    (next-expected-sequence: 6)

Ouch! This means that something bad happened and our peer has somehow lost not
only the two messages that we sent in the last batch, but also the last three
messages of the former batch :(

Luckly we've kept enough old messages around that we can try to send them
again, we update our sequence and pending offset and the store looks like::

    sequence: 6
    messages: D, E, F, G, H, I, J
                    ^

We can now start again sending messages using the same strategy.

Note however that in the worst case scenario we could receive from our peer
a next-expected-sequence value which is so old to be outside our buffer
of already-sent messages. In that case there is now way we can recover the
lost messages, and we'll just send the oldest one that we have.

See L{MessageStore} for details about how messages are stored on the file
system and L{landscape.lib.message.got_next_expected} to check how the
strategy for updating the pending offset and the sequence is implemented.
iN(tDEFAULT_SERVER_API(tbpickle(tcreate_file(t
sort_versionstis_version_higherthtbtMessageStorecBseZdZeZddZdZdZdZdZ	dZ
dZd	Zd
Z
dZdZd
ZdZdZdZdZdZdZdZd+dZdZdZdZdZdZdZdZ dZ!dZ"d+dZ#d d!Z$d"Z%d#Z&d$Z'd%Z(d&Z)d'Z*d+d(Z+d)Z,d+d*Z-RS(,sRA message store which stores its messages in a file system hierarchy.

    Beside the "sequence" and the "pending offset" values described in the
    module docstring above, the L{MessageStore} also stores what we call
    "server sequence", which is the next message number expected by the
    *client* itself (because we are in turn the peer of a specular message
    system running in the server, which tries to deliver messages to us).

    The server sequence is entirely unrelated to the stored messages, but is
    incremented when successfully receiving messages from the server, in the
    very same way described above but with the roles inverted.

    @param persist: a L{Persist} used to save state parameters like the
        accepted message types, sequence, server uuid etc.
    @param directory: base of the file system hierarchy
    icCsh||_||_i|_||_|jd|_|j}tjj	|sdtj
|ndS(Ns
message-store(t
_directoryt_directory_sizet_schemast_original_persisttroot_att_persistt_message_dirtostpathtisdirtmakedirs(tselftpersistt	directorytdirectory_sizetmessage_dir((s:/usr/lib/python2.7/dist-packages/landscape/broker/store.pyt__init__s				cCs|jjdS(sPersist metadata to disk.N(Rtsave(R((s:/usr/lib/python2.7/dist-packages/landscape/broker/store.pytcommitscCsNt|tttfks!t|jjdtt||jdS(sSpecify the types of messages that the server will expect from us.

        If messages are added to the store which are not currently
        accepted, they will be saved but ignored until their type is
        accepted.
        saccepted-typesN(ttypettupletlisttsettAssertionErrorR
tsortedt_reprocess_holding(Rttypes((s:/usr/lib/python2.7/dist-packages/landscape/broker/store.pytset_accepted_typess!cCs|jjddS(s)Get a list of all accepted message types.saccepted-types((R
tget(R((s:/usr/lib/python2.7/dist-packages/landscape/broker/store.pytget_accepted_typesscCs||jkS(s>Return bool indicating if C{type} is an accepted message type.(R%(RR((s:/usr/lib/python2.7/dist-packages/landscape/broker/store.pytacceptsscCs|jjddS(sGet the current sequence.

        @return: The sequence number of the message that the server expects us
            to send on the next exchange.
        tsequencei(R
R$(R((s:/usr/lib/python2.7/dist-packages/landscape/broker/store.pytget_sequencescCs|jjd|dS(sSet the current sequence.

        Set the sequence number of the message that the server expects us to
        send on the next exchange.
        R'N(R
R(Rtnumber((s:/usr/lib/python2.7/dist-packages/landscape/broker/store.pytset_sequencescCs|jjddS(sGet the current server sequence.

        @return: the sequence number of the message that we will ask the server
            to send to us on the next exchange.
        tserver_sequencei(R
R$(R((s:/usr/lib/python2.7/dist-packages/landscape/broker/store.pytget_server_sequencescCs|jjd|dS(sSet the current server sequence.

        Set the sequence number of the message that we will ask the server to
        send to us on the next exchange.
        R+N(R
R(RR)((s:/usr/lib/python2.7/dist-packages/landscape/broker/store.pytset_server_sequencescCs|jjdS(s%Return the currently set server UUID.tserver_uuid(R
R$(R((s:/usr/lib/python2.7/dist-packages/landscape/broker/store.pytget_server_uuidscCs|jjd|dS(s=Change the known UUID from the server we're communicating to.R.N(R
R(Rtuuid((s:/usr/lib/python2.7/dist-packages/landscape/broker/store.pytset_server_uuidscCs|jjd|jS(sReturn the server API version that client has decided to speak.

        The client will always try to speak the highest server message API
        version that it knows and that the server declares to be capable
        of handling.
        t
server_api(R
R$t_api(R((s:/usr/lib/python2.7/dist-packages/landscape/broker/store.pytget_server_apiscCs|jjd|dS(sChange the server API version used to communicate with the server.

        All messages added to the store after calling this method will be
        tagged with the given server API version.
        R2N(R
R(RR2((s:/usr/lib/python2.7/dist-packages/landscape/broker/store.pytset_server_apiscCs|jjdS(s:Get the authentication token to use for the next exchange.texchange_token(R
R$(R((s:/usr/lib/python2.7/dist-packages/landscape/broker/store.pytget_exchange_tokenscCs|jjd|dS(s:Set the authentication token to use for the next exchange.R6N(R
R(Rttoken((s:/usr/lib/python2.7/dist-packages/landscape/broker/store.pytset_exchange_tokenscCs|jjddS(sGet the current pending offset.tpending_offseti(R
R$(R((s:/usr/lib/python2.7/dist-packages/landscape/broker/store.pytget_pending_offsetscCs|jjd|dS(sSet the current pending offset.

        Set the offset into the message pool to consider assigned to the
        current sequence number as returned by l{get_sequence}.
        R:N(R
R(Rtval((s:/usr/lib/python2.7/dist-packages/landscape/broker/store.pytset_pending_offsetscCs|j|j|dS(s/Increment the current pending offset by C{val}.N(R=R;(RR<((s:/usr/lib/python2.7/dist-packages/landscape/broker/store.pytadd_pending_offsetscCstd|jDS(s&Return the number of pending messages.css|]}dVqdS(iN((t.0tx((s:/usr/lib/python2.7/dist-packages/landscape/broker/store.pys	<genexpr>s(tsumt_walk_pending_messages(R((s:/usr/lib/python2.7/dist-packages/landscape/broker/store.pytcount_pending_messagesscCs	|j}|j}g}x|jD]}|dk	rSt||krSPn|j|j|}ytj|}Wn0t	k
r}t
j||j|t
q+X|d|k}	t||d}
|	s|
r|j|tq+|j|q+W|S(s;Get any pending messages that aren't being held, up to max.RtapiN(R%R4RBtNonetlent_get_contentRRtloadst
ValueErrortloggingt	exceptiont
_add_flagstBROKENRtHELDtappend(Rtmaxtaccepted_typesR2tmessagestfilenametdatatmessagetetunknown_typetunknown_api((s:/usr/lib/python2.7/dist-packages/landscape/broker/store.pytget_pending_messagess$
cCsyxrtj|jdtt|jD]H}tj|tjj	|d}tj
|s)tj|q)q)WdS(s>Delete messages which are unlikely to be needed in the future.texcludeiN(t	itertoolstislicet_walk_messagesRNRMR;RtunlinkRtsplittlistdirtrmdir(Rtfntcontaining_dir((s:/usr/lib/python2.7/dist-packages/landscape/broker/store.pytdelete_old_messagess
cCs5|jdx!|jD]}tj|qWdS(sRemove ALL stored messages.iN(R=R]RR^(RRS((s:/usr/lib/python2.7/dist-packages/landscape/broker/store.pytdelete_all_messagess
cCsA|jr|jn|j}|jj|ji}|||<dS(sAdd a schema to be applied to messages of the given type.

        The schema must be an instance of L{landscape.schema.Message}.
        N(RDR3R
t
setdefaultR(RtschemaRDtschemas((s:/usr/lib/python2.7/dist-packages/landscape/broker/store.pyt
add_schema#scCsd}|j}x|jdtD]n}|j|}t|ksR||krntj|j|krntSt|kr%t|kr%|d7}q%q%Wt	S(sReturn bool indicating if C{message_id} still hasn't been delivered.

        @param message_id: Identifier returned by the L{add()} method.
        iRZi(
R;R]RMt
_get_flagsRNRtstattst_inotTruetFalse(Rt
message_idtiR:RStflags((s:/usr/lib/python2.7/dist-packages/landscape/broker/store.pyt
is_pending,scCs$|jjd|jjddS(sRecord a successful exchange.sfirst-failure-timesblackhole-messagesN(R
tremove(Rt	timestamp((s:/usr/lib/python2.7/dist-packages/landscape/broker/store.pytrecord_success<scCs|jjds(|jjd|n||jjd}|jjdrTdS|dkr|jidd6|jjdttjd	ndS(
s
        Record a failed exchange, if all exchanges for the past week have
        failed then blackhole any future ones and request a full re-sync.
        sfirst-failure-timesblackhole-messagesNi<iit
resynchronizeRsaUnable to succesfully communicate with Landscape server for more than a week. Waiting for resync.iiQi:	(R
thasRR$taddRmRJtwarning(RRttcontinued_failure_time((s:/usr/lib/python2.7/dist-packages/landscape/broker/store.pytrecord_failureAs
c
Cs4d|kst|jjdr5tjddS|j}d|krZ||d<n|j|d}x7t|jD]#}t	||r~||}Pq~q~W|j
|}tj|}|j
}|d}t||tj|||j|ds|j|t}ntj|j}	|	S(s?Queue a message for delivery.

        @param message: a C{dict} with a C{type} key and other keys conforming
            to the L{Message} schema for that specific message type.

        @return: message_id, which is an identifier for the added
                 message or C{None} if the message was rejected.
        Rsblackhole-messagess!Dropped message, awaiting resync.NRDs.tmp(RR
R$RJtdebugR4R
RtkeysRtcoerceRtdumpst_get_next_message_filenameRRtrenameR&t
_set_flagsRNRkRl(
RRUR2RhRDRgtmessage_dataRSt	temp_pathRo((s:/usr/lib/python2.7/dist-packages/landscape/broker/store.pyRxUs,	




cCs|j}|r|d}ntj|jdd}|j|}|se|j|d}nt||jkrtt|djddd}|j||}nA|jtt|d}tj|tj	j
|d}|S(Nit0t_ii(t_get_sorted_filenamesRRRRFR	tstrtintR_Rtjoin(Rtmessage_dirst
newest_dirtmessage_filenamesRS((s:/usr/lib/python2.7/dist-packages/landscape/broker/store.pyRs
'
ccsQ|j}x>t|jdttD] \}}||kr)|Vq)q)WdS(s,Walk the files which are definitely pending.RZN(R;t	enumerateR]RNRM(RR:RpRS((s:/usr/lib/python2.7/dist-packages/landscape/broker/store.pyRBs
#ccs|rt|}n|j}xc|D][}xR|j|D]A}t|j|}|sk||@r>|j||Vq>q>Wq(WdS(N(RRRjR(RRZRRRSRq((s:/usr/lib/python2.7/dist-packages/landscape/broker/store.pyR]s
tcCsQgtj|j|D]}|jds|^q}|jdd|S(Ns.tmptkeycSst|jddS(NRi(RR_(R@((s:/usr/lib/python2.7/dist-packages/landscape/broker/store.pyt<lambda>s(RR`Rtendswithtsort(RtdirR@t
message_files((s:/usr/lib/python2.7/dist-packages/landscape/broker/store.pyRscGstjj|j|S(N(RRRR(Rtargs((s:/usr/lib/python2.7/dist-packages/landscape/broker/store.pyRscCs,t|}z|jSWd|jXdS(N(topentreadtclose(RRStfile((s:/usr/lib/python2.7/dist-packages/landscape/broker/store.pyRGsc
CsDd}|j}|j}x|jD]}|j|}ytj|j|}Wn9tk
r}tj	|t
|kr<|d7}q<q+X|d|k}t
|kr|r<|j}	tj
||	|j|	t|tt
q<q+|r2||kr2|j|t|tt
Bn|d7}q+WdS(se
        Unhold accepted messages left behind, and hold unaccepted
        pending messages.
        iiRN(R;R%R]RjRRHRGRIRJRKRNRRRRR(
RtoffsetR:RQtold_filenameRqRURVtacceptedtnew_filename((s:/usr/lib/python2.7/dist-packages/landscape/broker/store.pyR!s(
&#cCs3tjj|}d|kr/|jddSdS(NRiR(RRtbasenameR_(RRR((s:/usr/lib/python2.7/dist-packages/landscape/broker/store.pyRjscCsztjj|\}}tjj||jdd}|rf|ddjtt|7}ntj|||S(NRiR(RRR_RR RR(RRRqtdirnameRtnew_path((s:/usr/lib/python2.7/dist-packages/landscape/broker/store.pyRs"&cCs!|j||j||dS(N(RRj(RRRq((s:/usr/lib/python2.7/dist-packages/landscape/broker/store.pyRLscCsu|jjdi}x*|jD]\}}||kr"|Sq"Wttj}|||<|jjd||S(snGenerate a unique session identifier, persist it and return it.

        See also L{landscape.broker.server.BrokerServer.get_session_id} for
        more information on what this is used for.

        @param scope: A string identifying the scope of interest of requesting
            object. Currently this is unused but it has been implemented in
            preparation for a fix for bug #300278 so that we don't have to
            change the persisted structure later.  When that fix is in place
            this will allow us to re-synchronise only certain types of
            information, limited by scope.
        ssession-ids(R
R$t	iteritemsRR0tuuid4R(Rtscopetsession_idst
session_idtstored_scope((s:/usr/lib/python2.7/dist-packages/landscape/broker/store.pytget_session_ids

cCs||jjdikS(si
        Returns L{True} if the provided L{session_id} is known by this
        L{MessageStore}.
        ssession-ids(R
R$(RR((s:/usr/lib/python2.7/dist-packages/landscape/broker/store.pytis_valid_session_idscCsqi}|rZ|jjdi}x6|jD]%\}}||kr.|||<q.q.Wn|jjd|dS(sDrop all session ids.ssession-idsN(R
R$RR(Rtscopestnew_session_idsRRt
session_scope((s:/usr/lib/python2.7/dist-packages/landscape/broker/store.pytdrop_session_ids
sN(.t__name__t
__module__t__doc__RR3RRR#R%R&R(R*R,R-R/R1R4R5R7R9R;R=R>RCRERYRdReRiRrRuR{RxRRBR]RRRGR!RjRRLRRR(((s:/usr/lib/python2.7/dist-packages/landscape/broker/store.pyRnsT
																												0		
							cOsAddlm}t||}x|D]}|j|q&W|S(sP
    Get a L{MessageStore} object with all Landscape message schemas added.
    i(tmessage_schemas(tlandscape.message_schemasRRRi(RtkwargsRtstoreRg((s:/usr/lib/python2.7/dist-packages/landscape/broker/store.pytget_default_message_stores

(RR[RJRR0t	landscapeRt
landscape.libRtlandscape.lib.fsRtlandscape.lib.versioningRRRNRMtobjectRR(((s:/usr/lib/python2.7/dist-packages/landscape/broker/store.pyt<module>]s