o /h-@sjdZddgZddlZddlZddlZddlmZmZmZddl m Z iZ GdddZ d d Z d d ZdS) zNew CSS Tokenizer (a generator) TokenizerCSSProductionsN)MACROS PRODUCTIONSr) normalizec@s~eZdZdZejejejejej ej dZ dZ e djZe djZddd Zd d Zd d ZddZddZdddZdS)rz^ generates a list of Token tuples: (Tokenname, value, startline, startcolumn) )z @font-facez@importz@mediaz @namespacez@pagez @variables z*\\[0-9a-fA-F]{1,6}(?:\r\n|[\t\r\n\f\x20])?z\\((\r\n)|[\n\r\f])NTc Cst|tr t|}n|}t||f}|tvr t|\}}}n*|s$t}|s(t}|| ||}dd|Dd}dd|Dd}|||ft|<||_ ||_ ||_ ||_ g|_dS)z inits tokenizer with given macros and productions which default to cssutils own macros and productions cS g|] }|ddkr|dqS)rCOMMENTr.0xr r F/var/www/html/myenv/lib/python3.10/site-packages/cssutils/tokenize2.py 5 z&Tokenizer.__init__..rcSr )rURIrr r r r rr6rN) isinstancedictsorteditemsstr_TOKENIZER_CACHErr_compile_productions_expand_macros tokenmatchescommentmatcher urimatcher _doComments_pushed) selfmacros productions doCommentsmacros_hash_keyhash_keyrrrr r r__init__!s*    zTokenizer.__init__csVfdd}g}|D]\}}td|r!td||}td|s|||fq |S)z@returns macro expanded productions, order of productions is keptcsd|dS)N(?:%s)macro) groupdict)mr!r r macro_valueCsz-Tokenizer._expand_macros..macro_valuez{[a-zA-Z][a-zA-Z0-9-]*}z"{(?P[a-zA-Z][a-zA-Z0-9-]*)})researchsubappend)r r!r"r,expandedkeyvaluer r+rr@s    zTokenizer._expand_macroscCs4g}|D]\}}||td|tjjfq|S)z>compile productions into callable match objects, order is keptr')r0r-compileUmatch)r expanded_productionscompiledr2r3r r rrOs  zTokenizer._compile_productionscGst||j|_dS)z:Push back tokens which have been pulled but not processed.N) itertoolschainr)r tokensr r rpushVszTokenizer.pushcCs g|_dSN)r)r r r rclearZs zTokenizer.clearFc #sddfdd}d}}d}jdjdd\}}} |||} | r:| d} || ||fV|t| 7}t||drVd} tj| ||fV|t| 7}|t| 7}t|} |} || krjEdH||}|d vrd |||fV|d7}|d7}n*| D]&\}}|r|d krt||d rd ||d}|} | rjrd |||fV| }n|||} | r| d} |dkr| dkr| dt|kr|| ddkrq|r!d|krt ||| rd| | d}} n.d|kr!d|| kr!dD]}||d|} |} | rd| d}} nq|dvr8 | }|dvr7d|}n5d|krkz j|| }Wn$tyjd| krft||t| drftj}| d7} nd}Ynw| }jszjs|d kr||||fV|t| 7}| j}||7}|rt| | jd}n|t| 7}nq|| usJ|| ksa|rdd||fVdSdS)a;Generator: Tokenize text and yield tokens, each token is a tuple of:: (name, value, line, col) The token value will contain a normal string, meaning CSS unicode escapes have been resolved to normal characters. The serializer escapes needed characters back to unicode escapes depending on the stylesheet target encoding. text to be tokenized fullsheet if ``True`` appends EOF token as last one and completes incomplete COMMENT or INVALID (to STRING) tokens cSs4t|dddd}|tjkrt|S|dS)zused by unicodesubrrN)intgroupsys maxunicodechr)r*numr r r_replos  z!Tokenizer.tokenize.._replcst|S)znormalize and do unicodesub)r unicodesub)r3rFr r r _normalizewsz&Tokenizer.tokenize.._normalizerrNz @charset z,:;{}>[]CHARz/*z%s*/r IDENTand(INVALIDSTRINGFUNCTIONzurl()z')z"))r) DIMENSIONrKrOrHASHr rPrNz UNICODE-RANGE)rOrN ATKEYWORDz@charset EOF)rrAlenhas_atr CHARSET_SYMrrrlowerend suffix_eqrrG cleanstring _atkeywordsKeyErrorcount_lineseprfind)r text fullsheetrIlinecolposBOMmatcherr"r6found _len_text _orig_textcnamepossiblecommentr\ possibleurir3nlsr rHrtokenize]s                          TozTokenizer.tokenize)NNT)F)__name__ __module__ __qualname____doc__r FONT_FACE_SYM IMPORT_SYM MEDIA_SYM NAMESPACE_SYMPAGE_SYM VARIABLES_SYMr_rbr-r4r/rGr^r&rrr<r>rsr r r rrs$   cCs|||t||kS)aCheck if text has substring string as position pos. This is effectively equal to ``text[pos:pos+len(string)] == string``. If ``pos`` is 0, this is equal to ``text.startswith(string)``. :param str text: The text to check against. :param int pos: The starting position where the substring should be searched. :param str string: The string to search. )rX)rdrhstringr r rrYs rYcsBskSttko tfddttDS)ayCheck if the suffix of text starting at pos equals expected. This is effectively equivalent to ``text[pos:] == expected``, but avoids allocating a huge substring. If ``pos`` is 0, this is equal to ``text == expected``. :param str text: Text to compare. :param int pos: Starting position for the suffix. :param str expected: String to compare to. c3s$|] }||kVqdSr=r )r iexpectedrhrdr r "s zsuffix_eq..)rXallrange)rdrhrr rrr]s  $ r])rw__all__r9r-rBcssproductionsrrrhelperrrrrYr]r r r rs w