o n_0@sdZddlZddlZddlZddlmZddlmZmZm Z ddl m Z ddl m Z ddlmZdd lmZGd d d eZdS) a Module containing the UniversalDetector detector class, which is the primary class a user of ``chardet`` should use. :author: Mark Pilgrim (initial port to Python) :author: Shy Shalom (original C code) :author: Dan Blanchard (major refactoring for 3.0) :author: Ian Cordasco N)CharSetGroupProber) InputStateLanguageFilter ProbingState)EscCharSetProber) Latin1Prober)MBCSGroupProber)SBCSGroupProberc @sneZdZdZdZedZedZedZ dddd d d d d dZ e j fddZ ddZddZddZdS)UniversalDetectoraq The ``UniversalDetector`` class underlies the ``chardet.detect`` function and coordinates all of the different charset probers. To get a ``dict`` containing an encoding and its confidence, you can simply run: .. code:: u = UniversalDetector() u.feed(some_bytes) u.close() detected = u.result g?s[-]s(|~{)s[-]z Windows-1252z Windows-1250z Windows-1251z Windows-1256z Windows-1253z Windows-1255z Windows-1254z Windows-1257)z iso-8859-1z iso-8859-2z iso-8859-5z iso-8859-6z iso-8859-7z iso-8859-8z iso-8859-9z iso-8859-13cCsNd|_g|_d|_d|_d|_d|_d|_||_t t |_ d|_ | dS)N)_esc_charset_prober_charset_probersresultdone _got_data _input_state _last_char lang_filterlogging getLogger__name__logger_has_win_bytesreset)selfrr;/usr/lib/python3/dist-packages/chardet/universaldetector.py__init__Qs  zUniversalDetector.__init__cCsVdddd|_d|_d|_d|_tj|_d|_|jr|j |j D]}| q"dS)z Reset the UniversalDetector and all of its probers back to their initial states. This is called by ``__init__``, so you only need to call this directly in between analyses of different documents. Nencoding confidencelanguageF) rrrrr PURE_ASCIIrrr rr )rproberrrrr^s   zUniversalDetector.resetcCs6|jrdSt|s dSt|tst|}|jso|tjr%dddd|_n;|tj tj fr6dddd|_n*|drCdddd|_n|d rPd ddd|_n|tj tj fr`d ddd|_d |_|jd durod |_dS|j tjkr|j|rtj|_ n|j tjkr|j|j|rtj|_ |dd|_|j tjkr|jst|j|_|j|tjkr|jj|j|jjd|_d |_dSdS|j tjkr|jst |jg|_|jt!j"@r|j#t$|j#t%|jD]}||tjkr |j||jd|_d |_nq|j&|rd |_'dSdSdS)a Takes a chunk of a document and feeds it through all of the relevant charset probers. After calling ``feed``, you can check the value of the ``done`` attribute to see if you need to continue feeding the ``UniversalDetector`` more data, or if it has made a prediction (in the ``result`` attribute). .. note:: You should always call ``close`` when you're done feeding in your document if ``done`` is not already ``True``. Nz UTF-8-SIG?rzUTF-32szX-ISO-10646-UCS-4-3412szX-ISO-10646-UCS-4-2143zUTF-16Tr )(rlen isinstance bytearrayr startswithcodecsBOM_UTF8r BOM_UTF32_LE BOM_UTF32_BEBOM_LEBOM_BErrr$HIGH_BYTE_DETECTORsearch HIGH_BYTE ESC_DETECTORr ESC_ASCIIr rrfeedrFOUND_IT charset_nameget_confidencer"r r rNON_CJKappendr rWIN_BYTE_DETECTORr)rbyte_strr%rrrr8os                zUniversalDetector.feedc Csf|jr|jSd|_|js|jdnY|jtjkr!dddd|_nK|jtjkrld}d}d}|j D]}|s5q0| }||krA|}|}q0|rl||j krl|j }|j }| }|d rd|jrd|j||}|||jd|_|jtjkr|jd dur|jd |j D]+}|sqt|tr|jD]}|jd |j |j| qq|jd |j |j| q|jS) z Stop analyzing the current document and come up with a final prediction. :returns: The ``result`` attribute, a ``dict`` with the keys `encoding`, `confidence`, and `language`. Tzno data received!asciir&r'rNrziso-8859r z no probers hit minimum thresholdz%s %s confidence = %s)rrrrdebugrrr$r5r r;MINIMUM_THRESHOLDr:lowerr,r ISO_WIN_MAPgetr"getEffectiveLevelrDEBUGr*rprobers) rprober_confidencemax_prober_confidence max_proberr%r:lower_charset_namer! group_proberrrrclosesn          zUniversalDetector.closeN)r __module__ __qualname____doc__rBrecompiler3r6r>rDrALLrrr8rNrrrrr 3s&      mr )rQr-rrRcharsetgroupproberrenumsrrr escproberr latin1proberrmbcsgroupproberr sbcsgroupproberr objectr rrrrs