jellyfin-kodi/resources/lib/libraries/requests/packages/chardet/hebrewprober.py

284 lines
13 KiB
Python
Raw Normal View History

Version 4.0.0 (#182) * Adjust refresh behavior * Fix favorites * Add option to mask info * Fix keymap delete * Fix empty show * Version bump 3.1.14 * Reset rescan flag * Fix subtitles encoding * Fix path verification * Fix update library Plug in remove library percentage * Fix unauthorized behavior Reprompt user with login * Fix series pooling * Version bump 3.1.15 * Fix for additional users Return all users, not just public users * Fix http potential errors Prevent from going further if {server} or {userid} is requested but not filled to avoid 401 errors * Fix extra fanart * Fix patch make a case insensitive search * Version bump 3.1.16 Additional logging, fix kodi source. * Fix library tags on update * Version bump 3.1.17 * Fix season artwork * Fix season artwork * Fix logging * Fix blank files sources * Add backup option * Fix userdata song * Transfer data.txt to data.json Use default port for webserver caching * Fix mixed content shortcut * Fix path encoding for patch Hopefully this works... * Fix source nonetype error Just incase, wrap in a try/except because it's not important. * Base fast sync on server time Try to fix music video refresh to prevent cursor from moving up. * Prep subfolders for dynamic Support homevideos for now * Fix empty artist, missing Title * Version bump 3.1.18a * Version bump for objects 171076013 * Notify user of large updates Give option to back out if the user wants to manually update the libraries * Fix sources.xml verification * Prevent error in monitor Put in place try/except in case data is None * Remember sync position for manual triggers Allow to resume sync on restart for manual user triggers (update, repair). Automatically refresh boxsets if movie library is selected. use waitForAbort and emby_should_stop prop to terminate threads * Update string for sync later * Add subfolders for dynamic movies * Small fixes * Version bump 3.1.19 * Fix fast sync try/except, default back to previous behavior. * Fix artwork * Change settings name To ensure it takes default value instead of previous value set in 3.0.34 * Fix transcode flac and live tv * Fix episodes for series pooling * Add live tv support * Version bump 3.1.20 * Revert "Small fixes" This reverts commit 9ec1fa35853352bb6c8a56da94e8c8bcc3843c07. * Version bump 3.1.21 * Fix playback starting server connection instance * Fix show update * Fix boxsets * Fix lastplayed * Patch to support pre 3.6 libraries * Fix slowness * Plug in settings for threading * Plug in settings for threading * Adjust sleep behavior * Version bump 3.1.22 * Fix server detection in monitor * Version bump 3.1.23 * Fix potential error with checksum * Fix missing new artists * Fix library sync Adjust lock, re-add screensaver deactivated during sync, prep compare sync, stop library updates from being processed before startup sync is completed * Version bump 3.1.25 * Fix local trailers * Adjust lock modification * Check db version * Prevent error from creating nodes The addon automatically creates nodes at startup with prefilled information. Prevent errors in the event something goes wrong. It will fix itself down the line, after user has logged in. * Version bump 3.1.26 * Revert "Version bump 3.1.26" This reverts commit c583a69a4b9bb8af160cfb564485cc59da20fca7. * Fix screensaver toggle * Fix source selection for direct stream * Version bump 3.1.26 * Add progress for updates * Revise progress bar Fix typos and subsetting * content notification * Remove content with update library Now remove irrelevant content as well * Fix slowness * Version bump 3.1.27 * Stop trying to get items if server offline * Fix content type for dynamic music * Fix resume sync Now save progress, unless exited due to path validation * Fix artwork for shortcuts on profile switch * Add force transcode settings * Fix audiobooks back to video type Add shortcuts. Audiobook can't be music type otherwise it break resume behavior and it won't play the right item. Has to be video type. * Update general info To finish, download and installation * Update README.md * Move welcome message to service * Prevent patch loop Try once, then let it go, to avoid locking user in a restart loop * Review library threads * Prep for audiobook transcode Still need to implement universal for audio transcode * Version bump 3.1.28 * Fix emby database locked * Fix regression to welcome message * Version bump 3.1.29 * Adjust playback Allow direct play for http streams * Ensure all threads are terminated correctly * Fix empty results due to error 500 * Fix boxset refresh * Fix resume sync behavior Allow to complete the startup sync in the event user backs out of resume sync * Version bump 3.1.30 * Update patch Move patch from cache to addon_data. No longer need to restart Kodi to apply the first patch. * Fix inital sync leading to fast sync * Fix user settings Due to api change in 3.6.0.55 * krypton update * Adjust for resume settings With .55 the resume setting is set per library. Instead query server to see if the item is played to offer delete * Restart service upon check for updates To reload the new objects module. * Fix update library Only do the compare when user selects update library, also add a restart service option in the add-on settings > advanced * Version bump 3.1.31 * Update dependencies * Update FR translation * Update DE translation * Add translation * Support up next * Small service adjustment * Krypton update to support upnext * Add a verification onwake Somehow, Kodi can trigger OnWake without first trigger OnSleep. * Fix loading if special char in path * Add logging and small fixes Prepare userdata by date modified * Version bump 3.1.32 * Change default behavior of startup dialog In case it is forced closed by Kodi, allow the sync to proceed * Ensure deliveryurl is an actual url * Update README.md * Fix nextup * Fix dynamic widgets * Detect coreelect, etc * Fix progress report Silent RefreshProgress in websocket * Follow emby settings for subtitles * Version bump 3.1.33 * Add Italian translation * Fix playback for server 3.6.0.61 * Version bump 3.1.34a * Add silent catch for errors * Adjust playback progress monitor Only track progress report if the item is an emby item * Fix subtitles not following server settings * Add remove libraries, fix mixed libraries * Fix live tv For now, use transcode since direct play returns a 127.0.0.1 unusable address as the path. * Allow live tv to direct stream * Fix LiveTV * Add setting to sync during playback * Fix updates * Fix encoding error * Add optional rotten tomatoes option * Version bump 3.1.35 * Fix emby connect auth string Was preventing proper device detection when using emby connect, play to, etc. * Add setup RT * Fix audio/sub change Only for addon playback * Add developer mode * Update patch Check for updates + dev = forced grab from github * Fix RT string * Fix patch Allow dev mode to redownload zip * Fix patch ugh sleep!! * Verify patch connection * Version bump 3.1.36 * Fix libraries being wiped Catch errors to prevent false positive * Add dateutil library * Prep convert to local time * Fix string * Prep for multi db version support * Fix service restart * Add shortcut restart addon Add notification * Add database discovery * Ensure previous playback terminated * Update translation New: Polish, Dutch Updated: German, French, Italian * Version bump 3.1.37 * Quick fix for new library dateutil * Catch error for dateutil In the event the server has some weird date that can't be converted * Version bump 3.1.38 * Fix dateutil import * Fix db discovery Ignore emby.db * Version bump 3.1.39 * Add a delay if setup not completed Avoid crash from everything loading at once. * Fix database discovery Add table verification + date modified verification * Container optional playutils * Version bump 3.1.40 * Adjust database discovery Compare loaded vs discovered to avoid loading old databases by accident. * Version bump 3.1.41 * Fix discovery toggle * Version bump 3.1.42 * Add webservice for playback prep * Fix service restart * Version bump 3.1.43 * Update default sync indicator Based on overall feedback * Fix check update * Fix if server is selected but unavailable * Support songs without albums * Fix encode and params * Increase retry timeout * Fix update generating duplicates * Add manage libraries Too many entries * Fix database discovery * Fixed transcode via context menu * Fix context transcode * Quiet webservice * Update Krypton objects * Fix database discovery prompt * fixed video listitem issues for krypton * load all item details for playlists * Fix playlist * Version bump 3.1.44 * Fix force hi10p transcoding behavior Fixes the "Force Hi10p transcoding" option to only apply to h264 video codecs * Clear playlist on player.onstop * Don't clear playlist if busy spinner is active * Fix case sensitive issue at calling the log function * fix db stuff (#164) * Reload objects upon initial setup * Fix database discovery ignore db-journal * Update translation German, Italian * Use LastConnectionMode for server test * Fix compare sync * Version bump 3.1.45 * Ensure widgets get updated Container.Refresh alone doesn't seem to work * Update database discovery * Re-add texture to database discovery * Add option to enable/disable service * Remove unused strings * Fix object reload upon restart service * Update Krypton objects * Update translation Dutch, Polish * Version bump 3.1.46 * Adjust client api * Adjust subtitles behavior * Fix string typo * Only run one full sync instance Prevent user from launching multiple syncs and freezing the add-on. * added "playlists" to wnodes * Disable Audiobooks Server doesn't have a set structure yet. This feature is broken atm. * Version bump 4.0.0 * License GPL v3 * Update readme
2019-01-24 13:04:48 +00:00
######################## BEGIN LICENSE BLOCK ########################
# The Original Code is Mozilla Universal charset detector code.
#
# The Initial Developer of the Original Code is
# Shy Shalom
# Portions created by the Initial Developer are Copyright (C) 2005
# the Initial Developer. All Rights Reserved.
#
# Contributor(s):
# Mark Pilgrim - port to Python
#
# This library is free software; you can redistribute it and/or
# modify it under the terms of the GNU Lesser General Public
# License as published by the Free Software Foundation; either
# version 2.1 of the License, or (at your option) any later version.
#
# This library is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public
# License along with this library; if not, write to the Free Software
# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA
# 02110-1301 USA
######################### END LICENSE BLOCK #########################
from .charsetprober import CharSetProber
from .constants import eNotMe, eDetecting
from .compat import wrap_ord
# This prober doesn't actually recognize a language or a charset.
# It is a helper prober for the use of the Hebrew model probers
### General ideas of the Hebrew charset recognition ###
#
# Four main charsets exist in Hebrew:
# "ISO-8859-8" - Visual Hebrew
# "windows-1255" - Logical Hebrew
# "ISO-8859-8-I" - Logical Hebrew
# "x-mac-hebrew" - ?? Logical Hebrew ??
#
# Both "ISO" charsets use a completely identical set of code points, whereas
# "windows-1255" and "x-mac-hebrew" are two different proper supersets of
# these code points. windows-1255 defines additional characters in the range
# 0x80-0x9F as some misc punctuation marks as well as some Hebrew-specific
# diacritics and additional 'Yiddish' ligature letters in the range 0xc0-0xd6.
# x-mac-hebrew defines similar additional code points but with a different
# mapping.
#
# As far as an average Hebrew text with no diacritics is concerned, all four
# charsets are identical with respect to code points. Meaning that for the
# main Hebrew alphabet, all four map the same values to all 27 Hebrew letters
# (including final letters).
#
# The dominant difference between these charsets is their directionality.
# "Visual" directionality means that the text is ordered as if the renderer is
# not aware of a BIDI rendering algorithm. The renderer sees the text and
# draws it from left to right. The text itself when ordered naturally is read
# backwards. A buffer of Visual Hebrew generally looks like so:
# "[last word of first line spelled backwards] [whole line ordered backwards
# and spelled backwards] [first word of first line spelled backwards]
# [end of line] [last word of second line] ... etc' "
# adding punctuation marks, numbers and English text to visual text is
# naturally also "visual" and from left to right.
#
# "Logical" directionality means the text is ordered "naturally" according to
# the order it is read. It is the responsibility of the renderer to display
# the text from right to left. A BIDI algorithm is used to place general
# punctuation marks, numbers and English text in the text.
#
# Texts in x-mac-hebrew are almost impossible to find on the Internet. From
# what little evidence I could find, it seems that its general directionality
# is Logical.
#
# To sum up all of the above, the Hebrew probing mechanism knows about two
# charsets:
# Visual Hebrew - "ISO-8859-8" - backwards text - Words and sentences are
# backwards while line order is natural. For charset recognition purposes
# the line order is unimportant (In fact, for this implementation, even
# word order is unimportant).
# Logical Hebrew - "windows-1255" - normal, naturally ordered text.
#
# "ISO-8859-8-I" is a subset of windows-1255 and doesn't need to be
# specifically identified.
# "x-mac-hebrew" is also identified as windows-1255. A text in x-mac-hebrew
# that contain special punctuation marks or diacritics is displayed with
# some unconverted characters showing as question marks. This problem might
# be corrected using another model prober for x-mac-hebrew. Due to the fact
# that x-mac-hebrew texts are so rare, writing another model prober isn't
# worth the effort and performance hit.
#
#### The Prober ####
#
# The prober is divided between two SBCharSetProbers and a HebrewProber,
# all of which are managed, created, fed data, inquired and deleted by the
# SBCSGroupProber. The two SBCharSetProbers identify that the text is in
# fact some kind of Hebrew, Logical or Visual. The final decision about which
# one is it is made by the HebrewProber by combining final-letter scores
# with the scores of the two SBCharSetProbers to produce a final answer.
#
# The SBCSGroupProber is responsible for stripping the original text of HTML
# tags, English characters, numbers, low-ASCII punctuation characters, spaces
# and new lines. It reduces any sequence of such characters to a single space.
# The buffer fed to each prober in the SBCS group prober is pure text in
# high-ASCII.
# The two SBCharSetProbers (model probers) share the same language model:
# Win1255Model.
# The first SBCharSetProber uses the model normally as any other
# SBCharSetProber does, to recognize windows-1255, upon which this model was
# built. The second SBCharSetProber is told to make the pair-of-letter
# lookup in the language model backwards. This in practice exactly simulates
# a visual Hebrew model using the windows-1255 logical Hebrew model.
#
# The HebrewProber is not using any language model. All it does is look for
# final-letter evidence suggesting the text is either logical Hebrew or visual
# Hebrew. Disjointed from the model probers, the results of the HebrewProber
# alone are meaningless. HebrewProber always returns 0.00 as confidence
# since it never identifies a charset by itself. Instead, the pointer to the
# HebrewProber is passed to the model probers as a helper "Name Prober".
# When the Group prober receives a positive identification from any prober,
# it asks for the name of the charset identified. If the prober queried is a
# Hebrew model prober, the model prober forwards the call to the
# HebrewProber to make the final decision. In the HebrewProber, the
# decision is made according to the final-letters scores maintained and Both
# model probers scores. The answer is returned in the form of the name of the
# charset identified, either "windows-1255" or "ISO-8859-8".
# windows-1255 / ISO-8859-8 code points of interest
FINAL_KAF = 0xea
NORMAL_KAF = 0xeb
FINAL_MEM = 0xed
NORMAL_MEM = 0xee
FINAL_NUN = 0xef
NORMAL_NUN = 0xf0
FINAL_PE = 0xf3
NORMAL_PE = 0xf4
FINAL_TSADI = 0xf5
NORMAL_TSADI = 0xf6
# Minimum Visual vs Logical final letter score difference.
# If the difference is below this, don't rely solely on the final letter score
# distance.
MIN_FINAL_CHAR_DISTANCE = 5
# Minimum Visual vs Logical model score difference.
# If the difference is below this, don't rely at all on the model score
# distance.
MIN_MODEL_DISTANCE = 0.01
VISUAL_HEBREW_NAME = "ISO-8859-8"
LOGICAL_HEBREW_NAME = "windows-1255"
class HebrewProber(CharSetProber):
def __init__(self):
CharSetProber.__init__(self)
self._mLogicalProber = None
self._mVisualProber = None
self.reset()
def reset(self):
self._mFinalCharLogicalScore = 0
self._mFinalCharVisualScore = 0
# The two last characters seen in the previous buffer,
# mPrev and mBeforePrev are initialized to space in order to simulate
# a word delimiter at the beginning of the data
self._mPrev = ' '
self._mBeforePrev = ' '
# These probers are owned by the group prober.
def set_model_probers(self, logicalProber, visualProber):
self._mLogicalProber = logicalProber
self._mVisualProber = visualProber
def is_final(self, c):
return wrap_ord(c) in [FINAL_KAF, FINAL_MEM, FINAL_NUN, FINAL_PE,
FINAL_TSADI]
def is_non_final(self, c):
# The normal Tsadi is not a good Non-Final letter due to words like
# 'lechotet' (to chat) containing an apostrophe after the tsadi. This
# apostrophe is converted to a space in FilterWithoutEnglishLetters
# causing the Non-Final tsadi to appear at an end of a word even
# though this is not the case in the original text.
# The letters Pe and Kaf rarely display a related behavior of not being
# a good Non-Final letter. Words like 'Pop', 'Winamp' and 'Mubarak'
# for example legally end with a Non-Final Pe or Kaf. However, the
# benefit of these letters as Non-Final letters outweighs the damage
# since these words are quite rare.
return wrap_ord(c) in [NORMAL_KAF, NORMAL_MEM, NORMAL_NUN, NORMAL_PE]
def feed(self, aBuf):
# Final letter analysis for logical-visual decision.
# Look for evidence that the received buffer is either logical Hebrew
# or visual Hebrew.
# The following cases are checked:
# 1) A word longer than 1 letter, ending with a final letter. This is
# an indication that the text is laid out "naturally" since the
# final letter really appears at the end. +1 for logical score.
# 2) A word longer than 1 letter, ending with a Non-Final letter. In
# normal Hebrew, words ending with Kaf, Mem, Nun, Pe or Tsadi,
# should not end with the Non-Final form of that letter. Exceptions
# to this rule are mentioned above in isNonFinal(). This is an
# indication that the text is laid out backwards. +1 for visual
# score
# 3) A word longer than 1 letter, starting with a final letter. Final
# letters should not appear at the beginning of a word. This is an
# indication that the text is laid out backwards. +1 for visual
# score.
#
# The visual score and logical score are accumulated throughout the
# text and are finally checked against each other in GetCharSetName().
# No checking for final letters in the middle of words is done since
# that case is not an indication for either Logical or Visual text.
#
# We automatically filter out all 7-bit characters (replace them with
# spaces) so the word boundary detection works properly. [MAP]
if self.get_state() == eNotMe:
# Both model probers say it's not them. No reason to continue.
return eNotMe
aBuf = self.filter_high_bit_only(aBuf)
for cur in aBuf:
if cur == ' ':
# We stand on a space - a word just ended
if self._mBeforePrev != ' ':
# next-to-last char was not a space so self._mPrev is not a
# 1 letter word
if self.is_final(self._mPrev):
# case (1) [-2:not space][-1:final letter][cur:space]
self._mFinalCharLogicalScore += 1
elif self.is_non_final(self._mPrev):
# case (2) [-2:not space][-1:Non-Final letter][
# cur:space]
self._mFinalCharVisualScore += 1
else:
# Not standing on a space
if ((self._mBeforePrev == ' ') and
(self.is_final(self._mPrev)) and (cur != ' ')):
# case (3) [-2:space][-1:final letter][cur:not space]
self._mFinalCharVisualScore += 1
self._mBeforePrev = self._mPrev
self._mPrev = cur
# Forever detecting, till the end or until both model probers return
# eNotMe (handled above)
return eDetecting
def get_charset_name(self):
# Make the decision: is it Logical or Visual?
# If the final letter score distance is dominant enough, rely on it.
finalsub = self._mFinalCharLogicalScore - self._mFinalCharVisualScore
if finalsub >= MIN_FINAL_CHAR_DISTANCE:
return LOGICAL_HEBREW_NAME
if finalsub <= -MIN_FINAL_CHAR_DISTANCE:
return VISUAL_HEBREW_NAME
# It's not dominant enough, try to rely on the model scores instead.
modelsub = (self._mLogicalProber.get_confidence()
- self._mVisualProber.get_confidence())
if modelsub > MIN_MODEL_DISTANCE:
return LOGICAL_HEBREW_NAME
if modelsub < -MIN_MODEL_DISTANCE:
return VISUAL_HEBREW_NAME
# Still no good, back to final letter distance, maybe it'll save the
# day.
if finalsub < 0.0:
return VISUAL_HEBREW_NAME
# (finalsub > 0 - Logical) or (don't know what to do) default to
# Logical.
return LOGICAL_HEBREW_NAME
def get_state(self):
# Remain active as long as any of the model probers are active.
if (self._mLogicalProber.get_state() == eNotMe) and \
(self._mVisualProber.get_state() == eNotMe):
return eNotMe
return eDetecting