You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
removeRepeatedCharacters is too aggressive with whitespaces #591
I use your code to scrape my own bank statements (via the python wrapper) and then feed them into a NER pipeline for analytics. As a result, whitespaces matter a bit to me for tokenisation.
Currently my credit card statements contain padding to line up some vendor details. These get removed entirely on the as-is package:
and i lose the word boundary.
if I up the removeRepeatedCharacters call (in TextChunk.groupByLines) from 3 to 20, I get the original padding that statement pdf intended.
I think going forward, it would be nice to either
make this minRunLength a parameter of sorts so people can choose for themselves
OR update TextChunk,squeeze to leave a single whitespace if all chars being squeezed are whitespaces just to not lose a word boundary as it's a little different to other spelling things.
Activity