Commits

Breadcrumbs

History for

onmaster

User selector

All users

All time

Commit History

Commits on Mar 13, 2021

• Add tokenisation support for the Tetun language

  raphaelmerx

  committed
  75d4c67

  Copy full SHA for 75d4c67

Commits on Jul 31, 2020

• adding rules for Catalan

  Show description for 8d78dae
  cristinae

  authored
  8d78dae

  Copy full SHA for 8d78dae

Commits on Feb 20, 2020

• Revert "line buffering for tokeniser and truecaser"

  Show description for 22923dd
  kpu

  committed
  22923dd

  Copy full SHA for 22923dd

Commits on Feb 17, 2020

• line buffering for tokeniser and truecaser

  wwaites

  committed
  691717c

  Copy full SHA for 691717c

Commits on Nov 7, 2018

• tokenizer.perl: split final dots unconditionally

  Show description for 9fc964d
  ozancaglayan

  committed
  9fc964d

  Copy full SHA for 9fc964d

Commits on Sep 6, 2018

• Handle glottal stops in Somalian

  bhaddow

  committed
  06f519d

  Copy full SHA for 06f519d

Commits on Feb 14, 2018

• add fi/sv-specific colon handling in tokenizer.perl

  

  Scherrer Yves

  committed
  4a7f16b

  Copy full SHA for 4a7f16b

Commits on Dec 23, 2016

• Merge pull request #168 from tofula/master

  Show description for 59119c0
  hieuhoang

  authored
  59119c0

  Copy full SHA for 59119c0

Commits on Jul 31, 2016

• Separate comma after a number end sentence

  Show description for d04bdc7
  a455bcd9

  authored
  d04bdc7

  Copy full SHA for d04bdc7

Commits on Feb 23, 2016

• .' at end of sentence is missed

  jimregan

  committed
  5d5bf18

  Copy full SHA for 5d5bf18

Commits on Oct 12, 2015

• Named group added for the safer 'protected patterns' recognition regexp.

  Show description for dd9eb54
  tofula

  committed
  dd9eb54

  Copy full SHA for dd9eb54

Commits on Sep 23, 2015

• ga (mostly) behaves more like fr/it

  jimregan

  committed
  36e4951

  Copy full SHA for 36e4951

Commits on May 29, 2015

• Add license notices to scripts.

  Show description for ef02844
  

  Jeroen Vermeulen

  committed
  ef02844

  Copy full SHA for ef02844

Commits on May 17, 2015

• Fix a lot of lint, mostly trailing whitespace.

  Show description for a25193c
  

  Jeroen Vermeulen

  committed
  a25193c

  Copy full SHA for a25193c

Commits on Apr 13, 2015

• add use warnings to all perl scripts

  hieuhoang

  committed
  6162223

  Copy full SHA for 6162223

Commits on Apr 2, 2015

• consistently use 'env perl' command for environments where the 1st perl in PATH isn't the default perl. Which is kinda stupid

  hieuhoang

  committed
  2d1da32

  Copy full SHA for 2d1da32

• Conditional import of Thread package for perl installations that don't support threads

  

  Hieu Hoang

  committed
  e76247e

  Copy full SHA for e76247e

Commits on Jan 29, 2015

• "just put it in. I'll verify it if i can be bovvered" --Hieu /usr/bin/env

  kpu

  committed
  925565a

  Copy full SHA for 925565a

• Revert "env perl shebang"

  Show description for 449d9b2
  

  Matthias Huck

  committed
  449d9b2

  Copy full SHA for 449d9b2

Commits on Jan 27, 2015

• env perl shebang

  kpu

  committed
  34f2801

  Copy full SHA for 34f2801

Commits on Jan 16, 2015

• move normalisation of quotes into normalize-punctuation.perl /Tom Hoar

  hieuhoang

  committed
  19d7c44

  Copy full SHA for 19d7c44

Commits on Aug 3, 2014

• changes to protecting specified patterns (with example patterns)

  phikoehn

  committed
  7f41bbb

  Copy full SHA for 7f41bbb

Commits on Feb 21, 2014

• - added option -no-escape to skip escaping of special characters

  Show description for d3d14cc
  ugermann

  committed
  d3d14cc

  Copy full SHA for d3d14cc

Commits on Oct 17, 2013

• correct usage

  bhaddow

  committed
  3b99636

  Copy full SHA for 3b99636

Commits on Oct 7, 2013

• Add option to protect expressions - eg URLs

  bhaddow

  committed
  67c294a

  Copy full SHA for 67c294a

Commits on Aug 16, 2013

• Fixed bug in tokenizer.perl where comma separated lists of single

  Show description for a5fb4d1
  jgwinnup

  committed
  a5fb4d1

  Copy full SHA for a5fb4d1

Commits on Jul 24, 2013

• Add option to do Penn Treebank style tokenization

  Show description for 1238041
  pjwilliams

  committed
  1238041

  Copy full SHA for 1238041

Commits on Feb 27, 2013

• fixed typo

  

  amittai

  committed
  7ca271b

  Copy full SHA for 7ca271b

Commits on Jun 30, 2012

• add multi-threading feature to the tokenizer.perl

  

  Pidong Wang

  committed
  eecefbe

  Copy full SHA for eecefbe

Commits on Jun 26, 2012

• lock m_vocab variable access in Encode() and Lookup(). Other functions are still not threadsafe

  

  Hieu Hoang

  committed
  93bff3f

  Copy full SHA for 93bff3f

• Change Bin to RealBin. Thanks to Tom Hoar

  

  Hieu Hoang

  committed
  debe090

  Copy full SHA for debe090

Commits on Jun 25, 2012

• escape bar character with proper html escape sequence

  phikoehn

  committed
  135e38d

  Copy full SHA for 135e38d

Commits on May 29, 2012

• more escaping in tokenizer; wrapper for berkeley parser (german)

  phikoehn

  committed
  2e370ed

  Copy full SHA for 2e370ed

Commits on May 25, 2012

• minor fixes

  phikoehn

  committed
  561b9ac

  Copy full SHA for 561b9ac

Commits on Mar 23, 2012

• bug fix to filter hierarchical

  phikoehn

  committed
  4d0fc99

  Copy full SHA for 4d0fc99

Pagination

Previous

Next