Merge pull request #26 from pharo-ai/merge-ngram

Merge ngram in NgramModel

Author: jecisc

.github/workflows/test.yml

@@ -8,14 +8,16 @@ env:
 on:
   push:
     branches:
-      - master
+      - '**'
+  pull_request:
+    types: [assigned, opened, synchronize, reopened]
 
 jobs:
   build:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        smalltalk: [ Pharo64-9.0, Pharo64-10 ]
+        smalltalk: [ Pharo64-9.0, Pharo64-10, Pharo64-11 ]
     name: ${{ matrix.smalltalk }}
     steps:
       - uses: actions/checkout@v2

README.md

@@ -4,6 +4,9 @@
 [![Coverage Status](https://coveralls.io/repos/github/pharo-ai/NgramModel/badge.svg?branch=master)](https://coveralls.io/github/pharo-ai/NgramModel?branch=master)
 [![License](https://img.shields.io/badge/license-MIT-blue.svg)](https://raw.githubusercontent.com/pharo-ai/NgramModel/master/LICENSE)
 
+`Ngram` package provides basic [n-gram](https://en.wikipedia.org/wiki/N-gram) functionality for Pharo. This includes `Ngram` class as well as `String` and `SequenceableCollection` extension that allow you to split text into unigrams, bigrams, trigrams, etc. Basically, this is just a simple utility for splitting texts into sequences of words.
+This project also provides 
+
 ## Installation
 
 To install the packages of NgramModel, go to the Playground (Ctrl+OW) in your Pharo image and execute the following Metacello script (select it and press Do-it button or Ctrl+D):
@@ -12,9 +15,97 @@ To install the packages of NgramModel, go to the Playground (Ctrl+OW) in your Ph
 Metacello new
   baseline: 'NgramModel';
   repository: 'github://pharo-ai/NgramModel/src';
-  load.
+  load
+```
+
+## How to depend on it?
+
+If you want to add a dependency to this project to your own project, include the following lines into your baseline method:
+
+```Smalltalk
+spec
+  baseline: 'NgramModel'
+  with: [ spec repository: 'github://pharo-ai/NgramModel/src' ].
+```
+
+If you are new to baselines and Metacello, check out the [Baselines](https://github.com/pharo-open-documentation/pharo-wiki/blob/master/General/Baselines.md) tutorial on Pharo Wiki.
+
+## What are n-grams?
+
+[N-gram](https://en.wikipedia.org/wiki/N-gram) is a sequence of n elements, usually words. Number n is called the order of n-gram The concept of n-grams is widely used in [natural language processing (NLP)](https://en.wikipedia.org/wiki/Natural_language_processing). A text can be split into n-grams - sequences of n words. Consider the following text:
+```
+I do not like green eggs and ham
+```
+We can split it into **unigrams** (n-grams with n=1):
+```
+(I), (do), (not), (like), (green), (eggs), (and), (ham)
+```
+Or **bigrams** (n-grams with n=2):
+```
+(I do), (do not), (not like), (like green), (green eggs), (eggs and), (and ham)
+```
+Or **trigrams** (n-grams with n=3):
+```
+(I do not), (do not like), (not like green), (like green eggs), (green eggs and), (eggs and ham)
+```
+And so on (tetragrams, pentagrams, etc.).
+
+### Applications
+
+N-grams are widely applied in [language modeling](https://en.wikipedia.org/wiki/Language_model). For example, take a look at the implementation of [n-gram language model](https://github.com/olekscode/NgramModel) in Pharo.
+
+### Structure of n-gram
+
+Each n-gram can be separated into:
+
+* **last word** - the last element in a sequence;
+* **history** (context) - n-gram of order n-1 with all words except the last one.
+
+Such separation is useful in probabilistic modeling when we want to estimate the probability of word given n-1 previous words (see [n-gram language model](https://github.com/olekscode/NgramModel)).
+
+## Ngram class
+
+This package provides only one class - `Ngram`. It models the n-gram.
+
+### Instance creation
+
+You can create n-gram from any `SequenceableCollection`:
+
+```Smalltalk
+trigram := Ngram withElements: #(do not like).
+tetragram := #(green eggs and ham) asNgram.
+```
+
+Or by explicitly providing the history (n-gram of lower order) and last element:
+
+```Smalltalk
+hist := #(green eggs and) asNgram.
+w := 'ham'.
+
+ngram := Ngram withHistory: hist last: w.
 ```
 
+You can also create a zerogram - n-gram of order 0. It is an empty sequence with no history and no last word:
+
+```Smalltalk
+Ngram zerogram.
+```
+
+### Accessing
+
+You can access the order of n-gram, its history and last element:
+
+```Smalltalk
+tetragram. "n-gram(green eggs and ham)"
+tetragram order. "4"
+tetragram history. "n-gram(green eggs and)"
+tetragram last. "ham"
+```
+
+## String extensions
+
+> TODO
+
 ## Example of text generation
 
 #### 1. Loading Brown corpus

src/BaselineOfNgramModel/BaselineOfNgramModel.class.st

@@ -6,16 +6,15 @@ Class {
 
 { #category : #baselines }
 BaselineOfNgramModel >> baseline: spec [
+
 	<baseline>
-	spec for: #common do: [	
-		"External dependencies"
+	spec for: #common do: [
 		spec
-			baseline: 'Ngram'
-			with: [ spec repository: 'github://pharo-ai/Ngram/src' ].
-				
-		"Packages"
-		spec 
-			package: 'NgramModel' with: [ spec requires: #('Ngram') ];
-			package: 'NgramModel-Tests' with: [ spec requires: #('NgramModel') ];
-			package: 'NgramModelTextGenerator' with: [ spec requires: #('NgramModel') ] ].
+			package: 'Ngram';
+			package: 'NgramModel' with: [ spec requires: #( 'Ngram' ) ];
+			package: 'NgramModelTextGenerator' with: [ spec requires: #( 'NgramModel' ) ];
+			package: 'Ngram-Tests' with: [ spec requires: #( 'Ngram' ) ];
+			package: 'NgramModel-Tests' with: [ spec requires: #( 'NgramModel' ) ].
+
+		spec group: 'Ngram-core' with: #( 'Ngram' 'Ngram-Tests' ) ]
 ]