Feature/deprecate guppy

README.md

@@ -12,7 +12,11 @@ wrong, let us know so we can add you to the GOTCHA hall of fame!
 > We also have more detailed documentation for your perusal at https://looselab.github.io/readfish
 
 > [!NOTE]  
-> Now also see our cool [FAQ](docs/FAQ.md).
+> Now also see our cool [FAQ](docs/FAQ.md). 
+
+> [!WARNING]
+Breaking for any version of `MinKNOW <= 6.0.0`
+As of `readfish >=2024.3.0` we no longer support guppy.
 
 readfish is a Python package that integrates with the
 [Read Until API](https://github.com/nanoporetech/read_until_api).
@@ -23,13 +27,13 @@ way most fit for purpose, and a return call can be made to the server to unblock
 the read in progress and so direct sequencing capacity towards reads of interest.
 
 
-**This implementation of readfish requires Guppy version >= 6.0.0 and MinKNOW version core >= 5.0.0 . It will not work on earlier versions.**
+**This implementation of readfish requires Dorado server version >= 7.3.9 and MinKNOW version core >= 6.0.0 . It will not work on earlier versions.**
 
-**Since MinKNOW version core >=5.9.0 and Dorado server version >=7.3.9, Dorado requires an alternate library, `ont-pybasecall-client-lib`. We have introduced a new`dorado` module to handle this.**
+**To run with earlier versions of MinKNOW please use an earlier version of readfish.**
 
 
-The code here has been tested with Guppy in GPU mode using GridION Mk1 and
-NVIDIA RTX2080 on live sequencing runs and an NVIDIA GTX1080 using playback
+The code here has been tested with Dorado in GPU mode using GridION Mk1 and
+NVIDIA RTX4090s on live sequencing runs and on MacOSX M2Max using playback
 on a simulated run (see below for how to test this).
 This code is run at your own risk as it DOES affect sequencing output. You
 are **strongly** advised to test your setup prior to running (see below for
@@ -41,6 +45,7 @@ The following platforms are supported:
 
 - **PromethION** Big Boy
 - **P2Solo** Smol Big Boy
+- **P2i** Not so Smol Big Boy
 - **GridION** Box
 - **MinION** Smol Boy
 
@@ -50,7 +55,7 @@ The following platforms are supported:
  The following OSs are supported:
 
  - **Linux** yay
- - **MacOS** boo (Apple Silicon, Only with Dorado)
+ - **MacOS** boo (Apple Silicon Only)
 
 
 > [!NOTE]  
@@ -128,7 +133,7 @@ conda activate readfish_dev
 | MinKNOW is transitioning from Guppy to Dorado. Until MinKNOW version 5.9 both Guppy and Dorado used ont-pyguppy-client-lib.<br/>As of MinKNOW version 5.9 and Dorado server version 7.3.9 and greater Dorado requires an alternate library, `ont-pybasecall-client-lib`.<br/>The listed `ont-pyguppy-client-lib` or `ont-pybasecaller-client-lib` version may not match the version installed on your system. To fix this, Please see this [issue](https://github.com/LooseLab/readfish/issues/221#issuecomment-1381529409), using the appropriate library. |
 
 
-[ONT's Guppy GPU](https://community.nanoporetech.com/downloads) should be installed and running as a server.
+[ONT's Dorado Basecall Server GPU](https://community.nanoporetech.com/downloads) should be installed and running as a server.
 
 <details style="margin-top: 10px">
 <summary><span id="py-ve">Alternatively, install readfish into a python virtual-environment</span></summary>
@@ -142,8 +147,8 @@ pip install --upgrade pip
 # Install our readfish Software
 pip install readfish[all]
 
-# Install ont_pyguppy_client_lib that matches your guppy server version. E.G.
-pip install ont_pyguppy_client_lib==6.3.8
+# Install ont_pybasecall_client that matches your dorado basecall server version. E.G.
+pip install ont_pybasecall_client_lib==7.1.2
 ```
 
 </details>
@@ -298,8 +303,9 @@ Steps:
     ``` 
     Change the text between the quotes to point to your downloaded bulk FAST5 file.
     <!-- end-obsolete -->
-1. Optional, If running GUPPY in GPU mode, set the parameter `break_reads_after_seconds = 1.0`
+1. Optional, If running Dorado in GPU mode,  you can set the parameter `break_reads_after_seconds = 1.0`
 to `break_reads_after_seconds = 0.4`. This results in a smaller read chunk. For R10.4 this is not required but can be tried. For adaptive sampling on PromethION, this should be left at 1 second.
+1. In MinKNOW >= 6.0.0 this value defaults to 0.8 which is a reasonable balance.
 1. In the MinKNOW GUI, right click on a sequencing position and select `Reload Scripts`.
 Your version of MinKNOW will now playback the bulkfile rather than live sequencing.
 1. Start a sequencing run as you would normally, selecting the corresponding flow
@@ -335,13 +341,10 @@ Note: The plots here are generated from running readfish unblock-all on an Apple
 <details style="margin-top: 10px">
 <summary id="testing-basecalling-and-mapping"><h3 style="display: inline;">Testing base-calling and mapping</h3></summary>
 
-To test selective sequencing you must have access to either a
-[guppy basecall server](https://community.nanoporetech.com/downloads/guppy/release_notes) (>=6.0.0) or a [dorado basecall server](https://community.nanoporetech.com/downloads/dorado/release_notes). 
+To test selective sequencing you must have access to a [dorado basecall server](https://community.nanoporetech.com/downloads/dorado/release_notes). 
 
 and a readfish TOML configuration file.
 
-NOTE: guppy and dorado are used here interchangeably as the basecall server. Dorado is gradually replacing guppy. All readfish code is compatible with Guppy >=6.0.0 and dorado >=0.4.0
-
 1. First make a local copy of the example TOML file:
     ```console
     curl -O https://raw.githubusercontent.com/LooseLab/readfish/master/docs/_static/example_tomls/human_chr_selection.toml
@@ -559,6 +562,14 @@ And for our Awesome Logo please checkout out [@tim_bassford](https://twitter.com
 
 <!-- start-changelog -->
 # Changelog
+## 2024.3.0
+### This release is breaking for ALL versions of `MinKNOW <= 6` and no longer supports Guppy.
+
+1. Introducing support for MinKNOW >=6.0.0 and deprecating support for earlier versions.
+1. Removing support for legacy guppy base caller and only supporting Dorado in future.
+1. Optimising batch sending to the base caller through the use of `pass_reads` rather than `pass_read`
+1. Adding the new strand classifications as used by MinKNOW, including strand2 and short.
+
 ## 2024.2.0
 1. Add a dorado base-caller which addressed issue [#347](https://github.com/LooseLab/readfish/issues/347) - chiefly in Dorado 7.3.9 ONT have moved to `ont-pybasecall-client-lib`, 
   and connections from `ont_pyguppy_client_lib` raise `Connection error. ...  LOAD_CONFIG. Reply: INVALID_PROTOCOL` [(#344)](https://github.com/LooseLab/readfish/pull/344)

docs/basecalling-params.md

@@ -0,0 +1,24 @@
+# Base-calling server parameters
+
+The default settings for `Dorado` GPU should be sufficient for most use cases.
+
+The one trick we sometimes use, on a _really_ high performance flow cell is to split the GPUs between `readfish` and `dorado`.
+
+This of course assumes that you have a multiple GPU machine, such as PromethION tower.
+In the new P24 towers, there are 4x NVIDIA A100 GPUs.
+
+1. Stop dorado `sudo systemctl stop doradod`
+1. View the available CUDA devices. `nvidia-smi`
+1. Restart two dorado instances, splitting the devices between each.
+
+Dorado:
+
+```console
+/opt/ont/dorado/bin/dorado_basecall_server --log_path /var/log/dorado --config dna_r10.4.1_e8.2_400bps_fast.cfg --ipc_threads 3 --port /tmp/.guppy/5555 --dorado_download_path /opt/ont/dorado-models --device cuda:0
+```
+
+```console
+/opt/ont/dorado/bin/dorado_basecall_server --log_path /var/log/dorado --config dna_r10.4.1_e8.2_400bps_fast.cfg --ipc_threads 3 --port /tmp/.guppy/5556 --dorado_download_path /opt/ont/dorado-models --device cuda:1
+```
+
+1. Start readfish, base calling on port 5556

docs/developers-guide.md

@@ -32,8 +32,8 @@ dev = ["readfish[all,docs,tests]"]
 # Running dependencies, this is a little bit clunky but works for now
 mappy = ["mappy"]
 mappy-rs = ["mappy-rs"]
-guppy = ["ont_pyguppy_client_lib"]
-all = ["readfish[mappy,mappy-rs,guppy]"]
+dorado = ["ont-pybasecall-client-lib"]
+all = ["readfish[mappy,mappy-rs,dorado]"]
 ```
 
 An example install command would be `pip install readfish[all]`.
@@ -159,11 +159,11 @@ Plugins are loaded as instances of `_PluginModules` - see source of `_PluginModu
 This would try to load the `foo` module from the `bar` package.
 
 ```toml
-[caller_settings.readfish.plugins.guppy]
+[caller_settings.readfish.plugins.dorado]
 ```
 
-This would load the readfish guppy `Caller` plugin explicitly.
-There are instances of "builtin" plugins, which are the included `mappy`, `mappy_rs` and `guppy` plugins.
+This would load the readfish dorado `Caller` plugin explicitly.
+There are instances of "builtin" plugins, which are the included `mappy`, `mappy_rs` and `dorado` plugins.
 See the source of `readfish._config._PluginModule.load_module` for more details.
 ```{eval-rst}
 .. automethod:: readfish._config._PluginModule.load_module
@@ -209,7 +209,7 @@ This next method is important. `validate()`
 
 The validate function is intended to be called in the __init__ method, before the actual `Aligner` or `Caller` is initialised. The contents of this method are left up to the author, however we suggest that people check for the things listed above.
 
-The purpose of `validate` is to check that the given parameters will create a valid `Aligner` or `Caller`. For example, in the `guppy.py` `Caller` plugin, we check the permissions of the provided `Guppy` socket. If these are insufficient, Guppy only errors out after a 10 minute timeout. However of this is caught in `validate`, everyone ends up being left a lot happier.
+The purpose of `validate` is to check that the given parameters will create a valid `Aligner` or `Caller`. For example, in the `dorado.py` `Caller` plugin, we check the permissions of the provided `dorado` socket. If these are insufficient, dorado only errors out after a 10 minute timeout. However if this is caught in `validate`, everyone ends up being left a lot happier.
 
 ```{literalinclude} ../src/readfish/plugins/_mappy.py
 :language: python

docs/index.md

@@ -6,7 +6,7 @@ hidden: true
 getting-started
 toml
 readfish.console
-guppy-params
+basecalling-params
 FAQ
 post-analysis
 developers-guide

docs/questions/creating_readfish-environment.question.md

@@ -8,17 +8,21 @@ We recommend using conda to manage your readfish environments, especially on ONT
 To install conda instructions can be found [here](https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html#regular-installation).
 An explanation is found in an answer to this [issue](https://github.com/LooseLab/readfish/issues/124#issuecomment-759599319) as well.
 
-Once Conda is installed, copy the following code snippet into a file on your system, and replace the X.X.X of the `ont-pyguppy-client-lib` with the version output of
+Once Conda is installed, copy the following code snippet into a file on your system, and replace the X.X.X of the `ont-pybasecall-client-lib` with the version output of
 
 ```console
- guppy_basecall_server --version
+ dorado_basecall_server --version
 ```
 
 You should get something like this back
 
-    Guppy Basecall Service Software, (C) Oxford Nanopore Technologies plc. Version 6.1.5+446c35524, client-server API version 11.0.0
+    : Dorado Basecall Service Software, (C)Oxford Nanopore Technologies plc. Version 7.4.12+0e5e75c49, client-server API version 20.0.0
 
-The important part is the first three numbers of the version, in this case `6.1.5`.
+    Use of this software is permitted solely under the terms of the end user license agreement (EULA).
+    By running, copying or accessing this software, you are demonstrating your acceptance of the EULA.
+    The EULA may be found in /opt/ont/dorado/bin
+
+The important part is the first three numbers of the version, in this case `7.4.12`.
 
 ```yaml
 name: readfish
@@ -27,10 +31,10 @@ channels:
   - conda-forge
   - defaults
 dependencies:
-  - python=3.9
+  - python=3.11
   - pip
   - pip:
     - git+https://github.com/nanoporetech/read_until_api@3.4.1
-    - ont-pyguppy-client-lib==X.X.X
+    - ont-pybasecall-client-lib==X.X.X
     - git+https://github.com/LooseLab/readfish@main
 ```

docs/questions/py-guppy-client-error.question.md

@@ -2,24 +2,33 @@
 title: "Connection error. \\[timed_out\\] Timeout waiting for reply to request: LOAD_CONFIG"
 alt_titles:
   - "Error connecting to Guppy"
+  - "Error connecting to Dorado"
   - "LOAD_CONFIG error when connecting to py-guppy-client"
   - "LOAD_CONFIG error when connecting to Guppy"
 ---
 
 N.B. The answer below was taken from this [issue](https://github.com/LooseLab/readfish/issues/221#issuecomment-1375673490)
 Theoretically this should be caught in the latest readfish release, but if you are still having issues, please try the below.
 
-This error often stems from a couple of sources. To test your Guppy Connection you can run
+This error often stems from a couple of sources. To test your Guppy/Dorado Connection you can run
 
 ```console
 python -c 'from pyguppy_client_lib.pyclient import PyGuppyClient as PGC; \
            c = PGC("ipc:///tmp/.guppy/5555", "dna_r9.4.1_450bps_fast.cfg"); \
            c.connect(); print(c)'
 ```
 
-Replacing the `5555` (The default guppy port) with whichever port Guppy may be running on.
+And for Dorado:
 
-The following tends to be the problem only on Computers which ar not provided by ONT, and have been set up manually. Guppy creates the socket file on which is listens as the `MinKNOW ` User, which doens't allow your User account to read/write to the socket.
+```console
+python -c 'from pybasecall_client_lib.pyclient import PyBasecallClient as PBC; \
+           c = PBC("ipc:///tmp/.guppy/5555", "dna_r9.4.1_450bps_fast.cfg"); \
+           c.connect(); print(c)'
+```
+
+Replacing the `5555` (The default Guppy/Dorado port) with whichever port Guppy/Dorado may be running on.
+
+The following tends to be the problem only on Computers which ar not provided by ONT, and have been set up manually. Guppy/Dorado creates the socket file on which is listens as the `MinKNOW ` User, which doens't allow your User account to read/write to the socket.
 
 To fix this, you can either add yourself to the `minknow` group and give the group write permission , or you can give everyone write permissions to the socket which is a bit less secure (Quick, but maybe try the group thing first).
 
@@ -38,7 +47,7 @@ sudo chmod 775 /tmp/.guppy/5555
 
 Which means any User has all permissions on this file.
 
-Once you have done that if you run the above Python command changing the IPC port to whatever port Guppy is listening on. This is usually found in `/tmp/.guppy`, and can be seen by running `ls /tmp/.guppy`
+Once you have done that if you run the above Python command changing the IPC port to whatever port Guppy/Dorado is listening on. This is usually found in `/tmp/.guppy`, and can be seen by running `ls /tmp/.guppy`
 
 ```console
 python -c 'from pyguppy_client_lib.pyclient import PyGuppyClient as PGC; \

docs/questions/py-guppy-client-version.question.md

@@ -4,8 +4,12 @@ alt_titles:
   - "Error connecting to Guppy"
   - "LOAD_CONFIG error when connecting to py-guppy-client"
   - "INVALID_PROTOCOL error when connecting to Guppy"
+  - "INVALID PROTOCOL"
 ---
 
+## UPDATE 20/08/2024
+The same all holds true for `Dorado`, although it is necessary to replace `ont-pyguppy-client-lib` with `ont-pybasecall-client-lib`
+
 This is most likely a version mismatch between `ont-pyguppy-client-lib`, the python library that enables readfish to talk to Guppy and the installed version of Guppy.
 
 If you open a terminal and run

docs/readfish.plugins.rst

@@ -22,10 +22,10 @@ readfish plugins API
 
 ----
 
-``readfish.plugins.guppy`` module
+``readfish.plugins.dorado`` module
 ---------------------------------
 
-.. automodule:: readfish.plugins.guppy
+.. automodule:: readfish.plugins.dorado
    :members:
    :undoc-members:
    :show-inheritance: