Skip to content
This repository was archived by the owner on Jul 7, 2026. It is now read-only.

Retire in favor of sqlite-utils Migrations #19

Merged
simonw merged 4 commits into from
Jul 7, 2026
Merged

Retire in favor of sqlite-utils Migrations #19

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
7ad5926
Retire in favor of sqlite-utils Migrations, refs #18
simonw Jul 7, 2026
570e23a
No need for conftest.py
simonw Jul 7, 2026
d7d273f
Link to sqlite-utils migrations docs
simonw Jul 7, 2026
4219a50
pip install . --group dev
simonw Jul 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/publish.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ jobs:
cache-dependency-path: pyproject.toml
- name: Install dependencies
run: |
pip install '.[test]'
pip install . --group dev
- name: Run tests
run: |
pytest
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ jobs:
cache-dependency-path: pyproject.toml
- name: Install dependencies
run: |
pip install '.[test]'
pip install . --group dev
- name: Run tests
run: |
pytest
Expand Down
178 changes: 10 additions & 168 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,188 +5,30 @@
[![Tests](https://github.com/simonw/sqlite-migrate/workflows/Test/badge.svg)](https://github.com/simonw/sqlite-migrate/actions?query=workflow%3ATest)
[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/sqlite-migrate/blob/main/LICENSE)

A simple database migration system for SQLite, based on [sqlite-utils](https://sqlite-utils.datasette.io/).
Deprecated compatibility package for `sqlite-utils` migrations.

**This project is an early alpha. Expect breaking changes.**

## Installation

This tool works as a plugin for `sqlite-utils`. First [install that](https://sqlite-utils.datasette.io/en/stable/installation.html):
Migration support is now built into `sqlite-utils` 4. Install and use `sqlite-utils` directly for new projects:

```bash
pip install sqlite-utils
```
Then install this plugin like so:
```bash
sqlite-utils install sqlite-migrate
```
## Migration files

This tool works against migration files. A migration file looks like this:
New code should import `Migrations` from `sqlite_utils`:

```python
from sqlite_migrate import Migrations

# Pick a unique name here - it must not clash with other migration sets that
# the user might run against the same database.

migration = Migrations("creatures")

# Use this decorator against functions that implement migrations
@migration()
def create_table(db):
# db is a sqlite-utils Database instance
db["creatures"].create(
{"id": int, "name": str, "species": str},
pk="id"
)

@migration()
def add_weight(db):
# db is a sqlite-utils Database instance
db["creatures"].add_column("weight", float)
from sqlite_utils import Migrations
```
Here is [documentation on the Database instance](https://sqlite-utils.datasette.io/en/stable/python-api.html) passed to each migration function.

## Running migrations
This package depends on `sqlite-utils>=4` and re-exports that class so existing migration files can continue to use their old import:

Running this command will execute those migrations in sequence against the specified database file.

Call `migrate` with a path to your database and a path to the migrations file you want to apply:
```bash
sqlite-utils migrate creatures.db path/to/migrations.py
```
Running this multiple times will have no additional affect, unless you add more migration functions to the file.

If you call it without arguments it will search for and apply any `migrations.py` files in the current directory or any of its subdirectories.

You can also pass the path to a directory, in which case all `migrations.py` files in that directory and its subdirectories will be applied:

```bash
sqlite-utils migrate creatures.db path/to/parent/
```
When applying a single migrations file you can use the `--stop-before` option to apply all migrations up to but excluding the specified migration:

```bash
sqlite-utils migrate creatures.db path/to/migrations.py --stop-before add_weight
```

## Listing migrations

Add `--list` to list migrations without running them, for example:

```bash
sqlite-utils migrate creatures.db --list
```
The output will look something like this:
```
Migrations for: creatures

Applied:
create_table - 2023-07-23 04:09:40.324002
add_weight - 2023-07-23 04:09:40.324649
add_age - 2023-07-23 04:09:44.441616
cleanup_columns - 2023-07-23 04:09:44.443394

Pending:
drop_table
```python
from sqlite_migrate import Migrations
```

## Verbose mode

Add `-v` or `--verbose` for verbose output, which will show the schema before and after the migrations were applied along with a diff:
Run migrations using the `sqlite-utils migrate` command:

```bash
sqlite-utils migrate creatures.db --verbose
```
Example output:

<!-- [[[cog
import cog
from sqlite_utils.cli import cli
import sqlite_utils
import textwrap
from click.testing import CliRunner
runner = CliRunner()
with runner.isolated_filesystem():
# First migration creates the table
open("migrations.py", "w").write(textwrap.dedent("""
from sqlite_migrate import Migrations
migration = Migrations("demo")
@migration()
def create_table(db):
db["creatures"].create(
{"id": int, "name": str, "species": str, "weight": float},
pk="id",
)
"""))
runner.invoke(cli, ["migrate", "creatures.db"])
# Second migration adds some columns
open("migrations.py", "a").write("\n\n" + textwrap.dedent("""
@migration()
def add_columns(db):
db["creatures"].add_column("age", int)
db["creatures"].add_column("shoe_size", int)
db["creatures"].transform()
"""))
result = runner.invoke(cli, ["migrate", "creatures.db", "--verbose"])
cog.out(
"```\n{}\n```".format(result.output.strip())
)
]]] -->
sqlite-utils migrate creatures.db path/to/migrations.py
```
Migrating creatures.db

Schema before:

CREATE TABLE [_sqlite_migrations] (
[id] INTEGER PRIMARY KEY,
[migration_set] TEXT,
[name] TEXT,
[applied_at] TEXT
);
CREATE UNIQUE INDEX [idx__sqlite_migrations_migration_set_name]
ON [_sqlite_migrations] ([migration_set], [name]);
CREATE TABLE [creatures] (
[id] INTEGER PRIMARY KEY,
[name] TEXT,
[species] TEXT,
[weight] FLOAT
);

Schema after:

CREATE TABLE [_sqlite_migrations] (
[id] INTEGER PRIMARY KEY,
[migration_set] TEXT,
[name] TEXT,
[applied_at] TEXT
);
CREATE UNIQUE INDEX [idx__sqlite_migrations_migration_set_name]
ON [_sqlite_migrations] ([migration_set], [name]);
CREATE TABLE "creatures" (
[id] INTEGER PRIMARY KEY,
[name] TEXT,
[species] TEXT,
[weight] FLOAT,
[age] INTEGER,
[shoe_size] INTEGER
);

Schema diff:

);
CREATE UNIQUE INDEX [idx__sqlite_migrations_migration_set_name]
ON [_sqlite_migrations] ([migration_set], [name]);
-CREATE TABLE [creatures] (
+CREATE TABLE "creatures" (
[id] INTEGER PRIMARY KEY,
[name] TEXT,
[species] TEXT,
- [weight] FLOAT
+ [weight] FLOAT,
+ [age] INTEGER,
+ [shoe_size] INTEGER
);
```
<!-- [[[end]]] -->
See the [`sqlite-utils` migrations documentation](https://sqlite-utils.datasette.io/en/stable/migrations.html) for the full Python API and CLI usage.
9 changes: 3 additions & 6 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,17 +1,17 @@
[project]
name = "sqlite-migrate"
version = "0.1b0"
description = "A simple database migration system for SQLite, based on sqlite-utils"
description = "Compatibility package for sqlite-utils migrations"
readme = "README.md"
requires-python = ">=3.10"
authors = [{name = "Simon Willison"}]
license = "Apache-2.0"
classifiers = [
"Development Status :: 2 - Pre-Alpha"
"Development Status :: 7 - Inactive"
]

dependencies = [
"sqlite-utils!=4.0,<4.0.post0"
"sqlite-utils>=4"
]

[project.urls]
Expand All @@ -20,9 +20,6 @@ Changelog = "https://github.com/simonw/sqlite-migrate/releases"
Issues = "https://github.com/simonw/sqlite-migrate/issues"
CI = "https://github.com/simonw/sqlite-migrate/actions"

[project.entry-points.sqlite_utils]
migrate = "sqlite_migrate.sqlite_utils_plugin"

[dependency-groups]
dev = ["pytest", "mypy", "black", "ruff", "cogapp"]

Expand Down
121 changes: 2 additions & 119 deletions sqlite_migrate/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,120 +1,3 @@
from dataclasses import dataclass
import datetime
from typing import cast, Callable, List, Optional
from typing import TYPE_CHECKING
from sqlite_utils import Migrations

if TYPE_CHECKING:
from sqlite_utils.db import Database, Table


class Migrations:
migrations_table = "_sqlite_migrations"

@dataclass
class _Migration:
name: str
fn: Callable

@dataclass
class _AppliedMigration:
name: str
applied_at: datetime.datetime

def __init__(self, name: str):
"""
:param name: The name of the migration set. This should be unique.
"""
self.name = name
self._migrations: List[Migrations._Migration] = []

def __call__(self, *, name: Optional[str] = None) -> Callable:
"""
:param name: The name to use for this migration - if not provided,
the name of the function will be used
"""

def inner(func: Callable) -> Callable:
self._migrations.append(self._Migration(name or func.__name__, func))
return func

return inner

def pending(self, db: "Database") -> List["Migrations._Migration"]:
"""
Return a list of pending migrations.
"""
self.ensure_migrations_table(db)
already_applied = {
r["name"]
for r in db[self.migrations_table].rows_where(
"migration_set = ?", [self.name]
)
}
return [
migration
for migration in self._migrations
if migration.name not in already_applied
]

def applied(self, db: "Database") -> List["Migrations._AppliedMigration"]:
"""
Return a list of applied migrations.
"""
self.ensure_migrations_table(db)
return [
self._AppliedMigration(name=row["name"], applied_at=row["applied_at"])
for row in db[self.migrations_table].rows_where(
"migration_set = ?", [self.name]
)
]

def apply(self, db: "Database", *, stop_before: Optional[str] = None):
"""
Apply any pending migrations to the database.
"""
self.ensure_migrations_table(db)
for migration in self.pending(db):
name = migration.name
if name == stop_before:
return
migration.fn(db)
_table(db, self.migrations_table).insert(
{
"migration_set": self.name,
"name": name,
"applied_at": str(datetime.datetime.now(datetime.timezone.utc)),
}
)

def ensure_migrations_table(self, db: "Database"):
"""
Ensure _sqlite_migrations table exists and has the correct schema
"""
table = _table(db, self.migrations_table)
if not table.exists():
table.create(
{
"id": int,
"migration_set": str,
"name": str,
"applied_at": str,
},
pk="id",
)
table.create_index(["migration_set", "name"], unique=True)
elif table.pks != ["id"]:
# This has an older primary key scheme, upgrade it
table.transform(pk="id")
unique_indexes = {tuple(index.columns) for index in table.indexes}
if ("migration_set", "name") not in unique_indexes:
table.create_index(["migration_set", "name"], unique=True)

def __repr__(self):
return "<Migrations '{}': [{}]>".format(
self.name, ", ".join(m.name for m in self._migrations)
)


def _table(db: "Database", name: str) -> "Table":
# mypy workaround
return cast("Table", db[name])
__all__ = ["Migrations"]
Loading