Pattern for *PublishLogEntry to enable easy reverts

[ormsbee]

Right now, ComponentPublishLogEntry has a reference to the ComponentVersion being published, but not the one being replaced. This makes it harder to do reverts because we have to scan back through the table for the last version published. In a large library with many publishes over a long period of time, this could get expensive. But if we stored it as a separate field in ComponentPublishLogEntry, we would know what to revert back to in a time proportional to the size of the most recent publish.

[ormsbee]

Ideally, we could have some kind of model builder that let you specify what the versioned and versionless thing is? On the one hand, that might be a little too magical/complex. On the other hand, it could make it so that revert logic could be entirely contained within the publishing app, which would be pretty nice.

[ormsbee]

@kdmccormick, @bradenmacdonald, @feanil

[ormsbee]

Maybe a Manager that takes parameters? Then you could attach it to your models, assuming you conform to a certain set of field names or pass it in as parameters to the manager. Is there a built-in way for managers track every instance of themselves? Is it weird to do so manually with a class variable?

I would like to have an API function where you can say revert({uuid_of_publish}), and it would find all the publishable things, read their prev/next version values and create a new publish entry that basically reverses everything that happened during the publish in question.

[feanil]

I don't fully understand what you're trying to get at here. The API function you want to get out at the other end makes sense, but why do you need a Model manager? What does that give you?

[ormsbee]

@feanil: Every publishable thing (Component, Unit, etc.) is going to have a small number of models that follow the same pattern. For Components, that's currently ComponentPublishLogEntry and PublishedComponent.

The act of reverting is going to mostly involve a fixed set of operations across those models, all wrapped in a transaction. I would like for the default logic of that to be centralized in the publishing app, and not re-implemented in separate apps for Components, Units, etc.

We could implement the logic as an inheritable, abstract model, but we couldn't inherit field definitions from it because those fields point to different models. ComponentPublishLogEntry is going to have foreign keys to Component and ComponentVersion, while a future UnitPublishLogEntry is going to have foreign keys to Unit and UnitVersion.

At this point, I was dangerously close to pondering a metaclass for creating models, but I thought that maybe a custom manager could do most of that. So you might have:

class ComponentPublishLogEntry(models.Model):
    publishing = PublishingManager(unversioned='component', versioned='component_version')

And then we could call ComponentPublishLogEntry.publishing.revert(...). But maybe I should just be doing a custom mixin with a classmethod instead, since Managers come with other expectations and query behaviors.

[kdmccormick]

A custom model manager would still have methods get, filter, all, etc., right? Would it be sensical for someone to ever write use those standard manager methods, like:

ComponentPublishLogEntry.publishing.filter(...)

?

If not, then I think a mixin w/ classmethod would be a better way to go. Something like this, right?

class PublishingMixin:
    unversioned_field = None  # override in subclass, eg 'component'
    versioned_field= None  # override in subclass, eg 'component_version'

    @classmethod
    def revert(cls, uuid):
        ...

[ormsbee]

I wonder if it even needs to be behavior/methods set on the individual model. If the API calls that publish/revert are done by the publishing app, maybe it's sufficient to make a decorator that just registers the model so that publishing can "see" it later to do the appropriate calls.

So for instance:

@publish_log_entry("publish_log_entry", "component", "component_version")
class ComponentPublishLogEntry(models.Model):
    publish_log_entry = models.ForeignKey(PublishLogEntry, on_delete=models.CASCADE)
    component = models.ForeignKey(Component, on_delete=models.RESTRICT)
    component_version = models.ForeignKey(
        ComponentVersion, on_delete=models.RESTRICT, null=True
    )

[kdmccormick]

Nice! I like the decorator approach a lot because I think it signals to the code reader: "hey, we will be doing some metaprogramming magic with this class, but you don't have to worry about it if it's not relevant to you, and the class will otherwise behave as you'd expect!" Mixins, on the other hand, always give me this feeling that I'll need to read the entire mixin to understand how the derived class will behave.

[ormsbee]

I'll run with this then, and maybe try for a kind of model-centric API, e.g.

def publish(*versioned_model_qset: QuerySet) -> PublishLogEntry:
    ...

So you'd pass in QuerySets that yield the versioned things you want published, and the publishing app wraps it all in a transaction.

In terms of the reverts, maybe a function called rewind_to makes more sense... unlike the previous system where we capture a full snapshot at every point, we can't just examine one entry to restore the state from four weeks ago–we'd have to construct a revert entry that examines and undoes all the publishes since that time. Which isn't hard, but "revert" might be confusing.

So maybe:

• publish
• revert_last_publish
• rewind_to

?

[kdmccormick]

I want to make sure I'm understanding the semantics right here, so let me know if this is correct:

A "node" (for lack of a better word) is a versioned thing, eg Component or Unit.

Each node model has an associated version model, eg ComponentVersion or UnitVersion. The versions form a linear chain. Versions are immutable and cannot be deleted, so each node's version chain can only grow. The latest version of a node indicates its latest draft state.

Each node model also has an associated publishing log model, eg ComponentPublishLogEntry or UnitPublishLogEntry. The log forms a linear chain. Log entries are immutable and cannot be deleted, so each node's publishing log can only grow. The latest publishing log entry indicates the node's latest published state.

For example, a node might be at version 7, with the latest published version 6 (indicated by log entry d):

INTIAL:

Versions:     0 1 2 3 4 5 6 7
              ^   ^     ^ ^
              |   |     | |
Log entries:  a   b     c d    

publish would create a new log entry pointing to the latest version of a node, eg:

INITIAL + `publish()`:

Versions:     0 1 2 3 4 5 6 7
              ^   ^     ^ ^ ^
              |   |     | | |
Log entries:  a   b     c d e    

rewind_to would create a new log entry pointing to the same version as an older log entry, eg:

INITIAL + `rewind_to(b)`:

Versions:     0 1 2 3 4 5 6 7
              ^   ^     ^ ^
              |   |     | |
Log entries:  a   b     c d    
                  e

and revert_last_publish is equivalent to rewind_to(<previous log entry>), eg:

INITIAL + `revert_last_publish()`:

Versions:     0 1 2 3 4 5 6 7
              ^   ^     ^ ^
              |   |     | |
Log entries:  a   b     c d    
                        e

[ormsbee]

@kdmccormick: Yeah, that's mostly my intent. The only minor thing I'd mention is that there are a few different models involved for performance reasons. So PublishedComponent would only have one row for each Component with a pointer to the currently published version. There would be a central PublishLogEntry for each publish operation (which might have published any number of Components). There would additionally be a separate table that has a foreign key to PublishLogEntry and would have a row for every Component published, and that row would have fields for both the previous version and new version.

[ormsbee]

So I tried the decorator approach, and while it was workable, it was requiring more and more magic to make things work. This was particularly frustrating in the bookkeeping sorts of tasks–e.g. keeping the Draft tables updated.

I tried taking a very different approach and extracted most of the actual base publishing tables themselves into the publishing app, and this feels like it's a much more reasonable path. The special bookkeeping tables around publishes, the current published version, and drafts all live only in publishing and don't have any companion tables that need to live at the app. The only publishing-related models that the components app needs now are Component and ComponentVersion.

While this makes the join relationship a bit more awkward (and I can sprinkle a few convenience methods there), it means that the apps can focus exclusively on what actually makes their publishable thing distinct, and not have to worry about getting all the field types right to match some assumed interface that publishing expects. For instance, in my current (still very broken) branch, ComponentVersion just looks like this:

class ComponentVersion(models.Model):
    """
    A particular version of a Component.

    This holds the title (because that's versioned information) and the contents
    via a M:M relationship with RawContent via ComponentVersionRawContent.

    * Each ComponentVersion belongs to one and only one Component.
    * ComponentVersions have a version_num that should increment by one with
      each new version.
    """
    publishable_version = models.OneToOneField(
        PublishableVersion, on_delete=models.CASCADE, primary_key=True
    )

    # The raw_contents hold the actual interesting data associated with this
    # ComponentVersion.
    raw_contents = models.ManyToManyField(
        "RawContent",
        through="ComponentVersionRawContent",
        related_name="component_versions",
    )

    def __str__(self):
        return f"v{self.version_num}: {self.title}"

I'll hack more on this tomorrow and put the results here.

[ormsbee]

@bradenmacdonald: ... and clearly my statement about how my next PR was going to be smaller was hilariously inaccurate. 😛

[ormsbee]

Closing this ticket. The general approach is going to be to do the extraction of models and logic into the publishing app as described in the above comment, and the specifics of that are being worked out in discussions on the the implementing PR: #41