Lastgenre: New config option keep_existing

[JOJ0]

Description (moved)

Initially this PR included fixes which moved to a separate PR #5582

Description

• Fix the behavior of theforce option. Previously disabling the option had "incomplete" behaviour:
  • If content was found, a whitelist check was issued and if valid the plugin exited early and logged ("keep").
  • This whitelist check was not aware of multiple genres (separated typically by a string like , ), thus it failed erased all existing genres and overwrote with new ones.

This didn't feel like a typical behaviour of a force option, which this PR tries to improve as follows...

• String-separated multi-genres are now compiled into a list and depending on the whitelist option are kept and enriched with freshly fetched last.fm genres.

• If force is off, pre-populated tags are not touched.

• A lot of refactoring was done, some absolutely required, some as a preparation for future work on the plugin.

• The main processing function _get_genre was massively overhauled and got a new pytest.mark.parametrize test which includes much more test cases.

Details & Docs

Back in 2023-09 we decided on an additional option named keep_allowed, details on what we came up with: #4982 (comment)):

My final conclusion is to change that option name to keep_existing, which feels slightly more self-explanatory. I also decided on Setup 3 (see below) as the default because:

• force always was the plugin's default.
• with keep-existing also enabled by default it feels like a pretty common use-case.

Setup 1

Overwrite all. Only fresh last.fm genres remain.

force: yes
keep_existing: no

Setup 2

Add new last.fm genres when empty. Present tags stay untouched.

force: no
keep_existing: no

Setup 3 (default)

Add new last.fm genres. Combine genres in present tags with new ones
(depending on the whitelist setting, allowed or any).

force: yes
keep_existing: yes

To Do

• Documentation
• Changelog.
• Fix existing tests.
• Refactor _get_genre tests using pytest.mark.parametrize and add new test-cases.
• Implement Case 1
• Implement Case 2
• Implement Case 3
• Implement Case 4

[JOJ0]

I'd request a review from you @sampsyo since I think you initially created it. Also @rain0r would be good since 5 years ago they added the -A option. Hi @rain0r , you wanna take a look? :-)

In short: I think I fixed the plugin to now really reflect what's documented. Any nitpicking in my code or functionality-wise is appreciated.

One question already. Here we do not state that a -a/--album option exists: https://beets.readthedocs.io/en/latest/plugins/lastgenre.html#running-manually

When I started out with using this plugin I was confused a verry long time about this option. As far as I understand it now: It doesn't do anything since it is default. So why keep it? Or is having a -a option that is the default anyway a common thing in beets? I know we have a lot of -a commands which streamlines usablity, and that is a very good thing! Usuall they change behaviour to not do something with items but with albums. I'm just not sure about this one....do we have such a pattern anywhere else? So, just leave it? Should I add some words to the docs?

I think the both of you decided these options should look like that around here: #3220 (comment)

[sampsyo]

Thanks for the extra context, @JOJ0!

About the existence of -a (the default mode) specifically: it's not too uncommon… for example, the beet import command has several flags that are opposites of each other, one of which is the default. Of course, it's important in that case because the default mode can be set in the config, so the user needs a way to override the default in either direction. That's not the case here, so maybe it at least makes sense to add "(default)" to the -a option's help string, or to remove it altogether?

[sampsyo]

Thanks for the ping!! Here are a couple of straightforward comments.

[JOJ0]

I'd like to pull out this conversation #4982 (comment) into a new thread, to make it more obvious for others as well. I think it could be a broader discussion of where this plugin should go. Basically we were talking about the current force: no behaviour being weird as well as the new behaviour I am initially proposing with this PR. I gave all this some thought and came up with this idea. Let's discuss it:

So from my point of view, the main problem with the current behaviour when force is disabled, is that it's not really what a user would typically expect. So what could we do to make force: no more predictable?

The following idea would require a new config setting as well as a whole new branch of behaviour (Case 3):

Case 1

force: yes
overwrite all, only fresh last.fm genres remain

Case 2

force: no

keep any string in present genre tag, only write last.fm genres when empty

Case 3

force: yes
keep_allowed: yes

keep present genres when whitelisted and add new last.fm genres (this is a new branch of behaviour and needs to be coded, I think there is open feature requests for it. Update: Something was feature-requested, but it might not be exactly as I'm proposing here: #4750)

Case 4

force: no
keep_allowed: yes

cleanup only - keep present genres when whitelisted but don't add new last.fm genres; Only when genre is empty, add last.fm genres.

That last combination is weird though....but it's what I proposed for force:no before!

Which of these would now make sense to be the new default? The new force: no (Case 2) would be the least invasive IMO...

@sampsyo brainstorming request 🧐

[JOJ0]

Some more context / cross-linking:

The initial reason why I got my hands dirty with this plugin was when I realised that comma separated multi-genres where not recognized: #4751 (comment)

Here @arsaboo requests a feature that goes in direction of Case 3 above: #4750

[arsaboo]

So, we have two config options - force and keep_allowed, i.e., 4 options in all. Given that, keep_allowed is no in cases 1 and 2. Thus, here's a slightly modified behavior in the 4 cases above:

Case 1: overwrite all, only fresh last.fm genres remain

force: yes
keep_allowed: no

Case 2: Since keep_allowed is no, we only write last.fm genres when empty. There may be incorrect genres in pre-existing tags even after this, as this option is not touching pre-existing tags

force: no
keep_allowed: no

Case 3: keep present genres when whitelisted and add new last.fm genres

force: yes
keep_allowed: yes

Case 4: keep any string in the present genre tag; only write last.fm genres when empty. This will not touch pre-existing genre tags.

force: no
keep_allowed: yes

Thus, Case 4 seems like the best default choice. It does not affect existing genre tags and updates the empty ones. Case 3, on the other hand, is the most useful one (at least for me).

[sampsyo]

This brainstorming honestly sounds great, y'all. It is indeed really weird that the force: no mode can still update old genres; keeping all nonempty genres seems like it should at least be an option. I feel less specific about what the default should be, but I like your idea about decoupling the two aspects of the behavior (when to override existing, nonempty data and what to do to old data) into two different options.

[JOJ0]

Ähem I might be slow or too tired already. Which of those 4 cases are now different from my proposal @arsaboo ? Sorry I must have missed it! Help! :-)

[arsaboo]

Not different....just a little more explicit about the force and keep_allowed config options. I think we have an agreement about the options.

[github-actions]

Thank you for the PR! The changelog has not been updated, so here is a friendly reminder to check if you need to add an entry.

[JOJ0]

Hi @arsaboo! I finally managed to find time to almost finish this PR. The general behaviour and docs of the new config options combinations are finished. If you want to, an "early" review would be super helpful. Since it probably also for you is a long time ago it might be interesting what you think if you read through the docs. Is it 100% clear what force/keep_allowed options do? Certainly but only if you have the time, some playing around and checking if it also really works that way would be great. Thanks a ton!

[arsaboo]

@JOJ0 this is AWESOME 🎉🎉

The docs look reasonably clear. I will play with this. The debug logs are great to see what is going on.

[JOJ0]

This something that hunts me for a while now already and were I would need your help @snejus: bea5aa2

Why is the plugin's self.whitelist truthy. I think it has something to do with things being cached. In the tests configuration I have explicitely set whitelist to False. The default whitelist is loaded though. Why?

Run the test with

pytest --disable-warnings -vvv 'test/plugins/test_lastgenre.py::test_get_genre[config_values5--mock_genres5-expected_result5]'

[JOJ0]

This something that hunts me for a while now already and were I would need your help @snejus: bea5aa2

Why is the plugin's self.whitelist truthy. I think it has something to do with things being cached. In the tests configuration I have explicitely set whitelist to False. The default whitelist is loaded though. Why?

Oh man....found it...of course, when setting the configuratoin after LastGenrePlugin is initialized self.whitelist is set up with default values already (which is True by default). This was giving me headache for a long time now 😮‍💨 So nevermind and thanks anyway :-)

[snejus]

I see what you mean regarding the flags - I think we simply have different use cases for this, so it's understandable we will see things differently here 😅

I'm happy with how things look right now (just have a look at my last comment) and I think it's time we move forward with it. It's gotten so large that it's painful to use GitHub UI to interact with things here 🤣

[JOJ0]

I see what you mean regarding the flags - I think we simply have different use cases for this, so it's understandable we will see things differently here 😅

Yes, it looks like it. I do hope though that the final outcome is acceptable for you. If I understood correctly what we have in common is, we prefer altering behaviour via the CLI instead of editing configuration files. I spent some time on nitpicking how --help should be worded and hope this is a concise enough result which might be already pointing in the right direction without reading the docs:

  -f, --force           modify existing genres
  -F, --no-force        don't modify existing genres
  -k, --keep-existing   combine with existing genres when modifying
  -K, --no-keep-existing
                        don't combine with existing genres when modifying

I'm happy with how things look right now (just have a look at my last comment) and I think it's time we move forward with it. It's gotten so large that it's painful to use GitHub UI to interact with things here 🤣

Yes at some point it get's pretty confusing with all the inline comments. Anyway, I would like to thank you again for thoroughly reading through it over and over. There were so many good suggestions. Nitpicking is good, and encouraged me to refactor things that I wouldn't have touched otherwise. There is so many ideas and room for improvement still for this plugin and this was a first step! ❤️

[arsaboo]

@JOJ0 Thanks for getting this done. Now that we have the basic code ready, it will be easier to improve upon. I will try improving the LastFM search (especially for international music).

[JOJ0]

Just FYI @snejus and @arsaboo , something broke in this wonderful third party plugin: mgoltzsche/beets-autogenre#5

[snejus]

@JOJ0 it's a private method that starts with an underscore, so feel free to do what needs to be done with it. Plugin maintainers accept this risk whenever they choose to reuse them.

I can only see positives - you've given some junior developer a chance to contribute and fix it there!

[JOJ0]

Sure, justaheads up, no worries it's easy to fix and since I finally want to use that plugin I'll open a PR over there :-)