Repeating calendar event shifted by 1 day when RRule is processed

[morozgrafix]

I found a bug in MagicMirror

Please make sure to only submit reproducible issues. You can safely remove everything above the dividing line.
When submitting a new issue, please supply the following information:

Platform: Raspberry Pi 2 - Raspbian GNU/Linux 12 (bookworm). Also reproduced and debugged on OS X - Sonoma 14.5 with Chrome 125.0.6422.142

Node Version: v20.14.0

MagicMirror² Version: 2.27.0

Description: Repeating weekly calendar event is offest by 1 day when processed by Magic Mirror. For example when event is scheduled on Thursday in the calendar, it appears to be scheduled for Friday when displayed on MM. I ran through the debug.js and it seems that RRules are affecting it. Full logs and steps to reproduce are below. Possibly related to #3342 and #3291 fixes. @jkriegshauser Not sure if this is an edge case that wasn't tested. My MM installations are set to America/Los_Angeles timezone.
The calendar is set in the same timezone, event is set in the same timezone. modules/default/calendar/calendarfetcherutils.js around line 300 seems to be adding an extra day.

Steps to Reproduce:
1 . Add following ics file to the calendar config or as url variable in debug.js Gist for basic.ics
2. Launch MM and observe the Test Entries to show up on Fridays at 17:30 or run modules/default/calendar/debug.js and observe the output.

Expected Results: Repeating calendar events to be displayed to occur on Thursday at 17:30

Actual Results: Events are displayed to occur on Fridays at 17:30

Configuration: Plain vanilla config.js based on config.js.sample with only addition of the sample calendar:

/* Config Sample
 *
 * For more information on how you can configure this file
 * see https://docs.magicmirror.builders/configuration/introduction.html
 * and https://docs.magicmirror.builders/modules/configuration.html
 *
 * You can use environment variables using a `config.js.template` file instead of `config.js`
 * which will be converted to `config.js` while starting. For more information
 * see https://docs.magicmirror.builders/configuration/introduction.html#enviromnent-variables
 */
let config = {
	address: "localhost",	// Address to listen on, can be:
	// - "localhost", "127.0.0.1", "::1" to listen on loopback interface
	// - another specific IPv4/6 to listen on a specific interface
	// - "0.0.0.0", "::" to listen on any interface
	// Default, when address config is left out or empty, is "localhost"
	port: 8080,
	basePath: "/",	// The URL path where MagicMirror² is hosted. If you are using a Reverse proxy
	// you must set the sub path here. basePath must end with a /
	ipWhitelist: ["127.0.0.1", "::ffff:127.0.0.1", "::1"],	// Set [] to allow all IP addresses
	// or add a specific IPv4 of 192.168.1.5 :
	// ["127.0.0.1", "::ffff:127.0.0.1", "::1", "::ffff:192.168.1.5"],
	// or IPv4 range of 192.168.3.0 --> 192.168.3.15 use CIDR format :
	// ["127.0.0.1", "::ffff:127.0.0.1", "::1", "::ffff:192.168.3.0/28"],

	useHttps: false,			// Support HTTPS or not, default "false" will use HTTP
	httpsPrivateKey: "",	// HTTPS private key path, only require when useHttps is true
	httpsCertificate: "",	// HTTPS Certificate path, only require when useHttps is true

	language: "en",
	locale: "en-US",
	logLevel: ["INFO", "LOG", "WARN", "ERROR"], // Add "DEBUG" for even more logging
	timeFormat: 24,
	units: "metric",

	modules: [
		{
			module: "alert",
		},
		{
			module: "updatenotification",
			position: "top_bar"
		},
		{
			module: "clock",
			position: "top_left"
		},
		{
			module: "calendar",
			header: "US Holidays",
			position: "top_left",
			config: {
				calendars: [
					{
						fetchInterval: 7 * 24 * 60 * 60 * 1000,
						symbol: "calendar-check",
						url: "https://gist.githubusercontent.com/morozgrafix/71379538d328e565130a89d6eb6d3137/raw/32ad67a4d0e76ceaab54e1d3d5bd55d65dc494b4/basic.ics"
					}
				]
			}
		},
		{
			module: "compliments",
			position: "lower_third"
		},
		{
			module: "weather",
			position: "top_right",
			config: {
				weatherProvider: "openweathermap",
				type: "current",
				location: "New York",
				locationID: "5128581", //ID from http://bulk.openweathermap.org/sample/city.list.json.gz; unzip the gz file and find your city
				apiKey: "YOUR_OPENWEATHER_API_KEY"
			}
		},
		{
			module: "weather",
			position: "top_right",
			header: "Weather Forecast",
			config: {
				weatherProvider: "openweathermap",
				type: "forecast",
				location: "New York",
				locationID: "5128581", //ID from http://bulk.openweathermap.org/sample/city.list.json.gz; unzip the gz file and find your city
				apiKey: "YOUR_OPENWEATHER_API_KEY"
			}
		},
		{
			module: "newsfeed",
			position: "bottom_bar",
			config: {
				feeds: [
					{
						title: "New York Times",
						url: "https://rss.nytimes.com/services/xml/rss/nyt/HomePage.xml"
					}
				],
				showSourceTitle: true,
				showPublishDate: true,
				broadcastNewsFeeds: true,
				broadcastNewsUpdates: true
			}
		},
	]
};

/*************** DO NOT EDIT THE LINE BELOW ***************/
if (typeof module !== "undefined") { module.exports = config; }

Additional Notes:
Debug Output is a bit too long, here is a link to a gist of the debug output: Full Debug Output
Small relevant part

[2024-06-14 21:34:25.408] [LOG]   Create fetcher ... 
[2024-06-14 21:34:25.474] [LOG]   Create fetcher done!  
[2024-06-14 21:34:25.590] [DEBUG] parsed data={"b847cf40-679a-4a77-b814-c9163692a785":{"type":"VTIMEZONE","params":[],"tzid":"America/Los_Angeles","LIC-LOCATION":"America/Los_Angeles","2bfb2433-6745-4d57-b55d-1a1e3ec778c3":{"type":"DAYLIGHT","params":[],"tzoffsetfrom":"-0800","tzoffsetto":"-0700","tzname":"PDT","start":"1970-03-08T10:00:00.000Z","datetype":"date-time","rrule":"RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=2SU","end":"1970-03-08T10:00:00.000Z"},"ca3f7f5e-7e4a-4405-99a5-4050700d3a9b":{"type":"STANDARD","params":[],"tzoffsetfrom":"-0700","tzoffsetto":"-0800","tzname":"PST","start":"1970-11-01T10:00:00.000Z","datetype":"date-time","rrule":"RRULE:FREQ=YEARLY;BYMONTH=11;BYDAY=1SU","end":"1970-11-01T10:00:00.000Z"},"end":"2024-06-16T04:34:25.576Z","method":"PUBLISH"},"78C53B9F-2152-4824-838D-F52000D3332B":{"type":"VEVENT","params":[],"start":"2023-08-04T00:30:00.000Z","datetype":"date-time","end":"2023-08-04T02:00:00.000Z","rrule":{"_cache":{"all":false,"before":[],"after":[],"between":[]},"origOptions":{"tzid":"America/Los_Angeles","dtstart":"2023-08-04T00:30:00.000Z","freq":2},"options":{"freq":2,"dtstart":"2023-08-04T00:30:00.000Z","interval":1,"wkst":0,"count":null,"until":null,"tzid":"America/Los_Angeles","bysetpos":null,"bymonth":null,"bymonthday":[],"bynmonthday":[],"byyearday":null,"byweekno":null,"byweekday":[4],"bynweekday":null,"byhour":[0],"byminute":[30],"bysecond":[0],"byeaster":null}},"exdate":[],"dtstamp":"2024-06-14T04:11:22.000Z","uid":"78C53B9F-2152-4824-838D-F52000D3332B","created":"2023-08-01T19:36:17.000Z","lastmodified":"2024-03-21T19:14:15.000Z","sequence":"0","status":"CONFIRMED","summary":"Test Event","transparency":"OPAQUE","method":"PUBLISH"},"vcalendar":{"type":"VCALENDAR","prodid":"-//Google Inc//Google Calendar 70.9054//EN","version":"2.0","calscale":"GREGORIAN","method":"PUBLISH","WR-CALNAME":"Family Calendar","WR-TIMEZONE":"America/Los_Angeles","WR-CALDESC":"Family Calendar"}} 
[2024-06-14 21:34:25.591] [DEBUG] There are 3 calendar entries. 
[2024-06-14 21:34:25.592] [DEBUG] Processing entry... 
[2024-06-14 21:34:25.592] [DEBUG] Processing entry... 
[2024-06-14 21:34:25.592] [DEBUG] Event:
{"type":"VEVENT","params":[],"start":"2023-08-04T00:30:00.000Z","datetype":"date-time","end":"2023-08-04T02:00:00.000Z","rrule":{"_cache":{"all":false,"before":[],"after":[],"between":[]},"origOptions":{"tzid":"America/Los_Angeles","dtstart":"2023-08-04T00:30:00.000Z","freq":2},"options":{"freq":2,"dtstart":"2023-08-04T00:30:00.000Z","interval":1,"wkst":0,"count":null,"until":null,"tzid":"America/Los_Angeles","bysetpos":null,"bymonth":null,"bymonthday":[],"bynmonthday":[],"byyearday":null,"byweekno":null,"byweekday":[4],"bynweekday":null,"byhour":[0],"byminute":[30],"bysecond":[0],"byeaster":null}},"exdate":[],"dtstamp":"2024-06-14T04:11:22.000Z","uid":"78C53B9F-2152-4824-838D-F52000D3332B","created":"2023-08-01T19:36:17.000Z","lastmodified":"2024-03-21T19:14:15.000Z","sequence":"0","status":"CONFIRMED","summary":"Test Event","transparency":"OPAQUE","method":"PUBLISH"} 
[2024-06-14 21:34:25.592] [DEBUG] start: Thu Aug 03 2023 17:30:00 GMT-0700 (Pacific Daylight Time) 
[2024-06-14 21:34:25.593] [DEBUG] end:: Thu Aug 03 2023 19:00:00 GMT-0700 (Pacific Daylight Time) 
[2024-06-14 21:34:25.593] [DEBUG] duration: 5400000 
[2024-06-14 21:34:25.593] [DEBUG] title: Test Event 
[2024-06-14 21:34:25.593] [DEBUG] Search for recurring events between: Fri Jun 14 2024 21:34:25 GMT-0700 (Pacific Daylight Time) and Fri Jun 13 2025 23:59:59 GMT-0700 (Pacific Daylight Time) 
[2024-06-14 21:34:25.594] [DEBUG] RRule: DTSTART;TZID=America/Los_Angeles:20230804T003000
RRULE:FREQ=WEEKLY 
[2024-06-14 21:34:25.599] [DEBUG] Title: Test Event, with dates: ["2024-06-21T00:30:00.000Z","2024-06-28T00:30:00.000Z","2024-07-05T00:30:00.000Z","2024-07-12T00:30:00.000Z","2024-07-19T00:30:00.000Z","2024-07-26T00:30:00.000Z","2024-08-02T00:30:00.000Z","2024-08-09T00:30:00.000Z","2024-08-16T00:30:00.000Z","2024-08-23T00:30:00.000Z","2024-08-30T00:30:00.000Z","2024-09-06T00:30:00.000Z","2024-09-13T00:30:00.000Z","2024-09-20T00:30:00.000Z","2024-09-27T00:30:00.000Z","2024-10-04T00:30:00.000Z","2024-10-11T00:30:00.000Z","2024-10-18T00:30:00.000Z","2024-10-25T00:30:00.000Z","2024-11-01T00:30:00.000Z","2024-11-08T00:30:00.000Z","2024-11-15T00:30:00.000Z","2024-11-22T00:30:00.000Z","2024-11-29T00:30:00.000Z","2024-12-06T00:30:00.000Z","2024-12-13T00:30:00.000Z","2024-12-20T00:30:00.000Z","2024-12-27T00:30:00.000Z","2025-01-03T00:30:00.000Z","2025-01-10T00:30:00.000Z","2025-01-17T00:30:00.000Z","2025-01-24T00:30:00.000Z","2025-01-31T00:30:00.000Z","2025-02-07T00:30:00.000Z","2025-02-14T00:30:00.000Z","2025-02-21T00:30:00.000Z","2025-02-28T00:30:00.000Z","2025-03-07T00:30:00.000Z","2025-03-14T00:30:00.000Z","2025-03-21T00:30:00.000Z","2025-03-28T00:30:00.000Z","2025-04-04T00:30:00.000Z","2025-04-11T00:30:00.000Z","2025-04-18T00:30:00.000Z","2025-04-25T00:30:00.000Z","2025-05-02T00:30:00.000Z","2025-05-09T00:30:00.000Z","2025-05-16T00:30:00.000Z","2025-05-23T00:30:00.000Z","2025-05-30T00:30:00.000Z","2025-06-06T00:30:00.000Z","2025-06-13T00:30:00.000Z"] 
[2024-06-14 21:34:25.599] [DEBUG] Rule has byweekday, checking for correction 
[2024-06-14 21:34:25.600] [DEBUG] West of GMT (tzOffset: 7) and hour=17 >= 24-7, Adding 1 day to Thu Jun 20 2024 17:30:00 GMT-0700 (Pacific Daylight Time) 
[2024-06-14 21:34:25.600] [DEBUG] West of GMT (tzOffset: 7) and hour=17 >= 24-7, Adding 1 day to Thu Jun 27 2024 17:30:00 GMT-0700 (Pacific Daylight Time) 
[2024-06-14 21:34:25.600] [DEBUG] West of GMT (tzOffset: 7) and hour=17 >= 24-7, Adding 1 day to Thu Jul 04 2024 17:30:00 GMT-0700 (Pacific Daylight Time) 
...

[2024-06-14 21:34:25.604] [DEBUG] initial tz=America/Los_Angeles 
[2024-06-14 21:34:25.604] [DEBUG] corrected tz=America/Los_Angeles 
[2024-06-14 21:34:25.604] [DEBUG] start date/time=Thu Aug 03 2023 17:30:00 GMT-0700 (Pacific Daylight Time) 
[2024-06-14 21:34:25.604] [DEBUG] start offset=-420 
[2024-06-14 21:34:25.605] [DEBUG] start date/time w tz =Thu Aug 03 2023 17:30:00 GMT-0700 (Pacific Daylight Time) 
[2024-06-14 21:34:25.605] [DEBUG] event date=Fri Jun 21 2024 17:30:00 GMT-0700 (Pacific Daylight Time) 
[2024-06-14 21:34:25.605] [DEBUG] event offset=-420 hour=17 event date=Fri Jun 21 2024 17:30:00 GMT-0700 (Pacific Daylight Time) 
[2024-06-14 21:34:25.605] [DEBUG] adjustHours=0 
[2024-06-14 21:34:25.605] [DEBUG] initial tz=America/Los_Angeles 
[2024-06-14 21:34:25.605] [DEBUG] corrected tz=America/Los_Angeles 
[2024-06-14 21:34:25.605] [DEBUG] start date/time=Thu Aug 03 2023 17:30:00 GMT-0700 (Pacific Daylight Time) 
[2024-06-14 21:34:25.605] [DEBUG] start offset=-420 
[2024-06-14 21:34:25.605] [DEBUG] start date/time w tz =Thu Aug 03 2023 17:30:00 GMT-0700 (Pacific Daylight Time) 
[2024-06-14 21:34:25.605] [DEBUG] event date=Fri Jun 28 2024 17:30:00 GMT-0700 (Pacific Daylight Time) 
[2024-06-14 21:34:25.606] [DEBUG] event offset=-420 hour=17 event date=Fri Jun 28 2024 17:30:00 GMT-0700 (Pacific Daylight Time) 
[2024-06-14 21:34:25.606] [DEBUG] adjustHours=0 
[2024-06-14 21:34:25.606] [DEBUG] initial tz=America/Los_Angeles 
[2024-06-14 21:34:25.606] [DEBUG] corrected tz=America/Los_Angeles 
[2024-06-14 21:34:25.606] [DEBUG] start date/time=Thu Aug 03 2023 17:30:00 GMT-0700 (Pacific Daylight Time) 
[2024-06-14 21:34:25.606] [DEBUG] start offset=-420 
[2024-06-14 21:34:25.606] [DEBUG] start date/time w tz =Thu Aug 03 2023 17:30:00 GMT-0700 (Pacific Daylight Time) 
[2024-06-14 21:34:25.606] [DEBUG] event date=Fri Jul 05 2024 17:30:00 GMT-0700 (Pacific Daylight Time) 
[2024-06-14 21:34:25.606] [DEBUG] event offset=-420 hour=17 event date=Fri Jul 05 2024 17:30:00 GMT-0700 (Pacific Daylight Time) 
....

[2024-06-14 21:34:25.652] [DEBUG] Processing entry... 
[2024-06-14 21:34:25.653] [INFO]  Calendar-Fetcher: Broadcasting 52 events from https://gist.githubusercontent.com/morozgrafix/71379538d328e565130a89d6eb6d3137/raw/32ad67a4d0e76ceaab54e1d3d5bd55d65dc494b4/basic.ics. 
[2024-06-14 21:34:25.656] [LOG]   [
  {
    title: 'Test Event',
    startDate: '1719016200000',
    endDate: '1719021600000',
    fullDayEvent: false,
    recurringEvent: true,
    class: undefined,
    firstYear: 2023,
    location: false,
    geo: false,
    description: false
  },
  {
    title: 'Test Event',
    startDate: '1719621000000',
    endDate: '1719626400000',
    fullDayEvent: false,
    recurringEvent: true,
    class: undefined,
    firstYear: 2023,
    location: false,
    geo: false,
    description: false
  },
  {
    title: 'Test Event',
    startDate: '1720225800000',
    endDate: '1720231200000',
    fullDayEvent: false,
    recurringEvent: true,
    class: undefined,
    firstYear: 2023,
    location: false,
    geo: false,
    description: false
  }
 ....
] 
[2024-06-14 21:34:25.656] [LOG]   ------------------------------------------------------------ 

Calendar file (copy of the gist linked above)

BEGIN:VCALENDAR
PRODID:-//Google Inc//Google Calendar 70.9054//EN
VERSION:2.0
CALSCALE:GREGORIAN
METHOD:PUBLISH
X-WR-CALNAME:Family Calendar
X-WR-TIMEZONE:America/Los_Angeles
X-WR-CALDESC:Family Calendar
BEGIN:VTIMEZONE
TZID:America/Los_Angeles
X-LIC-LOCATION:America/Los_Angeles
BEGIN:DAYLIGHT
TZOFFSETFROM:-0800
TZOFFSETTO:-0700
TZNAME:PDT
DTSTART:19700308T020000
RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=2SU
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0700
TZOFFSETTO:-0800
TZNAME:PST
DTSTART:19701101T020000
RRULE:FREQ=YEARLY;BYMONTH=11;BYDAY=1SU
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTART;TZID=America/Los_Angeles:20230803T173000
DTEND;TZID=America/Los_Angeles:20230803T190000
RRULE:FREQ=WEEKLY
EXDATE;TZID=America/Los_Angeles:20230817T173000
EXDATE;TZID=America/Los_Angeles:20230914T173000
EXDATE;TZID=America/Los_Angeles:20230921T173000
EXDATE;TZID=America/Los_Angeles:20230928T173000
EXDATE;TZID=America/Los_Angeles:20231026T173000
EXDATE;TZID=America/Los_Angeles:20231123T173000
EXDATE;TZID=America/Los_Angeles:20231207T173000
EXDATE;TZID=America/Los_Angeles:20240215T173000
EXDATE;TZID=America/Los_Angeles:20240222T173000
EXDATE;TZID=America/Los_Angeles:20240321T173000
DTSTAMP:20240614T041122Z
UID:78C53B9F-2152-4824-838D-F52000D3332B
CREATED:20230801T193617Z
LAST-MODIFIED:20240321T191415Z
SEQUENCE:0
STATUS:CONFIRMED
SUMMARY:Test Event
TRANSP:OPAQUE
END:VEVENT
END:VCALENDAR

[sdetweil]

yes the rrule processor has trouble with some date combinations.

not our code. and we can't tell as we don't process the ics file in MagicMirror code

[morozgrafix]

I did a little bit more digging and the extra day is added in calendarfetcherutils.js around line 300:

// RRule can generate dates with an incorrect recurrence date. Process the array here and apply date correction.
					if (hasByWeekdayRule) {
						Log.debug("Rule has byweekday, checking for correction");
						dates.forEach((date, index, arr) => {
							// NOTE: getTimezoneOffset() is negative of the expected value. For America/Los_Angeles under DST (GMT-7),
							// this value is +420. For Australia/Sydney under DST (GMT+11), this value is -660.
							const tzOffset = date.getTimezoneOffset() / 60;
							const hour = date.getHours();
							if ((tzOffset < 0) && (hour < -tzOffset)) { // east of GMT
								Log.debug(`East of GMT (tzOffset: ${tzOffset}) and hour=${hour} < ${-tzOffset}, Subtracting 1 day from ${date}`);
								arr[index] = new Date(date.valueOf() - oneDayInMs);
							} else if ((tzOffset > 0) && (hour >= (24 - tzOffset))) { // west of GMT
								Log.debug(`West of GMT (tzOffset: ${tzOffset}) and hour=${hour} >= 24-${tzOffset}, Adding 1 day to ${date}`);
								arr[index] = new Date(date.valueOf() + oneDayInMs);
							}
						});
			 

if I’m not mistaken the time zone is set to null just before that code is executed. I think if calendar/event is matching the local time zone the adjustment shouldn’t be made. I tried to comment out the code above and it displays the date correctly, but of course it breaks some of the existing tests. I think that it needs additional checks before execution. Maybe checking that event timezone doesn’t match local timezone (although I didn’t dig into the code much yet)

thanks

[sdetweil]

that's my code trying to deal with this problem. no good fix yet

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[morozgrafix]

Bump, too keep it open as it’s still an issue. Hopefully @sdetweil has some time to look at it and wontfix tag is just labeled by a bot.

[Pollard-J]

yea ive just updated mine and it is still displaying full day events as 2 days.

[sdetweil]

see this.. waiting on user test for submitting PR here and node-ical
all testcases in both run successfully..

here is a test version of the fixes for all kinds of date problems.

best to make a new folder and git clone there

git  clone https://github.com/sdetweil/MagicMirror
cd ~/MagicMirror 
git checkout fixcaldates
npm run install-mm

copy your config.js and custom.css from the prior folder
and the non-default modules you have installed..

this ONLY changes the default calendar
but DOES ship an updated node-ical library too

if you need to fall back, just rename the folders around again so that
your original is called MagicMirror

all the testcases for node-ical and MagicMirror execute successfully.

the 'BIG' change here is to get the local NON-TZ dates for the
rrule.between()

all the checking and conversion code is commented out or not used
the node-ical fixes are for excluded dates (exdate) values being adjusted for DST/STD time.. waiting to submit that PR

one fix in calendar.js for checking if a past date was too far back,
but it never checked to see IF the event date was in the past.. (before today) so it chopped off too many

and one change in calendarfetcher.js to put out a better diagnostic message of the parsed data.. (exdate was excluded cause JSON stringify couldn't convert the complex structure)

[sdetweil]

@Pollard-J can u try my new fork?

and post a calendar event I can test with.. thx
BEGIN:VEVENT
...
END:VEVENT

[Pollard-J]

@sdetweil i just tried this but i'm not that good when i come into problems. I cloned your file of MagicMirror, but it wouldn't execute the next 2 lines properly.
git checkout fixcaldates
npm run install-mm

ill just wait till its released. thank you for trying to help me.

[sdetweil]

sorry i left out a critical step
after the git clone

cd ~/MagicMirror 

fixed above too

[morozgrafix]

@sdetweil thanks for the fix. I just tested the issue and fixcaldates branch is displaying events correctly. I only tested with config.js from original description of this bug. Repeated events should occur every Thursday at 17:30. Below are screenshots from master (not fixed) and fixcaldates (fixed) branches. Thank you for digging into this issue, hopefully other issues can also be tested on that branch and your PR goes through soon.

Before:

After: