Thursday, 2020-01-09

*** SpeedEvil is now known as Guest5453101:56
*** zbenjamin_ is now known as zbenjamin02:39
*** frinring_ is now known as frinring05:15
flypigdcaliste, could we have a quick chat if you're available some time today?09:08
dcalisteflypig: sure, whenever you want.09:50
dcaliste(sorry, missed your ping before)09:50
flypigdcaliste, thanks, I got distracted myself too.10:11
flypigNow?10:11
dcalisteYes, I've one hour before colleagues will ask to go to lunch ;)10:11
flypigOkay great, should be plenty :)10:13
flypigI did some work on folder sync to extend to EAS+Google. Would you be willing to review it?10:13
dcalisteflypig: yes, within a one week time frame though, because I don't much about this code.10:14
dcalisteI mean, no answer guaranteed before one week ;)10:14
flypigAh, okay. I don't think that's going to be a problem. Let me check, one second...10:15
flypigYes, that's no problem.10:18
flypigYou have access to the internal repos, right?10:19
flypigBut not the bug reports?10:19
dcalisteOk, when your MRs are ready give me the URLs, but we need to ask jpetrell to give me read access on these repos first.10:19
dcalisteRight, I cannot read the bug reports.10:20
flypigOkay, I'll ask jpetrell about access to the repos. I don't think access to the bug report will help, but the repos are crucial (obviously!)10:21
dcalisteIndeed. I'll do my best to try to understand these parts and do the review thoroughly.10:23
flypigdcaliste, that part I'm not at all worried about :) I'd really appreciate your -- always helpful -- comments.10:24
dcalisteIf I'm not anymore on IRC, I'll check on bitbucket to see when new repo appears in my panel and give a look to the pending MR to find yours.10:25
flypigIf you can access this on bitbucket, then I'll add links to all the other repos there: jolla/ui-jolla-email/pull-requests/460/10:26
dcalisteYes, thanks, I can see this one.10:27
dcalisteOk I've seen your list. I'm currently missing access to qmf-eas-plugin and base-sailfish-eas10:29
dcalisteIt's ok for the account setting repo and the sailfishos ones (obviously).10:29
flypigOkay, I'll get your account set up to access those two as well.10:38
flypigdcaliste, I've asked for you to have access and will ping you to let you know when it's sorted.10:40
dcalisteThanks flypig.10:41
flypigThanks again with this. Concerning the caldav stuff, thanks for that ics file.10:41
flypigI've been trying it this morning. I may have questions, but I should check further about it now first. If you're around after lunch, I may query you if that's okay.10:42
dcalisteflypig, about that exdate issue I don't know if you see my comments a bit before https://git.sailfishos.org/mer-core/buteo-sync-plugin-caldav/merge_requests/59#note_3918610:42
flypigBut hopefully I'll just be able to merge it today.10:42
dcalisteThe last part about how upstream they are handling this.10:42
flypigYes, I did read those.10:42
dcalisteI'm still wondering if we should patch caldav plugin, like I'm proposing in this MR.10:43
dcalisteOr go the upstream solution of relaxing the test in entry of dissociateSingleOccurrence in mkcal…10:43
flypigSo the two approaches are: 1. fix up the ical data before it reaches mkcal; or 2. Relax the mkcal check.10:44
dcalisteBesides, no problem to ping me after lunch if you have questions.10:44
flypigIn the case of 1, everything works correctly, but the data gets changed on the server.10:44
dcalisteNot exactly, the 1. solution is to fix the stored parent incidence to remove of offending EXDATE.10:45
flypigIn the case of 2, no data is changed, but technically it may store non RFC-compliant data.10:45
dcalisteExact for case 2.10:45
flypigSo for case 1, what is the downside?10:45
flypigThe result is basically the same, no?10:46
flypig(I mean, for the user)10:46
dcalisteThe downside I see is additional code in caldav that is not strictly related too, and become useless if we move upstream to the new dissociate routine from kf5-kcalendarcore.10:46
dcalisteFor the user, both are fine ;)10:46
dcalisteHopefully.10:46
flypigOkay, so I agree that architecturally, and in the longterm, case 2 is nicer.10:47
flypigOn the other hand, case 1 is working, offers limited scope for regression, and can be deployed more quickly.10:48
dcalisteI'm sorry, I need to leave for lunch. And I agree with your positive points for solution 1.10:48
flypigOkay, enjoy your lunch :) Hopefully I can catch you later.10:49
dcalisteParticularly, it has been "tested" by the reporter in TJC and he feels fine with it.10:49
dcalisteI'll keep my IRC window on during the afternoon.10:49
dcalisteSee you.10:49
flypigThanks :) Speak later then.10:49
dcalisteHello again flypig, we can resume our discussion whenever you prefer.12:08
flypigGreat, thanks. I hope you had a good lunch :) I do have more questions though I'm afraid.12:08
dcalisteSo let's start.12:08
flypigJust to let you know, jpetrell has asked for you to have access, but has to check some licence-related issue for sailfish-eas.12:09
flypigSo that may take a little longer.12:10
dcalisteOk, no problem. Thanks for the info.12:10
flypigConcerning caldav, some things came to mind.12:10
flypigMy feeling is that, it's probably better to get the fix in. Especially given the dissociate code is already "in the wrong place".12:11
flypigSo this doesn't seem to be making it worse.12:12
flypigHowever, I'm still having trouble testing it. Can I ask how you got the ics file onto your phone?12:12
dcalisteI agree. I was undecided on the matter, and it was better to discussed it a bit. I agree with your conclusions.12:13
flypigMy conclusions aren't by any means the final word. But if you're happy with that, then it seems Chris is too (from what he says on the bug).12:13
dcalisteFor the ics, let me remember…12:14
flypigSorry: when I say "what he says on the bug", I mean the "on the MR".12:15
dcalisteAbout the ICS, I can't find the exact way I tested it. I think I highjacked some of the ICS data in tests/noteboosyncagent/data/, let the test run and see the crash with the error message from the TJC logs.12:18
flypigOkay, that makes sense. I'm having trouble because Thunderbird refuses to import it. If there's no obviously 'natural' way to do it, I should also hack around a bit to make it happen.12:19
flypigHowever, when trying to import it, it raised the issue that this won't fix the problem for importing the file directly.12:20
flypigDo you have any thoughts about a solution for that, and whether it's worth worrying about.12:21
dcalisteThen, I figure out the issue was with this EXDATE defined in the parent event. I accidentaly change a earlier ignored error https://git.sailfishos.org/mer-core/buteo-sync-plugin-caldav/commit/2692a7daf5698ed3759b032848bfd9a8beafebff#25b531d153d0472485d380353df2c41c0cb5532c_1058_988 (critical error flag is not set) into a failing error.12:22
dcalisteAbout your question:12:22
dcalisteI don't think it will becomes an issue during upsync, because all EXDATE for exceptions are removed before upsyncing (to respect RFC).12:23
dcalisteAnd because these EXDATEs are supposed to be added by us for mkcal own reasons.12:23
dcalisteThe fact that the EXDATE already exists and will be removed during upload is a side effect.12:23
flypigBut when the EXDATES are removed, that'll change the result, won't it?12:24
dcalisteThe lines for the removal : https://git.sailfishos.org/mer-core/buteo-sync-plugin-caldav/blob/master/src/incidencehandler.cpp#L37112:25
flypigI'm unclear as to why it does that.12:25
dcalisteNo, it should not change the result according to my understanding of the RFC. But it may change the server behaviour if it is expecting an EXDATE for an exception…12:25
dcalisteOk, let's start from the beginning on EXDATEs (from my understanding):12:26
flypigPlease!12:26
dcaliste- recurring events can have exceptions.12:26
dcaliste- a recurring event is defined by its recurring dates.12:26
dcaliste- you can define recurring dates by various means, including rules or explicit RDATEs.12:27
dcaliste- you can exclude some dates from the recurrence with rules or explicit EXDATEs.12:27
flypigOkay, all makes sense so far.12:28
dcaliste- at the end, you have a "parent" event that have occurrences on given dates.12:28
flypigOne second, please: question12:28
dcalistesure go ahead.12:29
flypigDoes a defined EXDATE recur?12:29
dcalisteNo but you can defined EXRULES.12:29
flypigOkay, that's what I thought. Thanks.12:29
flypigPlease go ahead!12:29
dcalisteThe RFC says then that you can defined exceptions, or let say modify a specific occurrence using a new event.12:31
dcalisteThis event MUST define a RECURRENCE-ID entry with the DTSTART value of the occurrence it is made to replace.12:32
dcalisteThen, it can have various other value for new DTSTART, DTEND, SUMMARY…12:32
flypigOkay.12:33
dcalisteSo if you want to define a week-day meeting, you create an event with a RRULE:byWeekDay12:33
dcaliste(I don't remember the excat syntax)12:33
flypigSure, n.p. :)12:33
dcalisteThen, if one of the meeting is during a public holyday, you can remove this specific occurrence.12:34
dcalisteTo do it, add an EXDATE in the parent incidence with the public holyday time stamp.12:34
flypigI can see how you do that using EXDATE. But can you do that using RECURRENCE-ID too?12:34
dcalisteThis is what happens in the calendar app, when you remove an occurrence of a recurring event.12:35
dcalisteIndeed, you can also define another event with the same UID and a RECURRENCE-ID set for this public holyday.12:35
flypigBut without any DTSTART/DTEND defined?12:36
dcalisteIn that case, the RFC says that the original event of that holyday is _replaced_ by this new event.12:36
dcalisteI don't know if DTSTART is mandatory, I think so.12:36
flypigBut replaced isn't the same as removed...?12:36
dcalisteIndeed.12:36
dcalisteTo remove, you use EXDATE.12:36
dcalisteTo modify locally, you use RECURRENCE-ID.12:37
dcalisteThen mKCal is arriving…12:37
flypigOkay, that all makes sense.12:37
flypigBut, if EXDATE is the only way to remove a repeating occurrence, then it's necessary for some arrangeents?12:37
flypig*arrangements12:37
dcaliste… and by a clever idea is introducing a bit of confusion (no blame intended).12:37
dcalisteSorry ?12:38
flypigSorry, that's a sticking point for me. Can we hold the thought about clever ideas for a second.12:38
dcalisteSure, of course.12:39
dcalisteAn event with RECURRENCE-ID defined will replace its original occurrence, the original occurrence should not be visible on any calendar UI.12:39
flypigSuppose I have a recurring event every Friday. I want to remove it from Christmas Day 2020. Then an EXDATE is used for that. If the EXDATE is stripped on upload, then there's no way to represent this arrangement.12:40
dcalisteAh, ok.12:40
dcalisteWe're removing EXDATEs only for EXDATEs matching RECURRENCE-IDs, not the others.12:40
dcalisteSo on upload in your case, the EXDATE is kept.12:41
dcalisteYour next question may be : but why do we have EXDATE for RECURRENCE-IDs in the database ?12:42
flypigYes, that's a good question. Let's pretend I was going to ask that ;)12:42
dcalisteThat's because of mKCal.12:42
flypigAre we back to the clever idea?12:43
dcalisteIt's not present in kcalcore, current SFOS version or upstream.12:43
dcalisteYes, it's the clever idea.12:43
dcalisteWhen you want to display your occurrences in a calendar UI, you need a function to expand your recurrences.12:43
dcalistein mKCal, it's rawExpandedEvents().12:44
dcalisteThere is no one in current kcalcore. But they introduce a nice iterator in upstream for this.12:44
dcalistekcalcore is providing functions to get list of occurrences from given recurrence rules and exceptions.12:45
dcalisteSo in mKCal, we're building a list of occurrences from the list of incidences (recurring or not).12:46
flypigThis is the function you recently refactored.12:47
dcalisteIndeed.12:47
flypigFor the record, it made my head hurt.12:47
flypig(but the blame lies with the iCal format)12:47
dcalisteThe initial thoughts of mKCal authors, I guess, was : "how to exclude all the RECURRING-IDS from this list ?"12:47
flypigOkay.12:48
dcalisteAnd, instead of doing it by hand, they suggest to use the EXDATEs !12:48
dcalisteThat's the clever idea.12:48
dcalisteSo kcalcore will automatically remove the replaced occurrences.12:48
flypigI think I'm following, but let me check.12:49
dcalisteAnd the list will contain only the occurrences, not the replaced ones, and the replacing events.12:49
flypigRather than removing items, they add exceptions.12:50
dcalisteThat's it.12:50
flypigBut kcalcore doesn't normalise this.12:50
dcalisteExact.12:50
dcalistemKCa is adding the exception when using the dissociateSingleOccurrence() rotuine.12:50
dcalisteSee the end of the routine in extendedcalendar.cpp.12:51
dcalisteBut of course, conversion to iCal format in kcalcore is not aware of this.12:51
flypigOne second...12:51
dcalisteSo every user of kcalcore exporter (plugins, calendar tool in nemo-qml-plugin-calendar) have to remove the offending EXDATEs.12:52
dcalisteSure…12:52
flypigRECURRING-IDS can reference VEVENTS that contain complex exceptions. These can't always be translatable into EXDATEs.12:53
flypigSo sometimes something will have to be added.12:54
dcalisteIt can. a RECURRENCE-ID is the DTSTART of the replaced occurrence. So this DTSTART can always be used a EXDATE if one wants.12:54
dcalisteThe EXRULEs and EXDATEs are used to create the occurrences list in addition to the RRULEs and RDATEs of course. For the parent event.12:55
flypigOkay, got it. And the DTSTART it replaces is only ever a single event.12:56
dcalisteYou can defined for instance a recurrence that works every week day with the appropriated RRULE, then remove every Wednesday with a right EXRULE.12:56
flypigBut you wouldn't create a child for that?12:57
flypigJust include it in the parent.12:57
dcalisteRight.12:57
dcalisteA child is to *modify* an occurrence.12:57
dcalisteThen, you can replace (or modify) any occurrence with as many new incidences with RECURRENCE-IDs.12:58
dcalisteBut RECURRENCE-ID is unique per child and define a unique time stamp.12:58
dcalisteAnd mKCal is adding these unique tiestamps as EXDATEs. to the parent.12:58
flypigBut I can see why they convert RECURRENCE-IDs into exclusions. Otherwise they might have to turn it into two separate RRULES with some complexity.12:58
flypigthey = mkcal devs.12:59
dcalisteSo the generation of the occurrence list is indeed *replacing* the exceptions12:59
dcalisteWithout additional work on the occurrence list.12:59
dcalisteAnd yes, they can only use EXDATE because there is no pattern to the exceptions (they are created one by one in the UI by saying "modify just this event not the whole series").13:00
dcalisteSo, to summarise:13:01
flypigOkay, this all seems to makes sense.13:01
dcaliste- parent events can recur, with complex recurrence rules, there can be "removed" occurrence in the pattern, using EXRULE and EXDATE.13:02
dcaliste- some child events can be individually modified using the RECURRENCE-ID, the original occurrence should not appear then in a calendar UI.13:02
dcaliste- mKCal is adding EXDATEs for each RECURRENCE-ID to avoid the pain of removing the replaced occurrences in expandRawEvents().13:03
dcaliste- these spurious EXDATEs must then be removed by hand befaore upload.13:03
dcaliste(in every client using the API…)13:04
dcalisteTo come back to the bug at hand:13:04
flypigIt seems to me that there's a mismatch between ical functionality and kcalcore functionality.13:04
flypigRECURRENCE-ID is redundant in the spec, and kcalcore doesn't support attaching exceptions to parents.13:05
dcalisteWhy is it redundant ?13:05
flypigAll it does is remove an event, and EXDATE does that already. There's no reason to tie an exception to a removal.13:06
flypigIf I change an event, I actually don't care which particular piece I changed. It may as well be an EX with an R.13:06
dcalisteAh, sorry a misunderstanding manybe :13:06
dcalisteRECURRENCE-ID is to be added in the child event.13:06
flypigYes, but it could also be replaced by an EXDATE in the child event. I'm not sure I've ever seen a client that exposes the deleted event that it replaces.13:07
flypig(exposes to the user, that is)13:07
flypigThe mkcal conversion is technically losing data.13:08
dcalisteNo, the child event is not recurring, so cannot have an EXDATE.13:08
flypigSorry, my mistake: the EXDATE could be in the parent.13:08
dcalisteAh, ok.13:08
flypigAnyway, that's a digression. We should return to the bug in hand.13:09
dcalisteAbout the bug:13:10
dcalistethe coming ICS is faulty in my opinion because the parent is _not_ recurring on the date the child event is defined by its RECURRENCE-ID.13:10
dcalistein the beginning of dissociateSingleOccurrence(à in mKCal, there is a test saying13:11
dcaliste"do the child event I'm receiving is happening at a recurring time of the parent?"13:11
dcalisteSince the EXDATE is set to the value of the RECURRENCE-ID, the answer to this questionis *no*.13:12
dcalisteBefore my rework of the caldav incidence update code, this negative return was ignored.13:12
dcalisteBut, I didn't notice that I changed the behaviour by making this negative return a fatal error in the plugin.13:13
dcalisteSo a user with such a faulty event is seeing that before 3.2.1.20 it was syncing (ignoring the exception but syncing anyway), while now it refuses to sync.13:14
flypigYes, this makes sense. Looking at the ical spec, this looks ambiguous as an error to me.13:14
dcalisteMy suggestion in this MR, is to remove the offending EXDATE if I'm receiving a RECURRENCE-ID on a given EXDATE.13:15
dcalisteNow, the question is what happen on export ?13:15
dcalisteBefore 3.2.1, the EXDATE was sent back, but not the exception (since it was ignored on importation).13:16
dcalisteWith the MR, the EXDATE is removed and the exception is sent back.13:16
dcalisteSo we're modifying the event with respect to what was sent initially.13:17
dcalisteBut at least we're sending the exception back…13:17
flypigOne second.13:19
dcalisteSure…13:21
flypigIt looks like there are three different approaches here:13:22
flypig1. On import remove exception EXDATE if it matches a RECURRENCE-ID (new behaviour).13:22
flypig2. On import ignore exception if EXDATE matches RECURRENCE-ID (old behaviour).13:22
flypig3. On import/export, keep EXDATE and RECURRENCE-ID (non-destructive behaviour).13:22
flypigIf this is correct, is it also correct to say that: 1 and 2 look different to the user (1. excepation retained vs. 2. excpeation lost). Whereas...13:23
flypigBoth 1 and 3 looks the same to the user, but technically data is lost by 1?13:24
dcalisteI think you sumarize it in a nice and correct way.13:25
dcalisteI agree on all your points.13:25
flypigGreat, then that's a good sign I'm following.13:25
dcalisteSolution 3 would be the best, but is "hard" to implement.13:26
dcalisteIn the sense, that the way I can see is to add a flag 'keep my EXDATE please' to the child event.13:26
dcalisteOn export, remove the EXDATEs for every children, except if they have the flag. (and then remove the flag also, otherwise it will be upsynced also).13:27
dcalisteThis would add a lot of boiler plate code… But everything is kept in that case, which ensure less probability of bug or error on upsync.13:28
flypigOn export, how does mkcal tie children to parents currently?13:28
dcalisteThere is a kcalcore method called Calendar::instances(), which is returning all children of a given incidence.13:29
dcalisteThen, we're creating a memory calendar with the parent and all the instances. and export this memory calendar using the kcalcore ical formater.13:29
flypigSo the RECURRENCE-ID can be reconstructed from this.13:30
flypigPresumably then kcalcore could filter out all events with RECURRENCE-IDs.13:30
dcalisteWhat do you mean ?13:31
flypigYou mentioned above that mkcal adds an exception for every RECURRENCE-ID, yes?13:32
dcalisteYes, it is adding to the parent an EXDATE for every RECURRENCE-ID of its children.13:32
flypigAnd the RECURRENCE-ID is what captures the child-parent relationship.13:33
flypig(in an ical file, that is).13:33
dcalisteYes (for a given UID value).13:33
flypigSo kcalcore could filter out both EXDATES *and* events that RECURRENCE-IDs refer to, just like the ical spec does.13:36
dcalisteSure, that's what we're doing at the moment.13:37
dcalisteOr, I'm not understanding the *and* part…13:38
dcalisteLet's take an example:13:38
flypigI think I have the wrong component. I mean that this could happen in rawExpandedEvents()13:38
flypigWhich is mkcal.13:39
dcalisteAh, great indeed, that is my proposition ;) let me find it somewhere…13:39
dcalistehttps://git.sailfishos.org/mer-core/buteo-sync-plugin-caldav/merge_requests/58#note_3911913:40
dcalisteYes, that would solve our problem by isolating the issue in mKCal.13:40
flypigOkay, yes, makes sense. And then no data loss in either direction.13:41
dcalisteI need though to check that only expandRawEvents() is used with respect to this, or if there are other routine that needs filtering.13:41
flypigYes, I can see it's not without risk.13:42
dcalisteIf it is working, we would have something that is much closer to upstream and ical format, which is great in my opinion, and sorry for the clever idea…13:43
flypig:) Yes, that would be nice.13:43
flypigPresumably there's an argument that we should adopt the newer upstream, rather than fix old upstream.13:45
dcalisteImplementing this is creating a transition issue also because currently stored events have been poluted with a lot of EXDATEs.13:45
dcalisteAh, but this is mKCal and we're upstream for this package…13:45
flypigYes, good point.13:46
flypigI guess that pollution doesn't really do any harm to the user.13:47
flypigEven on transition.13:47
flypigSorry: even after transition.13:47
dcalisteThe point about being closer to upstream is standing though, because they have added in kf5-CalendarCore many functions from extendedcalendar in mKCal have been re-implemented, so when moving to kcalcore upstream, we will have to clean later on to transition to the new routines.13:48
dcalisteabout the EXDATEs polution, yes, you're right, but the code to handle it on export will have to be kept eternally n every client…13:48
flypigYes, the import code can go, but not the export code. I suppose there could be a one-pass fix-up of the database.13:49
flypigSend everything through the export code, then import it all back again ;)13:50
dcalisteThat's true. We can figure out a oneshot for this. But when doing we'll have to be sure of ourselves, otherwise we will loot all user's databases…13:51
flypigYes, it's easy to discuss it consequence-free in the abstract.13:52
flypigOkay, thank you for the in-depth explanations. That's really helpful, but... I think this shouldn't affect the merging of your MR!58 or MR!59.!13:53
dcalisteI agree. MR!58 and MR!59 are fixing two different bugs even if related to this annoying EXDATE list added by mKCal.13:54
flypigThis also helps me better understand what I ought to be seeing when importing the example file. I'm afraid I still want to see it in action: your explanation makes perfect sense, but it's just a matter of due diligence on my part.13:56
dcalisteI fully agree.13:58
flypigOkay, thanks again dcaliste, that covers things for me. I'll let you know once the EAS repo is sorted out, but if you want to already comment on the rest of them, I'd be very pleased for your input.14:03
dcalisteI'll begin to review it and continue during the coming days and next week.14:03
dcalisteI'm adding at the moment the ICS data for the faulty case in the test to see if I can demonstrate it and check it properly.14:04
dcalisteAh, I remember also now looking at the code, that I added a test (not from ICS data but from handly crafted events) in updateIncidence_data() in notebooksyncagent test.14:06
flypigWill you update the MR?14:06
flypig... no need by the sound of it.14:06
dcalisteDepends, because the already added test is enough to test the behaviour.14:06
dcalisteBut I would be glad to see the ICS data work also, just for confidence !14:08
flypigIt's fine either way by me then. However, would you mind updating the title to add "Contributes to JB#48484" and rebasing if necessary?14:08
dcalisteSure, no problem.14:08
flypigThank you: all your work is really very much appreciated.14:08
dcalisteflypig, ok, I've just force pushed a modified MR with the addition of the ICS data and a test to ensure that this kind of ICS is imported without issue. This test is failing with master branch.14:32
flypigdcaliste: great, thank you for doing that. Very nice :)15:01
dcaliste@flypig, you're welcome15:06
zaggynlIs there an official wireguard package for sailfish? I found https://together.jolla.com/question/182324/wish-wireguard/ but not sure if is proper way to install16:01
x2sdoesn't look like there is any16:06
enigma9o7I'm new to sfos so this may be a dumb question.  How do i search packages to find which contains a file I want?18:01
enigma9o7(specifically, I'm tryiing to make something that requires 'header files' and believe I need mmem.h)18:02
r0kk3rzenigma9o7: sounds like you need kernel sources, which im not sure are packaged21:47

Generated by irclog2html.py 2.17.1 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!