How to handle persistent chat rooms / conversations

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

How to handle persistent chat rooms / conversations

David Woodhouse
I'm working on a new prpl for a protocol where conversations are
persistent — all the historical messages are available.

For chat rooms, I've handled this by fetching all old messages when the
room is joined. The first time, I fetch everything; then I remember the
latest message (using purple_account_set_string(…"last-msg-$roomid"…)
and subsequent joins will only fetch the messages since then.

This isn't perfect, but it seems to work OK. Anyone got better
suggestions? Is there a way to mark chats as persistent from the back
end, rather than just in the UI? That would help.

It gets slightly more complex when I get to other "conversations" in
this protocol. Those are mostly two-party conversations which are
basically IM, but there are also multi-party conversations.

The simplest conversation has two parties, and should basically be
represented as IM. But how do I "insert" the history when the account
is first opened, including both incoming and outgoing messages? Is
there a way to get those to be represented as 'chat' conversations in
Pidgin? I could automatically call serv_got_joined_chat() for incoming
messages and even on outgoing... but the normal IM window would still
exist. Any other ideas? 

Then there's multi-party conversations, which I suppose will need to be
handled as chat conversations — will I need to do my own UI actions to
allow for creating, listing and joining those? 




_______________________________________________
Devel mailing list
[hidden email]
https://pidgin.im/cgi-bin/mailman/listinfo/devel

smime.p7s (6K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: How to handle persistent chat rooms / conversations

Eion Robb-3
Hi David,

Really excited to see you start this conversation.  It's something that I've discussed with people on a case-by-case basis but never somewhere a bit more permanent and open like the devel mailing list.

As you might know, I maintain a couple of protocol plugins, and the trend for these is to have a persistent history.  The trouble is that libpurple isn't entirely suited to this kind of setup.  Various protocol plugins have tried various things over the years to be able to represent this, but as of yet I haven't found anything that solves the issue perfectly.  Group chat's are probably the worst when it comes to the UX for this.

In terms of keeping track of the group conversation itself, my personal favourite is to just automatically add the group chat to the buddy list.  Unfortunately, not all libpurple UI's support this (bitlbee is the main one that comes to mind, but I believe spectrum2 is similar), and so a workaround for them is to use the Tools->Room List api to keep track of what open conversations are available.

In terms of the actual history of those conversations, I've had a few issues.  Downloading all conversation history since the last time you opened a chat room is easy enough but you'll run into issues for very busy rooms or if you've connected to an account/room after several days/weeks/months/years of absence; although the Telegram plugin worksaround this fairly well with a setting for the max number of days of history to download.  Similarly, downloading the entirety of history for a conversation could cause problems for long-running conversations.  It would be easy to fill up a HDD with history of conversations in busy rooms.

Another issue is keeping track of the conversation state when closing an IM or Chat.  The History plugin in Pidgin helps a lot with this (although damn it gets slow when it's trying to find the newest conversation!), and I quite like the trick that the Matrix plugin does to mark a Chat as 'persistent' (although sadly this is a specific for Pidgin 'gtk-persistent').  In the Skype plugin, I tried experimenting with downloading the latest 20 messages whenever opening/joining a conversation - this was well-received by users, but made a real mess of logs, and could cause timing issues if a message was received while the history was loading.  This also caused headaches for people who were writing bots that used libpurple as it caused the bots to respond to very old messages.

The latest suggestion (that I haven't tried implementing yet) was to create a /history slash-command that could download X many messages in a conversation.  This feels a little clumsy though, and is hard for a user to know when a conversation does/doesn't support it - it also could be implemented a million different ways, depending on the whims of the plugin author at the time.


What I'd love to see in libpurple is a couple of things:
First of all, a way to mark a group-chat as persistent, in a non-UI-specific way - better API calls around this and adding a persistent group chat to the UI would be really helpful.
Secondly (and I was thinking of trialling this as a Pidgin plugin to begin with) is a signal or api hook for the UI to request the previous X number of lines or Y minutes of conversation from a certain point.  Where this gets tricky is that individual messages in libpurple don't have ID's so prpls that use ID's for messages (to get the previous 5 message since message ABC) who are being told 'give me all messages since the time Y' will have a tough time converting.  The other issue, is that without sub-second message precision, it's more difficult to say "get me all messages from before 2017-06-25 17:38:05" and not run into any duplicates.


I'd love to see other's opinions on this problem (especially people from a UI point of view), as it's something that keeps recurring with requests from users, that I'm not totally sure how to solve.

Apologies for the wall of text, but hopefully there's enough history in there (pun intented) to get an idea of what the problem is and why it's a problem.

Cheers,
Eion

On 25 June 2017 at 04:46, David Woodhouse <[hidden email]> wrote:
I'm working on a new prpl for a protocol where conversations are
persistent — all the historical messages are available.

For chat rooms, I've handled this by fetching all old messages when the
room is joined. The first time, I fetch everything; then I remember the
latest message (using purple_account_set_string(…"last-msg-$roomid"…)
and subsequent joins will only fetch the messages since then.

This isn't perfect, but it seems to work OK. Anyone got better
suggestions? Is there a way to mark chats as persistent from the back
end, rather than just in the UI? That would help.

It gets slightly more complex when I get to other "conversations" in
this protocol. Those are mostly two-party conversations which are
basically IM, but there are also multi-party conversations.

The simplest conversation has two parties, and should basically be
represented as IM. But how do I "insert" the history when the account
is first opened, including both incoming and outgoing messages? Is
there a way to get those to be represented as 'chat' conversations in
Pidgin? I could automatically call serv_got_joined_chat() for incoming
messages and even on outgoing... but the normal IM window would still
exist. Any other ideas? 

Then there's multi-party conversations, which I suppose will need to be
handled as chat conversations — will I need to do my own UI actions to
allow for creating, listing and joining those? 




_______________________________________________
Devel mailing list
[hidden email]
https://pidgin.im/cgi-bin/mailman/listinfo/devel


_______________________________________________
Devel mailing list
[hidden email]
https://pidgin.im/cgi-bin/mailman/listinfo/devel
Reply | Threaded
Open this post in threaded view
|

Re: How to handle persistent chat rooms / conversations

Kuba Niewiarowski
2017-06-25 7:42 GMT+02:00 Eion Robb <[hidden email]>:
Where this gets tricky is that individual messages in libpurple don't have ID's so prpls that use ID's for messages (to get the previous 5 message since message ABC) who are being told 'give me all messages since the time Y' will have a tough time converting.

According to [1], [2], [3], we should have messages UID with 3.0.


_______________________________________________
Devel mailing list
[hidden email]
https://pidgin.im/cgi-bin/mailman/listinfo/devel
Reply | Threaded
Open this post in threaded view
|

Re: How to handle persistent chat rooms / conversations

Eion Robb-3
Good point. The trick there is that those IDs are assigned by libpurple, not by the prpl, there's no API for the prpl to attach an arbitary (and persistent) ID to a received message

On 25 June 2017 at 20:23, Kuba Niewiarowski <[hidden email]> wrote:
2017-06-25 7:42 GMT+02:00 Eion Robb <[hidden email]>:
Where this gets tricky is that individual messages in libpurple don't have ID's so prpls that use ID's for messages (to get the previous 5 message since message ABC) who are being told 'give me all messages since the time Y' will have a tough time converting.

According to [1], [2], [3], we should have messages UID with 3.0.



_______________________________________________
Devel mailing list
[hidden email]
https://pidgin.im/cgi-bin/mailman/listinfo/devel
Reply | Threaded
Open this post in threaded view
|

Re: How to handle persistent chat rooms / conversations

David Woodhouse
On Sun, 2017-06-25 at 20:49 +1200, Eion Robb wrote:
> Good point. The trick there is that those IDs are assigned by
> libpurple, not by the prpl, there's no API for the prpl to attach an
> arbitary (and persistent) ID to a received message

That's hardly a new thing; we have to maintain a table to convert. We
have the same problem for various other things like buddies, where we
can't use a unique ID from the prpl and have to have other methods to
work it out.

It'll be good to be able to handle message receipts... but I'm not sure
this really addresses the 'persistence' problem especially across
multiple invocations of the application.

I was pondering a new message flag PURPLE_MESSAGE_HISTORICAL (or
_NO_NOTIFY) which would indicate that an incoming message isn't
actually new and shouldn't trigger a notification.

So when I first establish the account, I'd fetch all the historical
messages in each conversation and call serv_got_im() with this new
flag. The messages would get put into any log that the application is
keeping, but *wouldn't* cause a window to pop up immediately. And
if/when I later communicate with the same buddy, I can see the
historical conversation just as if it'd really happened using this
client.

I'm still unhappy with the way the Pidgin UI handles those separate
sessions as completely different logs, each time you close and reopen
the window, but that's mostly a UI thing and probably not something I
can address at the prpl level.

Eion, you mentioned something about logs growing too much if we fetch
historical messages... but that isn't a new problem; it can happen even
now if there's a lot of traffic. They need to be rotated/expired
anyway. So I'm not too worried about that.
_______________________________________________
Devel mailing list
[hidden email]
https://pidgin.im/cgi-bin/mailman/listinfo/devel

smime.p7s (6K) Download Attachment