Mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Arthur <>
Subject Re: Offset commit api
Date Tue, 18 Dec 2012 00:06:23 GMT
Currently, consumers must deal with ZooKeeper directly and implement the 
consumer re-balancing algorithm (correctly). As this is a rather 
difficult and error-prone process, the intent of this new API is to 
provide an easy mechanism for storing offsets for non-Scala clients.

At least, that's how I view it. Jay and others may have insights as to 
how this plays into the overall strategy of the project.

On 12/17/12 5:27 PM, Milind Parikh wrote:
> Perhaps I don't understand the motivation well enough and perhaps I am 
> misreading the intent.
> But I thought that the design principle behind kafka is for state  
> (from a consumer standpoint) was to be managed by consumer and not 
> broker. I understand that "These APIs are optional, clients can store 
> offsets another way if they like."
> So three questions :
> (a) Is this a change from the original intent of Kafka?
> (b) If it is a change, why not make it such that there is no need for 
> clients to roll their own?
>      autocommit=false-> no storage of offsets
>      autocommit=true -> store offsets
> (c) I suppose that the use case of multiple sets of consumer groups 
> wanting to use the offsets for different purposes could be one of the 
> use cases for the clients to roll their own. That corner case could be 
> handled through handing out a uuid for a set of consumer group to 
> operate against.Any other use cases for the clients to absolutely roll 
> their own?
> Regards
> Milind
> On Mon, Dec 17, 2012 at 10:45 AM, Jay Kreps < 
> <>> wrote:
>     Hey Guys,
>     David has made a bunch of progress on the offset commit api
>     implementation.
>     Since this is a public API it would be good to do as much thinking
>     up-front
>     as possible to minimize future iterations.
>     It would be great if folks could do the following:
>     1. Read the wiki here:
>     2. Check out the code David wrote here:
>     In particular our hope is that this API can act as the first step in
>     scaling the way we store offsets (ZK is not really very
>     appropriate for
>     this). This of course requires having some plan in mind for offset
>     storage.
>     I have written (and then after getting some initial feedback,
>     rewritten) a
>     section in the above wiki on how this might work.
>     If no one says anything I will be taking a slightly modified patch
>     that
>     adds this functionality on trunk as soon as David gets in a few minor
>     tweaks.
>     -Jay

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message