Mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jay Kreps <jay.kr...@gmail.com>
Subject Re: order guarantee failure: better/best effort?
Date Thu, 13 Dec 2012 19:10:29 GMT
So to expand on that, as a result my belief is that since we only allow one
outstanding request per socket at a time it should be impossible to
re-order requests. But we haven't rigorously tested this. If someone has a
script that doesn't use the scala client and makes lots of requests I think
we could test this easily. If not I can write a test that directly uses a
socket and the network server to validate this, which might be a good thing
to do anyway.

-Jay


On Thu, Dec 13, 2012 at 11:06 AM, Jay Kreps <jay.kreps@gmail.com> wrote:

> That is correct, but the network server only enqueues a single request
> from each socket at any given time.
>
> -Jay
>
>
> On Thu, Dec 13, 2012 at 9:40 AM, Prashanth Menon <
> prashanth.menon1@gmail.com> wrote:
>
>> From my understanding (correct me if I'm wrong here, Jay), if a client
>> does
>> non-blocking writes on a single socket connection, then it's entirely
>> possible to see out-of-order writes in the server from the client's
>> perspective.  The requests are serialized/ordered on a connection only
>> until it hits the request channel, after that they may be processed in
>> different threads, and therefore, in an unpredictable order.
>>
>> I suspect the Node client does indeed do non-blocking sends, but can you
>> confirm this, Ben?
>>
>> - Prashanth
>>
>> On Thu, Dec 13, 2012 at 11:42 AM, Jay Kreps <jay.kreps@gmail.com> wrote:
>>
>> > So my belief is that we handle this correctly and that I was mistaken in
>> > filing that bug. However, since, as the bug points out, the problem
>> can't
>> > exist in the scala client this may be mistaken and a problem may well
>> > remain.
>> >
>> > There are two reasons that requests should have enforced ordering:
>> > 1. The scala client blocks until it gets a response. As you say, this
>> > doesn't help you.
>> > 2. The server only processes a single request per socket at a time.
>> >
>> > Can you confirm that (1) these messages are sent to the same partition
>> > (guarantee is only within a partition), (2) the requests are sent over
>> the
>> > same socket (order can only be guaranteed over a single connection). If
>> > these are true and we are re-ordering I think we have a bug...if so if
>> you
>> > could give a small portable program of some kind that would reproduce
>> the
>> > problem that would be awesome since I don't think the scala client will.
>> >
>> > -Jay
>> >
>> >
>> > On Thu, Dec 13, 2012 at 12:01 AM, ben fleis <ben.fleis@gmail.com>
>> wrote:
>> >
>> > > Hello all,
>> > >
>> > > While running my own system tests against 0.8/HEAD, I was seeing
>> repeated
>> > > ordering failures, and tracked it down a bit further.  In simple
>> > summary, I
>> > > am sending out 2 consecutive ProduceRequests, X and Y.  Y gets
>> written to
>> > > disk before X.  Consumer sees Y before X.
>> > >
>> > > Bug #382 <https://issues.apache.org/jira/browse/KAFKA-382> discusses
>> > this
>> > > possibility, and it appears to describe my issue.  My client is, like
>> any
>> > > normal client written in Node, asynchronous by nature.  I can add
>> waits
>> > to
>> > > make it synchronous, but this feels like it fails the "best effort"
>> test.
>> > >  I.e., it seems reasonable that a pipelining client running without
>> > errors
>> > > ought to maintain order, all the time.
>> > >
>> > > Thoughts?
>> > >
>> > > b
>> > >
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message