ArchiveOrangemail archive

nginx mailing list English


nginx.nginx.org
(List home) (Recent threads) (4 other Nginx lists)

Subscription Options

  • RSS or Atom: Read-only subscription using a browser or aggregator. This is the recommended way if you don't need to send messages to the list. You can learn more about feed syndication and clients here.
  • Conventional: All messages are delivered to your mail address, and you can reply. To subscribe, send an email to the list's subscribe address with "subscribe" in the subject line, or visit the list's homepage here.
  • Moderate traffic list: up to 30 messages per day
  • This list contains about 44,222 messages, beginning Feb 2005
  • 2 messages added yesterday
Report the Spam
This button sends a spam report to the moderator. Please use it sparingly. For other removal requests, read this.
Are you sure? yes no

sending data in "chunks"

Ad
Manlio Perillo 1189964209Sun, 16 Sep 2007 17:36:49 +0000 (UTC)
Hi.

I need to send to the client some data in chunks.
The data is read from a Python file like object.

Right now I'm using a static ngx_chain_t variable, and for each cycle I 
read data from the "file" object in a buffer and do:

  // allocate buf and b
  out.buf = b;
  out.next = NULL;

  b->pos = buf;
  b->last = buf + n;

  b->memory = 1;
  b->last_buf = 0;

  rc = ngx_http_output_filter(r, &out);
  if (rc != NGX_OK) {
    goto error;


Then at the end of the cycle, then, I do a

  return ngx_http_output_filter(r, NULL);


Unfortunately this does not works.
What is the right method for sending data in chunks?



Thanks   Manlio Perillo
Igor Sysoev 1189964626Sun, 16 Sep 2007 17:43:46 +0000 (UTC)
On Sun, Sep 16, 2007 at 07:36:49PM +0200, Manlio Perillo wrote:

> I need to send to the client some data in chunks.
> The data is read from a Python file like object.
> 
> Right now I'm using a static ngx_chain_t variable, and for each cycle I 
> read data from the "file" object in a buffer and do:
> 
>  // allocate buf and b
>  out.buf = b;
>  out.next = NULL;
> 
>  b->pos = buf;
>  b->last = buf + n;
> 
>  b->memory = 1;
>  b->last_buf = 0;
> 
>  rc = ngx_http_output_filter(r, &out);
>  if (rc != NGX_OK) {
>    goto error;
> 
> 
> Then at the end of the cycle, then, I do a
> 
>  return ngx_http_output_filter(r, NULL);
> 
> 
> Unfortunately this does not works.
> What is the right method for sending data in chunks?You should set b->last_buf in last buf.
NULL is simply pump. It is used for exmple by default write handler
to pass all buffered data to client.
Manlio Perillo 1189966089Sun, 16 Sep 2007 18:08:09 +0000 (UTC)
Igor Sysoev ha scritto:
> On Sun, Sep 16, 2007 at 07:36:49PM +0200, Manlio Perillo wrote:
> 
>> I need to send to the client some data in chunks.
>> The data is read from a Python file like object.
>>
>> Right now I'm using a static ngx_chain_t variable, and for each cycle I 
>> read data from the "file" object in a buffer and do:
>>
>>  // allocate buf and b
>>  out.buf = b;
>>  out.next = NULL;
>>
>>  b->pos = buf;
>>  b->last = buf + n;
>>
>>  b->memory = 1;
>>  b->last_buf = 0;
>>
>>  rc = ngx_http_output_filter(r, &out);
>>  if (rc != NGX_OK) {
>>    goto error;
>>
>>
>> Then at the end of the cycle, then, I do a
>>
>>  return ngx_http_output_filter(r, NULL);
>>
>>
>> Unfortunately this does not works.
>> What is the right method for sending data in chunks?
> 
> You should set b->last_buf in last buf.
> NULL is simply pump. It is used for exmple by default write handler
> to pass all buffered data to client.
>Ok, thanks.
I have also found the real problem: one of the call to 
ngx_http_output_filter fails with a return value -2.

Now I have to found the problem...


Regards  Manlio Perillo
Igor Sysoev 1189966454Sun, 16 Sep 2007 18:14:14 +0000 (UTC)
On Sun, Sep 16, 2007 at 08:08:09PM +0200, Manlio Perillo wrote:

> Igor Sysoev ha scritto:
> >On Sun, Sep 16, 2007 at 07:36:49PM +0200, Manlio Perillo wrote:
> >
> >>I need to send to the client some data in chunks.
> >>The data is read from a Python file like object.
> >>
> >>Right now I'm using a static ngx_chain_t variable, and for each cycle I 
> >>read data from the "file" object in a buffer and do:
> >>
> >> // allocate buf and b
> >> out.buf = b;
> >> out.next = NULL;
> >>
> >> b->pos = buf;
> >> b->last = buf + n;
> >>
> >> b->memory = 1;
> >> b->last_buf = 0;
> >>
> >> rc = ngx_http_output_filter(r, &out);
> >> if (rc != NGX_OK) {
> >>   goto error;
> >>
> >>
> >>Then at the end of the cycle, then, I do a
> >>
> >> return ngx_http_output_filter(r, NULL);
> >>
> >>
> >>Unfortunately this does not works.
> >>What is the right method for sending data in chunks?
> >
> >You should set b->last_buf in last buf.
> >NULL is simply pump. It is used for exmple by default write handler
> >to pass all buffered data to client.
> 
> Ok, thanks.
> I have also found the real problem: one of the call to 
> ngx_http_output_filter fails with a return value -2.
> 
> Now I have to found the problem...It's NGX_AGAIN.
If you have got all your data ready you may send them at once in one chain.
But if you are getting then gradually, then after NGX_AGAIN you should
set event handlers and timer and return control to nginx.
Manlio Perillo 1189967763Sun, 16 Sep 2007 18:36:03 +0000 (UTC)
Igor Sysoev ha scritto:
> [...]
>
> It's NGX_AGAIN.
> If you have got all your data ready you may send them at once in one chain.
> But if you are getting then gradually, then after NGX_AGAIN you should
> set event handlers and timer and return control to nginx.
>Ok, thanks.
I can send all data in one chain.

Where can I find an example on how to handle NGX_AGAIN?



Regards  Manlio Perillo
Igor Sysoev 1189968209Sun, 16 Sep 2007 18:43:29 +0000 (UTC)
On Sun, Sep 16, 2007 at 08:36:03PM +0200, Manlio Perillo wrote:

> Igor Sysoev ha scritto:
> >[...]
> >
> >It's NGX_AGAIN.
> >If you have got all your data ready you may send them at once in one chain.
> >But if you are getting then gradually, then after NGX_AGAIN you should
> >set event handlers and timer and return control to nginx.
> 
> Ok, thanks.
> I can send all data in one chain.
> 
> Where can I find an example on how to handle NGX_AGAIN?It's complex thing. You need to call

ngx_add_timer()
ngx_handle_read_event()
ngx_handle_write_event()

The simple examples are:

ngx_http_init_connection() in src/http/ngx_http_request.c
and
ngx_mail_proxy_handler() in src/mail/ngx_mail_proxy_module.c
Adrian Perez de Castro 1189974942Sun, 16 Sep 2007 20:35:42 +0000 (UTC)
El Sun, 16 Sep 2007 22:14:14 +0400
Igor Sysoev  escribi?:

> On Sun, Sep 16, 2007 at 08:08:09PM +0200, Manlio Perillo wrote:
> 
> > Igor Sysoev ha scritto:
> > >On Sun, Sep 16, 2007 at 07:36:49PM +0200, Manlio Perillo wrote:
> > >
> > >>I need to send to the client some data in chunks. [...]
> > Ok, thanks.
> >
> > [...]
> >
> > I have also found the real problem: one of the call to 
> > ngx_http_output_filter fails with a return value -2.
> > 
> > Now I have to found the problem...
> 
> It's NGX_AGAIN.
> If you have got all your data ready you may send them at once in one
> chain. But if you are getting then gradually, then after NGX_AGAIN
> you should set event handlers and timer and return control to nginx.Mmmh, this affects me as well in the fancyindex module. When doing
subrequests for header inclusion the call to ngx_http_output_filter
returns NGX_AGAIN. I have just assumed NGX_AGAIN does not need extra
work, and the rest of the data I send goes to the client...

So I suppose I am making the worker process block too much, isn't it?

I can only send all data in one chain when not using header/footer
(they are subrequests), so I think it will be better to fix my code :D

Cheers,
Igor Sysoev 1190140169Tue, 18 Sep 2007 18:29:29 +0000 (UTC)
On Sun, Sep 16, 2007 at 10:35:42PM +0200, Adrian Perez de Castro wrote:

> El Sun, 16 Sep 2007 22:14:14 +0400
> Igor Sysoev  escribi??:
> 
> > On Sun, Sep 16, 2007 at 08:08:09PM +0200, Manlio Perillo wrote:
> > 
> > > Igor Sysoev ha scritto:
> > > >On Sun, Sep 16, 2007 at 07:36:49PM +0200, Manlio Perillo wrote:
> > > >
> > > >>I need to send to the client some data in chunks. [...]
> > > Ok, thanks.
> > >
> > > [...]
> > >
> > > I have also found the real problem: one of the call to 
> > > ngx_http_output_filter fails with a return value -2.
> > > 
> > > Now I have to found the problem...
> > 
> > It's NGX_AGAIN.
> > If you have got all your data ready you may send them at once in one
> > chain. But if you are getting then gradually, then after NGX_AGAIN
> > you should set event handlers and timer and return control to nginx.
> 
> Mmmh, this affects me as well in the fancyindex module. When doing
> subrequests for header inclusion the call to ngx_http_output_filter
> returns NGX_AGAIN. I have just assumed NGX_AGAIN does not need extra
> work, and the rest of the data I send goes to the client...
> 
> So I suppose I am making the worker process block too much, isn't it?
> 
> I can only send all data in one chain when not using header/footer
> (they are subrequests), so I think it will be better to fix my code :DIn your case, when you subrequest local files you can treat NGX_AGIAN
as NGX_OK. If you subrequest remote sites, then you need some handling
of this subrequests.
Adrian Perez de Castro 1190213987Wed, 19 Sep 2007 14:59:47 +0000 (UTC)
El Tue, 18 Sep 2007 22:29:29 +0400
Igor Sysoev  escribi?:

> On Sun, Sep 16, 2007 at 10:35:42PM +0200, Adrian Perez de Castro
> wrote:
>
> > [...]
> >
> > Mmmh, this affects me as well in the fancyindex module. When doing
> > subrequests for header inclusion the call to ngx_http_output_filter
> > returns NGX_AGAIN. I have just assumed NGX_AGAIN does not need extra
> > work, and the rest of the data I send goes to the client...
> > 
> > So I suppose I am making the worker process block too much, isn't
> > it?
> > 
> > I can only send all data in one chain when not using header/footer
> > (they are subrequests), so I think it will be better to fix my
> > code :D
> 
> In your case, when you subrequest local files you can treat NGX_AGIAN
> as NGX_OK. If you subrequest remote sites, then you need some handling
> of this subrequests.Thanks for the clarification. I will continue handling NGX_AGAIN as
NGX_OK and warn in the module README file about the issue with remote
sites. If there is demand for loading footer/header parts from
non-local URLs, I can try to add better handling for NGX_AGAIN.

Best regards,
Manlio Perillo 1189975870Sun, 16 Sep 2007 20:51:10 +0000 (UTC)
Igor Sysoev ha scritto:
> [...]
> It's NGX_AGAIN.
> If you have got all your data ready you may send them at once in one chain.
> But if you are getting then gradually, then after NGX_AGAIN you should
> set event handlers and timer and return control to nginx.
>As suggested, I have accumulate all the data in a buffer chain.
The data in my test is about 3.7 MB (an mp3) and, finally, the whole 
content is sent to the client.

However ngx_http_output_filter still returns NGX_AGAIN, and I have noted 
that Firefox when loading the data does not see the end of the stream.


P.S. I can just copy the data in a temporary file, if the size
      is too big"


Thanks and regards  Manlio Perillo
Igor Sysoev 1190140350Tue, 18 Sep 2007 18:32:30 +0000 (UTC)
On Sun, Sep 16, 2007 at 10:51:10PM +0200, Manlio Perillo wrote:

> Igor Sysoev ha scritto:
> >[...]
> >It's NGX_AGAIN.
> >If you have got all your data ready you may send them at once in one chain.
> >But if you are getting then gradually, then after NGX_AGAIN you should
> >set event handlers and timer and return control to nginx.
> >
> 
> As suggested, I have accumulate all the data in a buffer chain.
> The data in my test is about 3.7 MB (an mp3) and, finally, the whole 
> content is sent to the client.
> 
> However ngx_http_output_filter still returns NGX_AGAIN, and I have noted 
> that Firefox when loading the data does not see the end of the stream.Have you set last_buf in last buf ?
Manlio Perillo 1190143005Tue, 18 Sep 2007 19:16:45 +0000 (UTC)
Igor Sysoev ha scritto:
> On Sun, Sep 16, 2007 at 10:51:10PM +0200, Manlio Perillo wrote:
> 
>> Igor Sysoev ha scritto:
>>> [...]
>>> It's NGX_AGAIN.
>>> If you have got all your data ready you may send them at once in one chain.
>>> But if you are getting then gradually, then after NGX_AGAIN you should
>>> set event handlers and timer and return control to nginx.
>>>
>> As suggested, I have accumulate all the data in a buffer chain.
>> The data in my test is about 3.7 MB (an mp3) and, finally, the whole 
>> content is sent to the client.
>>
>> However ngx_http_output_filter still returns NGX_AGAIN, and I have noted 
>> that Firefox when loading the data does not see the end of the stream.
> 
> Have you set last_buf in last buf ?
>Yes.

It seems that the cause was a return NGX_AGAIN, instead of NGX_OK.

So, please let me know if I'm right:

- when a call to ngx_http_output_filter with a buffer chain of *only
   one* buffer returns with NGX_OK, it means that the *entire* buffer
   has been sent to the client.

   If nginx return NGX_AGAIN then I have to set the
   request->write_event_handler so that my handler is executed when the
   socket is ready to send data again.

   I have to setup a timer, so that I can timeout the connection when
   I can't write after a certain amount of time?

- when I pass a "full" buffer chain to ngx_http_output_filter,
   all the low level stuff (sending piece of data when the socket is
   ready) is done by nginx.

   This means that I can treat NGX_AGAIN as NGX_OK, however when
   ngx_http_output_filter returns I have no guarantee the all the buffers
   has been sent to the client.

   This also means that the ngx_chain_t variable must be allocate
   dynamically?

   What happens in case of errors?


P.S.
The source code of what I'm doing can be found here:
http://hg.mperillo.ath.cx/nginx/mod_pg/


Its a module for a direct interface with a PostgreSQL database.
The current version implemente a limited support for large objects.


The purpose of this module, beside offer access to PostgreSQL large 
objects for people who like to keep files in the database, is to offer 
the possibility to run queries without an intermediate web application.

The query result will be returned encoded in JSON format, so that AJAX 
applications can use it.



Thanks and regards   Manlio Perillo
Igor Sysoev 1190143863Tue, 18 Sep 2007 19:31:03 +0000 (UTC)
On Tue, Sep 18, 2007 at 09:16:45PM +0200, Manlio Perillo wrote:

> Igor Sysoev ha scritto:
> >On Sun, Sep 16, 2007 at 10:51:10PM +0200, Manlio Perillo wrote:
> >
> >>Igor Sysoev ha scritto:
> >>>[...]
> >>>It's NGX_AGAIN.
> >>>If you have got all your data ready you may send them at once in one 
> >>>chain.
> >>>But if you are getting then gradually, then after NGX_AGAIN you should
> >>>set event handlers and timer and return control to nginx.
> >>>
> >>As suggested, I have accumulate all the data in a buffer chain.
> >>The data in my test is about 3.7 MB (an mp3) and, finally, the whole 
> >>content is sent to the client.
> >>
> >>However ngx_http_output_filter still returns NGX_AGAIN, and I have noted 
> >>that Firefox when loading the data does not see the end of the stream.
> >
> >Have you set last_buf in last buf ?
> >
> 
> Yes.
> 
> It seems that the cause was a return NGX_AGAIN, instead of NGX_OK.
> 
> So, please let me know if I'm right:
> 
> - when a call to ngx_http_output_filter with a buffer chain of *only
>   one* buffer returns with NGX_OK, it means that the *entire* buffer
>   has been sent to the client.Yes. Or if gzipping is enabled the entire buffer is buffered inside zlib.>   If nginx return NGX_AGAIN then I have to set the
>   request->write_event_handler so that my handler is executed when the
>   socket is ready to send data again.Yes.>   I have to setup a timer, so that I can timeout the connection when
>   I can't write after a certain amount of time?Yes.> - when I pass a "full" buffer chain to ngx_http_output_filter,
>   all the low level stuff (sending piece of data when the socket is
>   ready) is done by nginx.Yes. nginx buffers the chain on several levels. They may be copy filter,
gzip filter, SSI filer, and finally write filter. Then standard writer
will pump this data using ngx_http_output_filter(r, NULL) to client.>   This means that I can treat NGX_AGAIN as NGX_OK, however when
>   ngx_http_output_filter returns I have no guarantee the all the buffers
>   has been sent to the client.Yes.>   This also means that the ngx_chain_t variable must be allocate
>   dynamically?Every buffering filter allocates its own ngx_chain_t's.
Only ngx_buf_t are the same. So you may safely use ngx_chain_t on stack.

>   What happens in case of errors?

NGX_ERROR.> P.S.
> The source code of what I'm doing can be found here:
> http://hg.mperillo.ath.cx/nginx/mod_pg/
> 
> 
> Its a module for a direct interface with a PostgreSQL database.
> The current version implemente a limited support for large objects.
> 
> 
> The purpose of this module, beside offer access to PostgreSQL large 
> objects for people who like to keep files in the database, is to offer 
> the possibility to run queries without an intermediate web application.
> 
> The query result will be returned encoded in JSON format, so that AJAX 
> applications can use it.I will look. Unfortunatly, now I have too few productive time.
Manlio Perillo 1190144671Tue, 18 Sep 2007 19:44:31 +0000 (UTC)
Igor Sysoev ha scritto:
> [...]
>>
>> So, please let me know if I'm right:
>>
>Thanks, this means that I can try to add NGX_AGAIN support without too 
much trouble.>>   What happens in case of errors?
> 
> NGX_ERROR.
>The problem is that the error can happen in a write_event handler.

This means, if I'm right, that I have no way to know if the data has 
been sent without problems.>> P.S.
>> The source code of what I'm doing can be found here:
>> http://hg.mperillo.ath.cx/nginx/mod_pg/
>>
> [...]
>
> I will look. Unfortunatly, now I have too few productive time.
>Thanks, however note that mod_pg is still in the early stage of 
development, so it may be not worth of looking, and you could be 
disappointed on how bad I'm treating nginx :-).

> 



Regards  Manlio Perillo
Manlio Perillo 1190404959Fri, 21 Sep 2007 20:02:39 +0000 (UTC)
Igor Sysoev ha scritto:
> [...]
>> So, please let me know if I'm right:
>>
>> - when a call to ngx_http_output_filter with a buffer chain of *only
>>   one* buffer returns with NGX_OK, it means that the *entire* buffer
>>   has been sent to the client.
> 
> Yes. Or if gzipping is enabled the entire buffer is buffered inside zlib.
> 
>>   If nginx return NGX_AGAIN then I have to set the
>>   request->write_event_handler so that my handler is executed when the
>>   socket is ready to send data again.
> 
> Yes.
>I have done some tests with mod_wsgi.

If a WSGI application returns a big buffer, ngx_http_output_filter 
returns NGX_AGAIN, *however* the entire buffer is sent to the client.

This is the same behaviour with a "full" buffer chain, with the 
difference that even if I return NGX_AGAIN from my request handler, the 
client reads the full response (with a "full" buffer chain in a previous 
test, I have to return NGX_OK, when receiving NGX_AGAIN).


So, it seems that nginx will send the entire content of a buffer, as 
soon as control returns to it.

This is important for me: when ngx_http_output_filter returns NGX_AGAIN, 
should I have to send again the previous buffer when my handler is 
called from the write event, or I can safely send the next buffer?



Thanks and regards  Manlio Perillo
Ad
Home | About | Privacy