phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Isart Montane <isart.mont...@gmail.com>
Subject Re: Phoenix and NodeJS
Date Mon, 18 May 2015 16:06:37 GMT
Hi Eli,

thanks a lot for your comments. I think you are right. I found the client
code that's causing the issue. Do you have an example I can use to patch
it? is that the recommended way to access phoenix? I've seen on the web
that there's also a query server available, is it worth a try?


public String[] query(String sql)
  {
    List<String> lsResults = new ArrayList();
    Connection conn = null;
    try
    {
      conn = this.dataSource.getConnection();
      ResultSet rs = conn.createStatement().executeQuery(sql);
      ResultSetMetaData data = rs.getMetaData();
      int numberOfColumns = data.getColumnCount();
      List<String> lsRows = new ArrayList();
      for (int i = 1; i <= numberOfColumns; i++) {
        lsRows.add(data.getColumnName(i));
      }
      lsResults.add(join("\t", lsRows));
      lsRows.clear();
      while (rs.next())
      {
        for (int i = 1; i <= numberOfColumns; i++) {
          lsRows.add(rs.getString(i));
        }
        lsResults.add(join("\t", lsRows));
        lsRows.clear();
      }
      rs.close();
      conn.close();
    }
    catch (Exception e)
    {
      e.printStackTrace();
      return null;
    }
    return (String[])lsResults.toArray(new String[lsResults.size()]);
  }

On Mon, May 18, 2015 at 5:43 PM, Eli Levine <elilevine@gmail.com> wrote:

> I don't have info on what your app does with results from Phoenix. If the
> app is constructing some sort of object representations from Phoenix
> results and holding on to them, I would look at what the memory footprint
> of that is. I know this isn't very helpful but at this point I would try to
> dig deeper into your app and the NodeJS driver rather than Phoenix, since
> you mentioned the same queries run fine in sqlline.
>
> On Mon, May 18, 2015 at 7:30 AM, Isart Montane <isart.montane@gmail.com>
> wrote:
>
>> Hi Eli,
>>
>> thanks a lot for your answer. That might be a workaround but I was hoping
>> to get a more generic answer I can apply to the driver/phoenix since that
>> will require me lots of changes to the code.
>>
>> Any clue on why it works with sqline but not trough the node driver?
>>
>> On Mon, May 18, 2015 at 4:20 PM, Eli Levine <elilevine@gmail.com> wrote:
>>
>>> Have you looked at paging [1] using Phoenix's row-value constructors
>>> together with the LIMIT clause? That might be what you are looking for.
>>>
>>> [1] http://phoenix.apache.org/paged.html
>>>
>>> Eli
>>>
>>>
>>> On Mon, May 18, 2015 at 6:46 AM, Isart Montane <isart.montane@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> the company I work for is performing some tests on Phoenix with NodeJS.
>>>> For simple queries I didn't had any problem, but as soon as I start to use
>>>> our app I'm getting "process out of memory" errors on the client when I
>>>> runs queries that return a big number of rows (i.e. 400k) . I think the
>>>> problem is that the client tries to buffer all the results in RAM and that
>>>> kills it. The same query runs fine when I run it with sqline.
>>>>
>>>> So, is there a way to tell the client to stream the results (or batch
>>>> them) instead of buffering them all? is raising the client memory the only
>>>> solution?
>>>>
>>>> I'm using phoenix-4.3.1 and https://github.com/gaodazhu/phoenix-client
>>>> as the NodeJS driver
>>>>
>>>> Thanks,
>>>>
>>>> Isart Montane
>>>>
>>>
>>>
>>
>

Mime
View raw message