phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vaclav Loffelmann <vaclav.loffelm...@socialbakers.com>
Subject Re: Phoenix and NodeJS
Date Tue, 19 May 2015 12:30:52 GMT
Hi,
currently we are migrating all read services to Java. For Phoenix 3.0
we used to use Phoenix proxy server [1, 2].

If you want use your existing code and native Java Phoenix client, you
can consider interpreting JS on Nashorn (Java 8) with Avatar.js.

Vaclav;

[1] https://github.com/falsecz/phoenix-proxy-server
[2] https://github.com/falsecz/node-phoenix-proxy

On 05/19/2015 02:46 AM, Eli Levine wrote:
> Yeah, so you can see that code creates a String array containing
> the whole result set. Usually a very bad idea for 400K-row result
> sets. You want to process results incrementally, probably via
> paging using row-value constructors and LIMIT.
> 
> On Mon, May 18, 2015 at 12:00 PM, Isart Montane
> <isart.montane@gmail.com> wrote:
> 
>> Thanks James.
>> 
>> That code is from the node driver, I will try to get some advice
>> from it's developer.
>> 
>> Thanks,
>> 
>> 
>> On Mon, May 18, 2015 at 6:34 PM, James Taylor
>> <jamestaylor@apache.org> wrote:
>> 
>>> Hi Isart, That code isn't Phoenix code. This sounds like a Node
>>> JS issue. Vaclav has done a lot with Node JS, so he may be able
>>> to give you some tips. Thanks, James
>>> 
>>> On Mon, May 18, 2015 at 9:06 AM, Isart Montane
>>> <isart.montane@gmail.com> wrote:
>>>> Hi Eli,
>>>> 
>>>> thanks a lot for your comments. I think you are right. I
>>>> found the
>>> client
>>>> code that's causing the issue. Do you have an example I can
>>>> use to
>>> patch it?
>>>> is that the recommended way to access phoenix? I've seen on
>>>> the web that there's also a query server available, is it
>>>> worth a try?
>>>> 
>>>> 
>>>> public String[] query(String sql) { List<String> lsResults =
>>>> new ArrayList(); Connection conn = null; try { conn =
>>>> this.dataSource.getConnection(); ResultSet rs =
>>>> conn.createStatement().executeQuery(sql); ResultSetMetaData
>>>> data = rs.getMetaData(); int numberOfColumns =
>>>> data.getColumnCount(); List<String> lsRows = new
>>>> ArrayList(); for (int i = 1; i <= numberOfColumns; i++) { 
>>>> lsRows.add(data.getColumnName(i)); } lsResults.add(join("\t",
>>>> lsRows)); lsRows.clear(); while (rs.next()) { for (int i = 1;
>>>> i <= numberOfColumns; i++) { lsRows.add(rs.getString(i)); } 
>>>> lsResults.add(join("\t", lsRows)); lsRows.clear(); } 
>>>> rs.close(); conn.close(); } catch (Exception e) { 
>>>> e.printStackTrace(); return null; } return
>>>> (String[])lsResults.toArray(new String[lsResults.size()]); }
>>>> 
>>>> On Mon, May 18, 2015 at 5:43 PM, Eli Levine
>>>> <elilevine@gmail.com>
>>> wrote:
>>>>> 
>>>>> I don't have info on what your app does with results from
>>>>> Phoenix. If
>>> the
>>>>> app is constructing some sort of object representations
>>>>> from Phoenix
>>> results
>>>>> and holding on to them, I would look at what the memory
>>>>> footprint of
>>> that
>>>>> is. I know this isn't very helpful but at this point I
>>>>> would try to dig deeper into your app and the NodeJS driver
>>>>> rather than Phoenix, since
>>> you
>>>>> mentioned the same queries run fine in sqlline.
>>>>> 
>>>>> On Mon, May 18, 2015 at 7:30 AM, Isart Montane <
>>> isart.montane@gmail.com>
>>>>> wrote:
>>>>>> 
>>>>>> Hi Eli,
>>>>>> 
>>>>>> thanks a lot for your answer. That might be a workaround
>>>>>> but I was
>>> hoping
>>>>>> to get a more generic answer I can apply to the
>>>>>> driver/phoenix since
>>> that
>>>>>> will require me lots of changes to the code.
>>>>>> 
>>>>>> Any clue on why it works with sqline but not trough the
>>>>>> node driver?
>>>>>> 
>>>>>> On Mon, May 18, 2015 at 4:20 PM, Eli Levine
>>>>>> <elilevine@gmail.com>
>>> wrote:
>>>>>>> 
>>>>>>> Have you looked at paging [1] using Phoenix's row-value
>>>>>>> constructors together with the LIMIT clause? That might
>>>>>>> be what you are looking
>>> for.
>>>>>>> 
>>>>>>> [1] http://phoenix.apache.org/paged.html
>>>>>>> 
>>>>>>> Eli
>>>>>>> 
>>>>>>> 
>>>>>>> On Mon, May 18, 2015 at 6:46 AM, Isart Montane <
>>> isart.montane@gmail.com>
>>>>>>> wrote:
>>>>>>>> 
>>>>>>>> Hi,
>>>>>>>> 
>>>>>>>> the company I work for is performing some tests on
>>>>>>>> Phoenix with
>>> NodeJS.
>>>>>>>> For simple queries I didn't had any problem, but as
>>>>>>>> soon as I start
>>> to use
>>>>>>>> our app I'm getting "process out of memory" errors on
>>>>>>>> the client
>>> when I runs
>>>>>>>> queries that return a big number of rows (i.e. 400k)
>>>>>>>> . I think the
>>> problem
>>>>>>>> is that the client tries to buffer all the results in
>>>>>>>> RAM and that
>>> kills it.
>>>>>>>> The same query runs fine when I run it with sqline.
>>>>>>>> 
>>>>>>>> So, is there a way to tell the client to stream the
>>>>>>>> results (or
>>> batch
>>>>>>>> them) instead of buffering them all? is raising the
>>>>>>>> client memory
>>> the only
>>>>>>>> solution?
>>>>>>>> 
>>>>>>>> I'm using phoenix-4.3.1 and
>>> https://github.com/gaodazhu/phoenix-client
>>>>>>>> as the NodeJS driver
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> 
>>>>>>>> Isart Montane
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> 
> 

Mime
View raw message