phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Randy Hu <ruw...@gmail.com>
Subject Re: Create View of Existing HBase table
Date Mon, 19 Jun 2017 23:23:03 GMT
You are very welcome. Glad it helped.

For people using Phoenix with existing HBase tables for a nice SQL
interface, they would very likely experience the same issue again.
Wondering if the Phoenix team can change the strategy to use maximum
timestamp for select on views. This would not cause problem with UPSERT
SELECT since view is not updatable from Phoenix.

Randy

On Mon, Jun 19, 2017 at 10:52 AM, M. Aaron Bossert [via Apache Phoenix User
List] <ml+s1124778n3691h70@n5.nabble.com> wrote:

> Thanks for that pointer!  When I got to work this morning, I redid my
> ingest with all timestamps in milliseconds and the view populate nicely.
> Thanks again!
>
> Sent from my iPhone
>
> > On Jun 18, 2017, at 01:12, Randy Hu <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=3691&i=0>> wrote:
> >
> > That's likely the problem. The value in HBase/Java is in millisecond (
> > System.currentTimeMillis()) as explained in the link:
> >
> > https://currentmillis.com/tutorials/system-currentTimeMillis.html#unix-
> timestamp
> >
> > If the packet time stamp is a long value in microsecond, then it's 1000
> > times of the value expected in HBase, which would be a future time stamp
> > for HBase.
> >
> > Randy
> >
> > On Sat, Jun 17, 2017 at 8:40 PM, M. Aaron Bossert [via Apache Phoenix
> User
> > List] <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=3691&i=1>> wrote:
> >
> >> One potential difference might be resolution.  The network packets have
> >> precision down to microsecond...could that cause an issue?
> >>
> >> Sent from my iPhone
> >>
> >>> On Jun 17, 2017, at 20:32, Randy Hu <[hidden email]
> >> <http:///user/SendEmail.jtp?type=node&node=3687&i=0>> wrote:
> >>>
> >>> Yes, the problem I had experienced was the future time stamp in cells.
> >>> Setting "CurrentSCN" to maximal value would enforce the scan/get to
> use
> >>> this as the timestamp, so any cells would be retuned.
> >>>
> >>> Not sure if the network packet time stamp you copied is the same as
> Unix
> >>> timestamp:
> >>>
> >>> https://currentmillis.com/tutorials/system-
> currentTimeMillis.html#unix-
> >> timestamp
> >>>
> >>> For testing purpose, maybe you can insert some cells without explicit
> >>> timestamp to confirm whether timestamp is the issue.
> >>>
> >>> Randy
> >>>
> >>> On Sat, Jun 17, 2017 at 6:21 PM, M. Aaron Bossert [via Apache Phoenix
> >> User
> >>> List] <[hidden email]
> >> <http:///user/SendEmail.jtp?type=node&node=3687&i=1>> wrote:
> >>>
> >>>> I looked through the discussion and it seems like their issue was
> >>>> timestamps in the future?  I do use explicit timestamps for the data.
> >> I am
> >>>> storing network traffic and am using the packet's timestamp as the
> >> Hbase
> >>>> timestamp, which is often some time in the past when ingesting bulk
> >>>> collection files for forensic-type analysis.
> >>>>
> >>>> How would setting the SCN value help?  Would I set it in the past to
> >> match
> >>>> the oldest or newest packets?
> >>>>
> >>>> Sent from my iPhone
> >>>>
> >>>>> On Jun 17, 2017, at 18:01, Randy Hu <[hidden email]
> >>>> <http:///user/SendEmail.jtp?type=node&node=3684&i=0>>
wrote:
> >>>>>
> >>>>> Not sure if you have custom time stamp set when the values were
> >>>> persisted.
> >>>>> Here is the time stamp issue I discovered with view. The solution
> >> (work
> >>>>> around) is in the last post:
> >>>>>
> >>>>> http://apache-phoenix-user-list.1124778.n5.nabble.com/
> >>>> View-timestamp-on-existing-table-potential-defect-td3475.html
> >>>>>
> >>>>> Randy
> >>>>>
> >>>>> On Fri, Jun 16, 2017 at 2:07 PM, M. Aaron Bossert [via Apache
> Phoenix
> >>>> User
> >>>>> List] <[hidden email]
> >>>> <http:///user/SendEmail.jtp?type=node&node=3684&i=1>>
wrote:
> >>>>>
> >>>>>> *I have an existing HBase table with the following
> characteristics:*
> >>>>>>
> >>>>>> hbase(main):032:0> describe 'bro'
> >>>>>>
> >>>>>> Table bro is ENABLED
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> bro, {TABLE_ATTRIBUTES => {coprocessor$1 => '|org.apache.phoenix.
> >>>>>> coprocessor.ScanRegionObserver|805306366|', coprocessor$2 =>
> >>>>>> '|org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver|805306366|',
>
> >>
> >>>>
> >>>>>> coprocessor
> >>>>>>
> >>>>>> $3 => '|org.apache.phoenix.coprocessor.
> GroupedAggregateRegionObserver|805306366|',
> >>
> >>>>
> >>>>>> coprocessor$4 => '|org.apache.phoenix.coprocessor.
> >>>>>> ServerCachingEndpointImpl|805306366|', METADATA =>
> >>>> {'COMPACTION_ENABLED'
> >>>>>> => 'true'}
> >>>>>>
> >>>>>> }
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> COLUMN FAMILIES DESCRIPTION
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> {NAME => 'bm', BLOOMFILTER => 'ROW', VERSIONS => '1',
IN_MEMORY =>
> >>>>>> 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING
=>
> >> 'NONE',
> >>>> TTL
> >>>>>> => 'FOREVER', COMPRESSION => 'LZ4', MIN_VERSIONS =>
'0', BLOCKCACHE
> >> =>
> >>>> 'tr
> >>>>>>
> >>>>>> ue', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> {NAME => 'uid', BLOOMFILTER => 'ROW', VERSIONS => '1',
IN_MEMORY =>
> >>>>>> 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING
=>
> >> 'NONE',
> >>>> TTL
> >>>>>> => 'FOREVER', COMPRESSION => 'LZ4', MIN_VERSIONS =>
'0', BLOCKCACHE
> >> =>
> >>>> 't
> >>>>>>
> >>>>>> rue', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> 2 row(s) in 0.0300 seconds
> >>>>>>
> >>>>>>
> >>>>>> *When I try to create a Phoenix VIEW (Phoenix 4.7) as such:*
> >>>>>>
> >>>>>>
> >>>>>> 0: jdbc:phoenix:pocc2.net:2181> CREATE VIEW "bro" (pk VARCHAR
> >> PRIMARY
> >>>>>> KEY, "bm"."id.orig_h" VARCHAR, "bm"."id.orig_p" VARCHAR,
> >>>> "bm"."id_resp_h"
> >>>>>> VARCHAR, "bm"."id_resp_p" VARCHAR, "bm"."proto" VARCHAR) AS
SELECT
> *
> >>>> FROM
> >>>>>> "bro";
> >>>>>>
> >>>>>> No rows affected (0.019 seconds)
> >>>>>>
> >>>>>>
> >>>>>> *I see no errors at all, but when I try to query, there are
no
> >>>> results.  I
> >>>>>> verified that the columns in question are entered using the
> >>>>>> Bytes.toBytes(String) method in my code, thus my choice of
> VARCHAR.*
> >>>>>>
> >>>>>>
> >>>>>> 0: jdbc:phoenix:pocc2.net:2181> select * from "bro";
> >>>>>>
> >>>>>> *+-----+------------+------------+------------+------------+--------+*
>
> >>
> >>>>>>
> >>>>>> *| **PK ** | **id.orig_h ** | **id.orig_p ** | **id_resp_h **
| *
> >>>>>> *id_resp_p ** | **proto ** |*
> >>>>>>
> >>>>>> *+-----+------------+------------+------------+------------+--------+*
>
> >>
> >>>>>>
> >>>>>> *+-----+------------+------------+------------+------------+--------+*
>
> >>
> >>>>>>
> >>>>>> No rows selected (0.078 seconds)
> >>>>>>
> >>>>>>
> >>>>>> *Any assistance would be greatly appreciated...*
> >>>>>>
> >>>>>>
> >>>>>> ------------------------------
> >>>>>> If you reply to this email, your message will be added to the
> >>>> discussion
> >>>>>> below:
> >>>>>> http://apache-phoenix-user-list.1124778.n5.nabble.com/
> >>>>>> Create-View-of-Existing-HBase-table-tp3678.html
> >>>>>> To start a new topic under Apache Phoenix User List, email
> >>>>>> [hidden email] <http:///user/SendEmail.jtp?type=node&node=3684&i=2>
>
> >>>>>> To unsubscribe from Apache Phoenix User List, click here
> >>>>>> <
> >>>>>> .
> >>>>>> NAML
> >>>>>> <http://apache-phoenix-user-list.1124778.n5.nabble.com/
> >>>> template/NamlServlet.jtp?macro=macro_viewer&id=instant_
> >>>> html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.
> >>>> BasicNamespace-nabble.view.web.template.NabbleNamespace-
> >>>> nabble.view.web.template.NodeNamespace&breadcrumbs=
> >>>> notify_subscribers%21nabble%3Aemail.naml-instant_emails%
> >>>> 21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> View this message in context: http://apache-phoenix-user-
> >>>> list.1124778.n5.nabble.com/Create-View-of-Existing-HBase-
> >>>> table-tp3678p3682.html
> >>>>> Sent from the Apache Phoenix User List mailing list archive at
> >>>> Nabble.com.
> >>>>
> >>>>
> >>>> ------------------------------
> >>>> If you reply to this email, your message will be added to the
> >> discussion
> >>>> below:
> >>>> http://apache-phoenix-user-list.1124778.n5.nabble.com/
> >>>> Create-View-of-Existing-HBase-table-tp3678p3684.html
> >>>> To start a new topic under Apache Phoenix User List, email
> >>>> [hidden email] <http:///user/SendEmail.jtp?type=node&node=3687&i=2>
> >>>> To unsubscribe from Apache Phoenix User List, click here
> >>>> <
> >>>> .
> >>>> NAML
> >>>> <http://apache-phoenix-user-list.1124778.n5.nabble.com/
> >> template/NamlServlet.jtp?macro=macro_viewer&id=instant_
> >> html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.
> >> BasicNamespace-nabble.view.web.template.NabbleNamespace-
> >> nabble.view.web.template.NodeNamespace&breadcrumbs=
> >> notify_subscribers%21nabble%3Aemail.naml-instant_emails%
> >> 21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
> >>>>
> >>>
> >>>
> >>>
> >>>
> >>> --
> >>> View this message in context: http://apache-phoenix-user-
> >> list.1124778.n5.nabble.com/Create-View-of-Existing-HBase-
> >> table-tp3678p3685.html
> >>> Sent from the Apache Phoenix User List mailing list archive at
> >> Nabble.com.
> >>
> >>
> >> ------------------------------
> >> If you reply to this email, your message will be added to the
> discussion
> >> below:
> >> http://apache-phoenix-user-list.1124778.n5.nabble.com/
> >> Create-View-of-Existing-HBase-table-tp3678p3687.html
> >> To start a new topic under Apache Phoenix User List, email
> >> [hidden email] <http:///user/SendEmail.jtp?type=node&node=3691&i=2>
> >> To unsubscribe from Apache Phoenix User List, click here
> >> <
> >> .
> >> NAML
> >> <http://apache-phoenix-user-list.1124778.n5.nabble.com/
> template/NamlServlet.jtp?macro=macro_viewer&id=instant_
> html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.
> BasicNamespace-nabble.view.web.template.NabbleNamespace-
> nabble.view.web.template.NodeNamespace&breadcrumbs=
> notify_subscribers%21nabble%3Aemail.naml-instant_emails%
> 21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
> >>
> >
> >
> >
> >
> > --
> > View this message in context: http://apache-phoenix-user-
> list.1124778.n5.nabble.com/Create-View-of-Existing-HBase-
> table-tp3678p3688.html
> > Sent from the Apache Phoenix User List mailing list archive at
> Nabble.com.
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-phoenix-user-list.1124778.n5.nabble.com/
> Create-View-of-Existing-HBase-table-tp3678p3691.html
> To start a new topic under Apache Phoenix User List, email
> ml+s1124778n1h80@n5.nabble.com
> To unsubscribe from Apache Phoenix User List, click here
> <http://apache-phoenix-user-list.1124778.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=cnV3ZWloQGdtYWlsLmNvbXwxfC04OTI3ODY3NTc=>
> .
> NAML
> <http://apache-phoenix-user-list.1124778.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>




--
View this message in context: http://apache-phoenix-user-list.1124778.n5.nabble.com/Create-View-of-Existing-HBase-table-tp3678p3695.html
Sent from the Apache Phoenix User List mailing list archive at Nabble.com.

Mime
View raw message