phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "M. Aaron Bossert" <maboss...@gmail.com>
Subject Re: Create View of Existing HBase table
Date Mon, 19 Jun 2017 15:51:56 GMT
Thanks for that pointer!  When I got to work this morning, I redid my ingest with all timestamps
in milliseconds and the view populate nicely.  Thanks again!

Sent from my iPhone

> On Jun 18, 2017, at 01:12, Randy Hu <ruweih@gmail.com> wrote:
> 
> That's likely the problem. The value in HBase/Java is in millisecond (
> System.currentTimeMillis()) as explained in the link:
> 
> https://currentmillis.com/tutorials/system-currentTimeMillis.html#unix-timestamp
> 
> If the packet time stamp is a long value in microsecond, then it's 1000
> times of the value expected in HBase, which would be a future time stamp
> for HBase.
> 
> Randy
> 
> On Sat, Jun 17, 2017 at 8:40 PM, M. Aaron Bossert [via Apache Phoenix User
> List] <ml+s1124778n3687h46@n5.nabble.com> wrote:
> 
>> One potential difference might be resolution.  The network packets have
>> precision down to microsecond...could that cause an issue?
>> 
>> Sent from my iPhone
>> 
>>> On Jun 17, 2017, at 20:32, Randy Hu <[hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=3687&i=0>> wrote:
>>> 
>>> Yes, the problem I had experienced was the future time stamp in cells.
>>> Setting "CurrentSCN" to maximal value would enforce the scan/get to use
>>> this as the timestamp, so any cells would be retuned.
>>> 
>>> Not sure if the network packet time stamp you copied is the same as Unix
>>> timestamp:
>>> 
>>> https://currentmillis.com/tutorials/system-currentTimeMillis.html#unix-
>> timestamp
>>> 
>>> For testing purpose, maybe you can insert some cells without explicit
>>> timestamp to confirm whether timestamp is the issue.
>>> 
>>> Randy
>>> 
>>> On Sat, Jun 17, 2017 at 6:21 PM, M. Aaron Bossert [via Apache Phoenix
>> User
>>> List] <[hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=3687&i=1>> wrote:
>>> 
>>>> I looked through the discussion and it seems like their issue was
>>>> timestamps in the future?  I do use explicit timestamps for the data.
>> I am
>>>> storing network traffic and am using the packet's timestamp as the
>> Hbase
>>>> timestamp, which is often some time in the past when ingesting bulk
>>>> collection files for forensic-type analysis.
>>>> 
>>>> How would setting the SCN value help?  Would I set it in the past to
>> match
>>>> the oldest or newest packets?
>>>> 
>>>> Sent from my iPhone
>>>> 
>>>>> On Jun 17, 2017, at 18:01, Randy Hu <[hidden email]
>>>> <http:///user/SendEmail.jtp?type=node&node=3684&i=0>> wrote:
>>>>> 
>>>>> Not sure if you have custom time stamp set when the values were
>>>> persisted.
>>>>> Here is the time stamp issue I discovered with view. The solution
>> (work
>>>>> around) is in the last post:
>>>>> 
>>>>> http://apache-phoenix-user-list.1124778.n5.nabble.com/
>>>> View-timestamp-on-existing-table-potential-defect-td3475.html
>>>>> 
>>>>> Randy
>>>>> 
>>>>> On Fri, Jun 16, 2017 at 2:07 PM, M. Aaron Bossert [via Apache Phoenix
>>>> User
>>>>> List] <[hidden email]
>>>> <http:///user/SendEmail.jtp?type=node&node=3684&i=1>> wrote:
>>>>> 
>>>>>> *I have an existing HBase table with the following characteristics:*
>>>>>> 
>>>>>> hbase(main):032:0> describe 'bro'
>>>>>> 
>>>>>> Table bro is ENABLED
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> bro, {TABLE_ATTRIBUTES => {coprocessor$1 => '|org.apache.phoenix.
>>>>>> coprocessor.ScanRegionObserver|805306366|', coprocessor$2 =>
>>>>>> '|org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver|805306366|',
>> 
>>>> 
>>>>>> coprocessor
>>>>>> 
>>>>>> $3 => '|org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver|805306366|',
>> 
>>>> 
>>>>>> coprocessor$4 => '|org.apache.phoenix.coprocessor.
>>>>>> ServerCachingEndpointImpl|805306366|', METADATA =>
>>>> {'COMPACTION_ENABLED'
>>>>>> => 'true'}
>>>>>> 
>>>>>> }
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> COLUMN FAMILIES DESCRIPTION
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> {NAME => 'bm', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY
=>
>>>>>> 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING =>
>> 'NONE',
>>>> TTL
>>>>>> => 'FOREVER', COMPRESSION => 'LZ4', MIN_VERSIONS => '0',
BLOCKCACHE
>> =>
>>>> 'tr
>>>>>> 
>>>>>> ue', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> {NAME => 'uid', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY
=>
>>>>>> 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING =>
>> 'NONE',
>>>> TTL
>>>>>> => 'FOREVER', COMPRESSION => 'LZ4', MIN_VERSIONS => '0',
BLOCKCACHE
>> =>
>>>> 't
>>>>>> 
>>>>>> rue', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 2 row(s) in 0.0300 seconds
>>>>>> 
>>>>>> 
>>>>>> *When I try to create a Phoenix VIEW (Phoenix 4.7) as such:*
>>>>>> 
>>>>>> 
>>>>>> 0: jdbc:phoenix:pocc2.net:2181> CREATE VIEW "bro" (pk VARCHAR
>> PRIMARY
>>>>>> KEY, "bm"."id.orig_h" VARCHAR, "bm"."id.orig_p" VARCHAR,
>>>> "bm"."id_resp_h"
>>>>>> VARCHAR, "bm"."id_resp_p" VARCHAR, "bm"."proto" VARCHAR) AS SELECT
*
>>>> FROM
>>>>>> "bro";
>>>>>> 
>>>>>> No rows affected (0.019 seconds)
>>>>>> 
>>>>>> 
>>>>>> *I see no errors at all, but when I try to query, there are no
>>>> results.  I
>>>>>> verified that the columns in question are entered using the
>>>>>> Bytes.toBytes(String) method in my code, thus my choice of VARCHAR.*
>>>>>> 
>>>>>> 
>>>>>> 0: jdbc:phoenix:pocc2.net:2181> select * from "bro";
>>>>>> 
>>>>>> *+-----+------------+------------+------------+------------+--------+*
>> 
>>>>>> 
>>>>>> *| **PK ** | **id.orig_h ** | **id.orig_p ** | **id_resp_h ** | *
>>>>>> *id_resp_p ** | **proto ** |*
>>>>>> 
>>>>>> *+-----+------------+------------+------------+------------+--------+*
>> 
>>>>>> 
>>>>>> *+-----+------------+------------+------------+------------+--------+*
>> 
>>>>>> 
>>>>>> No rows selected (0.078 seconds)
>>>>>> 
>>>>>> 
>>>>>> *Any assistance would be greatly appreciated...*
>>>>>> 
>>>>>> 
>>>>>> ------------------------------
>>>>>> If you reply to this email, your message will be added to the
>>>> discussion
>>>>>> below:
>>>>>> http://apache-phoenix-user-list.1124778.n5.nabble.com/
>>>>>> Create-View-of-Existing-HBase-table-tp3678.html
>>>>>> To start a new topic under Apache Phoenix User List, email
>>>>>> [hidden email] <http:///user/SendEmail.jtp?type=node&node=3684&i=2>
>>>>>> To unsubscribe from Apache Phoenix User List, click here
>>>>>> <
>>>>>> .
>>>>>> NAML
>>>>>> <http://apache-phoenix-user-list.1124778.n5.nabble.com/
>>>> template/NamlServlet.jtp?macro=macro_viewer&id=instant_
>>>> html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.
>>>> BasicNamespace-nabble.view.web.template.NabbleNamespace-
>>>> nabble.view.web.template.NodeNamespace&breadcrumbs=
>>>> notify_subscribers%21nabble%3Aemail.naml-instant_emails%
>>>> 21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> View this message in context: http://apache-phoenix-user-
>>>> list.1124778.n5.nabble.com/Create-View-of-Existing-HBase-
>>>> table-tp3678p3682.html
>>>>> Sent from the Apache Phoenix User List mailing list archive at
>>>> Nabble.com.
>>>> 
>>>> 
>>>> ------------------------------
>>>> If you reply to this email, your message will be added to the
>> discussion
>>>> below:
>>>> http://apache-phoenix-user-list.1124778.n5.nabble.com/
>>>> Create-View-of-Existing-HBase-table-tp3678p3684.html
>>>> To start a new topic under Apache Phoenix User List, email
>>>> [hidden email] <http:///user/SendEmail.jtp?type=node&node=3687&i=2>
>>>> To unsubscribe from Apache Phoenix User List, click here
>>>> <
>>>> .
>>>> NAML
>>>> <http://apache-phoenix-user-list.1124778.n5.nabble.com/
>> template/NamlServlet.jtp?macro=macro_viewer&id=instant_
>> html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.
>> BasicNamespace-nabble.view.web.template.NabbleNamespace-
>> nabble.view.web.template.NodeNamespace&breadcrumbs=
>> notify_subscribers%21nabble%3Aemail.naml-instant_emails%
>> 21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>> 
>>> 
>>> 
>>> 
>>> 
>>> --
>>> View this message in context: http://apache-phoenix-user-
>> list.1124778.n5.nabble.com/Create-View-of-Existing-HBase-
>> table-tp3678p3685.html
>>> Sent from the Apache Phoenix User List mailing list archive at
>> Nabble.com.
>> 
>> 
>> ------------------------------
>> If you reply to this email, your message will be added to the discussion
>> below:
>> http://apache-phoenix-user-list.1124778.n5.nabble.com/
>> Create-View-of-Existing-HBase-table-tp3678p3687.html
>> To start a new topic under Apache Phoenix User List, email
>> ml+s1124778n1h80@n5.nabble.com
>> To unsubscribe from Apache Phoenix User List, click here
>> <http://apache-phoenix-user-list.1124778.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=cnV3ZWloQGdtYWlsLmNvbXwxfC04OTI3ODY3NTc=>
>> .
>> NAML
>> <http://apache-phoenix-user-list.1124778.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>> 
> 
> 
> 
> 
> --
> View this message in context: http://apache-phoenix-user-list.1124778.n5.nabble.com/Create-View-of-Existing-HBase-table-tp3678p3688.html
> Sent from the Apache Phoenix User List mailing list archive at Nabble.com.

Mime
View raw message