phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun Kumaran Sabtharishi <arun1...@gmail.com>
Subject Re: Undefined column. columnName=IS_ROW_TIMESTAMP
Date Wed, 20 Apr 2016 16:37:03 GMT
One more question to add,
Do we need to have 1000 versions, and KEEP_DELETED_CELL=true?

I have limited the scan in HBase and here is the following data.

\x00\x00TEST_TABLE_2 column=0:, timestamp=1460455162842, type=DeleteFamily

_0_1460354090089

\x00\x00TEST_TABLE_2 column=0:BASE_COLUMN_COUNT, timestamp=1460354090655,
value=\x7F\xFF\xFF\xFF

_0_1460354090089

\x00\x00TEST_TABLE_2 column=0:COLUMN_COUNT, timestamp=1460354090655,
value=\x80\x00\x00\x05

_0_1460354090089

\x00\x00TEST_TABLE_2 column=0:DATA_TABLE_NAME, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089

\x00\x00TEST_TABLE_2 column=0:DEFAULT_COLUMN_FAMILY,
timestamp=1460354090655, value=default

_0_1460354090089

\x00\x00TEST_TABLE_2 column=0:DISABLE_WAL, timestamp=1460354090655,
value=\x00

_0_1460354090089

\x00\x00TEST_TABLE_2 column=0:IMMUTABLE_ROWS, timestamp=1460354090655,
value=\x00

_0_1460354090089

\x00\x00TEST_TABLE_2 column=0:INDEX_STATE, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089

\x00\x00TEST_TABLE_2 column=0:INDEX_TYPE, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089

\x00\x00TEST_TABLE_2 column=0:MULTI_TENANT, timestamp=1460354090655,
value=\x00

_0_1460354090089

\x00\x00TEST_TABLE_2 column=0:PK_NAME, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089

\x00\x00TEST_TABLE_2 column=0:ROW_KEY_ORDER_OPTIMIZABLE,
timestamp=1460354090655, value=\x01

_0_1460354090089

\x00\x00TEST_TABLE_2 column=0:SALT_BUCKETS, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089

\x00\x00TEST_TABLE_2 column=0:STORE_NULLS, timestamp=1460354090655,
value=\x00

_0_1460354090089

\x00\x00TEST_TABLE_2 column=0:TABLE_SEQ_NUM, timestamp=1460354090655,
value=\x80\x00\x00\x00\x00\x00\x00\x00

_0_1460354090089

\x00\x00TEST_TABLE_2 column=0:TABLE_TYPE, timestamp=1460354090655, value=u

_0_1460354090089

\x00\x00TEST_TABLE_2 column=0:VIEW_INDEX_ID, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089

\x00\x00TEST_TABLE_2 column=0:VIEW_STATEMENT, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089

\x00\x00TEST_TABLE_2 column=0:VIEW_TYPE, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089

\x00\x00TEST_TABLE_2 column=0:_0, timestamp=1460354090655, type=DeleteColumn

_0_1460354090089

\x00\x00TEST_TABLE_2 column=0:, timestamp=1460455162842, type=DeleteFamily

_0_1460354090089\x00CDS_CLIENT_CORELATION_ID\

x00default

\x00\x00TEST_TABLE_2 column=0:ARRAY_SIZE, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089\x00CDS_CLIENT_CORELATION_ID\

x00default

\x00\x00TEST_TABLE_2 column=0:COLUMN_DEF, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089\x00CDS_CLIENT_CORELATION_ID\

x00default

\x00\x00TEST_TABLE_2 column=0:COLUMN_SIZE, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089\x00CDS_CLIENT_CORELATION_ID\

x00default

\x00\x00TEST_TABLE_2 column=0:DATA_TABLE_NAME, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089\x00CDS_CLIENT_CORELATION_ID\

x00default

\x00\x00TEST_TABLE_2 column=0:DATA_TYPE, timestamp=1460354090655,
value=\x80\x00\x00\x0C

_0_1460354090089\x00CDS_CLIENT_CORELATION_ID\

x00default

\x00\x00TEST_TABLE_2 column=0:DECIMAL_DIGITS, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089\x00CDS_CLIENT_CORELATION_ID\

x00default

\x00\x00TEST_TABLE_2 column=0:IS_ROW_TIMESTAMP, timestamp=1460354090655,
value=\x00

_0_1460354090089\x00CDS_CLIENT_CORELATION_ID\

x00default

\x00\x00TEST_TABLE_2 column=0:IS_VIEW_REFERENCED, timestamp=1460354090655,
value=\x00

_0_1460354090089\x00CDS_CLIENT_CORELATION_ID\

x00default

\x00\x00TEST_TABLE_2 column=0:KEY_SEQ, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089\x00CDS_CLIENT_CORELATION_ID\

x00default

\x00\x00TEST_TABLE_2 column=0:NULLABLE, timestamp=1460354090655,
value=\x80\x00\x00\x01

_0_1460354090089\x00CDS_CLIENT_CORELATION_ID\

x00default

\x00\x00TEST_TABLE_2 column=0:ORDINAL_POSITION, timestamp=1460354090655,
value=\x80\x00\x00\x04

_0_1460354090089\x00CDS_CLIENT_CORELATION_ID\

x00default

\x00\x00TEST_TABLE_2 column=0:PK_NAME, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089\x00CDS_CLIENT_CORELATION_ID\

x00default

\x00\x00TEST_TABLE_2 column=0:SORT_ORDER, timestamp=1460354090655,
value=\x80\x00\x00\x02

_0_1460354090089\x00CDS_CLIENT_CORELATION_ID\

x00default

\x00\x00TEST_TABLE_2 column=0:VIEW_CONSTANT, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089\x00CDS_CLIENT_CORELATION_ID\

x00default

\x00\x00TEST_TABLE_2 column=0:_0, timestamp=1460354090655, type=DeleteColumn

_0_1460354090089\x00CDS_CLIENT_CORELATION_ID\

x00default

\x00\x00TEST_TABLE_2 column=0:, timestamp=1460455162842, type=DeleteFamily

_0_1460354090089\x00CDS_PK

\x00\x00TEST_TABLE_2 column=0:ARRAY_SIZE, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089\x00CDS_PK

\x00\x00TEST_TABLE_2 column=0:COLUMN_DEF, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089\x00CDS_PK

\x00\x00TEST_TABLE_2 column=0:COLUMN_SIZE, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089\x00CDS_PK

\x00\x00TEST_TABLE_2 column=0:DATA_TABLE_NAME, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089\x00CDS_PK

\x00\x00TEST_TABLE_2 column=0:DATA_TYPE, timestamp=1460354090655,
value=\x80\x00\x00\x0C

_0_1460354090089\x00CDS_PK

\x00\x00TEST_TABLE_2 column=0:DECIMAL_DIGITS, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089\x00CDS_PK

\x00\x00TEST_TABLE_2 column=0:IS_ROW_TIMESTAMP, timestamp=1460354090655,
value=\x00

_0_1460354090089\x00CDS_PK

\x00\x00TEST_TABLE_2 column=0:IS_VIEW_REFERENCED, timestamp=1460354090655,
value=\x00

_0_1460354090089\x00CDS_PK

\x00\x00TEST_TABLE_2 column=0:KEY_SEQ, timestamp=1460354090655,
value=\x80\x01

_0_1460354090089\x00CDS_PK

\x00\x00TEST_TABLE_2 column=0:NULLABLE, timestamp=1460354090655,
value=\x80\x00\x00\x00

_0_1460354090089\x00CDS_PK

\x00\x00TEST_TABLE_2 column=0:ORDINAL_POSITION, timestamp=1460354090655,
value=\x80\x00\x00\x05

_0_1460354090089\x00CDS_PK

\x00\x00TEST_TABLE_2 column=0:PK_NAME, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089\x00CDS_PK

\x00\x00TEST_TABLE_2 column=0:SORT_ORDER, timestamp=1460354090655,
value=\x80\x00\x00\x02

_0_1460354090089\x00CDS_PK

\x00\x00TEST_TABLE_2 column=0:VIEW_CONSTANT, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089\x00CDS_PK

\x00\x00TEST_TABLE_2 column=0:_0, timestamp=1460354090655, type=DeleteColumn

_0_1460354090089\x00CDS_PK

\x00\x00TEST_TABLE_2 column=0:, timestamp=1460455162842, type=DeleteFamily

_0_1460354090089\x00CDS_SUB_ORG_ID\x00default

\x00\x00TEST_TABLE_2 column=0:ARRAY_SIZE, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089\x00CDS_SUB_ORG_ID\x00default

\x00\x00TEST_TABLE_2 column=0:COLUMN_DEF, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089\x00CDS_SUB_ORG_ID\x00default

\x00\x00TEST_TABLE_2 column=0:COLUMN_SIZE, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089\x00CDS_SUB_ORG_ID\x00default

\x00\x00TEST_TABLE_2 column=0:DATA_TABLE_NAME, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089\x00CDS_SUB_ORG_ID\x00default

\x00\x00TEST_TABLE_2 column=0:DATA_TYPE, timestamp=1460354090655,
value=\x80\x00\x00\x0C

_0_1460354090089\x00CDS_SUB_ORG_ID\x00default

\x00\x00TEST_TABLE_2 column=0:DECIMAL_DIGITS, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089\x00CDS_SUB_ORG_ID\x00default

\x00\x00TEST_TABLE_2 column=0:IS_ROW_TIMESTAMP, timestamp=1460354090655,
value=\x00

_0_1460354090089\x00CDS_SUB_ORG_ID\x00default

\x00\x00TEST_TABLE_2 column=0:IS_VIEW_REFERENCED, timestamp=1460354090655,
value=\x00

_0_1460354090089\x00CDS_SUB_ORG_ID\x00default

\x00\x00TEST_TABLE_2 column=0:KEY_SEQ, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089\x00CDS_SUB_ORG_ID\x00default

\x00\x00TEST_TABLE_2 column=0:NULLABLE, timestamp=1460354090655,
value=\x80\x00\x00\x01

_0_1460354090089\x00CDS_SUB_ORG_ID\x00default

\x00\x00TEST_TABLE_2 column=0:ORDINAL_POSITION, timestamp=1460354090655,
value=\x80\x00\x00\x01

_0_1460354090089\x00CDS_SUB_ORG_ID\x00default

\x00\x00TEST_TABLE_2 column=0:PK_NAME, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089\x00CDS_SUB_ORG_ID\x00default

\x00\x00TEST_TABLE_2 column=0:SORT_ORDER, timestamp=1460354090655,
value=\x80\x00\x00\x02

_0_1460354090089\x00CDS_SUB_ORG_ID\x00default

\x00\x00TEST_TABLE_2 column=0:VIEW_CONSTANT, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089\x00CDS_SUB_ORG_ID\x00default

\x00\x00TEST_TABLE_2 column=0:_0, timestamp=1460354090655, type=DeleteColumn

_0_1460354090089\x00CDS_SUB_ORG_ID\x00default

\x00\x00TEST_TABLE_2 column=0:, timestamp=1460455162842, type=DeleteFamily

_0_1460354090089\x00CDS_UPDATED_BY\x00default

\x00\x00TEST_TABLE_2 column=0:ARRAY_SIZE, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089\x00CDS_UPDATED_BY\x00default

\x00\x00TEST_TABLE_2 column=0:COLUMN_DEF, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089\x00CDS_UPDATED_BY\x00default

\x00\x00TEST_TABLE_2 column=0:COLUMN_SIZE, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089\x00CDS_UPDATED_BY\x00default

\x00\x00TEST_TABLE_2 column=0:DATA_TABLE_NAME, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089\x00CDS_UPDATED_BY\x00default

\x00\x00TEST_TABLE_2 column=0:DATA_TYPE, timestamp=1460354090655,
value=\x80\x00\x00\x0C

_0_1460354090089\x00CDS_UPDATED_BY\x00default

\x00\x00TEST_TABLE_2 column=0:DECIMAL_DIGITS, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089\x00CDS_UPDATED_BY\x00default

\x00\x00TEST_TABLE_2 column=0:IS_ROW_TIMESTAMP, timestamp=1460354090655,
value=\x00

_0_1460354090089\x00CDS_UPDATED_BY\x00default

\x00\x00TEST_TABLE_2 column=0:IS_VIEW_REFERENCED, timestamp=1460354090655,
value=\x00

_0_1460354090089\x00CDS_UPDATED_BY\x00default

\x00\x00TEST_TABLE_2 column=0:KEY_SEQ, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089\x00CDS_UPDATED_BY\x00default

\x00\x00TEST_TABLE_2 column=0:NULLABLE, timestamp=1460354090655,
value=\x80\x00\x00\x01

_0_1460354090089\x00CDS_UPDATED_BY\x00default

\x00\x00TEST_TABLE_2 column=0:ORDINAL_POSITION, timestamp=1460354090655,
value=\x80\x00\x00\x03

_0_1460354090089\x00CDS_UPDATED_BY\x00default

\x00\x00TEST_TABLE_2 column=0:PK_NAME, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089\x00CDS_UPDATED_BY\x00default

\x00\x00TEST_TABLE_2 column=0:SORT_ORDER, timestamp=1460354090655,
value=\x80\x00\x00\x02

_0_1460354090089\x00CDS_UPDATED_BY\x00default

\x00\x00TEST_TABLE_2 column=0:VIEW_CONSTANT, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089\x00CDS_UPDATED_BY\x00default

\x00\x00TEST_TABLE_2 column=0:_0, timestamp=1460354090655, type=DeleteColumn

_0_1460354090089\x00CDS_UPDATED_BY\x00default

\x00\x00TEST_TABLE_2 column=0:, timestamp=1460455162842, type=DeleteFamily

_0_1460354090089\x00CDS_UPDATED_DATE\x00defau

lt

\x00\x00TEST_TABLE_2 column=0:ARRAY_SIZE, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089\x00CDS_UPDATED_DATE\x00defau

lt

\x00\x00TEST_TABLE_2 column=0:COLUMN_DEF, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089\x00CDS_UPDATED_DATE\x00defau

lt

\x00\x00TEST_TABLE_2 column=0:COLUMN_SIZE, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089\x00CDS_UPDATED_DATE\x00defau

lt

\x00\x00TEST_TABLE_2 column=0:DATA_TABLE_NAME, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089\x00CDS_UPDATED_DATE\x00defau

lt

\x00\x00TEST_TABLE_2 column=0:DATA_TYPE, timestamp=1460354090655,
value=\x80\x00\x00\x0C

_0_1460354090089\x00CDS_UPDATED_DATE\x00defau

lt

\x00\x00TEST_TABLE_2 column=0:DECIMAL_DIGITS, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089\x00CDS_UPDATED_DATE\x00defau

lt

\x00\x00TEST_TABLE_2 column=0:IS_ROW_TIMESTAMP, timestamp=1460354090655,
value=\x00

_0_1460354090089\x00CDS_UPDATED_DATE\x00defau

lt

\x00\x00TEST_TABLE_2 column=0:IS_VIEW_REFERENCED, timestamp=1460354090655,
value=\x00

_0_1460354090089\x00CDS_UPDATED_DATE\x00defau

lt

\x00\x00TEST_TABLE_2 column=0:KEY_SEQ, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089\x00CDS_UPDATED_DATE\x00defau

lt

\x00\x00TEST_TABLE_2 column=0:NULLABLE, timestamp=1460354090655,
value=\x80\x00\x00\x01

_0_1460354090089\x00CDS_UPDATED_DATE\x00defau

lt

\x00\x00TEST_TABLE_2 column=0:ORDINAL_POSITION, timestamp=1460354090655,
value=\x80\x00\x00\x02

_0_1460354090089\x00CDS_UPDATED_DATE\x00defau

lt

\x00\x00TEST_TABLE_2 column=0:PK_NAME, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089\x00CDS_UPDATED_DATE\x00defau

lt

\x00\x00TEST_TABLE_2 column=0:SORT_ORDER, timestamp=1460354090655,
value=\x80\x00\x00\x02

_0_1460354090089\x00CDS_UPDATED_DATE\x00defau

lt

\x00\x00TEST_TABLE_2 column=0:VIEW_CONSTANT, timestamp=1460354090655,
type=DeleteColumn

_0_1460354090089\x00CDS_UPDATED_DATE\x00defau

lt

\x00\x00TEST_TABLE_2 column=0:_0, timestamp=1460354090655, type=DeleteColumn

_0_1460354090089\x00CDS_UPDATED_DATE\x00defau

lt


Thanks,

Arun


On Wed, Apr 20, 2016 at 11:31 AM, Arun Kumaran Sabtharishi <
arun1087@gmail.com> wrote:

> James,
>
> Table SYSTEM.CATALOG is ENABLED
> SYSTEM.CATALOG, {TABLE_ATTRIBUTES => {coprocessor$1 =>
> '|org.apache.phoenix.coprocessor.ScanRegionObserver|1|', coprocessor$2 =>
> '|org.apache
>
> .phoenix.coprocessor.UngroupedAggregateRegionObserver|1|', coprocessor$3
> => '|org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver|1
>
> |', coprocessor$4 =>
> '|org.apache.phoenix.coprocessor.ServerCachingEndpointImpl|1|',
> coprocessor$5 => '|org.apache.phoenix.coprocessor.MetaDa
>
> taEndpointImpl|1|', coprocessor$6 =>
> '|org.apache.phoenix.coprocessor.MetaDataRegionObserver|2|', METADATA =>
> {'SPLIT_POLICY' => 'org.apache.
>
> phoenix.schema.MetaDataSplitPolicy', 'UpgradeTo30' => 'true'}}
> COLUMN FAMILIES DESCRIPTION
> {NAME => '0', DATA_BLOCK_ENCODING => 'FAST_DIFF', BLOOMFILTER => 'ROW',
> REPLICATION_SCOPE => '0', COMPRESSION => 'NONE', VERSIONS => '1000',
>
> MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'true',
> BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}
>
> 1 row in 0.6060 seconds
>
> The above is for describe SYSTEM.CATALOG. The output for scan
> 'SYSTEM.CATALOG', {RAW=>true} is too huge.
>
> Thanks,
> Arun
>
>
> On Wed, Apr 20, 2016 at 11:19 AM, James Taylor <jamestaylor@apache.org>
> wrote:
>
>> Arun,
>> Please run the command Ankit mentioned in an HBase shell and post the
>> output back here.
>> Thanks,
>> James
>>
>>
>> On Wednesday, April 20, 2016, Arun Kumaran Sabtharishi <
>> arun1087@gmail.com> wrote:
>>
>>> !describe SYSTEM.CATALOG is not returning IS_ROW_TIMESTAMP column.
>>>
>>> But we do see this column from select statement:
>>>
>>> select * from SYSTEM.CATALOG where TABLE_NAME=’TEST_TABLE_1’ AND
>>> TABLE_SCHEM IS NULL AND TENANT_ID IS NULL ;
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Arun
>>>
>>> On Wed, Apr 20, 2016 at 1:37 AM, Ankit Singhal <ankitsinghal59@gmail.com
>>> > wrote:
>>>
>>>> Hi Arun,
>>>>
>>>> Do you see 'IS_ROW_TIMESTAMP' column in SYSTEM.CATALOG, by doing
>>>> !describe on system.catalog.
>>>>
>>>>
>>>> if not,
>>>> can you share the output of below command. As it seems SYSTEM.CATALOG
>>>> was updated with timestamp greater v4.6 timestamp , and which stopping
>>>> upgrade code to add a new column.
>>>>
>>>> scan 'SYSTEM.CATALOG', {RAW=>true}
>>>>
>>>>
>>>>
>>>> Regards,
>>>> Ankit Singhal
>>>>
>>>> On Wed, Apr 20, 2016 at 4:25 AM, Arun Kumaran Sabtharishi <
>>>> arun1087@gmail.com> wrote:
>>>>
>>>>> After further investigation, we found that Phoenix Upsert query
>>>>> SYSTEM.CATALOG has IS_ROW_TIMESTAMP column, but PTableImpl.getColumn()
is
>>>>> failing with error:"Undefined column. columnName=IS_ROW_TIMESTAMP" .
Does
>>>>> this mean that PTableImpl is reading from cached entity of  SYSTEM.CATALOG
>>>>>  before 4.6 upgrade?"
>>>>>
>>>>> We do see that clearCache() is being called for 4.7, and 4.7 upgrades
>>>>> from ConnectionQueryServicesImpl class, but not for 4.6
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Arun
>>>>>
>>>>> On Tue, Apr 19, 2016 at 10:22 AM, Arun Kumaran Sabtharishi <
>>>>> arun1087@gmail.com> wrote:
>>>>>
>>>>>> James,
>>>>>>
>>>>>> To add more information on this issue, this happens in new phoenix
>>>>>> views associated with brand new tables as well. So, this cannot be
an
>>>>>> upgrade/migration issue. Not figured out a specific way to reproduce
this
>>>>>> issue yet. Could you throw some ideas on what direction this problem
could
>>>>>> be approached from this point?
>>>>>>
>>>>>> Thanks,
>>>>>> Arun
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>

Mime
View raw message