phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Taylor <jamestay...@apache.org>
Subject Re: Migrate data between 2.2 and 4.0
Date Sun, 29 Jun 2014 07:50:35 GMT
The default column family (i.e. the name of the column family used for
your table when one is not explicitly specified) was changed from _0
to 0 between 2.2 and 3.0/4.0. You can override this in your CREATE
TABLE statement through the DEFAULT_COLUMN_FAMILY property. The
upgrade script modifies this property dynamically for any table being
upgraded.

Was the upgrade script not working for you?

Thanks,
James

On Fri, Jun 27, 2014 at 6:44 PM, Kristoffer Sjögren <stoffe@gmail.com> wrote:
> After copying data from column _0 to column 0 phoenix is able to read
> the data (didn't try the catalog trick).
>
> I suppose this is the way Phoenix does the upgrade internally also
> (move data between columns)?
>
> On Wed, Jun 25, 2014 at 9:48 PM, Jeffrey Zhong <jzhong@hortonworks.com> wrote:
>>
>> No, you have to create column family name 0 and copy the data from column
>> family "_0" to it or (personally I didn't try the following) you might
>> able to change meta data in system.catalog to use column family "_0"
>> instead.
>>
>> For normal upgrade, you should follow instructions at
>> http://phoenix.apache.org/upgrade_from_2_2.html
>>
>> On 6/25/14 7:11 AM, "Kristoffer Sjögren" <stoffe@gmail.com> wrote:
>>
>>>Yes, the import worked after creating the column family and I can see
>>>all the rows when doing scans.
>>>
>>>But I got nothing when using Phoenix 4.0 client, so after comparing
>>>old and new tables I saw that 4.0 tables have column family name 0
>>>instead of _0.
>>>
>>>Now as far as I know there is no way to rename a column qualifier,
>>>right? So I cant simply remove the 0 column and rename _0 to 0 right?
>>>
>>>
>>>
>>>On Wed, Jun 25, 2014 at 3:08 AM, Jeffrey Zhong <jzhong@hortonworks.com>
>>>wrote:
>>>>
>>>>
>>>> You can try to use hbase shell to manually add "_0" column family into
>>>> your destination hbase table. Phoenix 4.0 from Apache can't work on
>>>> hbase0.96. You can check discussions in
>>>> https://issues.apache.org/jira/browse/PHOENIX-848 to see if your hbase
>>>>is
>>>> good for phoenix 4.0.
>>>>
>>>> Thanks,
>>>> -Jeffrey
>>>>
>>>> On 6/24/14 5:32 AM, "Kristoffer Sjögren" <stoffe@gmail.com> wrote:
>>>>
>>>>>Hi
>>>>>
>>>>>We're currently running Phoenix 2.2 on HBase 0.94 CDH 4.4 and slowly
>>>>>preparing to move to Phoenix 4 and HBase 0.96 CDH 5.
>>>>>
>>>>>For my first tests I wanted to simply copy data from 0.94 to 0.96,
>>>>>which works fine for regular hbase table using the following commands:
>>>>>
>>>>>$ hbase org.apache.hadoop.hbase.mapreduce.Export table /tmp/table
>>>>>$ hadoop distcp hftp://hbase94:50070/tmp/table hdfs://hbase96/tmp/table
>>>>>$ hbase -Dhbase.import.version=0.94
>>>>>org.apache.hadoop.hbase.mapreduce.Import table /tmp/table
>>>>>
>>>>>This approach fail on the import for for phoenix tables tough (see
>>>>>below) - where I create an identical table in 0.96 using phoenix 4.0
>>>>>sqlline and then do the commands mentioned above. As I understand, the
>>>>>_0 column family was used to allow hbase empty rows.
>>>>>
>>>>>Are there any tricks that can be made to allow copy data between these
>>>>>two installations?
>>>>>
>>>>>Cheers,
>>>>>-Kristoffer
>>>>>
>>>>>
>>>>>2014-06-18 13:31:09,633 INFO  [main] mapreduce.Job: Task Id :
>>>>>attempt_1403015236309_0015_m_000004_1, Status : FAILED
>>>>>Error:
>>>>>org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
>>>>>Failed 6900 actions:
>>>>>org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException:
>>>>>Column family _0 does not exist in region
>>>>>TABLE,\x1F\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x
>>>>>00
>>>>>\x00\x00\x00\x00\x00\x00,1403090967713.51a3eefa9b92568a87223aff7878cdcf.
>>>>>in table 'TABLE', {TABLE_ATTRIBUTES => {coprocessor$1 =>
>>>>>'|org.apache.phoenix.coprocessor.ScanRegionObserver|1|', coprocessor$2
>>>>>=>
>>>>>'|org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver|1|',
>>>>>coprocessor$3 =>
>>>>>'|org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver|1|',
>>>>>coprocessor$4 =>
>>>>>'|org.apache.phoenix.coprocessor.ServerCachingEndpointImpl|1|',
>>>>>coprocessor$5 =>
>>>>>'|org.apache.phoenix.hbase.index.Indexer|1073741823|index.builder=org.ap
>>>>>ac
>>>>>he.phoenix.index.PhoenixIndexBuilder,org.apache.hadoop.hbase.index.codec
>>>>>.c
>>>>>lass=org.apache.phoenix.index.PhoenixIndexCodec'},
>>>>>{NAME => '0', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY
=>
>>>>>'false', KEEP_DELETED_CELLS => 'true', DATA_BLOCK_ENCODING =>
>>>>>'FAST_DIFF', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS
=>
>>>>>'0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE
=>
>>>>>'0'}
>>>>>at
>>>>>org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServ
>>>>>er
>>>>>.java:4056)
>>>>>at
>>>>>org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMuta
>>>>>ti
>>>>>on(HRegionServer.java:3361)
>>>>>at
>>>>>org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.j
>>>>>av
>>>>>a:3265)
>>>>>at
>>>>>org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.
>>>>>ca
>>>>>llBlockingMethod(ClientProtos.java:26935)
>>>>>at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2175)
>>>>>at
>>>>>org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1879)
>>>>>: 6900 times,
>>>>>at
>>>>>org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(As
>>>>>yn
>>>>>cProcess.java:187)
>>>>>at
>>>>>org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$500(Async
>>>>>Pr
>>>>>ocess.java:171)
>>>>>at
>>>>>org.apache.hadoop.hbase.client.AsyncProcess.getErrors(AsyncProcess.java:
>>>>>88
>>>>>2)
>>>>>at
>>>>>org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java
>>>>>:9
>>>>>40)
>>>>>at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:903)
>>>>>at org.apache.hadoop.hbase.client.HTable.put(HTable.java:864)
>>>>>at
>>>>>org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.wr
>>>>>it
>>>>>e(TableOutputFormat.java:126)
>>>>>at
>>>>>org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.wr
>>>>>it
>>>>>e(TableOutputFormat.java:87)
>>>>>at
>>>>>org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.
>>>>>ja
>>>>>va:635)
>>>>>at
>>>>>org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskIn
>>>>>pu
>>>>>tOutputContextImpl.java:89)
>>>>>at
>>>>>org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedM
>>>>>ap
>>>>>per.java:112)
>>>>>at
>>>>>org.apache.hadoop.hbase.mapreduce.Import$Importer.writeResult(Import.jav
>>>>>a:
>>>>>167)
>>>>>at
>>>>>org.apache.hadoop.hbase.mapreduce.Import$Importer.map(Import.java:136)
>>>>>at
>>>>>org.apache.hadoop.hbase.mapreduce.Import$Importer.map(Import.java:118)
>>>>>at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
>>>>>at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
>>>>>at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
>>>>>at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
>>>>>at java.security.AccessController.doPrivileged(Native Method)
>>>>>at javax.security.auth.Subject.doAs(Subject.java:422)
>>>>>at
>>>>>org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformatio
>>>>>n.
>>>>>java:1548)
>>>>>at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
>>>>
>>>>
>>>>
>>>> --
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or
>>>>entity to
>>>> which it is addressed and may contain information that is confidential,
>>>> privileged and exempt from disclosure under applicable law. If the
>>>>reader
>>>> of this message is not the intended recipient, you are hereby notified
>>>>that
>>>> any printing, copying, dissemination, distribution, disclosure or
>>>> forwarding of this communication is strictly prohibited. If you have
>>>> received this communication in error, please contact the sender
>>>>immediately
>>>> and delete it from your system. Thank You.
>>
>>
>>
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.

Mime
View raw message