phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Taylor <jamestay...@apache.org>
Subject Re: Migrate data between 2.2 and 4.0
Date Sun, 29 Jun 2014 08:20:03 GMT
FWIW, the upgrade script does not touch your data - it only modifies
the metadata in your SYSTEM.CATALOG table.

On Sun, Jun 29, 2014 at 10:18 AM, Kristoffer Sjögren <stoffe@gmail.com> wrote:
> Thanks James for the clarification. I haven't tried the upgrade procedure
> yet. I just wanted to migrate our existing data first to CDH5 and try out a
> few things on HBase 0.96.
>
>
> On Sun, Jun 29, 2014 at 9:50 AM, James Taylor <jamestaylor@apache.org>
> wrote:
>>
>> The default column family (i.e. the name of the column family used for
>> your table when one is not explicitly specified) was changed from _0
>> to 0 between 2.2 and 3.0/4.0. You can override this in your CREATE
>> TABLE statement through the DEFAULT_COLUMN_FAMILY property. The
>> upgrade script modifies this property dynamically for any table being
>> upgraded.
>>
>> Was the upgrade script not working for you?
>>
>> Thanks,
>> James
>>
>> On Fri, Jun 27, 2014 at 6:44 PM, Kristoffer Sjögren <stoffe@gmail.com>
>> wrote:
>> > After copying data from column _0 to column 0 phoenix is able to read
>> > the data (didn't try the catalog trick).
>> >
>> > I suppose this is the way Phoenix does the upgrade internally also
>> > (move data between columns)?
>> >
>> > On Wed, Jun 25, 2014 at 9:48 PM, Jeffrey Zhong <jzhong@hortonworks.com>
>> > wrote:
>> >>
>> >> No, you have to create column family name 0 and copy the data from
>> >> column
>> >> family "_0" to it or (personally I didn't try the following) you might
>> >> able to change meta data in system.catalog to use column family "_0"
>> >> instead.
>> >>
>> >> For normal upgrade, you should follow instructions at
>> >> http://phoenix.apache.org/upgrade_from_2_2.html
>> >>
>> >> On 6/25/14 7:11 AM, "Kristoffer Sjögren" <stoffe@gmail.com> wrote:
>> >>
>> >>>Yes, the import worked after creating the column family and I can see
>> >>>all the rows when doing scans.
>> >>>
>> >>>But I got nothing when using Phoenix 4.0 client, so after comparing
>> >>>old and new tables I saw that 4.0 tables have column family name 0
>> >>>instead of _0.
>> >>>
>> >>>Now as far as I know there is no way to rename a column qualifier,
>> >>>right? So I cant simply remove the 0 column and rename _0 to 0 right?
>> >>>
>> >>>
>> >>>
>> >>>On Wed, Jun 25, 2014 at 3:08 AM, Jeffrey Zhong <jzhong@hortonworks.com>
>> >>>wrote:
>> >>>>
>> >>>>
>> >>>> You can try to use hbase shell to manually add "_0" column family
>> >>>> into
>> >>>> your destination hbase table. Phoenix 4.0 from Apache can't work
on
>> >>>> hbase0.96. You can check discussions in
>> >>>> https://issues.apache.org/jira/browse/PHOENIX-848 to see if your
>> >>>> hbase
>> >>>>is
>> >>>> good for phoenix 4.0.
>> >>>>
>> >>>> Thanks,
>> >>>> -Jeffrey
>> >>>>
>> >>>> On 6/24/14 5:32 AM, "Kristoffer Sjögren" <stoffe@gmail.com>
wrote:
>> >>>>
>> >>>>>Hi
>> >>>>>
>> >>>>>We're currently running Phoenix 2.2 on HBase 0.94 CDH 4.4 and
slowly
>> >>>>>preparing to move to Phoenix 4 and HBase 0.96 CDH 5.
>> >>>>>
>> >>>>>For my first tests I wanted to simply copy data from 0.94 to
0.96,
>> >>>>>which works fine for regular hbase table using the following
>> >>>>> commands:
>> >>>>>
>> >>>>>$ hbase org.apache.hadoop.hbase.mapreduce.Export table /tmp/table
>> >>>>>$ hadoop distcp hftp://hbase94:50070/tmp/table
>> >>>>> hdfs://hbase96/tmp/table
>> >>>>>$ hbase -Dhbase.import.version=0.94
>> >>>>>org.apache.hadoop.hbase.mapreduce.Import table /tmp/table
>> >>>>>
>> >>>>>This approach fail on the import for for phoenix tables tough
(see
>> >>>>>below) - where I create an identical table in 0.96 using phoenix
4.0
>> >>>>>sqlline and then do the commands mentioned above. As I understand,
>> >>>>> the
>> >>>>>_0 column family was used to allow hbase empty rows.
>> >>>>>
>> >>>>>Are there any tricks that can be made to allow copy data between
>> >>>>> these
>> >>>>>two installations?
>> >>>>>
>> >>>>>Cheers,
>> >>>>>-Kristoffer
>> >>>>>
>> >>>>>
>> >>>>>2014-06-18 13:31:09,633 INFO  [main] mapreduce.Job: Task Id :
>> >>>>>attempt_1403015236309_0015_m_000004_1, Status : FAILED
>> >>>>>Error:
>> >>>>>org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
>> >>>>>Failed 6900 actions:
>> >>>>>org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException:
>> >>>>>Column family _0 does not exist in region
>>
>> >>>>> >>>>>TABLE,\x1F\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x
>> >>>>>00
>>
>> >>>>> >>>>>\x00\x00\x00\x00\x00\x00,1403090967713.51a3eefa9b92568a87223aff7878cdcf.
>> >>>>>in table 'TABLE', {TABLE_ATTRIBUTES => {coprocessor$1 =>
>> >>>>>'|org.apache.phoenix.coprocessor.ScanRegionObserver|1|',
>> >>>>> coprocessor$2
>> >>>>>=>
>>
>> >>>>> >>>>>'|org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver|1|',
>> >>>>>coprocessor$3 =>
>> >>>>>'|org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver|1|',
>> >>>>>coprocessor$4 =>
>> >>>>>'|org.apache.phoenix.coprocessor.ServerCachingEndpointImpl|1|',
>> >>>>>coprocessor$5 =>
>>
>> >>>>> >>>>>'|org.apache.phoenix.hbase.index.Indexer|1073741823|index.builder=org.ap
>> >>>>>ac
>>
>> >>>>> >>>>>he.phoenix.index.PhoenixIndexBuilder,org.apache.hadoop.hbase.index.codec
>> >>>>>.c
>> >>>>>lass=org.apache.phoenix.index.PhoenixIndexCodec'},
>> >>>>>{NAME => '0', BLOOMFILTER => 'ROW', VERSIONS => '1',
IN_MEMORY =>
>> >>>>>'false', KEEP_DELETED_CELLS => 'true', DATA_BLOCK_ENCODING
=>
>> >>>>>'FAST_DIFF', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS
=>
>> >>>>>'0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE
=>
>> >>>>>'0'}
>> >>>>>at
>>
>> >>>>> >>>>>org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServ
>> >>>>>er
>> >>>>>.java:4056)
>> >>>>>at
>>
>> >>>>> >>>>>org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMuta
>> >>>>>ti
>> >>>>>on(HRegionServer.java:3361)
>> >>>>>at
>>
>> >>>>> >>>>>org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.j
>> >>>>>av
>> >>>>>a:3265)
>> >>>>>at
>>
>> >>>>> >>>>>org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.
>> >>>>>ca
>> >>>>>llBlockingMethod(ClientProtos.java:26935)
>> >>>>>at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2175)
>> >>>>>at
>>
>> >>>>> >>>>>org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1879)
>> >>>>>: 6900 times,
>> >>>>>at
>>
>> >>>>> >>>>>org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(As
>> >>>>>yn
>> >>>>>cProcess.java:187)
>> >>>>>at
>>
>> >>>>> >>>>>org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$500(Async
>> >>>>>Pr
>> >>>>>ocess.java:171)
>> >>>>>at
>>
>> >>>>> >>>>>org.apache.hadoop.hbase.client.AsyncProcess.getErrors(AsyncProcess.java:
>> >>>>>88
>> >>>>>2)
>> >>>>>at
>>
>> >>>>> >>>>>org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java
>> >>>>>:9
>> >>>>>40)
>> >>>>>at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:903)
>> >>>>>at org.apache.hadoop.hbase.client.HTable.put(HTable.java:864)
>> >>>>>at
>>
>> >>>>> >>>>>org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.wr
>> >>>>>it
>> >>>>>e(TableOutputFormat.java:126)
>> >>>>>at
>>
>> >>>>> >>>>>org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.wr
>> >>>>>it
>> >>>>>e(TableOutputFormat.java:87)
>> >>>>>at
>>
>> >>>>> >>>>>org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.
>> >>>>>ja
>> >>>>>va:635)
>> >>>>>at
>>
>> >>>>> >>>>>org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskIn
>> >>>>>pu
>> >>>>>tOutputContextImpl.java:89)
>> >>>>>at
>>
>> >>>>> >>>>>org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedM
>> >>>>>ap
>> >>>>>per.java:112)
>> >>>>>at
>>
>> >>>>> >>>>>org.apache.hadoop.hbase.mapreduce.Import$Importer.writeResult(Import.jav
>> >>>>>a:
>> >>>>>167)
>> >>>>>at
>>
>> >>>>> >>>>>org.apache.hadoop.hbase.mapreduce.Import$Importer.map(Import.java:136)
>> >>>>>at
>>
>> >>>>> >>>>>org.apache.hadoop.hbase.mapreduce.Import$Importer.map(Import.java:118)
>> >>>>>at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
>> >>>>>at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
>> >>>>>at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
>> >>>>>at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
>> >>>>>at java.security.AccessController.doPrivileged(Native Method)
>> >>>>>at javax.security.auth.Subject.doAs(Subject.java:422)
>> >>>>>at
>>
>> >>>>> >>>>>org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformatio
>> >>>>>n.
>> >>>>>java:1548)
>> >>>>>at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
>> >>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>> CONFIDENTIALITY NOTICE
>> >>>> NOTICE: This message is intended for the use of the individual or
>> >>>>entity to
>> >>>> which it is addressed and may contain information that is
>> >>>> confidential,
>> >>>> privileged and exempt from disclosure under applicable law. If the
>> >>>>reader
>> >>>> of this message is not the intended recipient, you are hereby
>> >>>> notified
>> >>>>that
>> >>>> any printing, copying, dissemination, distribution, disclosure or
>> >>>> forwarding of this communication is strictly prohibited. If you
have
>> >>>> received this communication in error, please contact the sender
>> >>>>immediately
>> >>>> and delete it from your system. Thank You.
>> >>
>> >>
>> >>
>> >> --
>> >> CONFIDENTIALITY NOTICE
>> >> NOTICE: This message is intended for the use of the individual or
>> >> entity to
>> >> which it is addressed and may contain information that is confidential,
>> >> privileged and exempt from disclosure under applicable law. If the
>> >> reader
>> >> of this message is not the intended recipient, you are hereby notified
>> >> that
>> >> any printing, copying, dissemination, distribution, disclosure or
>> >> forwarding of this communication is strictly prohibited. If you have
>> >> received this communication in error, please contact the sender
>> >> immediately
>> >> and delete it from your system. Thank You.
>
>

Mime
View raw message