phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Elser <els...@apache.org>
Subject Re: Fwd: Apache Phoenix
Date Tue, 02 May 2017 14:25:56 GMT
Planning for unexpected outages with HBase is a very good idea. At a 
minimum, there will likely be points in time where you want to change 
HBase configuration, apply some patched jars, etc. A staging area that 
can buffer data for later processing and avoid dropping data on the 
floor makes this process much easier.

Apache Kafka is just one tool that can help with implementing such a 
staging area -- Apache NiFi is another you might want to look at. I'll 
avoid making any suggestions as to how you should do it because I don't 
know your requirements (nor really care to *wink*). There are lots of 
tools here, you'll need to do the research for your requirements and 
needs to evaluate what tools would work best.

Ash N wrote:
>
> Hello,
>
> We are building an Enterprise Datawarehouse on Phoenix(HBase)
> Please refer the diagram attached.
>
> The EDW supports an unified architecture that serves both Streaming and
> batch use cases.
>
> I am recommending a staging area that is source compliant (i.e. that
> mimics source structure)
> In the EDW path - data is always loaded into staging and then gets moved
> to EDW.
>
> Folks are not liking the idea due to an additional hop. They are saying
> the hop is unnecessary and will cause latency issues.
>
> I am saying latency can be handled in two ways:
>
> 1. The caching layer will take care
> 2. If designed properly, Latency is a function of hardware
>
> What are your thoughts?
>
> One other question -  is Kafka required at all???
> It is introduced in the architecture for replay messages in case kinesis
> connectivity issues.  So that we can replay messages.
> Is there a better way to do it?
>
> help as always is appreciated.
>
>
> Inline image 1
>
>
>
> thanks,
> -ash
>
>
>
>

Mime
View raw message