livy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Harsch, Tim" <Tim.Har...@Teradata.com>
Subject Re: Query: Local Path cannot read by Livy (Running with Docker)
Date Mon, 30 Jul 2018 16:26:59 GMT
I created a Jira to start tracking the issue:

https://issues.apache.org/jira/browse/LIVY-487


When you see the issue are you using the files parameter of sessions endpoint?


It would be helpful for the ticket if we could construct SSCCE ( http://sscce.org/ ) to demonstrate
the problem.  I don't have an environment going where I can do that easily at the moment.
 If you do, can you please comment on the ticket?


Thanks,

Tim

________________________________
From: Divya Arya <divya@cyberops.in>
Sent: Friday, July 27, 2018 11:17:30 PM
To: user@livy.incubator.apache.org
Subject: Re: Query: Local Path cannot read by Livy (Running with Docker)

[External Email]



Yes, that is the exact same scenario I have, just instead of jar files I am using CSV files
stored in my client machine's file system and no protocol is supporting to access those files.

On Sat, Jul 28, 2018 at 3:42 AM, Harsch, Tim <Tim.Harsch@teradata.com<mailto:Tim.Harsch@teradata.com>>
wrote:

When you say not able to use my files in Spark Session what do you mean?



I think I’m seeing a similar problem.   If I submit jars using a client machine that is
not the same as the server (so file system is not shared) and I submit the jar it’s looks
like it gets transmitted to the server but then I can’t reference any of the classes.  If
I look at the Spark UI that gets launched for the session then I can see references to paths
in my client machine, which aren’t resolvable on the server.   Is this the scenario you
have?   I think it might occur in a docker env if you are doing client stuff on the host os,
and livy is running in a container.







From: Divya Arya <divya@cyberops.in<mailto:divya@cyberops.in>>
Reply-To: "user@livy.incubator.apache.org<mailto:user@livy.incubator.apache.org>" <user@livy.incubator.apache.org<mailto:user@livy.incubator.apache.org>>
Date: Thursday, July 26, 2018 at 11:45 PM
To: "user@livy.incubator.apache.org<mailto:user@livy.incubator.apache.org>" <user@livy.incubator.apache.org<mailto:user@livy.incubator.apache.org>>
Subject: Re: Query: Local Path cannot read by Livy (Running with Docker)



[External Email]



Please help me out. I am really looking forward.



On Thu, Jul 26, 2018 at 5:49 PM, Divya Arya <divya@cyberops.in<mailto:divya@cyberops.in>>
wrote:

Hello Team,



I need a little support from you, I am trying to bundle up my application which is working
on Apache Livy, Spark (SCALA), so it can run on a Windows-based system, for that, I am using
Apache Docker.



I have set up Livy and Spark in that, which is working perfectly fine but I am not able to
use my Local path files (suppose files in C:/ or D:/) in SparkSession and SparkContext. Please
help me out here.





Thanks and Regards,



Divya Arya



divya@cyberops.in<mailto:divya@cyberops.in>

divya.arya69@gmail.com<mailto:divya.arya69@gmail.com>






Mime
View raw message