Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LIVY-245. Add support shared variables across Jobs #333

Closed
wants to merge 2 commits into from

Conversation

jerryshao
Copy link
Contributor

Currently we cannot share variables across different Jobs in Livy, so here propose to add a cache layer in RSC to store shared objects. This cache followed LRU, the least not used will be removed when exceeding limits.

This work is based on @alex-the-man 's work #248

jerryshao added 2 commits May 18, 2017 17:01
Change-Id: I6eccd5d2efebe153e3be4d94b80aa964ed095fc1
Change-Id: I14d688caf94950a4dcd9ad17bf7edfeba2339eed
@codecov-io
Copy link

codecov-io commented May 18, 2017

Codecov Report

Merging #333 into master will increase coverage by 0.18%.
The diff coverage is 88.46%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master     #333      +/-   ##
============================================
+ Coverage     70.46%   70.65%   +0.18%     
- Complexity      726      731       +5     
============================================
  Files            96       96              
  Lines          5123     5145      +22     
  Branches        774      776       +2     
============================================
+ Hits           3610     3635      +25     
+ Misses          996      994       -2     
+ Partials        517      516       -1
Impacted Files Coverage Δ Complexity Δ
...a/com/cloudera/livy/scalaapi/ScalaJobContext.scala 23.07% <0%> (-6.93%) 0 <0> (ø)
...rc/main/scala/com/cloudera/livy/repl/Session.scala 74.34% <100%> (ø) 1 <0> (ø) ⬇️
...c/src/main/java/com/cloudera/livy/rsc/RSCConf.java 87.75% <100%> (+0.12%) 7 <0> (ø) ⬇️
...a/com/cloudera/livy/rsc/driver/JobContextImpl.java 69.01% <100%> (+10.52%) 19 <4> (+4) ⬆️
...c/src/main/java/com/cloudera/livy/rsc/rpc/Rpc.java 78.61% <0%> (+0.62%) 12% <0%> (ø) ⬇️
...n/java/com/cloudera/livy/rsc/driver/RSCDriver.java 77.15% <0%> (+0.86%) 39% <0%> (ø) ⬇️
rsc/src/main/java/com/cloudera/livy/rsc/Utils.java 85.71% <0%> (+2.38%) 16% <0%> (ø) ⬇️
...va/com/cloudera/livy/rsc/rpc/KryoMessageCodec.java 98.11% <0%> (+3.77%) 19% <0%> (+1%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2abb8a3...2b09553. Read the comment docs.

@jerryshao jerryshao closed this Jul 24, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants