-
Notifications
You must be signed in to change notification settings - Fork 11
Zeppelin Related Issues
Please update your script using command
/scripts/update
And Start/Stop Zeppelin services using follow command:
/scripts/start-zeppelin.sh
/scripts/stop-zeppelin.sh
This is due to zeppelin would supposed to create a few temp files in “current directory”, may cause some permission issue if it was not started in a appropriate path.
We placed all environment initialization scripts in /etc/profile.d/bigbox.sh, you can also edit this file to make some revision.
If you edit this file, you may find a few lines as follow:
unset ZEPPELIN_MEM
unset ZEPPELIN_INTP_MEM
# export ZEPPELIN_MEM=" -Xms2048m -Xmx2048m -XX:MaxPermSize=1024m "
# export ZEPPELIN_INTP_MEM=" -Xms1024m -Xmx2048m -XX:MaxPermSize=1024m "
export ZEPPELIN_INTP_MEM=" -Xms512m -Xmx2048m "
A few lines are commented out, which will not effected. You can also modify those lines to meke zeppelin fit for your environment.
Please refer to this page for the explaination of those java options.
In brief, -Xms512m means sets the initial and minimum Java heap size as 512m. -Xmx2048m means sets the maximum Java heap size as 2g.
Please refer to this page for the introduction of zeppelin options.
In brief, once you started your zeppelin, it will start a JVM using ZEPPELIN_MEM as service manager, once your wrote some code, and click 'run', it will try to find the specific interpreter, which could be found in /usr/local/zeppelin/interpreter, and start it if it was not started, using ZEPPELIN_INTP_MEM as default jvm options. If ZEPPELIN_INTP_MEM does not exists, it will copy ZEPPELIN_MEM. Caution: each interpreter would reserve so much memory. If you start markdown, at least '-Xms' was gone, if you start spark, another at least '-Xms' was gone.
Due to our homework requires to load a few big files, and 1G in default for interpreter is not enough, my suggestion is please copy my configuration above (include 'unset'). If it is still not enough, please adjust it by yourself.
If you decide to destroy your container, but wish to reserve the downloaded zeppelins (hmm, it is really long long long downloading), you can make a backup from /usr/local/zeppelin to /mnt/host/plus/path/to/your/host/fs
Once you have created a new container, and copy your zeppelin back, you can execute the follow commands to reconfigure your zeppelin:
/scripts/config-zeppelin.sh
It is NOT required to reconfigure you zeppelin if it is your first time installing your zeppelin. You can do that but effected nothing.
docker-machine is a tool helping you to manage your hosts in Mac or Windows users, or some remote hosts.
If you are using docker-machine in macOS or Windows, please don't give to too much memory in creating docker machine. Please reserve at least half of your total RAM for your host machine. We may reduce the RAM usage somewhere else.
If you are a Linux user and running your docker service in localhost, you are not required to care about this. You are safe to reserve most of your memory to the docker.
Here are a few commands as reference:
docker-machine rm default # remove default machine, if you open your virtualbox, you will find one vm was erased
docker-machine create --driver virtualbox --virtualbox-memory 8192 --virtualbox-cpu-count -1 default # create a vm in virtualbox, and reserve 8192M RAM at most, which could use each of your CPUs
docker-machine ls # list the docker machine in managing
docker-machine start default # start your docker machine named as default
docker-machine stop default # start your docker machine named as defaultBesides, if you are a Linux user, you can simply visit 'localhost:8888' for your jupyter, however, you should find the ip of docker host to reach your jupyter if you are using docker-machine.
Finally, if you installed Docker.app from docker.com and not managed by your docker-machine, you can visit localhost:9530 directly.
For the purpose of getting the latest scripts, you’d better update your scripts using command:
/scripts/update
Please go to /mnt/host. DON'T miss '/'!
If you are a windows user, you may see a folder named as 'c', which would be your "C:" disk.
Otherwise you may meet some 'Connection Refused' exception.
If you meet any questions about zeppelin, please stop your zeppelin first, backup or cleanup folder /usr/local/zeppelin/logs, and try to restart your zeppelin.
And then, please reproduce your errors using as less steps as possible, and save the logs in follow
[root@bootcamp1 logs]# cd /usr/local/zeppelin/logs
[root@bootcamp1 logs]# pwd
/usr/local/zeppelin/logs
[root@bootcamp1 logs]# ls -al
total 704
drwxr-xr-x 2 zeppelin zeppelin 4096 Jan 25 23:25 .
drwxr-xr-x 13 zeppelin zeppelin 4096 Jan 25 23:26 ..
-rw-r--r-- 1 zeppelin zeppelin 1088 Jan 25 23:03 install-interpreter.log
-rw-r--r-- 1 zeppelin zeppelin 9331 Jan 25 23:26 zeppelin-interpreter-md-zeppelin-bootcamp1.docker.log
-rw-r--r-- 1 zeppelin zeppelin 649767 Jan 25 23:26 zeppelin-interpreter-spark-zeppelin-bootcamp1.docker.log
-rw-r--r-- 1 zeppelin zeppelin 36581 Jan 25 23:27 zeppelin-zeppelin-bootcamp1.docker.log
-rw-r--r-- 1 zeppelin zeppelin 5479 Jan 25 23:27 zeppelin-zeppelin-bootcamp1.docker.out
- install-interpreter.log is a log about interpreter installing message
- zeppelin-interpreter-md* logs your information about markdown in zeppelin
- zeppelin-interpreter-spark* logs spark interpreter
- zeppelin-zeppelin-* logs the zeppelin itself
Actually, you can debug by yourself sometimes according to your log.
[root@bootcamp1 /]# cat /proc/meminfo | grep MemTotal
MemTotal: 8164644 kB
Zeppelin is a manager to help you run those code online. It will take some time to start those interpreter. In this case, your first running may take some time. For the same reason, if you click 'run all' before interpreters initialized, it may crashed in boosting. Some of those processes are required to be singleton. If this interpreter was crashed, you can go to interpreter configuration (http://some-ip:9530/#/interpreter ) page, and try to restart spark over there.
Zeppelin service runs as user zeppelin, sometimes it may meet some permission issue when you are obtaining some files only visible to root or other users.
It requires a patch to execute python in zeppelin. Currently it supports python3.5 only.
/scripts/update
/scripts/install-prerequisites-for-py35-in-zeppelin.sh
and then restart your zeppelin.