Tag Archives: Jupyter

Jupyter notebook Failed to Switch to the Virual Environment: DLL load failed python.exe could not find the entry

I. Error reporting

I originally installed Anaconda3, with python3.7 and Jupyter-notebook, and installed tf2.0 environment.

Later I created a new virtual environment tf_1 based on tf2.0 environment and installed tf.14, so that

tf1.0 and tf2.0 versions can be switched flexibly on Jupyter-notebook.

If you log in to notebook directly with cmd, see the following:

At this point, I directly create a new python3, which means that the default tf2.0 environment is Ok, as shown below:

But I new a tf_1_jjupyter is reported the following error, also open a tf2.0 version of the notebook file in the change kenerl will also report the same error.

The error is reported as follows.

ImportError: DLL load failed: The specified module could not be found

II. Solution

Solve the Jupyter notebook startup error or running code error

1. ImportError: DLL load failed : The specified module could not be found
Solution.

cmd-windows console-enter conda activate virtual environment name

For example, the name of the virtual environment here is tf_1

If you don’t remember, you can find it in the Anaconda installation directory

D:\software\Anaconda_candy\envs\tf_1

2. solve python.exe can not find the entrance can not locate the program input point
After entering the virtual environment if it still reports an error as follows.

This error pops up when I enter jupyter notebook, but I can enter jupyter notebook to debug the code normally when I cross it out. Initially, I think there is a problem with the dll file. After reading some online solutions, the following is the solution:

Solution: pythoncom37.dll is a file of pywin32 located in the path Anaconda3\envs\your virtual environment\Lib\site-packages\pywin32_system32, the location of my file here is shown below.

And there is a file with the same name ythoncom37.dll in D:\python\Anaconda3\envs\tf_1\Library\binp.

After deleting this file, there will be no pop-up error!

After deleting the pythoncom37.dl file according to the file path in the pop-up box, the error is still reported as follows.

[premise conda activate tf_1 under virtual environment

We follow the file path given in the pop-up box to find pythoncom37.dl and delete it again, and that’s the end of it.

The above solution has been successfully solved as follows.

Switching kernel in the file will also not report errors

Select tf2.0 and virtual environment tf1.0 in the drop-down box by creating a new, and you can switch versions freely, or switch environments in the current file, as shown below:

[Solved] Jupyter Notebook Error: SparkException: Python worker failed to connect back

report errors

---------------------------------------------------------------------------
Py4JJavaError                             Traceback (most recent call last)
<ipython-input-24-bafca16b0526> in <module>
      8     return jobitem, ratingsRDD
      9 jobitem, jobRDD = preparJobdata(sc)
---> 10 jobRDD.collect() 

G:\Projects\python-3.6.4-amd64\lib\site-packages\pyspark\rdd.py in collect(self)
    947         """
    948         with SCCallSiteSync(self.context) as css:
--> 949             sock_info = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd())
    950         return list(_load_from_socket(sock_info, self._jrdd_deserializer))
    951 

G:\Projects\python-3.6.4-amd64\lib\site-packages\py4j\java_gateway.py in __call__(self, *args)
   1303         answer = self.gateway_client.send_command(command)
   1304         return_value = get_return_value(
-> 1305             answer, self.gateway_client, self.target_id, self.name)
   1306 
   1307         for temp_arg in temp_args:

G:\Projects\python-3.6.4-amd64\lib\site-packages\py4j\protocol.py in get_return_value(answer, gateway_client, target_id, name)
    326                 raise Py4JJavaError(
    327                     "An error occurred while calling {0}{1}{2}.\n".
--> 328                     format(target_id, ".", name), value)
    329             else:
    330                 raise Py4JError(

Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0) (192.168.101.68 executor driver): org.apache.spark.SparkException: Python worker failed to connect back.
	at org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:182)
	at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:107)
	at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:119)
	at org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:145)
	at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:65)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.run(Task.scala:131)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Accept timed out
	at java.net.DualStackPlainSocketImpl.waitForNewConnection(Native Method)
	at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:135)
	at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
	at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199)
	at java.net.ServerSocket.implAccept(ServerSocket.java:545)
	at java.net.ServerSocket.accept(ServerSocket.java:513)
	at org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:174)
	... 14 more

Driver stacktrace:
	at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2253)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2202)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2201)
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2201)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1078)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1078)
	at scala.Option.foreach(Option.scala:407)
	at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1078)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2440)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2382)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2371)
	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
	at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:868)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2202)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2223)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2242)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2267)
	at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1030)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
	at org.apache.spark.rdd.RDD.withScope(RDD.scala:414)
	at org.apache.spark.rdd.RDD.collect(RDD.scala:1029)
	at org.apache.spark.api.python.PythonRDD$.collectAndServe(PythonRDD.scala:180)
	at org.apache.spark.api.python.PythonRDD.collectAndServe(PythonRDD.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:238)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.spark.SparkException: Python worker failed to connect back.
	at org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:182)
	at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:107)
	at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:119)
	at org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:145)
	at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:65)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.run(Task.scala:131)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	... 1 more
Caused by: java.net.SocketTimeoutException: Accept timed out
	at java.net.DualStackPlainSocketImpl.waitForNewConnection(Native Method)
	at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:135)
	at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
	at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199)
	at java.net.ServerSocket.implAccept(ServerSocket.java:545)
	at java.net.ServerSocket.accept(ServerSocket.java:513)
	at org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:174)
	... 14 more

Solution:

The following variable environments are configured:

# Windows Hadoop variable environments are configured
HADOOP_HOME = F:\hadoop-common-2.2.0-bin-master\hadoop-common-2.2.0-bin-master

# Windows JDKvariable environments are configured
JAVA_HOME = F:\jdk-8u121-windows-x64_8.0.1210.13

# Windows Pysparkvariable environments are configured
PYSPARK_DRIVER_PYTHON = jupyter
PYSPARK_DRIVER_PYTHON_OPTS = notebook
PYSPARK_PYTHON = python

Remember to restart the computer after the configuration is completed!

jupyter notebook Sets Default Browser Error: SyntaxError: (unicode error) ‘utf-8‘ codec can‘t decode byte 0xd4

Jupiter notebook sets the default browser to open with an error
syntax error: (Unicode error) ‘UTF-8’ codec can’t decode byte 0xd4 in position 0: invalid continuation byte

The default code of the Notepad provided by the win system is ANSI. Just reopen the PY script in the notebook,
save it as a py script coded as UTF-8, and it’s OK to run the PY script.

Modify the name of the file: jupyter_notebook_config.py

Code reference

import webbrowser 
webbrowser.register( "Cent Browser", None, webbrowser.GenericBrowser(r"C:\Users\中文用户名\AppData\Local\CentBrowser\Application\chrome.exe"))
c.NotebookApp.browser = "Cent Browser"

Set default browser, CMD input

jupyter notebook --generate-config

Find the PY file of the default configuration file
open it with notepad and find this statement: # c.NotebookApp.browser =
at the bottom of this statement, enter the following statement: the following figure is the configuration of chrome, * * pay attention to double \\**

# c.NotebookApp.password = '' Here
import webbrowser
webbrowser.register('chrome', None, webbrowser.GenericBrowser(u'C:\\Program Files\\Google\\Chrome\\Application\\chrome.exe'))
c.NotebookApp.browser = 'chrome'

jupyter notebook Use pyLDAvis Error: AttributeError: module ‘pyLDAvis‘ has no attribute ‘gensim

Preparation
First, make sure you have the pyldavis and gensim libraries installed.

pip install gensim
pip install pyldavis

Code change
Secondly, if the error not attribute is because the version of pyldavis is different, the new version needs to be changed as follows: change all pyLDAvis.gensim to gensimvis。

import pyLDAvis.gensim as gensimvis
pyLDAvis.enable_notebook()
vis_sub_10 = gensimvis.prepare(lda_fst, corpus, dic, sort_topics = False)
vis_sub_20 = gensimvis.prepare(lda_snd, corpus, dic, sort_topics = False)
pyLDAvis.display(vis_sub_10)

Success! The effect is shown in the following figure.

Or you can install a lower version of pyldavis

pip install pyLDAvis==2.1.2

Codes:

import pyLDAvis.gensim
pyLDAvis.enable_notebook()
vis_sub_10 = pyLDAvis.gensimda_fst, corpus, dic, sort_topics = False)
vis_sub_20 = pyLDAvis.gensim.prepare(lda_snd, corpus, dic, sort_topics = False)
pyLDAvis.display(vis_sub_10)

Change Jupyter Notebook Default Directory

Change Jupyter Notebook Default Directory

There are three way to change the default (i.e., start-up) directory of jupyter notebook.

Solution #1
1. Use command line (aka, cmd), run the following command: this will generate a config file (jupyter_notebook_config.py), in your working location (C:\Users[Your Username].jupyter).
jupyter notebook --generate-config
2. Goto that location (C:\Users[Your Username].jupyter) and edit file jupyter_notebook_config.py as follow.
Find line:

## The directory to use for notebooks and kernels.
#c.NotebookApp.notebook_dir = ''

3. Delete the “#” and type your default directory inside the ”. (Make sure the directory is exist, or it may causes some troubles)
For example:

## The directory to use for notebooks and kernels.
c.NotebookApp.notebook_dir = 'E:\Default Jupyter Dir\'

Solution #2
1. Find the Jupyter Notebook execute file.
2. Right click the file
3. Find “properties” tab and click it; this will lead you to the “Jupyter Notebook Properties”
4. Check the pop-up window, and you should see “Start in” property.
5. Change the directory to your default directory, e.g., ‘E:\Default Jupyter Dir\’
Solution #3

This solution is based on Anaconda, since Jupyter Notebook load the profile from Anaconda.

You may see the following information from properties window if you had gone through Solution #2.

"Target: D:\Anaconda3\python.exe d:\Anaconda3\cwp.py d:\Anaconda3 "d:/Anaconda3/python.exe" "d:/Anaconda3/Scripts/jupyter-notebook-script.py" %USERPROFILE%"

1. Goto the Anaconda installed location.
2. Find the ‘etc’ directory in Anaconda.
3. Open file ‘jupyter_notebook_config.json’
4. Add your default directory to the ‘notebook_dir’.

Error converting jupyter notebook to PDF

nbconvert failed: PDF creating failed, captured latex output:
This is XeTeX, Version 3.14159265-2.6-0.99999 (TeX Live 2018/W32TeX) (preloaded format=xelatex)
restricted \write18 enabled.
entering extended mode
! Undefined control sequence.
< > .\notebook
.tex
?
! Emergency stop.
< > .\notebook
.tex
No pages of output.
Transcript written on ?.

solution:
1, if the file path with Chinese, remove the Chinese
2, find the installation path under the PDF. Py file open
… \site-packages\nbconvert\exporters\pdf.py

#writer = Instance("nbconvert.writers.FilesWriter", args=(), kw={'build_directory': '.'})
writer = Instance("nbconvert.writers.FilesWriter", args=(), kw={'build_directory': ''})

The

and 'build_directory' : '. ' to ' build_directory ':'

GitHub