Tag Archives: python

How to Solve Pytorch eval Stuck Error

Question

The single card training is very fast. When it comes to eval, it doesn’t move after running a batch, and there is no error.

Tried, still not moving

1, change the pin_memory of valid_loader to False. if it is True, it will automatically load the data into pin_memory, which speeds up the data
transfer speed to GPU.
2, change num_workers to 1, some people say too many workers may lead to multi-process interlock, can reduce or not

 

Final Solution:

valid_loader:
pin_memory = true # this is very important. Before, people on the Internet said that changing false might solve the problem. My experiment proved that if you do not work, you can run normally by changing back to true.
num_workers=4
batch_size=8

train_loader:
pin_memory=True
num_workers=4
batch_size = 8
these parameters are the same as valid_loader

In general, first of all, the pin_memory of valid_loader is kept True, which is well understood, the data is automatically loaded into pin_memory, which speeds up the data transfer to the GPU and naturally speeds up the inference process. Then, the number of workers and batch_size is reduced, and both valid_loader and train_loader are reduced. pin_memory of train_loader is also kept True.

[Solved] LeNet Script Train Error: AttributeError: ‘DictIterator’ object has no attribute ‘get_next’

My training environment:

Windows10 64bit;

MindSpore1.5.0-beta;

CPU;

python3.9;

When training Mnist data set with LeNet , the following error occurs

How to solve this problem??

It’s true that the version is a bit old – the above use case needs to be modified if implemented on a newer version – from the current implementation of Iterator’s code, it no longer has the get_next method: https://gitee.com/mindspore/mindspore/blob/master/mindspore/python/mindspore/dataset/engine/iterators.py#L59

But it has __next__ method, so the above line can be modified, you can try it: > original: data = ds.get_next() > modified: data = next(ds)

How to Solve wikiextractor Extract Wikipedia Corpus Error

When I extracted Wikipedia corpus, I first used the wikiextractor. Later, I found that it was always wrong, so it was useless. Since many people asked me how to extract the corpus, I now publish the code

I didn’t write the code, but I found it from a website. Because it took too long and I forgot the address of the website, I can’t post the original URL. If the author sees it, please send a private letter to my original URL

The author’s email address is: [email protected]

How to use: enter the command at the command line:

python data_pre_process.py zhwiki-latest-pages-articles.xml.bz2(Wikipedia Corpus) wiki.zh.text(Saved files)

Source code:

# -*- coding: utf-8 -*-
# Author: Pan Yang ([email protected])
# Copyrigh 2017
from __future__ import print_function

import logging
import os.path
import six
import sys

from IPython.core.page import page
from gensim.corpora import WikiCorpus

page.encoding = 'utf-8'

# Wrapping the Wikipedia xml corpus into txt format
# python data_pre_process.py zhwiki-latest-pages-articles.xml.bz2 wiki.zh.text
if __name__ == '__main__':
    program = os.path.basename(sys.argv[0])
    logger = logging.getLogger(program)

    logging.basicConfig(format='%(asctime)s: %(levelname)s: %(message)s')
    logging.root.setLevel(level=logging.INFO)
    logger.info("running %s" % ' '.join(sys.argv))

    # check and process input arguments
    if len(sys.argv) != 3:
        print("Using: python process_wiki.py enwiki.xxx.xml.bz2 wiki.en.text")
        sys.exit(1)
    inp, outp = sys.argv[1:3]
    space = " "
    i = 0

    output = open(outp, 'w', encoding='utf-8')
    wiki = WikiCorpus(inp, dictionary={})
    for text in wiki.get_texts():
        if six.PY3:
            output.write(bytes(' '.join(text), 'utf-8').decode('utf-8') + '\n')
            #   ###another method
            #   output.write(space.join(map(lambda x: x.decode("utf-8"), str(text))) + '\n')
        else:
            output.write(space.join(text) + "\n")
        i = i + 1
        if i % 10000 == 0:
            logger.info("Saved " + str(i) + " articles")

    output.close()
    logger.info("Finished Saved " + str(i) + " articles")

ModuleNotFoundError: No module named ‘requests‘ [How to Solve]

Problem description: After installing the requests module using pip under cmd, I can use import requests, but I can’t import it under Pycharm IDE with the following error: ModuleNotFoundError: No module named ‘requests ‘

Later I found out that my python was installed on E drive, but I installed it on C drive with the install requests command, without switching E drive. I reinstalled it on the E drive

Installation steps

①cmd window: switch to the Scripts file in the python installation directory on disk E

②Install command: pip install requests, but there is a yellow text error

③ Follow the Yellow prompt command and enter: python -m pip install --upgrade pip to complete the installation

④ Prove that the installation is successful: you can enter the command under Python: import requests command does not report an error

Pycharm page

[Solved] Appium Error: InvalidArgumentException: Message: invalid argument: invalid locator

Appium Error:
selenium.common.exceptions.InvalidArgumentException: Message: invalid argument: invalid locator

 

Solution: Add the following two parameters when you create a driver.

desired_caps[‘chromeOptions’] = {‘w3c’:False}
desired_caps[‘showChromedriverLog’] = True

 

Refer:
Fail to locate an web element using “findElement” on Android 9 using Appium 1.15.0 · Issue #13306 · appium/appium · GitHub

[Solved] Python Requests Error: simplejson.errors.JSONDecodeError: Expecting value

Problem: when running the interface automation script, the requested data is correct, but it keeps reporting error: simplejson.errors Jsondecodeerror: expecting value


reason:
the reason for the error reported by the author here is that this interface returns: response in the non-JSON format
while in the basic interface code, the return is res.json()

Solution:
Modify to return text, or add an exception capture

exception capture

Nltk Library Download error: [errno: 11004] getaddrinfo failed

Download the natural language processing nltk library and install it under win10. The error is as follows:

IP address of raw.githubusercontent.com could not found

since the hosts cannot be modified and saved in normal mode, it needs to be run in management mode

Solution:

  1. Run Power Shell as administrator
  2. Go to the hosts directory: C:\Windows\system32\drivers\etc
  3. Enter cmd, notepad hosts, and enter to modify the hosts file.
  4. Add the website ip address and URL in the last line of the hosts file and save it.


If the IP address of raw.github is changed frequently, you need to query the URL first: open the URL to query the IP address: https://www.ipaddress.com/ Enter raw.githubusercontent Com to get the IP address

Test succeeded

VScode: How to Solve Pylance Error (pip Library Files Installed)

View Python installation location

Enter in Terminal

where python

View PIP installation library file location

pip show <packagename>

Open vscode and set the library path

Ctrl+shift+p Open ` preference; Open Settings(JSON)

Add path

"python.analysis.extraPaths":[
	    "/root/miniconda3/lib/python3.9/site-packages",
        "/root/.local/lib/python3.9/site-packages",
        "...."
        ]

[Solved] Python Error: socket.error [Errno 9] Bad file descriptor

To learn Python Network programming, I wrote two small programs on the server and the client according to the book, and found that an error was reported:

Traceback (most recent call last):

File “./tsTserv.py”, line 20, in

data = tcpCliSock.recv(BUFSIZ)

File “/usr/lib/python2.6/socket.py”, line 165, in _dummy

raise error(EBADF, ‘Bad file descriptor’)

socket.error: [Errno 9] Bad file descriptor

The source code of the server side is as follows:

while True:
        print 'waiting for connection...'
        tcpCliSock,addr = tcpSerSock.accept()
        print '...connected from:',addr
        while True:
                data = tcpCliSock.recv(BUFSIZ)
                if not data:
                        break
                tcpCliSock.send('[%s] %s' %(ctime(),data))
        tcpCliSock.close()
tcpSerSock.close()

Solution:

tcpCliSock.close() is placed in the second while loop, causing tcpCliSock to be closed after receiving data once, and this statement should be placed in the outer loop

[Solved] D455 Depth Camera Error: keyerror: ‘frame_ device_ t‘

color_t = sens_frame.color_t[self.computespeed_t_type]
KeyError: ‘frame_device_t’
Probably because two values are not manually added in the registry
Enable Metadata: Metadata contains important timestamp information and needs to be enabled manually. Please refer to:

https://dev.intelrealsense.com/docs/compiling-librealsense-for-windows-guide

Modifying the Windows Registry:
For each interface found (Steps 2 and 3) perform
Using Registry Editing tool such as “regedit” navigate to HKLM\SYSTEM\CurrentControlSet\Control\DeviceClasses{e5323777-f976-4f5b-9b55-b94699c46e44} branch.
Browse into the subdirectory with the name identical to the Device instance path obtained from the previous step
Expand the entry into #GLOBAL -> Device Parameters
Add DWORD 32bit value named MetadataBufferSizeInKB0 with value 5.
Add an additional DWORD 32bit value named MetadataBufferSizeInKB1 with value 5 for RS400 device zero interface ##?##USB#VID_8086&PID… MI_00…

Repeat the previous step for HKLM\SYSTEM\CurrentControlSet\Control\DeviceClasses\{65E8773D-8F56-11D0-A3B9-00A0C9223196} branch

[Solved] No tf data. Actual error: Fixed Frame [world] does not exist

1. No tf data. Actual error: Fixed Frame [world] does not exist

When using rviz to view relevant coordinate information, you may encounter No tf data. Actual error: Fixed Frame [world] does not exist, the main reason is that there is no world coordinate system in the TF tree. The solution is to let rviz know where the world coordinate system is. You can manually publish using the following command

rosrun tf static_transform_publisher 0 0 0 0 0 0 1 map world 5

2. nvalid argument “/world” passed to canTransform argument source_frame in tf2 frame_ids cannot start with a ‘/’ like

After the first problem is solved, another problem may appear Invalid argument "/world" passed to canTransform argument source_frame in tf2 frame_ids cannot start with a '/' like: as shown in the figure:

this problem is mainly because “/ world” is used in the release code and “world” is used in our release It may be good to recompile the code from “/world” to “world”

[Solved] RuntimeError: “unfolded2d_copy“ not implemented for ‘Half‘

report errors

RuntimeError: "unfolded2d_copy" not implemented for 'Half'

reason

Parameters use_half=true passed in by the model, that is, the CPU is reasoned by using fp16 mixed precision calculation. Fp16 is used to speed up the speed, but the pytorch CPU does not support fp16,

Solution:

  1. Add use_half=False.
  2. Modify half() to float().

So that the model can be calculated;

Modification of my error report:


I hope this article is useful to you!

Thank you for your comments!