Tag Archives: python

Image input data check: Nan inf and do not exist

import os
import cv2
import numpy as np
from tqdm import tqdm

# read txt file content to list
def ReadTxt_to_list(rootdir):
    lines = []
    with open(rootdir, 'r') as file_to_read:
        while True:
            line = file_to_read.readline()
            if not line:
                break
            line = line.rstrip('\n')
            lines.append(line)
    return lines

def check_exist_nan(lstFile):
    rt = os.path.split(lstFile)[0]
    notExist = open(os.path.join(rt,'Lst_notExist.txt'),'w')
    bad = open(os.path.join(rt,'Lst_bad.txt'),'w')

    lines = ReadTxt_to_list(lstFile)
    notNum = 0
    badNum = 0
    newLines = []
    for line in tqdm(lines):
        info_ = line.split('\t')
        assert len(info_) == 3
        _, filePth, idNum = info_
        if not os.path.exists(filePth):
            print('Not exist:', line)
            notExist.write(line+'\n')
            notNum += 1
        else:
            img = cv2.imread(filePth)
            try:
                if np.any(np.isnan(img)) or not np.all(np.isfinite(img)):
                    print('Nan/Inf:', line)
                    badNum += 1
                    bad.write(line + '\n')
                else:
                    newLines.append(line)
            except:
                print('Error:', line)
                badNum += 1
                bad.write(line + '\terror\n')

    print('Not exist', notNum)
    print('Bad', badNum)
    if len(lines) != len(newLines):
        newLst = open(os.path.join(rt,'Lst_new.txt'), 'w')
        for line in newLines:
            newLst.write(line+'\n')
        newLst.close()

    notExist.close()
    bad.close()

if __name__ == '__main__':
    imgLst = '/home/img.lst'
    check_exist_nan(imgLst)

Keras-nightly Import package Error: cannot import name ‘Adam‘ from ‘keras.optimizers‘

Version keras nightly = 2.5.0.dev2021032900

Error information

    from keras.optimizers import Adam
ImportError: cannot import name 'Adam' from 'keras.optimizers' 

Solution

error code

from keras.optimizers import Adam
opt = Adam(lr=lr, decay=lr/epochs)

modify

from keras.optimizers import adam_v2
opt = adam_v2.Adam(learning_rate=lr, decay=lr/epochs)

reason

After the keras library is updated, the package cannot be imported in the original way. Open the optimizers.py source code and find the following two key codes. You can see that Adam import has changed, so it is modified as above.

from keras.optimizer_v2 import adam as adam_v2
'adam': adam_v2.Adam,

[Solved] Error: unrecognized arguments: — no site packages

Error: unrecognized arguments: — no site packages

virtualenv --no-site-packages ${G_VENV_DIR}
usage: virtualenv [--version] [--with-traceback] [-v | -q] [--app-data APP_DATA] [--clear-app-data] [--discovery {builtin}] [-p py] [--creator {builtin,cpython3-win,venv}] [--seeder {app-data,pip}] [--no-seed] [--activators comma_sep_list]
                  [--clear] [--system-site-packages] [--copies] [--download | --no-download] [--extra-search-dir d [d ...]] [--pip version] [--setuptools version] [--wheel version] [--no-pip] [--no-setuptools] [--no-wheel]
                  [--symlink-app-data] [--prompt prompt] [-h]
                  dest
virtualenv: error: unrecognized arguments: --no-site-package

If the version is greater than 20 and the virtualenv version is greater than 20, it is the default no site packages parameter

View virtualenv version

virtualenv --version

1. Downgrading

pip install --upgrade virtualenv==16.7.9

2. remove –no-site-package

virtualenv ${G_VENV_DIR}

Keras import a custom metric model error: unknown metric function: Please ensure this object is passed to`custom_object‘

Keras’s model defines metric or loss,
there is no problem when saving to H5, but when using load_ When importing the model, an error will be reported:

unknown metric function: HammingScore. Please ensure this object is passed to the custom_ objects argument.

This is because the custom parameters are not passed in. There are two solutions:

          1. if you only need to predict and no longer train, you can add

        compile = false directly

model = keras.models.load_model('model.h5', compile = False)

If you need further training or modification, add your own metrics code and compile it again

model.compile(loss='binary_crossentropy',
              optimizer=Ada,
              metrics=[HammingScore]) # 这里HammingScore是我自定义的metric
      1. when importing, the user-defined metric/loss is passed into

Custom as a key value_ objects

      1. :
model = keras.models.load_model('model.h5', custom_objects={'HammingScore': HammingScore} )

Note that the key value should be consistent

How to Solve PyInstaller Package Error: ModuleNotFoundError: No module named ‘xxxx‘

In the venv environment, there is no exception in the packaging process when the command line pyinstaler is used to execute the packaging command. However, after the packaging is successful, it is executed. In the venv environment, the packaging process is normal when the command line pyinstaler is used to execute the packaging command. After the packaging is successful, the following similar errors appear when the dist/xxx.exe file is executed:

Traceback (most recent call last):
  File "main.py", line 4, in <module>
    import jcw
  File "PyInstaller\loader\pyimod03_importers.py", line 531, in exec_module
  File "jcw.py", line 3, in <module>
    import pandas as pd
ModuleNotFoundError: No module named 'pandas'

Solution:

Because the running environment is venv virtual environment, the command line packaging may use the installation environment of Python in the computer, that is, the global environment and dependency library, which leads to the failure to include the required modules in the packaging.

Use another packaging method of pyinstaler official website, Python code to run the packaging command, such as:

#!/usr/bin/python3
# -*- coding: utf-8 -*-

import PyInstaller.__main__
import os


if __name__ == '__main__':
    pyi_args = [
        '--upx-dir=F:\\soft\\upx-3.96-win64',
        '--clean',
        '--add-data={0};.'.format(os.path.realpath('cfg.ini')),
        '--add-binary={0};driver'.format(os.path.realpath('driver/chromedriver.exe')),
        '--name=demo',
        'main.py',
        '-y'
    ]
    print("pyinstaller " + " ".join(pyi_args))
    PyInstaller.__main__.run(pyi_args=pyi_args)

Several ways of online search don’t work (you can try your own environment or not)

1.Move the import statement from the file header to the code block

2.Command line use — hidden import = missing module

 

 

FileNotFoundError: Could not find module ‘D:\Anaconda3\envs\labe\Library\bin\geos_c.dll‘ [Solved]

Solution:
FileNotFoundError: Could not find module
‘D:\Anaconda3\envs\LabelImg\label\bin\geos_c.dll’ (or one of its dependencies).
Try using the full path with constructor syntax.(or one of its dependencies). Try using the full path with constructor syntax.

1, most people are not importing the “shapely” dependency. You only need to import it

pip install shapely

2. Import this or report an error, basically it is wrong with your Python version.

To download the corresponding. WHL file, the link is as follows: https://www.lfd.uci.edu/~gohlke/pythonlibs/#shapely , after entering the website, Ctrl + F can find shapely, and select the version up or down according to your own version( The author is python3.8)

#First uninstall the downloaded shapely
pip uninstall shapely
#then download the local cp38 version of shapely
pip install D:\PyCharm\Pro\OCR\PaddleOCR\PPOCRLabel\BACK\Shapely-1.7.1-cp38-cp38-win_amd64.whl

Ctypes DLL Error: FileNotFoundError: Could not find module ‘***_dll.dll‘ (or one of its dependencies).

Generally speaking, a 32-bit DLL can run on a 64 bit computer, so it has nothing to do with the number of bits in the computer. First of all, make sure that the path of the DLL file is correct. If it is correct, you will still report this error, that is, the environment is missing. Baidu search Microsoft Visual C++   2019。

Installation of X86 x64 depends on the two environments

For the record, a lot of mistakes on the Internet delayed the whole day.

Httprunner v3. X generates HTML report in assure format

This document records the process and steps of the initial report generation with allure.

Install allure first, download and configure the environment
reference documents:

https://www.cnblogs.com/wsy1103/p/10530397.html

After installation, the following steps begin:

Step 1: generate report data with assure. The generated report data includes two files: one JSON file and one TXT file.

Switch to D: Python/Test & gt; The following is implemented:

hrun test_post_api.json --alluredir=report

Step 2: create the report data generated just now with assure, generate the report2 folder, – O means to specify the folder to generate the report, – C means to clean up the previous report directory before generating the report.

allure generate D:\Python\test\report -o D:\Python\test\report2 -c

Then open the assure report using the default browser to view it.

allure open D:\Python\test\report2

end. The effect of successful generation:


[frequently asked questions]

1. If the error prompt oserror: [winerror 6] handle is invalid, please refer to the document for solution:

https://ask.csdn.net/questions/3017114

Add some code to runner. Py’s test_ Start method, and import colorama, Sys at the beginning of the file

2. If the Chinese encoding in the return body fails, please refer to the document for solution:

https://blog.csdn.net/zjxht62/article/details/116305944

Add some code to the log of client. Py_ In the print method, the default ASCII encoding is used for Chinese when json.dumps is serialized. To output real Chinese, you need to specify ensure_ ascii=False


[allure grammar]

$Hrun — alluredir allure results — clean alluredir # when the use case format is py file, you can replace “Hrun” with “pytest”, which has the same effect
– alluredir: generate the original data of allure report
allure results: save location of the original data
– clean alluredir: clear the historical data of allure results

$assure generate assure results – O assure report
– O specify the folder to generate the report
– C clean up the previous report directory before generating the report

Assure open assure report: start the default browser to open the assure test report

[Python] pip.vendor.urllib3.exceptions.readtimeouterror: httpsconnectionpool protocol

When installing a python module with pip or pychar, you sometimes encounter a readtimeout error. The reason for this problem is probably the network speed

my default source is Tsinghua, sometimes the link network is not good ( the day after the college entrance examination today, it is estimated that there are too many visitors, everyone wants to go to Tsinghua, ha ha ), so I change the source.

pip --default-timeout=100 install pyspark -i http://pypi.douban.com/simple/ --trusted-host pypi.douban.com

Perfect solution

ModuleNotFoundError: No module named ‘MySQLdb‘

Install MySQL client to solve!

Install the following dependent packages in order

================================

yum install mysql-devel

yum install gcc

yum install python3-devel

pip3 install mysqlclient

================================
the two methods found before
install MySQL connector Python
or pymysql.install_ as_ Mysqldb ()
doesn’t work for me. If I use it, there will be other errors

[Solved] Django.core.exceptions.ImproperlyConfigured: SQLite 3.9.0 or later is required (found 3.7.17).

CentOS comes with 3.7.17 Django, which is not supported. It must be above 3.9.0. If you want to use SQLite3 as a database, there is no way to upgrade it. If it is me, I will not use SQLite3, but use mysql

The upgrade is simple:

1. Download the latest package of SQLite3

https://www.sqlite.org/download.html

wget https://www.sqlite.org/2021/sqlite-autoconf-3350500.tar.gz

2. Compile and install

tar xf https://www.sqlite.org/2021/sqlite-autoconf-3350500.tar.gz
cd sqlite-autoconf-3350500
./configure --prefix=/usr/local/ && make && make install 

3. Change the original SQLite3 command

First check which directory SQLite3 has executable files in

(python36) [[email protected] servermonitor]# whereis sqlite3
 sqlite3: /usr/bin/sqlite3 /usr/local/bin/sqlite3 /usr/include/sqlite3.h /usr/share/man/man1/sqlite3.1.gz (python36) [[email protected] servermonitor]#

It is found that/usr/bin/SQLite3 is old, 3.7.17. If you are not sure, just execute it

/usr/bin/sqlite3 –verssion

Then replace the old version

mv /usr/bin/sqlite3 /usr/bin/sqlite3_3.7.17
 ln -s /usr/local/bin/sqlite3 /usr/bin/sqlite3

4. Change the library path

Many small partners went to the above and tested it. They found that the same error was still reported when executing Python manage.py runserver 8080. The reason is that Django read the old library, and you can verify it yourself

(python36) [[email protected]ge servermonitor]# python 
Python 3.6.8 (default, Nov 16 2020, 16:55:22) [GCC 4.8.5 20150623 
(Red Hat 4.8.5-44)] on linux Type "help", "copyright", "credits" or "license" for more information.
 >>> import sqlite3
 > >>> sqlite3.sqlite_version 
 > '3.7.17'      
>>> .exit

Modifying library variables

export LD_LIBRARY_PATH="/usr/local/lib/"

Retest

(python36) [[email protected] servermonitor]# python Python 3.6.8
 (default, Nov 16 2020, 16:55:22) [GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux Type "help", "copyright", "credits" or "license" for more information.
  >>> import sqlite3
  > >>> sqlite3.sqlite_version
  '3.35.5'          
  >>> exit()

Execution returned to normal

(python36) [[email protected] servermonitor]# python manage.py runserver 8080
 Django version 3.2.4, using settings 'servermonitor.settings' Starting development server at
http://127.0.0.1:8080/ Quit the server with CONTROL-C.

Processing method of PIP exception no module named ‘pip’

1、 Pip is not installed
most of the current versions of python (2.7.9 + or 3.4 +) come with PIP. You can use the following command first:
PIP -- version (Python 2. + version)
PIP3 -- version (Python 3. + version)
try to check the version of PIP. If it can be displayed normally, PIP should have been installed.

If pip is not installed, the official download address of PIP is:
0 https://pypi.org/project/pip/

Installation:
Step 1: decompress
Step 2: open the console at the decompressed position
Step 3: use the command Python setup.py install
Step 4: check whether the PIP is installed again, if not, restart the computer

2、 The missing environment variable
may be that the option of add path was not checked when installing python, and the path of Python and pip.exe needs to be added to the environment variable manually
e.g:
C:\Users\Example\AppData\Local\Programs\Python\Python39-32;

and
C: users, example, appdata, local programs, python, python39-32, scripts Python comes with PIP, but when it is used to install packages, it will still report “no mouse named ‘pip'”<
use the command:
Python - M ensurepip

after the installation, PIP can be used normally.