Category Archives: Python

Pandas read_csv pandas.errors.ParserError: Error tokenizing data

What you will learn？
pandas read_csv escape commas and double qoutes
Prepare datas

# test.csv or test.txt
"1","123","4","\"data\""
"test","123","4","if(\"data\" = \"<test>\", (10*24))"

Wrong-way

import pandas as pd

datas = pd.read_csv('test.txt', header=None, skip_blank_lines=True)

You got

pandas\_libs\parsers.pyx in pandas._libs.parsers.TextReader.read()

pandas\_libs\parsers.pyx in pandas._libs.parsers.TextReader._read_low_memory()

pandas\_libs\parsers.pyx in pandas._libs.parsers.TextReader._read_rows()

pandas\_libs\parsers.pyx in pandas._libs.parsers.TextReader._tokenize_rows()

pandas\_libs\parsers.pyx in pandas._libs.parsers.raise_parser_error()

ParserError: Error tokenizing data. C error: Expected 4 fields in line 2, saw 5

Right way

import pandas as pd

datas = pd.read_csv('test.txt', header=None, skip_blank_lines=True, escapechar='\\')

Digression

Many people on the Internet encounter this problem and add a parameter: error_bad_Lines = false (tested, the second row will be lost for the above data). If the amount of data is not large, check the method of the specified row: cat – N filename | head – N end_line_no| tail -n +start_line_no

[Solved] HTTPError: 404 Client Error: Not Found for url: https://huggingface.co/saved_model

Cause:

Originally, there was a saved_model folder under the two folders sort_change_nlp and sort_nlp, but the names were different, one was saved_model and the other was saved_model_copy. Using saved_model in http_sever clock, the following error occurred:

model = YesOrNoModel.from_pretrained(model_name)


def is_model_answer(query):
    for rule in base_data.q_v_model_list:
        result1 = re.compile(rule).findall(query)
        if len(result1):
            return "saved_model"
    return ""
  model_name = is_model_answer(query)
    print(f"model_name={model_name}，answer={answer}")
    if len(model_name):
        model = YesOrNoModel.from_pretrained(model_name)

Error:

ssh://[email protected]:22/usr/bin/python -u /opt/program_files/python/local_map_python_work/judge/sort_proj/test.py
404 Client Error: Not Found for url: https://huggingface.co/saved_model/resolve/main/config.json
Traceback (most recent call last):
  File "/root/.local/lib/python3.6/site-packages/transformers/configuration_utils.py", line 520, in get_config_dict
    user_agent=user_agent,
  File "/root/.local/lib/python3.6/site-packages/transformers/file_utils.py", line 1371, in cached_path
    local_files_only=local_files_only,
  File "/root/.local/lib/python3.6/site-packages/transformers/file_utils.py", line 1534, in get_from_cache
    r.raise_for_status()
  File "/usr/local/python3/lib/python3.6/site-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://huggingface.co/saved_model/resolve/main/config.json
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/program_files/python/local_map_python_work/judge/sort_proj/test.py", line 22, in <module>
    model = YesOrNoModel.from_pretrained("saved_model")
  File "/root/.local/lib/python3.6/site-packages/transformers/modeling_utils.py", line 1196, in from_pretrained
    **kwargs,
  File "/root/.local/lib/python3.6/site-packages/transformers/configuration_utils.py", line 455, in from_pretrained
    config_dict, kwargs = cls.get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/root/.local/lib/python3.6/site-packages/transformers/configuration_utils.py", line 532, in get_config_dict
    raise EnvironmentError(msg)
OSError: Can't load config for 'saved_model'. Make sure that:
- 'saved_model' is a correct model identifier listed on 'https://huggingface.co/models'
- or 'saved_model' is the correct path to a directory containing a config.json file

 

Solution:
The reason is that http.server is a level folder with sort_change_nlp and sort_nlp, so the path passed in when calling the model should be: sort_change_nlp/saved_model or sort_nlp, saved_model_copy

Python Error: mongod: error while loading shared libraries: libcrypto.so.1.1

Reading guide

After installing mongodb in Python, the following error is reported when executing mongod

mongod: error while loading shared libraries: libcrypto.so.1.1: cannot open shared object file: No such file or directory

Solution:

Find the virtual environment used. For example, the name of my virtual environment is torch , and use the following command to check whether the libcrypto. So. 1.1 file exists in the environment

#Switch to the lib directory
cd /root/anaconda3/envs/torch/lib
# Check if the file exists
ls -lh | grep "libcrypto.so.1.1"
-rwxrwxrwx 1 root root 3.2M Jun 4 15:43 libcrypto.so.1.1
# If the file exists execute the following command
sudo ldconfig /root/anaconda3/envs/torch/lib

If there is no in the Lib directory, use the following command for global search

find/-name 'libcrypto.so.1.1'

Observe the search results and find the existing directory by using the above sudo ldconfig directory path

[Solved] C error: expected 1 fields in line 3, saw 2 processing method

1. There is a CSV file that I read directly with the pandas library, and an error is reported: error tokenizing data. C error: expected 1 fields in line 3, saw 2. There should be a problem with the format. It can only be opened after it is opened and saved again. However, because there are many files, I wonder if I can read the CSV file by other methods;

2. Read through the CSV library, traverse, and merge:

import pandas as pd
import csv


path = 'the location of file'

test = pd.DataFrame()
data = csv.reader(open(path, 'r'))
for d in data:
    # print(d)
    result = pd.DataFrame(d).T
    test = pd.concat([test,result])
test = test.reset_index(drop = True)

3. Finally, you will get the dataframe of test, which is the same as the file read by pandas after saving

Flash cli encountered a keyerror “migrate” [How to Solve]

Flash management command-line tool. I use flash cli

Then, using any command related to flash DB will cause the problem of keyerror “migrate”

No solution was found on the Internet

Therefore, I started to think step by step and finally determined that the biggest problem without the migrate command was that when I executed the flash DB XX command, I didn’t find the corresponding app. The command line didn’t know which app I wanted to run, so I didn’t find the app associated with DB, that is, I didn’t find it

Migrate = migrate (APP = app, DB = dB), so an error is reported

Therefore, to use the flash command, you first need to declare flash_ APP

That is, enter in terminal

export FLASK_APP=The name of the app you want to run

=Do not leave a space before or after, otherwise bad assignment will be reported

Then enter flash DB migrate again and it will be normal

[Solved] Python matplotlib Error: RuntimeError: In set_size: Could not set the fontsize…

Problem
Error when saving image:RuntimeError: In set_size: Could not set the fontsize

Traceback (most recent call last):
  File "/Users/robin/MLcode/Pycharm_Project/tensorflow/2021/0823_face_recognition_environment/0827_img_quality_analysis_v4.py", line 1556, in <module>
    image_cluster_analysis()
  File "/Users/robin/MLcode/Pycharm_Project/tensorflow/2021/0823_face_recognition_environment/0827_img_quality_analysis_v4.py", line 1549, in image_cluster_analysis
    image_showing(img_compressed)
  File "/Users/robin/MLcode/Pycharm_Project/tensorflow/2021/0823_face_recognition_environment/0827_img_quality_analysis_v4.py", line 1408, in image_showing
    plt.savefig(img_name)
  File "/Users/robin/software/anaconda3/envs/tensorflow/lib/python3.6/site-packages/matplotlib/pyplot.py", line 722, in savefig
    res = fig.savefig(*args, **kwargs)
  File "/Users/robin/software/anaconda3/envs/tensorflow/lib/python3.6/site-packages/matplotlib/figure.py", line 2180, in savefig
    self.canvas.print_figure(fname, **kwargs)
  File "/Users/robin/software/anaconda3/envs/tensorflow/lib/python3.6/site-packages/matplotlib/backend_bases.py", line 2082, in print_figure
    **kwargs)
  File "/Users/robin/software/anaconda3/envs/tensorflow/lib/python3.6/site-packages/matplotlib/backends/backend_agg.py", line 579, in print_jpg
    buf, size = self.print_to_buffer()
  File "/Users/robin/software/anaconda3/envs/tensorflow/lib/python3.6/site-packages/matplotlib/backends/backend_agg.py", line 535, in print_to_buffer
    FigureCanvasAgg.draw(self)
  File "/Users/robin/software/anaconda3/envs/tensorflow/lib/python3.6/site-packages/matplotlib/backends/backend_agg.py", line 388, in draw
    self.figure.draw(self.renderer)
  File "/Users/robin/software/anaconda3/envs/tensorflow/lib/python3.6/site-packages/matplotlib/artist.py", line 38, in draw_wrapper
    return draw(artist, renderer, *args, **kwargs)
  File "/Users/robin/software/anaconda3/envs/tensorflow/lib/python3.6/site-packages/matplotlib/figure.py", line 1709, in draw
    renderer, self, artists, self.suppressComposite)
  File "/Users/robin/software/anaconda3/envs/tensorflow/lib/python3.6/site-packages/matplotlib/image.py", line 135, in _draw_list_compositing_images
    a.draw(renderer)
  File "/Users/robin/software/anaconda3/envs/tensorflow/lib/python3.6/site-packages/matplotlib/artist.py", line 38, in draw_wrapper
    return draw(artist, renderer, *args, **kwargs)
  File "/Users/robin/software/anaconda3/envs/tensorflow/lib/python3.6/site-packages/matplotlib/axes/_base.py", line 2607, in draw
    self._update_title_position(renderer)
  File "/Users/robin/software/anaconda3/envs/tensorflow/lib/python3.6/site-packages/matplotlib/axes/_base.py", line 2556, in _update_title_position
    if title.get_window_extent(renderer).ymin < top:
  File "/Users/robin/software/anaconda3/envs/tensorflow/lib/python3.6/site-packages/matplotlib/text.py", line 890, in get_window_extent
    bbox, info, descent = self._get_layout(self._renderer)
  File "/Users/robin/software/anaconda3/envs/tensorflow/lib/python3.6/site-packages/matplotlib/text.py", line 291, in _get_layout
    ismath="TeX" if self.get_usetex() else False)
  File "/Users/robin/software/anaconda3/envs/tensorflow/lib/python3.6/site-packages/matplotlib/backends/backend_agg.py", line 210, in get_text_width_height_descent
    font = self._get_agg_font(prop)
  File "/Users/robin/software/anaconda3/envs/tensorflow/lib/python3.6/site-packages/matplotlib/backends/backend_agg.py", line 250, in _get_agg_font
    font.set_size(size, self.dpi)
RuntimeError: In set_size: Could not set the fontsize

Solution:

The source of the problem is figure_Size and DPI :

The following two codes are different:

figure_size = (6.40, 4.80)
plt.figure(figsize=figure_size, dpi=100)

And:

figure_size = (640, 480)
plt.figure(figsize=figure_size, dpi=1)

Note:

figsize : width, height in inches, default: (6.4, 4.8) , that is, the picture size in inches. The default value in Matplotlib is (6.4, 40.8) DPI : dots (or pixels) per inch, default: 100.0 , that is, the number of pixels per inch. The default value is 100

Although the above two representations seem to be the same, an error is reported by using DPI = 1. (I won’t delve into it for the time being)

[Solved] R Error: Python module tensorflow.keras was not found.

Scenario:

rstudio reported error R: Python module tensorflow.keras was not found.
at first, I suspected that I could not accurately locate the keras package of R, because I was using Anaconda to do other things, but later I saw other error reports and felt that they were not so complex… Later, I found that they were just simple and not installed properly… In short, the solution is as follows

Solution:

terminal input:

install.packages(tensorflow)
install_tensorflow()
library(tensorflow)
#Keras The R interface uses the TensorFlow backend engine.
# To install the core Keras library and TensorFlow backend, use the install_keras() function
install_keras()
library(keras)
# Run them all and then tune the package, if you have already done so, restart RStudio
-------------------------------------------
#Other solutions to this problem are as follows:
library(tensorflow)
library(keras)
use_condaenv("r-tensorflow")

Pip3 install Mysqlclient Error: Command “python setup.py egg_info“ failed with error

pip3 install mysqlclient error Command “python setup.py egg_info” failed with error
1. Software description

    centos 8.0python3.6

2. Problem description
Use the command

pip3 install mysqlclient
error
[root@z ~]# pip3 install mysqlclient
WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.
Collecting mysqlclient
  Using cached https://files.pythonhosted.org/packages/3c/df/59cd2fa5e48d0804d213bdcb1acb4d08c403b61c7ff7ed4dd4a6a2deb3f7/mysqlclient-2.0.3.tar.gz
    Complete output from command python setup.py egg_info:
    /bin/sh: mysql_config: Command not found
    /bin/sh: mariadb_config: Command not found
    /bin/sh: mysql_config: Command not found
    mysql_config --version
    mariadb_config --version
    mysql_config --libs
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-8k22usrq/mysqlclient/setup.py", line 15, in <module>
        metadata, options = get_config()
      File "/tmp/pip-build-8k22usrq/mysqlclient/setup_posix.py", line 70, in get_config
        libs = mysql_config("libs")
      File "/tmp/pip-build-8k22usrq/mysqlclient/setup_posix.py", line 31, in mysql_config
        raise OSError("{} not found".format(_mysql_config_path))
    OSError: mysql_config not found
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-8k22usrq/mysqlclient/

3. Solutions

yum install python3-devel mysql-devel

[Perfectly Solved] attributeerror: module ‘SciPy. Misc’ has no attribute ‘toimage’ error

Problem Description:

When learning the MNIST machine learning introductory course, the attributeerror: module ‘SciPy. Misc’ has no attribute ‘toimage’ error occurs when converting the numpy array into a picture and saving it
the execution code is as follows:

# Save the first 20 images
for i in range(20):
    image_array = train_images[i]
    #Save file as the format:mnist_train_0.jpg,mnist_train_1.jpg,...
    filename = save_dir + '/mnist_train_%d.jpg' %i
    #save image_array to image
    # use scipy.misc.toimage to convert to image and save
    scipy.misc.toimage(image_array, cmin=0.0, cmax=1.0).save(filename)

Crux attribution:
I checked the scene where Python 3 uses the toimage() function on the Internet and found that this function has been deprecated. Many tutorials recommend reducing the version of the third-party SciPy to match the use of the toimage() function. I feel that this is a bit “married down” and can not fundamentally solve the problem. Moreover, the iteration of technology update is inevitable, We need to follow the trend.

Scipy.misc.toimage official latest instructions.

Solution:

Turn the problem into how to save the array as an image?

Method 1: use the .Imwrite() function of CV2 module to convert the numpy array into an image and save it. The specific codes are as follows:

cv2.imwrite(filename, image_array)  #Mutual transformation of images and numpy arrays using cv2

Method 2: use the .Fromarray() function under Image in PIL module to convert the numpy array into an image and save it. The specific codes are as follows:

from PIL import Image
Image.fromarray((image_array)).save(filename)  #Mutual transformation of images and numpy arrays using PIL
# or
Image.fromarray((image_array*255).astype('uint8'), mode='L').convert('RGB').save(filename)  # For processing color images

Method 3: use the Matplotlib module to convert the numpy array into a picture and save it. (including two methods): (1) preferred recommendation:   Use the . Imsave() function under Image in the Matplotlib module to convert the numpy array into an image for saving. The specific codes are as follows:

from matplotlib import image
image.imsave(filename,image_array,cmap='gray')  # cmap is often used to change the drawing style, such as black and white gray, emerald green virdidis

(2) Not recommended:   Use the .Savefig() function under pyplot in Matplotlib module to convert the numpy array into a picture for saving. The generated picture contains coordinate axis and border information. The specific codes are as follows:

import matplotlib.pyplot as plt
# Drawing pictures
plt.imshow(image_array,cmap='gray')
# save image
plt.savefig(filename) # In this case, the picture variable is already specified when drawing the picture, so there is no need to specify it again when saving

Pytorch CUDA Error: UserWarning: CUDA initialization: CUDA unknown error…

After CUDA is installed, the following error is reported using pytorch

UserWarning: CUDA initialization: CUDA unknown error – this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_ VISIBLE_ DEVICES after program start.

Solution: after CUDA and pytorch are installed, add the following in. Bashrc

export  PATH=/usr/local/cuda-11.4/bin:$PATH
export  LD_LIBRARY_PATH=/usr/local/cuda-11.4/lib64$LD_LIBRARY_PATH
export CUDA_HOME=/usr/local/cuda-11.4/bin
export CUDA_VISIBLE_DEVICES=0,1

If there is still a problem, use sudo apt-get install NVIDIA modprobe to install it. After the installation, you can use it

Methods of checking CUDA

import torch
flag = torch.cuda.is_available()
print(flag)

Output is: True cuda normal

[Solved] error: when using the property decorator in Python, an error occurs: typeerror: descriptor ‘setter’ requires a ‘property’ object but

Error message

• When we use the property decorator, the following errors may be caused by the wrong writing of the decorator Name:

TypeError: descriptor 'setter' requires a 'property' object but received a 'function'

Problem analysis

• The reason for this error is that we write all decorator names as property instead of the same method name in our class:

Problem code

class AgeDemo(object):

    def __init__(self, age):
        self.age = age

    @property
    def age_test(self):
        return self.age

    @property.setter   # error
    def age_test(self, age):
        if not isinstance(age, int):
            raise TypeError('TypeError')
        self.age = age

Resolve error reporting

• Change the name of the decorator with an error in the figure to the same name set in our class to solve this error
the code is as follows:

class AgeDemo(object):

    def __init__(self, age):
        self.age = age

    @property
    def age_test(self):
        return self.age

    @age_test.setter # 修改的代码行
    def age_test(self, age):
        if not isinstance(age, int):
            raise TypeError('TypeError')
        self.age = age