UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0xd6 in position 3089: invalid continuation byte

The cause of
New Ubuntu16.04LTS, then ctags+cscope+taghighlight. Cscope-rb, and then open the source input :UpdateTypesFile and you encounter the following error.


  45 #define CREATE_TRACE_POINTS
  46 #include <trace/events/asoc.h>
  47 
  48 #define NAME_SIZE 32
  49 
  50 #ifdef CONFIG_DEBUG_FS
  51 struct dentry *snd_soc_debugfs_root;
  52 EXPORT_SYMBOL_GPL(snd_soc_debugfs_root);
  53 #endif
Error detected while processing function TagHighlight#Generation#UpdateAndRead[16]..TagHighlight#Generation#UpdateTypesFile[106]..TagHighlight#RunPython
Script#RunGenerator:
line   54:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/zz/.vim/plugin/TagHighlight/module/worker.py", line 52, in RunWithOptions
    GenerateTags(config)
  File "/home/zz/.vim/plugin/TagHighlight/module/ctags_interface.py", line 65, in GenerateTags
    tagLines = [line.strip() for line in tagFile]
  File "/home/zz/.vim/plugin/TagHighlight/module/ctags_interface.py", line 65, in <listcomp>
    tagLines = [line.strip() for line in tagFile]
  File "/usr/lib/python3.5/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd6 in position 3089: invalid continuation byte
Press ENTER or type command to continue

Analysis:
1. There was no problem when browsing the same code on the server of the company. The temporary exclusion is the problem of the code. 2. Taghighlight and python would have been taghighlight and python, amateurish, but not vim. 3. Notice the last line of the error:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd6 in position 3089: invalid continuation byte

Say it’s a character cannot be resolved as utf-8, although the position is given, but don’t know is which files, is/usr/lib/python3.5/codecs. Py this file?I opened it up and I looked at it and I said, well, I don’t know, I’m not good at Python, I don’t see anything wrong. Then take this string to search on the Internet, no search to useful, they encounter is the source file itself has problems, modify their source code is good, but I encounter here should not be the source file problem, the reason in the above analysis also said. However, in the process of searching, I felt that the python version was a problem, so I checked the explanatory document of TagHighlight, and it did introduce the configuration of changing the Python version:

 953     PythonVariantPriority                TagHL-PythonVariantPriority
 954         The python part of TagHighlight can be run in a number of ways,
 955         depending on your system configuration.  For most users, this should
 956         be handled completely automatically, but if you want to customise
 957         things, this option is the best way to start.  There are currently
 958         four supported python variants:
 959 
 960             if_pyth3: When using this variant, the python code will be run
 961                       using the Python 3.x interface built into Vim.  If using
 962                       a Vim that is compiled with +python/dyn and
 963                       +python3/dyn, be aware that this may cause problems if
 964                       you want to use the python 2.x interface.  See the
 965                       details on case 4 under :py3file.
 966 
 967             if_pyth:  When using this variant, the python code will be run
 968                       using the Python 2.x (at least version 2.6 required)
 969                       interface built into Vim.  See the comments above
 970                       regarding potential conflicts between the two python
 971                       versions.
 972 
 973             python:   When using this variant. the python code will be run
 974                       using the version of python that is in the path or
 975                       specified with the option TagHL-PathToPython.
 976 
 977             compiled: When using this variant, the compiled executable version
 978                       of the python part of the plugin will be used (so no
 979                       system version of python is required).  Note that this
 980                       requires the compiled version to be installed as it is
 981                       not part of the main distribution.  See the installation
 982                       guide for more information: TagHighlight-install.
 983 
 984         Option Type: List
 985         Default: ["if_pyth3", "if_pyth", "python", "compiled"]

When I checked this again and again, I found that Vim also specifies the Python version when it is compiled, so I wanted to see the version of Vim in question:

[email protected]:doc$ vim --version
VIM - Vi IMproved 7.4 (2013 Aug 10, compiled Nov 24 2016 16:44:48)
Included patches: 1-1689
Extra patches: 8.0.0056
Modified by [email protected]
Compiled by [email protected]
Huge version without GUI.  Features included (+) or not (-):
+acl             +farsi           +mouse_netterm   +tag_binary
+arabic          +file_in_path    +mouse_sgr       +tag_old_static
+autocmd         +find_in_path    -mouse_sysmouse  -tag_any_white
-balloon_eval    +float           +mouse_urxvt     -tcl
-browse          +folding         +mouse_xterm     +terminfo
++builtin_terms  -footer          +multi_byte      +termresponse
+byte_offset     +fork()          +multi_lang      +textobjects
+channel         +gettext         -mzscheme        +timers
+cindent         -hangul_input    +netbeans_intg   +title
-clientserver    +iconv           +packages        -toolbar
-clipboard       +insert_expand   +path_extra      +user_commands
+cmdline_compl   +job             -perl            +vertsplit
+cmdline_hist    +jumplist        +persistent_undo +virtualedit
+cmdline_info    +keymap          +postscript      +visual
+comments        +langmap         +printer         +visualextra
+conceal         +libcall         +profile         +viminfo
+cryptv          +linebreak       -python          +vreplace
+cscope          +lispindent      +python3         +wildignore
+cursorbind      +listcmds        +quickfix        +wildmenu
+cursorshape     +localmap        +reltime         +windows
+dialog_con      -lua             +rightleft       +writebackup
+diff            +menu            -ruby            -X11
+digraphs        +mksession       +scrollbind      -xfontset
-dnd             +modify_fname    +signs           -xim
-ebcdic          +mouse           +smartindent     -xsmp
+emacs_tags      -mouseshape      +startuptime     -xterm_clipboard
+eval            +mouse_dec       +statusline      -xterm_save
+ex_extra        +mouse_gpm       -sun_workshop    -xpm
+extra_search    -mouse_jsbterm   +syntax          
   system vimrc file: "$VIM/vimrc"
     user vimrc file: "$HOME/.vimrc"
 2nd user vimrc file: "~/.vim/vimrc"
      user exrc file: "$HOME/.exrc"
  fall-back for $VIM: "/usr/share/vim"
Compilation: gcc -c -I. -Iproto -DHAVE_CONFIG_H   -Wdate-time  -g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1      
Linking: gcc   -Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -o vim        -lm -ltinfo -lnsl  -lselinux  -lacl -lattr -lgpm -ldl     -L/usr/lib/python3.5/config-3.5m-x86_64-linux-gnu -lpython3.5m -lpthread -ldl -lutil -lm      
[email protected]:doc$ 

The obvious answer is that the version used is python3.5. A “+” in front of it means yes, and a “-” means no.
notice that python is preceded by “-” and python3 by “+”.
for comparison, check the vim version of the company’s server again:

[email protected]:~$ vim --version
VIM - Vi IMproved 7.4 (2013 Aug 10, compiled Nov 24 2016 16:42:52)
Included patches: 1-52
Extra patches: 8.0.0056
Modified by [email protected]
Compiled by [email protected]
Huge version with GTK2 GUI.  Features included (+) or not (-):
+acl             +farsi           +mouse_netterm   +syntax
+arabic          +file_in_path    +mouse_sgr       +tag_binary
+autocmd         +find_in_path    -mouse_sysmouse  +tag_old_static
+balloon_eval    +float           +mouse_urxvt     -tag_any_white
+browse          +folding         +mouse_xterm     +tcl
++builtin_terms  -footer          +multi_byte      +terminfo
+byte_offset     +fork()          +multi_lang      +termresponse
+cindent         +gettext         -mzscheme        +textobjects
+clientserver    -hangul_input    +netbeans_intg   +title
+clipboard       +iconv           +path_extra      +toolbar
+cmdline_compl   +insert_expand   +perl            +user_commands
+cmdline_hist    +jumplist        +persistent_undo +vertsplit
+cmdline_info    +keymap          +postscript      +virtualedit
+comments        +langmap         +printer         +visual
+conceal         +libcall         +profile         +visualextra
+cryptv          +linebreak       +python          +viminfo
+cscope          +lispindent      -python3         +vreplace
+cursorbind      +listcmds        +quickfix        +wildignore
+cursorshape     +localmap        +reltime         +wildmenu
+dialog_con_gui  +lua             +rightleft       +windows
+diff            +menu            +ruby            +writebackup
+digraphs        +mksession       +scrollbind      +X11
+dnd             +modify_fname    +signs           -xfontset
-ebcdic          +mouse           +smartindent     +xim
+emacs_tags      +mouseshape      -sniff           +xsmp_interact
+eval            +mouse_dec       +startuptime     +xterm_clipboard
+ex_extra        +mouse_gpm       +statusline      -xterm_save
+extra_search    -mouse_jsbterm   -sun_workshop    +xpm
   system vimrc file: "$VIM/vimrc"
     user vimrc file: "$HOME/.vimrc"
 2nd user vimrc file: "~/.vim/vimrc"
      user exrc file: "$HOME/.exrc"
  system gvimrc file: "$VIM/gvimrc"
    user gvimrc file: "$HOME/.gvimrc"
2nd user gvimrc file: "~/.vim/gvimrc"
    system menu file: "$VIMRUNTIME/menu.vim"
  fall-back for $VIM: "/usr/share/vim"
Compilation: gcc -c -I. -Iproto -DHAVE_CONFIG_H -DFEAT_GUI_GTK  -pthread -I/usr/include/gtk-2.0 -I/usr/lib/x86_64-linux-gnu/gtk-2.0/include -I/usr/include/atk-1.0 -I/usr/include/cairo -I/usr/include/gdk-pixbuf-2.0 -I/usr/include/pango-1.0 -I/usr/include/gio-unix-2.0/ -I/usr/include/freetype2 -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/usr/include/pixman-1 -I/usr/include/libpng12 -I/usr/include/harfbuzz     -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1     -I/usr/include/tcl8.6  -D_REENTRANT=1  -D_THREAD_SAFE=1  -D_LARGEFILE64_SOURCE=1  
Linking: gcc   -L. -Wl,-Bsymbolic-functions -Wl,-z,relro -L/build/buildd/ruby1.9.1-1.9.3.484/debian/lib -rdynamic -Wl,-export-dynamic -Wl,-E  -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,--as-needed -o vim   -lgtk-x11-2.0 -lgdk-x11-2.0 -latk-1.0 -lgio-2.0 -lpangoft2-1.0 -lpangocairo-1.0 -lgdk_pixbuf-2.0 -lcairo -lpango-1.0 -lfontconfig -lgobject-2.0 -lglib-2.0 -lfreetype   -lSM -lICE -lXpm -lXt -lX11 -lXdmcp -lSM -lICE  -lm -ltinfo -lnsl  -lselinux  -lacl -lattr -lgpm -ldl  -L/usr/lib -llua5.2 -Wl,-E  -fstack-protector -L/usr/local/lib  -L/usr/lib/perl/5.18/CORE -lperl -ldl -lm -lpthread -lcrypt -L/usr/lib/python2.7/config-x86_64-linux-gnu -lpython2.7 -lpthread -ldl -lutil -lm -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions  -L/usr/lib/x86_64-linux-gnu -ltcl8.6 -ldl -lz -lpthread -lieee -lm -lruby-1.9.1 -lpthread -lrt -ldl -lcrypt -lm  -L/usr/lib   

Looking at the server version again, the company server is Ubuntu 14.04.2lts (GNU/Linux 3.13.0-24-generic x86_64)
and my new one is 16.04, which is probably why 14.02 USES python2.7.
Python3.5 is supposed to be a bit more compatible than 2.7, but it turns out that python3.5 is more compatible than 2.7.
the next line of thought is clearer:
1. Either change my vim to use python2.7 as well; 2. Either install a python3.6 or later, and then figure out a way for vim to use this version;
obviously, method 1 is relatively easy to implement, and the way to add python2 to vim is as follows:
py2 package installed, under the command terminal input: sudo apt - get the install vim - nox - py2
please see the specific link https://blog.csdn.net/u012629369/article/details/53447555
I do the same thing, can use python2 results vim:

[email protected]:doc$ sudo apt-get install vim-nox-py2 
[sudo] password for zz: 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Suggested packages:
  vim-doc
The following NEW packages will be installed:
  vim-nox-py2
0 upgraded, 1 newly installed, 0 to remove and 23 not upgraded.
Need to get 0 B/1,100 kB of archives.
After this operation, 2,642 kB of additional disk space will be used.
Selecting previously unselected package vim-nox-py2.
(Reading database ... 220112 files and directories currently installed.)
Preparing to unpack .../vim-nox-py2_2%3a7.4.1689-3ubuntu1.2_amd64.deb ...
Unpacking vim-nox-py2 (2:7.4.1689-3ubuntu1.2) ...
Setting up vim-nox-py2 (2:7.4.1689-3ubuntu1.2) ...
update-alternatives: using /usr/bin/vim.nox-py2 to provide /usr/bin/vim (vim) in auto mode
update-alternatives: using /usr/bin/vim.nox-py2 to provide /usr/bin/vimdiff (vimdiff) in auto mode
update-alternatives: using /usr/bin/vim.nox-py2 to provide /usr/bin/rvim (rvim) in auto mode
update-alternatives: using /usr/bin/vim.nox-py2 to provide /usr/bin/rview (rview) in auto mode
update-alternatives: using /usr/bin/vim.nox-py2 to provide /usr/bin/vi (vi) in auto mode
update-alternatives: using /usr/bin/vim.nox-py2 to provide /usr/bin/view (view) in auto mode
update-alternatives: using /usr/bin/vim.nox-py2 to provide /usr/bin/ex (ex) in auto mode
[email protected]:doc$ 
[email protected]:doc$ 
[email protected]:doc$ vim --version
VIM - Vi IMproved 7.4 (2013 Aug 10, compiled Nov 24 2016 16:44:48)
Included patches: 1-1689
Extra patches: 8.0.0056
Modified by [email protected]
Compiled by [email protected]
Huge version without GUI.  Features included (+) or not (-):
+acl             +farsi           +mouse_netterm   +tag_binary
+arabic          +file_in_path    +mouse_sgr       +tag_old_static
+autocmd         +find_in_path    -mouse_sysmouse  -tag_any_white
-balloon_eval    +float           +mouse_urxvt     +tcl
-browse          +folding         +mouse_xterm     +terminfo
++builtin_terms  -footer          +multi_byte      +termresponse
+byte_offset     +fork()          +multi_lang      +textobjects
+channel         +gettext         -mzscheme        +timers
+cindent         -hangul_input    +netbeans_intg   +title
-clientserver    +iconv           +packages        -toolbar
-clipboard       +insert_expand   +path_extra      +user_commands
+cmdline_compl   +job             +perl            +vertsplit
+cmdline_hist    +jumplist        +persistent_undo +virtualedit
+cmdline_info    +keymap          +postscript      +visual
+comments        +langmap         +printer         +visualextra
+conceal         +libcall         +profile         +viminfo
+cryptv          +linebreak       +python          +vreplace
+cscope          +lispindent      -python3         +wildignore
+cursorbind      +listcmds        +quickfix        +wildmenu
+cursorshape     +localmap        +reltime         +windows
+dialog_con      +lua             +rightleft       +writebackup
+diff            +menu            +ruby            -X11
+digraphs        +mksession       +scrollbind      -xfontset
-dnd             +modify_fname    +signs           -xim
-ebcdic          +mouse           +smartindent     -xsmp
+emacs_tags      -mouseshape      +startuptime     -xterm_clipboard
+eval            +mouse_dec       +statusline      -xterm_save
+ex_extra        +mouse_gpm       -sun_workshop    -xpm
+extra_search    -mouse_jsbterm   +syntax          
   system vimrc file: "$VIM/vimrc"
     user vimrc file: "$HOME/.vimrc"
 2nd user vimrc file: "~/.vim/vimrc"
      user exrc file: "$HOME/.exrc"
  fall-back for $VIM: "/usr/share/vim"
Compilation: gcc -c -I. -Iproto -DHAVE_CONFIG_H   -Wdate-time  -g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1      
Linking: gcc   -L. -Wl,-Bsymbolic-functions -Wl,-z,relro -fstack-protector -rdynamic -Wl,-export-dynamic -Wl,-E  -Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -o vim        -lm -ltinfo -lnsl  -lselinux  -lacl -lattr -lgpm -ldl  -L/usr/lib -llua5.2 -Wl,-E  -fstack-protector-strong -L/usr/local/lib  -L/usr/lib/x86_64-linux-gnu/perl/5.22/CORE -lperl -ldl -lm -lpthread -lcrypt -L/usr/lib/python2.7/config-x86_64-linux-gnu -lpython2.7 -lpthread -ldl -lutil -lm -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions  -L/usr/lib/x86_64-linux-gnu -ltcl8.6 -ldl -lz -lpthread -lieee -lm -lruby-2.3 -lpthread -lgmp -ldl -lcrypt -lm     
[email protected]:doc$ 

The next step is to use :UpdateTypesFile.
The next step is to look at how to get vim and taghighlight to use python3.6 and see if there are any problems.


Read More: