Tag Archives: Programming

076_ OpenGL graphics program development practice

OpenGL Graphic Program Development Practice
Delphi tutorial series books (076) “OpenGL graphics program development practice” (state) finishing Email: [email protected]
Download Address:
Pdf

Xie Danrui Publishing House: Northwestern Polytechnical University Press ISBN: 7561219393 Available Date: 2006-9-6 Published Date: August 2005 Open Book: 16 Page Number: 293 Edition: 1-1
Content abstract
As a kind of graphics industry standard, OpenGL is widely used in practical work and occupies a very important position. This book USES the Delphi programming environment about OpenGL function principle, mainly introduced the development of computer graphics and development present situation, the related theory of OpenGL and OpenGL characteristics and working principle, the basic knowledge of OpenGL and advanced application techniques, the last instance system and how to using the OpenGL technology, integrated complex three-dimensional realistic graphics. This book is well-structured, informative, innovative and practical. It can be used as a reference for undergraduate students, graduate students, scientific researchers and graphic and image developers.
preface
Visualization of scientific computation, computer animation and virtual reality are three hot research directions in the field of computer graphics in recent years. Their technical core is the rendering of three-dimensional realistic graphics. At present, 3D realistic graphics technology has been successfully and widely used in military, aerospace, aviation, medicine, geological exploration, 3D games and industrial CAD design and other fields, so that people can directly operate the physical information in the 3D graphics world, and directly communicate with the computer. This technology has unified the power of man and machine in an intuitive and natural way, and this revolutionary change will undoubtedly greatly improve the quality and efficiency of people’s work. Therefore, it is of great theoretical and practical significance to actively promote the research of computer graphics. OpenGL is the abbreviation of Open Graphics Library, which is a set of high performance Graphics processing system developed by SGI company. It is the software interface of Graphics hardware, meaning the Open Graphics Library. With OpenGL, programmers can create interactive applications to achieve realistic three-dimensional graphics and images, which can be used in many fields that require a high degree of simulation of the real world. Due to its openness and high reusability, it has now become an industry standard. OpenGL is designed to be independent of hardware, independent from the window system, running a variety of operating systems can be used on a variety of computers, and can work in the network environment in the client/server mode, is a professional graphics processing, scientific computing and other high-end application areas of the standard graphics library. OpenGL as a graphics industry standard, almost all the visual application development tools support OpenGL integration, such as Delphi, VC++, VB, JAVA and so on. However, the current introduction of OpenGL principle of the reference resources are almost based on C or VC++, which is familiar with the Delphi programming environment for the majority of developers, is undoubtedly a pity. As we all know, Delphi with its simple and easy to use visual development environment, high efficiency compiler, powerful PASCAL language in the field of visual development tools occupies a very important position. Based on the above factors, this book uses the Delphi programming environment to tell the principle of OpenGL function. The book consists of 16 chapters. Chapter 1 briefly introduces the development of computer graphics and the development of graphics and images. Chapter 2 and Chapter 3 introduce the Delphi programming foundation and object-oriented theory; Chapter 4 introduces the basic situation of OpenGL graphics library, OpenGL characteristics and working principle; Chapter 5 introduces the foundation of entity modeling and describes the basic drawing commands of OpenGL, including the description of point entity, line entity and surface entity. Chapter 6 introduces the theory of view transformation in OpenGL, including model transformation, view transformation, projection transformation, view-area transformation and matrix stack operation. Chapters 7 and 8 cover colors and lighting in OpenGL; Chapters 9 to 13 introduce the advanced application techniques of OpenGL, including blending, anti-aliasing, fog, display lists, bitmaps, fonts, texture mapping, frame caching, and animation. Chapter 14 introduces the evaluation program and the principle of nonuniform rational B-splines. Chapter 15 explains how interactive OpenGL works, including selection mechanisms and feedback mechanisms. As a conclusion, Chapter 16 systematically describes how to comprehensively use OpenGL technology to draw complex 3D realistic graphics through several examples. This book is well-structured and informative. Much of the source code provided in the book is compiled under the Delphi 7.0 environment. This book can not only be used as a college undergraduate, graduate study of computer graphics course, but also for research institutes of researchers, computer graphics and image developers have a high reference value. Xue Huifeng, Wu Huixin and Xie Danrui participated in the compilation of the book. In addition, Xia Xiaoye, Gao Xiong, Xu Rongyue and Bai Linfei also participated in the compilation. Due to the limited level, there are omissions and mistakes in the book, please readers friends criticism and correction. Author April 2005
directory
The first chapter graphics and image development status
1.1 The booming development of computer graphics
1.2 OpenGL leads the trend of graphic image development
1.3 Selection of development tools
The second chapter is Delphi language foundation
2.1 Delphi Integrated Development Environment
2.2 Basic concepts of Delphi language
2.3 Data types
2.4 Variables and Constants
2.5 Operators and expressions
2.6 Statements
2.7 Process and Function
Chapter 3 Object-Oriented Programming
3.1 Classes and Objects
3.2 Class declarations
Members of the 3.3 class
3.4 Three features of OOP
3.5 Class Operators
3.6 Objects
3.7 Exception Handling
Chapter 4 is an overview of OpenGL
4.1 History of OpenGL
4.2 Basic features of OpenGL
4.3 Architecture of OpenGL
4.4 OpenGL workflow
4.5 OpenGL function library
4.6 Basic graphics functions of OpenGL
4.7 Future and prospect of OpenGL
The fifth chapter is the foundation of entity modeling
5.1 Preparation before drawing
5.2 OpenGL data types and function forms
5.3 Description of basic geometric primitives
5.4 Using glBegin() and glEnd()
5.5 Normal vector
5.6 Draw basic primitive instances
Chapter 6 View transformation
6.1 Mathematical basis of graph transformation
6.2 From 3D Space to 2D Plane
6.3 View transformation and model transformation
6.4 Projection transformation
6.5 Visual area transformation
6.6 Additional cut plane
6.7 View transform instance
Chapter 7 Colors in OpenGL
7.1 Color perception
7.2 Colors in the computer
7.3 RGBA mode and color index mode
7.4 Specify the shadow model
7.5 Examples of color usage
Chapter 8 Light
8.1 Real World and OpenGL Lighting
8.2 Illumination Sphere Case Analysis
8.3 Create Light Source..
8.4 Select the lighting model
8.5 Define material properties
8.6 Illumination math calculation
8.7 Lighting in color index mode
Chapter 9 Mixing, anti-aliasing, fog
9.1 Mixing
9.2 Anti-aliasing
9.3 the fog
Chapter 10 shows the list
10.1 The basic concept of displaying lists
10.2 Show the creation and execution of lists
10.3 Show nested use of lists
10.4 Management of display lists
10.5 Multiple operations to display lists
10.6 Use display lists to change encapsulation mode
Chapter 11 Bitmaps, Characters, and Images
11.1 Bitmap
11.2 Characters
11.3 Images
Chapter 12 Texture Mapping
12.1 Basic steps for texture mapping
12.2 Specify the texture
12.3 Texture Control
12.4 Texture coordinates
Chapter 13 Frame Caching and Animation
13.1 frame cache
13.2 pixel segment testing and operation
13.3 Accumulate the cache
13.4 Dual cache animation
Chapter 14 Evaluation Procedures and Nonuniform Rational B-Splines
14.1 Evaluation procedure
14.2 NURBS curves and surfaces
Chapter 15 Selection and Feedback
15.1 Selection
15.2 Feedback
Chapter 16 OpenCL advanced application skills
16.1 Flames dancing
16.2 Three-dimensional trees
16.3 Texture Synthesis Application
References…

OpenGL basic graphics programming – OpenGL and 3D graphics world

We live in a three-dimensional world full of three-dimensional objects. In order for a computer to reproduce these objects accurately, we must be able to depict these objects in three-dimensional space. We live in a world full of information, and the ability to understand and use this information as quickly as possible will have a direct impact on the success or failure of our career, so we need to express this information in the most direct form.

In recent years, the development of computer graphics has enabled the formation of 3D presentation technology. These 3D presentation technologies enable us to reproduce objects in the 3D world and express complex information with 3D shapes. This technology is visualization (
Visualization) technology. Visualization technology enables people to operate information with shapes directly in the world of three-dimensional graphics and communicate with computers directly. This kind of technology has unified the power of man and machine in an intuitive and natural way, and this revolutionary change will undoubtedly greatly improve people’s work efficiency. Visualization technology gives people a simulation, three-dimensional and real-time interaction ability, so that people can use previously unimaginable means in the three-dimensional graphic world to obtain information or play their creative thinking. The mechanical engineer can be liberated from the two-dimensional plan directly into the three-dimensional world, so that he can quickly get the three-dimensional model of the mechanical parts he has designed. A doctor can analyze a patient’s lesions from a three-dimensional scan. Military commanders can direct realistic 3D planes, warships and tanks to advance to targets and analyze the effect of combat plans in the face of battlefield terrain generated by 3D graphics technology.

Even more amazing is the development of virtual reality technology, which allows people to enter a three-dimensional, multimedia virtual world, people can visit ancient castles, can roam vast space. All these depend on the development of computer graphics and computer visualization technology. People’s research on computer visualization technology has gone through a long process, and many visualization tools have been formed, among which the GL 3D graphics library launched by SGI has outstanding performance, easy to use and powerful functions. The 3D application software developed by GL is quite popular with many professional and technical personnel, and these 3D application software have been involved in the fields of architecture, product design, medicine, earth science, fluid mechanics and so on. With the continue development of computer technology, GL has been further developed into OpenGL, OpenGL has been considered a high-performance graphics and interactive visual processing standards, currently including ATT company UNIX software lab, DEC, SUN, IBM, HP, Microsoft, SGI company and several in the computer market of leading large companies have adopted the standard OpenGL graphics.

it is worth mentioning that because Microsoft company provide OpenGL graphics standard in Windows NT, OpenGL will be widely used in computer, especially the OpenGL 3 d graphics accelerator card and computer graphics workstations, people can be implemented on microcomputer 3 d graphics applications, such as CAD design, simulation, 3 d games and so on, so as to have a better chance, make it easier to use OpenGL and application software to build their own 3 d graphics in the world.

1.2. OpenGL provides an intuitive 3D graphics development environment

OpenGL is actually a graphics and hardware interface. It includes 120 graphical functions that developers can use to create 3D models and 3D real-time interactions. Unlike other graphical programming interfaces, OpenGL provides very clear graphics functions, so novice programmers can take advantage of OpenGL’s graphics processing power and a color palette of 16.7 million colors to quickly design three-dimensional graphics and three-dimensional interactive software.

OpenGL’s powerful graphics functions do not require developers to write 3D object model data in a fixed data format, so developers can not only use their own data directly, but also can use other data sources in different formats. This flexibility greatly saves developers time and improves software development efficiency.

for a long time, the technical personnel engaged in the development of 3 d graphics have to write in their own applications matrix transformation, the external device access and other functions, such as the modulation is not very close relations with their own software development target function of bother, and OpenGL is to provide an intuitive programming environment, it provides a series of function is greatly simplify the 3 d graphics program. Such as:

> OpenGL provides a series of 3D graphics units for developers to call.
OpenGL provides a series of graph transformation functions. OpenGL provides a series of external device access functions, so that developers can easily access the mouse, keyboard, space ball, data gloves, such as this intuitive 3D graphics development environment reflects the OpenGL technical advantages, which is also the reason why many 3D graphics developers are keen on OpenGL.

In the early days of the development of computer, people began to engage in the development of computer graphics. Until the computer hardware and software and computer graphics highly developed in the 1990s, people found that complex data in the form of visual representation is the most easy to understand, so the rapid development of three-dimensional graphics, so a variety of three-dimensional graphics tool software package has been launched, such as Phigs, Pex, RenderMan. Some of these 3D graphics tool packages focus on ease of use, others on rendering effects or connection to application software, but none of them can match OpenGL in interactive 3D graphics modeling capabilities, external device management, and programming ease.
OpenGL after further development of GL, achieve 2D and 3D advanced graphics technology, in performance is extremely superior, it includes modeling, transformation, light processing, color processing, animation and more advanced capabilities, such as texture mapping, object motion blur, etc.. These capabilities of OpenGL provide excellent software tools for achieving realistic 3D renderings and building interactive 3D landscapes. OpenGL is independent of each other in terms of hardware, window and operating system.
Many computer companies have integrated OpenGL into a variety of Windows and operating systems, which operating systems include UNIX, Windows NT, DOS, etc. Window systems have X Windows, Windows, etc. In order to realize a fully functional graphics processing system, a system structure related to OpenGL is designed as follows: the bottom layer is the graphics hardware, the second layer is the operating system, the third layer is the window system, the fourth layer is OpenGL, the fifth layer is the application software. OpenGL is network transparent, allowing both local and remote drawing in a client-server architecture. So in the network system, OpenGL in the X window, Windows or other window system can appear as a separate graphics window.
OpenGL as a superior performance graphics application design interface (API) and suitable for a wide range of computing environments, from personal computers to workstations and supercomputers, OpenGL can achieve high performance of three-dimensional graphics functions. As many leading computer companies in the computer industry have adopted OpenGL as a 3D graphical application design interface, OpenGL applications have a wide range of portability. Therefore, OpenGL has become the current 3D graphics development standards, is engaged in 3D graphics development of technical personnel must master the development tools.

How to disable Network Manager on Linux

origin: http://xmodulo.com/disable-network-manager-linux.html

Network Manager is a feature-rich network configuration service which is used by default in most Linux desktop environments nowadays. It provides automatic configuration of (wired/wireless) network interfaces, as well as VPN, mobile broadband and even Bluetooth connections. Network Manager is smart enough to automatically switch to the best (or the most recent) connection network, and can inform other applications of up-to-date network events via D-Bus API.
While Network Manager comes with various sophisticated features, you might just prefer the old plain network service. There could be several reasons to do so. For one, you may wish to have more control over network management, instead of relying on purely automatic configuration and switch-over, which may or may not suit your needs. Besides, Network Manager is not suitable for advanced networking setup, such as multi-homing to more than one wired connections, forwarding network traffic, configuring Linux bridge or aliases, etc.
In general, Network Manager is designed to keep a typical end-user Linux box connected at all times with the best possible connection with minimum intervention by users. If such automation is not needed in your Linux environment, you can turn off Network Manager.
This tutorial describes how to disable Network Manager in various Linux desktop environments. Be careful if you are trying to disable Network Manager on a remote host, as you will lose connectivity if you fail to set up its networking correctly in some other means.
Check Which Network Interfaces are Managed by Network Manager
To check if Network Manager is managing any network interface, you can use nmcli, which is a command line utility that comes with Network Manager.

$ nmcli dev status

DEVICE     TYPE              STATE
eth1      802-3-ethernet   connected
eth0      802-3-ethernet   connected

The above command will list all existing network interfaces along with their STATE. If STATE is shown as “unmanaged”, this means Network Manager is NOT controlling a corresponding interface. If STATE displays any other values (e.g., “connected”), it implies that a given interface is managed by Network Manager.
Disable Network Manager Completely
Here is how to disable Network Manager completely, so that Network Manager stops running on your Linux system.
To disable Network Manager on Debian 8 or later:

$ sudo systemctl stop NetworkManager.service

$ sudo systemctl disable NetworkManager.service

To disable Network Manager on Debian 7 or earlier:

$ sudo /etc/init. d/network-manager stop

$ sudo update-rc. d network-manager remove

To disable Network Manager on Ubuntu or Linux Mint:

$ sudo stop network-manager

$ echo “manual” | sudo tee /etc/init/network-manager.override

After disabling Network Manager on Debian or Ubuntu, use /etc/network/interfaces to configure network interfaces.
To disable Network Manager on Fedora or CentOS/RHEL 7 or later:

$ sudo systemctl stop NetworkManager.service

$ sudo systemctl disable NetworkManager.service



To disable Network Manager on CentOS/RHEL 6 or earlier:

$ sudo service NetworkManager stop

$ sudo chkconfig NetworkManager off

After disabling Network Manager on Fedora or CentOS, use /etc/sysconfig/network-scripts/ifcfg-ethX files to configure network interfaces.

Disable Network Manager for a Particular Network Interface on Debian, Ubuntu or Linux Mint
To disable Network Manager only for eth1 on Debian, Ubuntu or Linux Mint, you can do the following.
First, open the Network Manager configuration file in /etc/NetworkManager with a text editor, and set “managed=false”, typically shown under [ifupdown].

$ sudo vi /etc/NetworkManager/NetworkManager.conf

[ifupdown]
managed=false

Then in /etc/network/interfaces, add information about the interface you want to disable Network Manager for. In this example, the interface is eth1, and we are using static IP configuration.

$ sudo vi /etc/network/interfaces

# The loopback network interface
auto lo
iface lo inet loopback

# network interface not managed by Network Manager
allow-hotplug eth1
iface eth1 inet static
address 10.0.0.10
netmask 255.255.255.0
gateway 10.0.0.1
dns-nameservers 8.8.8.8

Then Network Manager automatically ignore any interfaces specified in /etc/network/interfaces, and stop managing them.
After rebooting, verify that Network Manager is successfully disabled for eth1.

$ nmcli dev status


Disable Network Manager for a Particular Network Interface on Fedora or CentOS/RHEL
The following steps will disable Network Manager for eth1 on Fedora, CentOS or RHEL.
Create an interface configuration file for eth1 (if not exist), and add “NM_CONTROLLED=no”.

TYPE="Ethernet"
NAME="eth1"
BOOTPROTO="static"
IPADDR=10.0.0.10
NETMASK=255.255.255.0
GATEWAY=10.0.0.1
HWADDR="00:90:29:95:EA:57"
ONBOOT="yes"
DEFROUTE="yes"
NM_CONTROLLED=no

Now enable network service so that eth1 is activated by network service automatically upon boot.
On Fedora or CentOS/RHEL 7 or later:

$ sudo systemctl enable network.service

On CentOS/RHEL 6 or earlier:

$ sudo chkconfig network on

Upon rebooting, verify that Network Manager is successfully disabled for eth1 with nmcli command.

$ nmcli dev status

Error: current transaction is aborted, commands ignored until end of transaction blockp

In normal database programming, we encountered errors from time to time: Current Transaction is aborted, commands ignored until the end of transaction BlockP. How to solve it?We usually look it up on the Internet. If you’re a little bit better at English you can tell from the error message that there’s probably something wrong there. But if the experience is not very rich, or the Internet to find more convenient. The most basic error message can see from the official documentation (http://www.postgresql.org/docs/8.4/interactive/errcodes-appendix.html).
The error above is the cause of improper transaction control, which can be written as follows.
Improper writing 1:
Connection OPEN.
Try {
Try {
Database operation A.
} Catch (Exception e) {
The log. The error (” do something “);
}
 
Database operation B. // The above error may occur during this data operation.
The transaction commits commit.
} Catch (Exception ex) {
Transaction rollback rollback.
} the finally {
Close the connection.
}
 
Improper writing method 2:
Connection OPEN.
Try {
Database operation A.
Database operation B.
The transaction commits commit.
} Catch (Exception ex) {
Database operation C. // The above error may occur during this data operation.
The transaction commits commit.
} the finally {
Close the connection.
}
 
In a Postgres database, if there is an error in a database operation in the same transaction, all subsequent databases in that transaction will fail.
If there is an error in database operation A, the same database operation B will report an error when it is executed. If there is an error in database operation A or database operation B, the same database operation C will report an error.
To avoid errors, someone is using the re-open method, which in effect throws away all previous operations (not in the case of auto-commit) and adds overhead by re-opening, which is not recommended.
Depending on the cause of the error, we can treat the processing before the possible error as a transaction. The processing after the error is then treated as a transaction, depending on the specific logic. That way you can avoid making mistakes.
In this way, we can change the improper 1 to
Try {
Database operation A.
} Catch (Exception e) {
Transaction rollback or commit;
The log. The error (” do something “);
}
Database operation B.  
 
In this way we can change the improper notation 2 to
} Catch (Exception ex) {
Transaction rollback or commit;
Database operation C.   
The transaction commits commit.
} the finally {
Close the connection.
}
 
 
 

IIS 7.5, ASP.NET MVC. HTTP error 500 (internal server error), but debugging does not enter the background, the browser only reports 500 errors

Bugs, such as:
IIS 7.5, ASP.NET MVC. HTTP Error 500 (Internal Server Error) uploadify uploads images, the browser only reported 500 errors, the progress bar does not move, but debugging does not enter the background
After several days of searching, I could not debug the source of the error. I also wondered whether the maximum byte limit of the upload was limited. However, I finally found the prompt of foreign gods under Google and determined that it was this error.
http://stackoverflow.com/questions/17188930/iis-7-5-asp-net-mvc-http-error-500-internal-server-error-an-unexpected-cond
Solutions:
& lt; The httpRuntime requestValidationMode = “2.0” maxRequestLength = “100000”/& gt; The configuration file adds a maximum byte limit

C programming interface of SQLite database (6) result codes and error codes

SQlite database C programming interface (VI) Result Codes and Error Codes by QQ: 253786989 2012-02-07

Standard Codes
Here are the standard return values and error code definitions:

#define SQLITE_OK           0   /* Successful result */
/* beginning-of-error-codes */
#define SQLITE_ERROR        1   /* SQL error or missing database */
#define SQLITE_INTERNAL     2   /* Internal logic error in SQLite */
#define SQLITE_PERM         3   /* Access permission denied */
#define SQLITE_ABORT        4   /* Callback routine requested an abort */
#define SQLITE_BUSY         5   /* The database file is locked */
#define SQLITE_LOCKED       6   /* A table in the database is locked */
#define SQLITE_NOMEM        7   /* A malloc() failed */
#define SQLITE_READONLY     8   /* Attempt to write a readonly database */
#define SQLITE_INTERRUPT    9   /* Operation terminated by sqlite3_interrupt()*/
#define SQLITE_IOERR       10   /* Some kind of disk I/O error occurred */
#define SQLITE_CORRUPT     11   /* The database disk image is malformed */
#define SQLITE_NOTFOUND    12   /* Unknown opcode in sqlite3_file_control() */
#define SQLITE_FULL        13   /* Insertion failed because database is full */
#define SQLITE_CANTOPEN    14   /* Unable to open the database file */
#define SQLITE_PROTOCOL    15   /* Database lock protocol error */
#define SQLITE_EMPTY       16   /* Database is empty */
#define SQLITE_SCHEMA      17   /* The database schema changed */
#define SQLITE_TOOBIG      18   /* String or BLOB exceeds size limit */
#define SQLITE_CONSTRAINT  19   /* Abort due to constraint violation */
#define SQLITE_MISMATCH    20   /* Data type mismatch */
#define SQLITE_MISUSE      21   /* Library used incorrectly */
#define SQLITE_NOLFS       22   /* Uses OS features not supported on host */
#define SQLITE_AUTH        23   /* Authorization denied */
#define SQLITE_FORMAT      24   /* Auxiliary database format error */
#define SQLITE_RANGE       25   /* 2nd parameter to sqlite3_bind out of range */
#define SQLITE_NOTADB      26   /* File opened that is not a database file */
#define SQLITE_ROW         100  /* sqlite3_step() has another row ready */
#define SQLITE_DONE        101  /* sqlite3_step() has finished executing */
/* end-of-error-codes */

Some of these constants are returned only by a specific function, such as the SQLITE_RANGE, which is returned only by the SQlite3_bind_xxx function. Some constants, such as SQLITE_ERROR, only indicate that an error occurred during the execution of the function, but there is no way to know why the error occurred.
Sqlite_false stands for MISUSE of apis. For example, a bind function returns SQlite_ern when a statement is given another parameter after the sqlite3_step function has executed without being reset.
Extended Codes
Standard error codes provide less information about the cause of the error. So sometimes we use extended error codes. The extended error code is based on the standard error code, whose lower order byte is the original standard error code, and then attaches information to its higher order byte “or” to describe the details of the error.

int sqlite3_extended_result_codes(sqlite3*, int onoff);

The error codes for these extensions are not enabled by default due to compatibility issues with the client’s legacy programs. Programmers can enable or disable extended error codes by using the SQlite3_extended_result_CODES function.
Here are all the extended error codes (most of which describe SQLITE_IOERR) :

#define SQLITE_IOERR_READ              (SQLITE_IOERR | (1<<8))
#define SQLITE_IOERR_SHORT_READ        (SQLITE_IOERR | (2<<8))
#define SQLITE_IOERR_WRITE             (SQLITE_IOERR | (3<<8))
#define SQLITE_IOERR_FSYNC             (SQLITE_IOERR | (4<<8))
#define SQLITE_IOERR_DIR_FSYNC         (SQLITE_IOERR | (5<<8))
#define SQLITE_IOERR_TRUNCATE          (SQLITE_IOERR | (6<<8))
#define SQLITE_IOERR_FSTAT             (SQLITE_IOERR | (7<<8))
#define SQLITE_IOERR_UNLOCK            (SQLITE_IOERR | (8<<8))
#define SQLITE_IOERR_RDLOCK            (SQLITE_IOERR | (9<<8))
#define SQLITE_IOERR_DELETE            (SQLITE_IOERR | (10<<8))
#define SQLITE_IOERR_BLOCKED           (SQLITE_IOERR | (11<<8))
#define SQLITE_IOERR_NOMEM             (SQLITE_IOERR | (12<<8))
#define SQLITE_IOERR_ACCESS            (SQLITE_IOERR | (13<<8))
#define SQLITE_IOERR_CHECKRESERVEDLOCK (SQLITE_IOERR | (14<<8))
#define SQLITE_IOERR_LOCK              (SQLITE_IOERR | (15<<8))
#define SQLITE_IOERR_CLOSE             (SQLITE_IOERR | (16<<8))
#define SQLITE_IOERR_DIR_CLOSE         (SQLITE_IOERR | (17<<8))
#define SQLITE_IOERR_SHMOPEN           (SQLITE_IOERR | (18<<8))
#define SQLITE_IOERR_SHMSIZE           (SQLITE_IOERR | (19<<8))
#define SQLITE_IOERR_SHMLOCK           (SQLITE_IOERR | (20<<8))
#define SQLITE_IOERR_SHMMAP            (SQLITE_IOERR | (21<<8))
#define SQLITE_IOERR_SEEK              (SQLITE_IOERR | (22<<8))
#define SQLITE_LOCKED_SHAREDCACHE      (SQLITE_LOCKED |  (1<<8))
#define SQLITE_BUSY_RECOVERY           (SQLITE_BUSY   |  (1<<8))
#define SQLITE_CANTOPEN_NOTEMPDIR      (SQLITE_CANTOPEN | (1<<8))
#define SQLITE_CORRUPT_VTAB            (SQLITE_CORRUPT | (1<<8))
#define SQLITE_READONLY_RECOVERY       (SQLITE_READONLY | (1<<8))
#define SQLITE_READONLY_CANTLOCK       (SQLITE_READONLY | (2<<8))

Error Functions

int sqlite3_extended_result_codes(sqlite3*, int onoff);

Enable or disable the use of extended error codes for a database connection. Enable extended error codes by passing a non-zero value to the second parameter of the SQlite3_extended_result_CODES function. This function always returns SQLITE_OK, and there is no way to get an extension error code that is currently enabled or off.

int sqlite3_errcode(sqlite3 *db);

If a database function operation does not return SQLITE_OK, you can then call the function to get the error code. By default it returns the standard error code, and it may also return an extended error code if the current database connection is enabled.

int sqlite3_extended_errcode(sqlite3 *db);

Similar to the SQlite3_errcode function, except that it only returns the extended error code.

const char *sqlite3_errmsg(sqlite3*);
const void *sqlite3_errmsg16(sqlite3*);

Returns an error code string, encoded in UTF-8 or UTF-16. Programmers should either call these functions, get the error code information, or make a copy. The next database operation may invalidate the returned string pointer.
SQlite error handling cannot handle multiple errors simultaneously. For example, if an API function call goes awrong and the programmer fails to check for that error, the next API function call may well return SQlite_ern, indicating that the program is trying to use an invalid data structure. So programmers should check and handle any errors that might occur after each API function call.
In addition, if multiple threads share the same database connection, it is best to encapsulate the core API calls and error-handling code in the Critical section. Programmers can use the SQlite3_db_mutex function to get the mutex pointer to the database connection (a pointer to the SQlite3_mutex object).
Prepare for V2 Version
The following table compares the prepare function of the original version with that of v2 version:

The V2 version of prepare is more concise for error handling and has the schema advantages listed in the table above, so it is recommended to use the V2 version of prepare.
Transactions and Errors
Typically, SQlite operations are in auto-commit mode. SQlite automatically encapsulates each SQL command into a transaction. Error recovery is easy if each statement is encapsulated in its own transaction. Any time SQLite finds itself in the wrong state, it simply rolls back the current transaction. This effectively cancels the current SQL command and returns the database to the state it was in before the error occurred.
However, once the BEGIN TRANSACTION command is executed, SQlite is no longer in autocommit mode. A TRANSACTION is opened and will remain open until either the END TRANSACTION or the COMMIT TRANSACTION command is executed. This allows multiple SQL commands to be encapsulated into a single transaction, allowing a discrete set of commands to execute either all or none (atomic operations), but it also limits SQLite’s error recovery.
When a displayed (explicit) transaction encounters an error during execution, SQLite attempts to cancel the statement just executed. Unfortunately, this is not always possible. If things go badly, SOMETIMES SQlite can just roll back the entire current transaction, with no other option.
The most likely errors to cause a rollback are SQLITE_FULL (database or disk space is full), SQLITE_IOERR (disk IO error or file is locked), SQLITE_BUSY (database lock), SQLITE_NOMEM (out of memory), SQLITE_INTERRUPT (interrupt). If the program is executing a display transaction and receives one of these errors, be prepared to handle the possibility of a transaction being rolled back.

int sqlite3_get_autocommit(sqlite3*);

With this function, you can get the current commit status. If a non-zero value is returned, the database is in auto-commit (atutoconmit) mode. If 0 is returned, the database is currently inside an explicit transaction.
If the SQlite database is forced to do a full transaction rollback, the database will once again go into transaction autocommit mode. If the database is not in auto-commit mode, it must be in a transaction, indicating that no rollback is required.

SQlite database C programming interface (VI) Result Codes and Error Codes by QQ: 253786989 2012-02-07

error: expected declaration or statement at end of input

This error is sometimes reported in Linux programming, and it hurts a lot because the compiler doesn’t tell you exactly what went wrong…
There are several possible reasons for this error:
1) Where the program is missing a parenthesis.
The detection method is as follows. If you are programming with VIM, you can use % to match the parentheses one by one. Here is how I did it
2) the variable does not declare

matlab Error Subscript indices must either be real positive integers or logicals.

Matlab errors: Subscript Indices must either be real positive integers or Logicals.Subscript index must be of positive integer type or logical type

Cause of error: in the process of accessing the matrix (including vector, two-dimensional matrix, multidimensional array, same below), the index of the subscript either starts at 0 or has a negative number. Note: The syntax of MATLAB stipulates that the index of the matrix starts from 1, which is different from the habit of programming languages such as C.

Solution: Debug the program yourself and fix subscripts that are zero or negative.

I made a mistake when I was writing the program. I should have done the transpose of the matrix, but I forgot…

OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to github.com:443

The github project was suddenly unavailable with Git today, error reporting:
Fatal: unable to access “https://github.com/dmlc/xgboost.git/” : OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to github.com: 443
It doesn’t work in all kinds of ways. I found many ways on the Internet:
https://blog.csdn.net/daerzei/article/details/79528153
 
https://stackoverflow.com/questions/49345357/fatal-unable-to-access-https-github-com-xxx-openssl-ssl-connect-ssl-error
 
If that doesn’t work, try switching to a network, such as a hotspot or your own broadband connection, because you may be using a company Intranet and have a proxy set up.
If not, I don’t know what to do

Accelerating spark iterative computation with coalesce

problem: join causes partition number inflation

Spark divides each RDD into several partitions, and the calculation on each partition is an independent task. Each task has its own schedule when it executes, so it needs to complete the sorting, serialization, calculation and deserialization work once.

the overhead of a large part of the work is basically constant and does not vary with the amount of data in partition. So when there are too many partitions in an RDD, there will be a large overhead to compute.

in the process of calculation, we often need to put two or more variables together to do some calculation, we usually use join, cogroup operations. Take a.join(b) for example. In the process of execution, the mapb3 operations would be done to partitions (a and b) for the two RDDS respectively. This is because sharding is not guaranteed by default and b's method is not the same, so it is impossible to complete the join operation with only local data. After re-sharding, you can ensure that the keys of a and B on the same partition are the same. The join operation can then be performed locally.

Due to the existence of mapPartitions, operations such as joins would add partition Numbers. In an iterative computing task, the number of partitions would be added to an unimaginably large number, and the computation time for each round would continue to swell.

below is an example of iteratively calculating the shortest path, where we use the BFS algorithm. Among them, we use graph. Join (SSSP) to combine the graph and the current known shortest distance, and then calculate the distance from each node to other nodes through flatMap and compute calculation approach, and finally calculate the known shortest distance in the new round with reduceByKey(min).

# 随机生成一个有权图
# 格式:(src, [(dst, weight)*n])
n=10000
graph=[(k, [(i, random.random()) for i in random.sample(range(n),random.randint(1,n-1))]) for k in range(n)]
print(graph[0])
# 初始化从source到各个节点的已知最短距离
source=0
sssp=graph.map(lambda r:(r[0],0 if r[0]==source else math.inf)
# 定义计算函数
def compute(links, v):
	for d,w in links:
		yield (d,v+w)
# 迭代计算
for i in range(20):
	t=time.time()
	c=graph.join(sssp).flatMap(lambda kls: compute(kls[1][0],kls[1][1]))
	sssp=c.reduceByKey(min)
	#sssp=sssp.coalesce(4)
	p=sssp.aggregate((0,0), (lambda lr,v:(lr[0],lr[1]+1) if math.isinf(v[1]) else(lr[0]+v[1],lr[1])), (lambda a,b:(a[0]+b[0],a[1]+b[1])))
	print(i,time.time()-t,p,c.getNumPartitions(),sssp.getNumPartitions())

The getnumb2 function here called can see how many partitions are currently in an RDD.
the following data is the calculation result when I ran graph with 3 core, and cut graph into 3 pieces at the same time, and cut the initial SSSP into 6 pieces. It can be seen that, except for the first round, which was slower due to some initialization work, the calculation time of each round increased with the increase of partition number.
if WebUI is used, the corresponding stage information can be pointed out, and it can be seen clearly that the number of tasks is increasing gradually and overhead is also increasing. The resulting operation reduceByKey also took longer and longer to operate.

0 2.0179808139801025 (8745.09903, 1) 9 9
1 1.5022525787353516 (8745.099020000001, 0) 12 12
2 2.0394625663757324 (8745.09902, 0) 15 15
3 2.5546443462371826 (8745.099020000001, 0) 18 18
4 3.239337921142578 (8745.09902, 0) 21 21

Solution: merge partition

repartition function
The most direct merger of parition of

is the repartition function. It can convert an RDD into any number of Partitions.

a=sc.parallelize(range(100),5) # 初始为5个partition
print(a.getNumPartitions())
b=a.repartition(10) # 转为10个partition
print(b.getNumPartitions())
c=a.repartition(2) # 转为2个partition
print(c.getNumPartitions())

, but repartition will recalculate all the elements' partitions that they should be part of each execution, which may be very expensive.

coalesce function
The

function is different from the repartition function, and the coalesce function is used only to reduce the number of partitions. And it does not recompute partition ownership, it simply merges some data. Therefore, its parameters are only meaningful if they are less than the current number of partitions of the RDD.
its execution principle is not to carry out cross-communication, but simply to combine data blocks on the same core, so that the final partition quantity meets the given parameters.

a=sc.parallelize(range(100),5
b=a.coalesce(5)

Below is the result after I removed the comment in the above code for coalesce. The first run took a little longer because I did some other things with each other. But then the running time is very stable and no longer increases with the number of rounds.

0 2.5019025802612305 (8745.099020000001, 0) 7 4
1 0.9806034564971924 (8745.099020000001, 0) 7 4
2 0.8405632972717285 (8745.099020000001, 0) 7 4
3 0.795809268951416 (8745.099020000001, 0) 7 4
4 0.7986171245574951 (8745.099020000001, 0) 7 4