Author Archives: Robins

[Solved] M1 Chip MacBook Pro Error: snappy-java FAILED_TO_LOAD_NATIVE_LIBRARY

Solution:

Directly find the xml file that introduces the dependency (I am the xml file of spark-core), change the version of snappy-java in it to 1.1.8.4. just reload it.

    <dependency>
      <groupId>org.xerial.snappy</groupId>
      <artifactId>snappy-java</artifactId>
      <version>1.1.8.4</version>
      <scope>compile</scope>
    </dependency>

 

How to Solve: (Detailed exploration process)

The MacBook Pro with the new M1 pro chip arrived a few days ago, and I couldn’t wait to try it out. Because the chip uses the ARM instruction set, the previous Intel chip computers used the X86 instruction set, so there were inevitably some problems. However, after a year of adaptation, there are basically no major problems, so after a few days of trying, I decided to use the machine for work production.

Because I want to install Java, at first I used the traditional Intel version of jdk (because the main use is jdk1.8, and Oracle only did the adaptation of jdk17 for the arm version), and I feel that some programs in the development will not be executed so fast (and 18 MacBook Pro execution speed is about the same), because after all, it has to go through Rosetta2 translate. Therefore, after using it for a few days, I replaced the Intel version of jdk with the ARM version of jdk, but there is a problem in the documentation, the direct download link given in the documentation is not the ARM version, I found that the execution has been using the intel version of java after installation, then I found that the download link is wrong, you need to manually download the real ARM version of jdk from the official website.

After installing the arm version of jdk, the compilation speed is really improved, but when executing the spark program locally, I encountered such a problem, the detailed log is as follows:

Caused by: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=Mac and os.arch=aarch64
	at org.xerial.snappy.SnappyLoader.findNativeLibrary(SnappyLoader.java:331)
	at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:171)
	at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:152)
	at org.xerial.snappy.Snappy.<clinit>(Snappy.java:47)
	at org.apache.parquet.hadoop.codec.SnappyDecompressor.decompress(SnappyDecompressor.java:62)
	at org.apache.parquet.hadoop.codec.NonBlockedDecompressorStream.read(NonBlockedDecompressorStream.java:51)
	at java.io.DataInputStream.readFully(DataInputStream.java:195)
	at java.io.DataInputStream.readFully(DataInputStream.java:169)
	at org.apache.parquet.bytes.BytesInput$StreamBytesInput.toByteArray(BytesInput.java:205)
	at org.apache.parquet.column.values.dictionary.PlainValuesDictionary$PlainBinaryDictionary.<init>(PlainValuesDictionary.java:89)
	at org.apache.parquet.column.values.dictionary.PlainValuesDictionary$PlainBinaryDictionary.<init>(PlainValuesDictionary.java:72)
	at org.apache.parquet.column.Encoding$1.initDictionary(Encoding.java:90)
	at org.apache.parquet.column.Encoding$4.initDictionary(Encoding.java:149)
	at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.<init>(VectorizedColumnReader.java:103)
	at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.checkEndOfRowGroup(VectorizedParquetRecordReader.java:280)
	at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:225)
	at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.nextKeyValue(VectorizedParquetRecordReader.java:137)
	at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:39)
	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:105)
	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:177)
	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:105)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.scan_nextBatch$(Unknown Source)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)
	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
	at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:395)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.agg_doAggregateWithKeys$(Unknown Source)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)
	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
	at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:395)
	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
	at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:125)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
	at org.apache.spark.scheduler.Task.run(Task.scala:108)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
	... 3 more

org.xerial.snappy is a compressed/uncompressed library, I am using spark version 2.2.0, which depends on snappy 1.1.2.6. The contents of this version are as follows:

As you can see, this version of snappy only supports x86 and x86_64 for Mac systems, not arm64. Just find the dependency that introduces snappy-java and upgrade its version directly to the latest version.

Searching through IDEA’s dependency graph, I found that the library was imported by spark-core, my version of spark-core is 2.2.0 and the snappy-java introduced is 1.1.2.6. As shown in the figure:

The latest version of snappy-java is 1.1.8.4 in the mvnrepository, so replace the original version in the spark-core xml file directly, as shown in the figure

Reload the dependency and execute it again. The code executes smoothly and the problem is solved.

[Solved] Installation failed due to: ‘INSTALL_FAILED_SHARED_USER_INCOMPATIBLE: Package com.

1. No system signature is added

If you have not added the system signature, find the following codes at the header of the AndroidManifest.xml file and delete it:

android:sharedUserId="android.uid.system"

2. Added system signature

  1. Check whether there is any problem with the system signature itself or the joining method?
  2. If there is no problem with the system signature, there should already be an APK in the system, and the current APK should be added to the system uid. At this time, you should first delete the APK that exists in the system, and then reboot.
adb root
adb uninstall com.xxx.xxxx
adb reboot

[How to Solve] Task :app:compileDebugKotlin FAILED

Question:

After changing the app ID and the module name of the Android project, this problem occurred in the compilation.

Solution:

1. tried to unify kotlin version, Android Studio Tools -> Kotlin -> Configure Kotlin Plugin Updates
Check the current Kotlin version, and find kotlin-version in the project (different projects show different ways, usually in build.gradle), change the Kotlin version of the project to the version on Android Studio, and then recompile, it failed.

2. before reporting this error log, look up, you know the problem, is the program code has reported red place, just a matter of habit, each time the red part of the log down to see, resulting in a delay of a good while.

[Solved] JAVA HttpClient Send Https request certificate error:PKIX path building failed:

Recently, when using httpclient to connect with the third-party SMS interface, a certificate invalidation error was reported during the local test.

1. Post request of Encapsulated HttpClient:

public static Map<String, Object> postReq(String URL, Map<String, Object> paramMap, Map<String, String> headers) throws Exception {
        Map<String, Object> map = new HashMap<String, Object>();

        RequestConfig requestConfig = RequestConfig.custom()
                .setConnectTimeout(2000) // Set the connection timeout, in milliseconds
                .setConnectionRequestTimeout(1000)
                .setSocketTimeout(5000) // timeout for requesting data, in milliseconds
                .build();
        HttpRequestRetryHandler myRetryHandler = new HttpRequestRetryHandler() {
            @Override
            public boolean retryRequest(IOException exception, int executionCount, HttpContext context) {
                return false;
            }
        };

        try (CloseableHttpClient client = HttpClients.custom()
                .setDefaultRequestConfig(requestConfig)
                .setRetryHandler(myRetryHandler)
                .build()) {

            HttpPost httpPost = new HttpPost(URL);
            if (paramMap != null) {
                JSONObject paramJson = new JSONObject(paramMap);
                StringEntity paramEntity = new StringEntity(paramJson.toString(), "UTF-8");
                paramEntity.setContentType("application/json; charset=utf-8");
                httpPost.setEntity(paramEntity);
            }
            httpPost.setConfig(requestConfig);

            if (headers != null && !headers.isEmpty()) {
                for (String key : headers.keySet()) {
                    String value = headers.get(key);
                    httpPost.setHeader(key, value);
                }
            }

            CloseableHttpResponse response = client.execute(httpPost);

            HttpEntity entity = response.getEntity();

            if (entity != null) {
                String responseStr = EntityUtils.toString(entity, "UTF-8");
                if (responseStr.isEmpty()) {
                    responseStr = "{}";
                }

                int statusCode = response.getStatusLine().getStatusCode();
                if (HttpServletResponse.SC_OK == statusCode) {
                    try {
                        JSONObject dataJson = (JSONObject) JSONObject.parse(responseStr);
                        map = new HashMap<>(dataJson);
                    } catch (Exception e) {
                        map.put("reponse", responseStr);
                    }
                } else {
                    return map;
                }
            }
            response.close();
        }
        return map;
    }

However, an error will be reported when accessing some self signed HTTPS requests. This problem is caused by the invalid link certificate, because the self signed certificate will be recognized as an unsafe link.

sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

2. Solutions

  1. Set the security certificate of jdk according to the online method -> No work
  2. Set maven to ignore the certificate checksum -> no work
  3. Yml file configuration httpclient ignore SSL checksum, seems to have no effect.
  4. Finally used the method of modifying the code, the principle is also to ignore the certificate checksum, but the code produced the effect, it is estimated that the construction is related to the Httpclient.

 

Modify the original code

// Ignore SSL Security Certification
**SSLConnectionSocketFactory scsf = new SSLConnectionSocketFactory(
        SSLContexts.custom().loadTrustMaterial(null, new TrustSelfSignedStrategy()).build(),
        NoopHostnameVerifier.INSTANCE);**

try (CloseableHttpClient client = **HttpClients.custom().setSSLSocketFactory(scsf)**
        .setDefaultRequestConfig(requestConfig)
        .setRetryHandler(myRetryHandler)
        .build()) {

[Solved] socket failed: EPERM (Operation not permitted)

Reason & Solution:

1. The network permission is not enabled
2 HTTP is not supported

Add network permissions to AndroidManifest.xml

<uses-permission android:name="android.permission.INTERNET" />
<uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />

If the error is still reported, add the following to the Application in the AndroidManifest.xml file

android:usesCleartextTraffic="true"

Reason: On Android 9.0 machines, http access is not supported by default, all network access must use https. of course Android has supported https in several earlier versions, except that http is not supported by default using https on 9.0.

If the above operation can not be solved, please uninstall the application first, and then run it again.

[Solved] Docker Start nginx error: driver failed programming external connectivity on endpoint nginx

@[TOC](Docker start nginx error: Error response from daemon: driver failed programming external connectivity on endpoint nginx (6307a83f775bf316440e9b843841d808a84dded1b805b204484e41a73835f24a): (iptables failed: iptables –wait -t nat -A DOCKER -p tcp -d 0/0 –dport 80 -j DNAT –to-destination 172.17.0.6:80 ! -i docker0: iptables: No chain/target/match by that name.)

Problem Background
When starting nginx using docker, an error is reported

[root@localhost conf.d]# docker start nginx 
Error response from daemon: driver failed programming external connectivity on endpoint nginx (6307a83f775bf316440e9b843841d808a84dded1b805b204484e41a73835f24a):  (iptables failed: iptables --wait -t nat -A DOCKER -p tcp -d 0/0 --dport 80 -j DNAT --to-destination 172.17.0.6:80 ! -i docker0: iptables: No chain/target/match by that name.

Solution:

1. Before starting the nginx of docker, the firewall was closed, which led to an error in the link. You can restart docker to re-establish the link

systemctl restart docker

Restart nginx

docker start nginx

[Solved] Servers Install matlab 2018a error: license checkout failed -8

Servers Install matlab 2018a error: license checkout failed -8

 

Solution:

  • Located under the licence folder in the installation path
  • Open the license_standalone.lic file with Notepad
  • Replace “SIGH=” with “TS_OK SIGN=”

Refer to the official website solution

https://www.mathworks.com/matlabcentral/answers/91874-why-do-i-receive-license-manager-error-103

[Solved] E xx: xxxx:xx ERR thread 0x536753929472 setscheduler failed Operation not permitted

E xx: xxxx:xx ERR thread 0x536753929472 setscheduler failed Operation not permitted

The “setscheduler failed Operation not permitted” error occurred when setting the thread priority via sched_setscheduler.

int ret = sched_setscheduler(pthread_gettid_np((pthread_t)pHandle), SCHED_RR, &prio);

 

Solution:
add the following contents to the process startup parameters:

capabilities SYS_NICE

init.target.rc

service xxxxxx /system/vendor/bin/xxxxxx
    class core
    user system
    group camera input graphics
    disabled
    onrestart restart evs_driver
    onrestart restart qcarcam_hal
    capabilities SYS_NICE

[Solved] Linux Error: Failed to start bean ‘webServerStartStop‘; nested exception is org.springframework.

💛 Environment: windows10, centos7.5.1804

linux_debug：Failed to start bean ‘webServerStartStop’; nested exception is org. springframework. boot.

1. Error information

2022-07-08 18:12:25.088 ERROR 33600 --- [           main] o.s.boot.SpringApplication               : Application run failed

org.springframework.context.ApplicationContextException: Failed to start bean 'webServerStartStop'; nested exception is org.springframework.boot.web.server.WebServerException: Unable to start embedded Tomcat server
	at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:181) ~[spring-context-5.3.21.jar!/:5.3.21]
	at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:54) ~[spring-context-5.3.21.jar!/:5.3.21]
	at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:356) ~[spring-context-5.3.21.jar!/:5.3.21]
	at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_333]
	at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:155) ~[spring-context-5.3.21.jar!/:5.3.21]
	at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:123) ~[spring-context-5.3.21.jar!/:5.3.21]

2. Problem description

When running the jar package made by springboot project on Linux;

If the above problem occurs, port 8080 is occupied;

3. Solutions

Close the occupied 8080 port;

3.1 windows

Enter in CMD

# Find the corresponding pid
netstat -ano | findstr 8080

## out：TCP 0.0.0.0:8080 0.0.0.0:0 LISTENING 6148

Close the PID number corresponding to the port, assuming that port 8080 corresponds to the process number with process number 123

taskkill /pid 6148 -t -f

3.2 linux

Enter in CMD

# Find the corresponding pid
netstat -ano | grep 8080

## out：TCP 0.0.0.0:8080 0.0.0.0:0 LISTENING 6148

Close the PID number corresponding to the port, assuming that port 8080 corresponds to the process number with process number 123

kill -9 PID 123