Tag Archives: CDH

[Solved] CDH ipa: ERROR: Host ‘cdh-master-v02.yunes.com‘ does not have corresponding DNS A/AAAA record

1. Error environment

1. System version: CentOS7.7

2. IPA version: 4.6.8

2. Error occurred when installing Kerberos

1. Error description:

Execute global command Generate lost credentials
/opt/cloudera/cm/bin/gen_credentials_ipa.sh failed with exit code 1 and output of <<
+ CMF_REALM=YUNES.COM
+ export PATH=/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/usr/lib/mit/bin:/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
+ PATH=/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/usr/lib/mit/bin:/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
+ kinit -k -t /var/run/cloudera-scm-server/cmf8356082187139991822.keytab [email protected]
+ KEYTAB_OUT=/var/run/cloudera-scm-server/cmf1174813005448134353.keytab
+ PRINCIPAL=impala/[email protected]
+ MAX_RENEW_LIFE=432000
+ '[' -z /etc/krb5.conf ']'
+ echo 'Using custom config path '\''/etc/krb5.conf'\'', contents below:'
+ cat /etc/krb5.conf
+ PRINC=impala
++ echo impala/[email protected]
++ cut -d/-f 2
++ cut -d @ -f 1
+ HOST=cdh-master-v02.yunes.com
+ set +e
+ ipa host-find cdh-master-v02.yunes.com
+ ERR=0
+ set -e
+ [[ 0 -eq 0 ]]
+ echo 'Host cdh-master-v02.yunes.com exists'
+ set +e
+ ipa service-find impala/[email protected]
+ ERR=1
+ set -e
+ [[ 1 -eq 0 ]]
+ PRINC_EXISTS=no
+ echo 'Adding new principal: impala/[email protected]'
+ ipa service-add impala/[email protected]
ipa: ERROR: Host 'cdh-master-v02.yunes.com' does not have corresponding DNS A/AAAA record


2, the solution (may be version differences in their own choice of test)
1)

ipa dnsrecord-add yunes.com cdh-cm-v01 –a-rec 192.168.0.200;
ipa dnsrecord-add yunes.com cdh-master-v01 –a-rec 192.168.0.201;
ipa dnsrecord-add yunes.com cdh-master-v02 –a-rec 192.168.0.202;
ipa dnsrecord-add yunes.com cdh-datanode-v01 –a-rec 192.168.0.203;
ipa dnsrecord-add yunes.com cdh-datanode-v02 –a-rec 192.168.0.204;
ipa dnsrecord-add yunes.com cdh-datanode-v03 –a-rec 192.168.0.205;
ipa dnsrecord-add yunes.com cdh-client-v01 –a-rec 192.168.0.206;

[root@cdh-ipa-v01 ~]# ipa dnsrecord-add yunes.com cdh-cm-v01 --a-rec 192.168.0.200;
ipa dnsrecord-add yunes.com cdh-master-v01 --a-rec 192.168.0.201;
ipa dnsrecord-add yunes.com cdh-master-v02 --a-rec 192.168.0.202;
ipa dnsrecord-add yunes.com cdh-datanode-v01 --a-rec 192.168.0.203;
ipa dnsrecord-add yunes.com cdh-datanode-v02 --a-rec 192.168.0.204;
ipa dnsrecord-add yunes.com cdh-datanode-v03 --a-rec 192.168.0.205;
ipa dnsrecord-add yunes.com cdh-client-v01 --a-rec 192.168.0.206;  Record name: cdh-cm-v01
  A record: 192.168.0.200
  SSHFP record: 1 1 2EE47C060AD498FACE720384A62F5672A24F2B15, 1 2 E0F507FC5983919E80A81397167FE5B0A31247E55B9FE96D1F789534 35250808, 3 1
                182B18E515A1A9D4C7B434BA4775876709F6DF2A, 3 2 76C34C382E5060EF30D0545A82D8BC0DB3D18034849CCE3ECB601A37 08F8F36C, 4 1
                67CE1FEB39325B57790BB046035E53A3AF2B893C, 4 2 CBDE7A7845393E7C60713731DE0F18CA7670FDA37A8232E2F53AE401 527D1248
[root@cdh-ipa-v01 ~]# ipa dnsrecord-add yunes.com cdh-master-v01 --a-rec 192.168.0.201;
  Record name: cdh-master-v01
  A record: 192.168.0.201
  SSHFP record: 1 1 2EE47C060AD498FACE720384A62F5672A24F2B15, 1 2 E0F507FC5983919E80A81397167FE5B0A31247E55B9FE96D1F789534 35250808, 3 1
                182B18E515A1A9D4C7B434BA4775876709F6DF2A, 3 2 76C34C382E5060EF30D0545A82D8BC0DB3D18034849CCE3ECB601A37 08F8F36C, 4 1
                67CE1FEB39325B57790BB046035E53A3AF2B893C, 4 2 CBDE7A7845393E7C60713731DE0F18CA7670FDA37A8232E2F53AE401 527D1248
[root@cdh-ipa-v01 ~]# ipa dnsrecord-add yunes.com cdh-master-v02 --a-rec 192.168.0.202;
  Record name: cdh-master-v02
  A record: 192.168.0.202
  SSHFP record: 1 1 2EE47C060AD498FACE720384A62F5672A24F2B15, 1 2 E0F507FC5983919E80A81397167FE5B0A31247E55B9FE96D1F789534 35250808, 3 1
                182B18E515A1A9D4C7B434BA4775876709F6DF2A, 3 2 76C34C382E5060EF30D0545A82D8BC0DB3D18034849CCE3ECB601A37 08F8F36C, 4 1
                67CE1FEB39325B57790BB046035E53A3AF2B893C, 4 2 CBDE7A7845393E7C60713731DE0F18CA7670FDA37A8232E2F53AE401 527D1248
[root@cdh-ipa-v01 ~]# ipa dnsrecord-add yunes.com cdh-datanode-v01 --a-rec 192.168.0.203;
  Record name: cdh-datanode-v01
  A record: 192.168.0.203
  SSHFP record: 1 1 2EE47C060AD498FACE720384A62F5672A24F2B15, 1 2 E0F507FC5983919E80A81397167FE5B0A31247E55B9FE96D1F789534 35250808, 3 1
                182B18E515A1A9D4C7B434BA4775876709F6DF2A, 3 2 76C34C382E5060EF30D0545A82D8BC0DB3D18034849CCE3ECB601A37 08F8F36C, 4 1
                67CE1FEB39325B57790BB046035E53A3AF2B893C, 4 2 CBDE7A7845393E7C60713731DE0F18CA7670FDA37A8232E2F53AE401 527D1248
[root@cdh-ipa-v01 ~]# ipa dnsrecord-add yunes.com cdh-datanode-v02 --a-rec 192.168.0.204;
  Record name: cdh-datanode-v02
  A record: 192.168.0.204
  SSHFP record: 1 1 2EE47C060AD498FACE720384A62F5672A24F2B15, 1 2 E0F507FC5983919E80A81397167FE5B0A31247E55B9FE96D1F789534 35250808, 3 1
                182B18E515A1A9D4C7B434BA4775876709F6DF2A, 3 2 76C34C382E5060EF30D0545A82D8BC0DB3D18034849CCE3ECB601A37 08F8F36C, 4 1
                67CE1FEB39325B57790BB046035E53A3AF2B893C, 4 2 CBDE7A7845393E7C60713731DE0F18CA7670FDA37A8232E2F53AE401 527D1248
[root@cdh-ipa-v01 ~]# ipa dnsrecord-add yunes.com cdh-datanode-v03 --a-rec 192.168.0.205;
  Record name: cdh-datanode-v03
  A record: 192.168.0.205
  SSHFP record: 1 1 2EE47C060AD498FACE720384A62F5672A24F2B15, 1 2 E0F507FC5983919E80A81397167FE5B0A31247E55B9FE96D1F789534 35250808, 3 1
                182B18E515A1A9D4C7B434BA4775876709F6DF2A, 3 2 76C34C382E5060EF30D0545A82D8BC0DB3D18034849CCE3ECB601A37 08F8F36C, 4 1
                67CE1FEB39325B57790BB046035E53A3AF2B893C, 4 2 CBDE7A7845393E7C60713731DE0F18CA7670FDA37A8232E2F53AE401 527D1248
[root@cdh-ipa-v01 ~]# ipa dnsrecord-add yunes.com cdh-client-v01 --a-rec 192.168.0.206;
  Record name: cdh-client-v01
  A record: 192.168.0.206
  SSHFP record: 1 1 2EE47C060AD498FACE720384A62F5672A24F2B15, 1 2 E0F507FC5983919E80A81397167FE5B0A31247E55B9FE96D1F789534 35250808, 3 1
                182B18E515A1A9D4C7B434BA4775876709F6DF2A, 3 2 76C34C382E5060EF30D0545A82D8BC0DB3D18034849CCE3ECB601A37 08F8F36C, 4 1
                67CE1FEB39325B57790BB046035E53A3AF2B893C, 4 2 CBDE7A7845393E7C60713731DE0F18CA7670FDA37A8232E2F53AE401 527D1248
[root@cdh-ipa-v01 ~]#


2)

ipa dnsrecord-add 0.168.192.in-addr.arpa 200 –ptr-rec cdh-cm-v01.yunes.com.
ipa dnsrecord-add 0.168.192.in-addr.arpa 201 –ptr-rec cdh-master-v01.yunes.com.
ipa dnsrecord-add 0.168.192.in-addr.arpa 202 –ptr-rec cdh-master-v02.yunes.com.
ipa dnsrecord-add 0.168.192.in-addr.arpa 203 –ptr-rec cdh-datanode-v01.yunes.com.
ipa dnsrecord-add 0.168.192.in-addr.arpa 204 –ptr-rec cdh-datanode-v02.yunes.com.
ipa dnsrecord-add 0.168.192.in-addr.arpa 205 –ptr-rec cdh-datanode-v03.yunes.com.
ipa dnsrecord-add 0.168.192.in-addr.arpa 206 –ptr-rec cdh-client-v01.yunes.com.

ipa dnsrecord-add 0.168.192.in-addr.arpa 200 --ptr-rec cdh-cm-v01.yunes.com.
ipa dnsrecord-add 0.168.192.in-addr.arpa 201 --ptr-rec cdh-master-v01.yunes.com.
ipa dnsrecord-add 0.168.192.in-addr.arpa 202 --ptr-rec cdh-master-v02.yunes.com.
ipa dnsrecord-add 0.168.192.in-addr.arpa 203 --ptr-rec cdh-datanode-v01.yunes.com.
ipa dnsrecord-add 0.168.192.in-addr.arpa 204 --ptr-rec cdh-datanode-v02.yunes.com.
ipa dnsrecord-add 0.168.192.in-addr.arpa 205 --ptr-rec cdh-datanode-v03.yunes.com.
ipa dnsrecord-add 0.168.192.in-addr.arpa 206 --ptr-rec cdh-client-v01.yunes.com.

 

scm_prepare_database.sh CDH initial data script execute error

Environment

CDH 5
MySQL 5.7.30

check the manual that corresponds to your MySQL server version for the right synta to use near '@ default character  
set utf8 at line 1

reason

For the problem of script execution mode, – P does not write a password after it. Execute it directly, and then enter the password. The result returns successful

/usr/share/cmf/schema/scm_prepare_database.sh mysql -h Database_ip -u username -p --scm-host cdhip scm scm scm

Install additional stage package for streamsets – error in cdh6.3.0 package rest API call error: java.io.eofexception

Version
streamsets3.16.1 (core)
cdh6.3.2

1、 Question

The name of the streamsets installation package is streamsets-datacollector-core-3.16.1. Tgz , and an error is reported when downloading the package of cdh6.3 after installation

1 operation

Install the package of cdh6.3.0 through the streamsets UI and report an error

click Show error

2. Complete error reporting contents

java.io.EOFException
	at org.apache.commons.compress.compressors.gzip.GzipCompressorInputStream.read(GzipCompressorInputStream.java:303)
	at org.apache.commons.compress.archivers.tar.TarArchiveInputStream.read(TarArchiveInputStream.java:608)
	at java.io.InputStream.read(InputStream.java:101)
	at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1792)
	at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1769)
	at org.apache.commons.io.IOUtils.copy(IOUtils.java:1744)
	at com.streamsets.datacollector.restapi.StageLibraryResource.installLibraries(StageLibraryResource.java:363)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:160)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)
	at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473)
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:760)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1617)
	at com.streamsets.datacollector.http.GroupsInScopeFilter.lambda$doFilter$0(GroupsInScopeFilter.java:82)
	at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:34)
	at com.streamsets.datacollector.http.GroupsInScopeFilter.doFilter(GroupsInScopeFilter.java:81)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
	at com.streamsets.datacollector.restapi.rbean.rest.RestResourceContextFilter.doFilter(RestResourceContextFilter.java:42)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
	at org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:310)
	at org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:264)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
	at com.streamsets.datacollector.http.LocaleDetectorFilter.doFilter(LocaleDetectorFilter.java:39)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
	at org.eclipse.jetty.servlets.HeaderFilter.doFilter(HeaderFilter.java:117)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
	at com.streamsets.pipeline.http.MDCFilter.doFilter(MDCFilter.java:47)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:717)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:501)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1592)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1296)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1562)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1211)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:500)
	at com.streamsets.lib.security.http.LimitedMethodServer.handle(LimitedMethodServer.java:41)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:386)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:562)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:378)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
	at java.lang.Thread.run(Thread.java:748)

3 sdc.log key error

2021-11-02 11:02:04,278 [user:admin] [pipeline:] [runner:] [thread:webserver-48] [stage:] INFO  StageLibraryResource - Installing stage library streamsets-datacollector-cdh_6_3-lib from http://archives.streamsets.com/datacollector/3.16.1/tarball/streamsets-datacollector-cdh_6_3-lib-3.16.1.tgz
2021-11-02 11:21:13,324 [user:admin] [pipeline:] [runner:] [thread:webserver-48] [stage:] ERROR ExceptionToHttpErrorProvider - REST API call error: java.io.EOFException
java.io.EOFException
        at org.apache.commons.compress.compressors.gzip.GzipCompressorInputStream.read(GzipCompressorInputStream.java:303)
        at org.apache.commons.compress.archivers.tar.TarArchiveInputStream.read(TarArchiveInputStream.java:608)
        at java.io.InputStream.read(InputStream.java:101)
        at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1792)
        at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1769)

2、 Positioning

1 idea a

According to the log, it is from http://archives.streamsets.com/datacollector/3.16.1/tarball/streamsets-datacollector-cdh_ 6_ 3-lib-3.16.1. Tgz Download streamsets DataCollector CDH_ 6_ 3-lib
try the browser to visit this address and find that it can be downloaded. Try to manually download the package and put it in the target directory

SSH connects to the server and finds that the network is very slow. Try downloading this package on a PC. after decompression, you find streamsets-datacollector-3.16.1 \ streamsets LIBS \ streamsets DataCollector CDH_ 6_ There are jar packages in the 3-lib \ lib directory. Put these in the target directory of streamsets download streamsets-datacollector-3.16.1/streamsets-libs/streamsets-datacollector-cdh_ 6_ When looking at the installed packages in 3-lib/lib , it is found that there is still no

this method is not very good. It may be that some other configurations will be written in addition to the jar package during the installation process, which we can’t do manually. Change your mind

2 idea B (feasible)

The installation package of this version is the core version. Consider reinstalling the full version. Download streamsets-datacollector-all-3.22.3. Tgz the installation steps refer to the official documents

You can see that the required package is already installed after reinstallation

reference material

Official documents