Spark ошибка во время передачи файла - ErrorsMaster.ru - большая энциклопедия ошибок и их решений

Редактировать | Профиль | Сообщение | Цитировать | Сообщить модератору shustersh
netdiag

Computer Name: SERVER-KIS
DNS Host Name: SERVER-KIS.kis.kz
System info : Windows 2000 Server (Build 3790)
Processor : x86 Family 15 Model 4 Stepping 3, GenuineIntel
List of installed hotfixes :
Q147222

Netcard queries test . . . . . . . : Passed

Per interface results:

Adapter : ╧юфъы■ўхэшх яю ыюъры№эющ ёхЄш

Netcard queries test . . . : Passed

Host Name. . . . . . . . . : SERVER-KIS
IP Address . . . . . . . . : 192.168.79.3
Subnet Mask. . . . . . . . : 255.255.255.0
Default Gateway. . . . . . : 192.168.79.8
Primary WINS Server. . . . : 192.168.79.3
Dns Servers. . . . . . . . : 192.168.79.3

AutoConfiguration results. . . . . . : Passed

Default gateway test . . . : Failed
No gateway reachable for this adapter.

NetBT name test. . . . . . : Passed
No remote names have been found.

WINS service test. . . . . : Passed

Domain membership test . . . . . . : Passed

NetBT transports test. . . . . . . : Passed
List of NetBt transports currently configured:
NetBT_Tcpip_ <8E45C675-BE4A-454D-92E2-E3FF45108475>
1 NetBt transport currently configured.

Autonet address test . . . . . . . : Passed

IP loopback ping test. . . . . . . : Passed

Default gateway test . . . . . . . : Failed

[FATAL] NO GATEWAYS ARE REACHABLE.
You have no connectivity to other network segments.
If you configured the IP protocol manually then
you need to add at least one valid gateway.

NetBT name test. . . . . . . . . . : Passed

Winsock test . . . . . . . . . . . : Passed

DNS test . . . . . . . . . . . . . : Failed
[FATAL] File confignetlogon.dns contains invalid DNS entries. [FATAL
] No DNS servers have the DNS records for this DC registered.

Redir and Browser test . . . . . . : Passed
List of NetBt transports currently bound to the Redir
NetBT_Tcpip_ <8E45C675-BE4A-454D-92E2-E3FF45108475>
The redir is bound to 1 NetBt transport.

List of NetBt transports currently bound to the browser
NetBT_Tcpip_ <8E45C675-BE4A-454D-92E2-E3FF45108475>
The browser is bound to 1 NetBt transport.

DC discovery test. . . . . . . . . : Passed

DC list test . . . . . . . . . . . : Passed

Trust relationship test. . . . . . : Skipped

Kerberos test. . . . . . . . . . . : Passed

LDAP test. . . . . . . . . . . . . : Passed

Bindings test. . . . . . . . . . . : Passed

WAN configuration test . . . . . . : Skipped
No active remote access connections.

Modem diagnostics test . . . . . . : Passed

IP Security test . . . . . . . . . : Skipped

Note: run «netsh ipsec dynamic show /?» for more detailed information

The command completed successfully

Domain Controller Diagnosis

Performing initial setup:
Done gathering initial info.

Doing initial required tests

Testing server: Default-First-Site-NameSERVER-KIS
Starting test: Connectivity
. SERVER-KIS passed test Connectivity

Doing primary tests

Testing server: Default-First-Site-NameSERVER-KIS
Starting test: Replications
. SERVER-KIS passed test Replications
Starting test: NCSecDesc
. SERVER-KIS passed test NCSecDesc
Starting test: NetLogons
. SERVER-KIS passed test NetLogons
Starting test: Advertising
. SERVER-KIS passed test Advertising
Starting test: KnowsOfRoleHolders
. SERVER-KIS passed test KnowsOfRoleHolders
Starting test: RidManager
. SERVER-KIS passed test RidManager
Starting test: MachineAccount
. SERVER-KIS passed test MachineAccount
Starting test: Services
. SERVER-KIS passed test Services
Starting test: ObjectsReplicated
. SERVER-KIS passed test ObjectsReplicated
Starting test: frssysvol
. SERVER-KIS passed test frssysvol
Starting test: frsevent
. SERVER-KIS passed test frsevent
Starting test: kccevent
. SERVER-KIS passed test kccevent
Starting test: systemlog
An Error Event occured. EventID: 0x00000457
Time Generated: 06/01/2007 14:34:58
(Event String could not be retrieved)
An Error Event occured. EventID: 0x00000457
Time Generated: 06/01/2007 14:34:58
(Event String could not be retrieved)
An Error Event occured. EventID: 0x00000457
Time Generated: 06/01/2007 14:34:58
(Event String could not be retrieved)
An Error Event occured. EventID: 0x00000457
Time Generated: 06/01/2007 14:39:10
(Event String could not be retrieved)
An Error Event occured. EventID: 0x00000457
Time Generated: 06/01/2007 14:39:11
(Event String could not be retrieved)
An Error Event occured. EventID: 0x00000457
Time Generated: 06/01/2007 14:39:12
(Event String could not be retrieved)
. SERVER-KIS failed test systemlog
Starting test: VerifyReferences
. SERVER-KIS passed test VerifyReferences

Running partition tests on : DomainDnsZones
Starting test: CrossRefValidation
. DomainDnsZones passed test CrossRefValidation

Starting test: CheckSDRefDom
. DomainDnsZones passed test CheckSDRefDom

Running partition tests on : ForestDnsZones
Starting test: CrossRefValidation
. ForestDnsZones passed test CrossRefValidation

Starting test: CheckSDRefDom
. ForestDnsZones passed test CheckSDRefDom

Running partition tests on : Schema
Starting test: CrossRefValidation
. Schema passed test CrossRefValidation
Starting test: CheckSDRefDom
. Schema passed test CheckSDRefDom

Running partition tests on : Configuration
Starting test: CrossRefValidation
. Configuration passed test CrossRefValidation
Starting test: CheckSDRefDom
. Configuration passed test CheckSDRefDom

Running partition tests on : kis
Starting test: CrossRefValidation
. kis passed test CrossRefValidation
Starting test: CheckSDRefDom
. kis passed test CheckSDRefDom

Running enterprise tests on : kis.kz
Starting test: Intersite
. kis.kz passed test Intersite
Starting test: FsmoCheck
. kis.kz passed test FsmoCheck

Добавлено:
все время так работало и никогда не было проблем с днс. nslookup отрабатывает нормально ошибок в журнале днс тоже нет. в домене все регатся тоже отлично.никаких траблов.

I had a similar problem to this. What version of openfire are you on? Also, when the file is being transferred, is the file also open?

If you have access to openfire, I would check the logs and see what the error is returning. On ours, I started to think the problem was with the file. I renamed it, stored it somewhere else and away it went. The problem could have been an issue with a corrupted temp file when the file was open.

Also with Windows 7 and above you need to make sure that the downloads folder location in the «settings» of the user receiving it is in an area they have write access to. When I first installed spark on a Windows 7, I found that the downloads folder was sitting under the admin profile instead of the user profile.

Hope this is of some help.

Was this post helpful?
thumb_up
thumb_down

#apache-spark #apache-spark-sql #jupyter-notebook

Вопрос:

Я нахожусь в EMR, получаю данные из каталога клея

когда я пытаюсь передать эти данные и прочитать их через Spark SQL, это выдает мне следующую ошибку:

 Caused by: org.apache.spark.SparkUpgradeException: 
You may get a different result due to the upgrading of Spark 3.0: reading dates before 1582-10-15 or timestamps 
before 1900-01-01T00:00:00Z from Parquet files can be ambiguous, as the files may be written by Spark 2.x or legacy versions of Hive, which uses a legacy hybrid calendar that is different from Spark 3.0 's Proleptic Gregorian calendar. 
See more details in SPARK-31404. You can set spark.sql.legacy.parquet.datetimeRebaseModeInRead to 'LEGACY' to rebase the datetime values w.r.t. the calendar difference during reading. Or set spark.sql.legacy.parquet.datetimeRebaseModeInRead to 'CORRECTED' to read the datetime values as it is.
    at org.apache.spark.sql.execution.datasources.DataSourceUtils$.newRebaseExceptionInRead(DataSourceUtils.scala:159)
    at org.apache.spark.sql.execution.datasources.DataSourceUtils$.$anonfun$creteTimestampRebaseFuncInRead$1(DataSourceUtils.scala:209)
    at org.apache.spark.sql.execution.datasources.parquet.ParquetRowConverter$anon$4.addLong(ParquetRowConverter.scala:330)
    at org.apache.parquet.column.impl.ColumnReaderImpl$2$4.writeValue(ColumnReaderImpl.java:268)
    at org.apache.parquet.column.impl.ColumnReaderImpl.writeCurrentValueToConverter(ColumnReaderImpl.java:367)
    at org.apache.parquet.io.RecordReaderImplementation.read(RecordReaderImplementation.java:406)
    at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:226)
    ... 21 more

Я попытался изменить следующие настройки в spark, но безуспешно

spark.conf.set(«spark.sql.legacy.parquet.datetimeRebaseModeInRead»,»ИСПРАВЛЕНО») и spark.conf.set(«spark.sql.legacy.parquet.datetimeRebaseModeInRead», «НАСЛЕДИЕ»)

Я также сделал выбор в представлении, созданном с помощью следующего кода, и он работал без проблем.

поэтому это заставляет меня думать, что проблема в том, что я использую% sql

почему это происходит? Я делаю что-то не так?

1. Было ли найдено решение этой проблемы?

2. Со своей стороны, код верен, и мы провели тесты с поддержкой aws для видеозвонков. Я собрал билет в службу поддержки Amazon. Он все еще находится на рассмотрении.

Spark Streaming in cluster mode is throwing FileNotFoundException with linux file system (GFS — shared file system across all nodes) but working fine with HDFS as input.

Data is actually available and accessible on this path from all the worker nodes.

JavaPairInputDStream<Text, Text> myDStream =
    jssc.fileStream(path, Text.class, Text.class, customInputFormat.class, new Function<Path, Boolean>() {
      @Override
      public Boolean call(Path v1) throws Exception {
        return Boolean.TRUE;
      }
    }, false);

error message:

14/06/03 21:33:40 WARN FileInputDStream: Error finding new files
java.io.FileNotFoundException: File /data/spark/input does not exist.
        at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:697)
        at org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:105)
        at org.apache.hadoop.hdfs.DistributedFileSystem$15.doCall(DistributedFileSystem.java:755)
        at org.apache.hadoop.hdfs.DistributedFileSystem$15.doCall(DistributedFileSystem.java:751)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:751)
        at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1485)
        at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1525)
        at org.apache.spark.streaming.dstream.FileInputDStream.findNewFiles(FileInputDStream.scala:176)
        at org.apache.spark.streaming.dstream.FileInputDStream.compute(FileInputDStream.scala:134)
        at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1.apply(DStream.scala:300)
        at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1.apply(DStream.scala:300)
        at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
        at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1.apply(DStream.scala:299)
        at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1.apply(DStream.scala:287)
        at scala.Option.orElse(Option.scala:257)
        at org.apache.spark.streaming.dstream.DStream.getOrCompute(DStream.scala:284)
        at org.apache.spark.streaming.dstream.MappedDStream.compute(MappedDStream.scala:35)
        at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1.apply(DStream.scala:300)
        at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1.apply(DStream.scala:300)
        at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
        at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1.apply(DStream.scala:299)
        at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1.apply(DStream.scala:287)
        at scala.Option.orElse(Option.scala:257)
        at org.apache.spark.streaming.dstream.DStream.getOrCompute(DStream.scala:284)
        at org.apache.spark.streaming.dstream.FlatMappedDStream.compute(FlatMappedDStream.scala:35)
        at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1.apply(DStream.scala:300)
        at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1.apply(DStream.scala:300)
        at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
        at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1.apply(DStream.scala:299)
        at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1.apply(DStream.scala:287)
        at scala.Option.orElse(Option.scala:257)
        at org.apache.spark.streaming.dstream.DStream.getOrCompute(DStream.scala:284)
        at org.apache.spark.streaming.dstream.FilteredDStream.compute(FilteredDStream.scala:35)
        at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1.apply(DStream.scala:300)
        at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1.apply(DStream.scala:300)
        at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
        at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1.apply(DStream.scala:299)
        at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1.apply(DStream.scala:287)
        at scala.Option.orElse(Option.scala:257)
        at org.apache.spark.streaming.dstream.DStream.getOrCompute(DStream.scala:284)
        at org.apache.spark.streaming.dstream.MappedDStream.compute(MappedDStream.scala:35)
        at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1.apply(DStream.scala:300)
        at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1.apply(DStream.scala:300)
        at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
        at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1.apply(DStream.scala:299)
        at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1.apply(DStream.scala:287)
        at scala.Option.orElse(Option.scala:257)
        at org.apache.spark.streaming.dstream.DStream.getOrCompute(DStream.scala:284)
        at org.apache.spark.streaming.dstream.FlatMappedDStream.compute(FlatMappedDStream.scala:35)
        at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1.apply(DStream.scala:300)
        at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1.apply(DStream.scala:300)
        at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
        at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1.apply(DStream.scala:299)
        at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1.apply(DStream.scala:287)
        at scala.Option.orElse(Option.scala:257)
        at org.apache.spark.streaming.dstream.DStream.getOrCompute(DStream.scala:284)
        at org.apache.spark.streaming.dstream.ForEachDStream.generateJob(ForEachDStream.scala:38)
        at org.apache.spark.streaming.DStreamGraph$$anonfun$1.apply(DStreamGraph.scala:116)
        at org.apache.spark.streaming.DStreamGraph$$anonfun$1.apply(DStreamGraph.scala:116)
        at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
        at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
        at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
        at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
        at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
        at org.apache.spark.streaming.DStreamGraph.generateJobs(DStreamGraph.scala:116)
        at org.apache.spark.streaming.scheduler.JobGenerator$$anonfun$2.apply(JobGenerator.scala:243)
        at org.apache.spark.streaming.scheduler.JobGenerator$$anonfun$2.apply(JobGenerator.scala:241)
        at scala.util.Try$.apply(Try.scala:161)
        at org.apache.spark.streaming.scheduler.JobGenerator.generateJobs(JobGenerator.scala:241)
        at org.apache.spark.streaming.scheduler.JobGenerator.org$apache$spark$streaming$scheduler$JobGenerator$$processEvent(JobGenerator.scala:177)
        at org.apache.spark.streaming.scheduler.JobGenerator$$anonfun$start$1$$anon$1$$anonfun$receive$1.applyOrElse(JobGenerator.scala:86)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
        at akka.actor.ActorCell.invoke(ActorCell.scala:456)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
        at akka.dispatch.Mailbox.run(Mailbox.scala:219)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
14/06/03 21:33:40 INFO FileInputDStream: New files at time 1433347420000 ms:

Note:
Spark shell works with this shared file system.

How to resolve this issue?

asked Jun 3, 2015 at 16:37

Vijay InnamuriVijay Innamuri

4,2326 gold badges42 silver badges67 bronze badges

JavaPairInputDStream<Text, Text> myDStream =
    jssc.fileStream(path, Text.class, Text.class, customInputFormat.class, new Function<Path, Boolean>() {
      @Override
      public Boolean call(Path v1) throws Exception {
        return Boolean.TRUE;
      }
    }, false);

resolved after directory path is prefixed with file:///

answered Jun 4, 2015 at 10:38

Vijay InnamuriVijay Innamuri

4,2326 gold badges42 silver badges67 bronze badges

My guess is that probably it’s permission problem.

Make sure, when you run the job, you are under the user with sufficient privilege (for master node or the machine you submit the job ) to ssh to worker nodes and r/w/x on worker file system.

answered Jun 4, 2015 at 3:11

Когда вы передаете путь к файлам с помощью —files, они сохраняются в локальном каталоге (временном) для каждого исполнителя. Поэтому, если имена файлов не меняются, вы можете просто использовать их следующим образом вместо использования полного пути, указанного в аргументах.

String connDetailsFile = "SalesforceConn.properties";
String mapFile = "columnMapping.prop";
String sourceColumnsFile = "sourceTableColumns.prop";

Если имена файлов меняются каждый раз, вам нужно удалить путь к файлу и просто использовать имя файла. Это связано с тем, что spark не распознает это как путь, а считает всю строку именем файла.
Например, /home/aiman/SalesforceConn.properties будет считаться именем файла, и искра выдаст вам исключение, сообщающее, что не может найти файл с именем /home/aiman/SalesforceConn.properties.

Итак, ваш код должен быть примерно таким.

String connDetailsFile = args[0].split("/").last
String mapFile = args[1].split("/").last
String sourceColumnsFile = args[2].split("/").last

Источник