Scala/Spark ClassCastException when accesing list element

I am new to spark/scala and am trying to work with(specifically average a subset of values from) data returned by a method

def getRDDForTask(sc: SparkContext, taskName: String, attributeName: String, numberOfPoints: Int): RDD[List[BigDecimal]] = { val Stats=getRDDForTask(sc, "example_name", "memory_usage", 200) val initList=Stats.toArray.head val initValue=initList.head }

Calling Stats.toArray.head returns: res100: List[BigDecimal] = List(1429505205000, 68400001, 210800640)

However, when I call initList.head, to access the first element of the List, I get an error. This message occurs when I try to run any operation (e.g. foreach or map) on values of the list. I've tried creating List[BigDecimal] from scratch and can do all standard operations on it. Am I missing something obvious? The specific error is:

java.lang.ClassCastException: scala.math.BigInt cannot be cast to scala.math.BigDecimal at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:42) at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:47) at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:49) at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:51) at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:53) at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:55) at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:57) at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:59) at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:61) at $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:63) at $iwC$$iwC$$iwC$$iwC.<init>(<console>:65) at $iwC$$iwC$$iwC.<init>(<console>:67) at $iwC$$iwC.<init>(<console>:69) at $iwC.<init>(<console>:71) at <init>(<console>:73) at .<init>(<console>:77) at .<clinit>(<console>) at .<init>(<console>:7) at .<clinit>(<console>) at $print(<console>) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke( at sun.reflect.DelegatingMethodAccessorImpl.invoke( at java.lang.reflect.Method.invoke( at org.apache.spark.repl.SparkIMain$ at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1338) at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840) at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871) at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819) at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:856) at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:901) at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:813) at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:656) at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:664) at$apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:669) at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:996) at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944) at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944) at$.savingContextLoader(ScalaClassLoader.scala:135) at$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:944) at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1058) at org.apache.spark.repl.Main$.main(Main.scala:31) at org.apache.spark.repl.Main.main(Main.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke( at sun.reflect.DelegatingMethodAccessorImpl.invoke( at java.lang.reflect.Method.invoke( at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Category:java Views:1 Time:2018-02-10

Related post

  • Executing code when a list elements become active 2011-11-17

    I have the following HTML: <ul class="tabs"> <li id="taba" class="active"><a href="#a">A</a></li> <li><a id="tabb" href="#b">B</a></li> <li><a id="tabc" href="#c">C</a></li

  • Scala - how to get a list element 2011-01-25

    I'm trying to get element from a list: data =List(("2001",13.1),("2009",3.1),("2004",24.0),("2011",1.11)) Any help? Task is to print separatly Strings and numbers like: print(x._1+" "+x._2) but this is not working. --------------Solutions------------

  • run a function when a user clicks on any list element 2010-12-28

    This is probably a very common question, but I was unable to find an answer myself; All my list elements call the function setQuery like this onClick="admin_stats.setQuery(this);" rather than [hardcode] add this to every list element, is there a way

  • jquery ui dialog: possible to create a list element on close? 2011-03-31

    When I close the dialog, I want a list element to be made on the fly. When the list element is clicked, the dialog is opened again. Is this possible? $('#dialog').dialog({ close: function() { var e = $(this).parent().find('.ui-dialog-title').text();

  • C++: how to track a pointer to a STL list element? 2011-07-23

    I would like to track a pointer to a list element for the next read access. This pointer would be advanced every time the list is read. Would it be bad practice to cache an iterator to the list as a member variable, assuming I take proper precautions

  • Getting a ConcurrentModificationException thrown when removing an element from a java.util.List during list iteration? 2011-02-25

    @Test public void testListCur(){ List<String> li=new ArrayList<String>(); for(int i=0;i<10;i++){ li.add("str"+i); } for(String st:li){ if(st.equalsIgnoreCase("str3")) li.remove("str3"); } System.out.println(li); } When I run this code,

  • When shall I use list-element in HTML 2011-06-11

    I have never used list before. I have created a website with over 30 000 LOC and never use a single list-element. But when I look at the source code of other say Twitter they use list all the time. What is the advantage of using list-elements? Most o

  • Re-usable dropdown list when clicking various elements, what's the technique? 2011-07-16

    Say I have various parts of a page, that when clicked, should display a dropdown list below the link I just clicked on. Since this drop down is the exact same, I noticed that some sites do this, where they have a single drop down list defined in the

  • Add dynamic elements with jquery when I choose element from list 2011-10-27

    I have list of elements and when I choose element from list, "value" from code bellow are changing. <div class="drop-down-cont zi-5 fleft"> <input class="dd-input" id="packaging_item_packaging_name" name="packaging_item[packaging_name]" titl

  • Python: Memory usage and optimization when modifying lists 2010-04-13

    The problem My concern is the following: I am storing a relativity large dataset in a classical python list and in order to process the data I must iterate over the list several times, perform some operations on the elements, and often pop an item ou

  • List Element not completely centered? 2012-03-29

    I am trying to create a simple web site that will scale based on the resolution of the device being used. I want it to scale based on whether it is viewed from a mobile device or a desktop monitor. I am using percentage width and height to size the c

  • ASP.NET/DataList: Client-side postback by clicking list elements 2008-12-29

    I have a DataList control that displays a set of elements. Can anyone point me in the right direction on how I can add some client-side functionality for submitting/doing a postback when the user clicks an element in the list (e.g. anywhere in the th

  • Directly modifying List elements 2009-01-06

    I have this struct: struct Map { public int Size; public Map ( int size ) { this.Size = size; } public override string ToString ( ) { return String.Format ( "Size: {0}", this.Size ); } } When using array, it works: Map [ ] arr = new Map [ 4 ] { new M

  • jQuery selectors for list elements 2009-02-22

    I am using jQuery and jQuery UI. Using the getJSON function, we are creating list elements and appending them to an OL element. Here's the code: $.getJSON("http://localhost/b/t.php", function(data){ $.each(data, function(i, thet){ //alert(thet.fname)

  • Is it possible to include block elements in un-ordered list elements horizontally laid out 2009-04-01

    I have a list of items that I want to be laid out horizontally. It's basically a row of pictures with captions. Something like picture1 picture2 picture3 caption1 caption2 caption3 So I figured that was a unordered list with List elements styled with

  • Can I configure the DataContractSerializer to not create optional (i.e. Nullable and List) elements in output XML? 2009-04-01

    I am using the new .NET 3.0 DataContractSerializer. I have both Nullable<> and List<> objects I am going to serialize. Example: [DataContract(Namespace = "")] class Test { public static void Go() { Test test = new Test(); var dcs = new Da

  • Using jQuery to limit the number of list elements 2009-06-20

    I have a list element containing a number of between 20-30 events. I only want to show 5 of those, and have a «More» link I can click to watch the the next five. It doesn't matter if the five next events are appended. Can't find a jQuery script that

  • jQuery sortable and AJAX question. (sortable not working when the list comes from AJAX) 2009-10-14

    I have 2 sortable lists. The first one is getting the elements with AJAX from database (like pagination, I do not want 200 item at once on that list). The second one is empty at first and I want to fill it with elements from the first. I'm very new t

  • ClassCastException when eclipse "fixed" my code 2009-10-19

    I get a ClassCastException when eclipse suggested that my code should be like that.. I have a class named Kort. ArrayList<Kort> kort = new ArrayList<Kort>(); then I use toArray(), and eclipse suggest it should look like: Kort[] array = (K

  • jQuery return position of list element with class on table column click 2010-01-25

    I was wondering how to return the position of a list element with a class of 'currentTab' in the following unordered list: <ul id="coverTabs"> <li class="currentTab"><a href="#">Individual</a></li> <li><a href="

  • Creating a javascript function to switch classes between list elements depending on which one has been clicked 2010-05-12

    I am trying to create a function to change the class on a series of 5 list elements depending on which has been clicked. The goal is to change the style of the element depending on which one is the current element. I had tried something like this: fu

  • Javascript function to add class to a list element based on # in url 2010-05-20

    I am trying to create a javascript function to add and remove a class to a list element based on the #tag at the end of the url on a page. The page has several different states, each with a different # in the url. I am currently using this script to

  • How to know when an DOM element moves or is resized 2010-08-09

    I'd like to know if there is a DOM event to listen for that tells me when a DOM element moves is repositioned or is resized on the page. I am not talking about an elements being drag/dropped. An example is if a have a list of three items on a page, a

  • Style the select list element in jEditable 2010-08-13

    How do I style the select list element? Here's my code as it is now... $('.editableSelect').editable(function(value, settings) { if (this.revert == value) { this.reset(); return; } setSaveButtonRed(this); setDescFromAccountCode(this, value); return (

Copyright (C), All Rights Reserved.

processed in 0.190 (s). 11 q(s)