Skip to content

LoadStatistics stats = job.getStatistics() yields java.lang.ClassCastException #2652

@kansuke231

Description

@kansuke231

Hi all,

I am trying to insert some json data into a BigQuery table with Spark. Here is a snippet of code that is supposed to be doing the insertion:

        ...
        TableId tableId = TableId.of(dataset_id, table_id);
        BigQuery bigQuery = createAuthorizedClient(credential_json, project_id);

        WriteChannelConfiguration writeChannelConfiguration =
                WriteChannelConfiguration.newBuilder(tableId)
                        .setFormatOptions(FormatOptions.json())
                        .build();
        TableDataWriteChannel writer = bigQuery.writer(writeChannelConfiguration);

        ArrayList<StatementResult> al = new ArrayList<StatementResult>();

        String jsonString = "";
        long count = 0;

        // concatenate all rows into one string.
        for (Iterator<String> iter = stringIterator; iter.hasNext();) {
            count++;
            jsonString += "\n" + iter.next();
        }

        try {

            if (count == 0) {
                StatementResult sr = new StatementResult("", 0,0);
                al.add(sr);
                return al.iterator();
            }

            writer = bigQuery.writer(writeChannelConfiguration);
            writer.write(ByteBuffer.wrap(jsonString.getBytes(Charsets.UTF_8)));

            writer.close();
            // Get load job
            Job job = writer.getJob();

            job = job.waitFor();

            LoadStatistics stats = job.getStatistics();
            long success =  stats.getOutputRows();

            StatementResult sr = new StatementResult(null, success,count - success);
            al.add(sr);


        }  catch (IOException | InterruptedException | TimeoutException e) {

            StatementResult sr = new StatementResult(e.getMessage(), 1,1);
            al.add(sr);


        } finally {
            writer.close();
        }
        return al.iterator();

However, the lineLoadStatistics stats = job.getStatistics(); yields an exception of java.lang.ClassCastException: shaded.guava.cloud.bigquery.JobStatistics$CopyStatistics cannot be cast to shaded.guava.cloud.bigquery.JobStatistics$LoadStatistics. The code structure is pretty much the same as examples I found online, so I don't know what's wrong with this. Any help would be highly appreciated. Thanks!

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions