Skip to content

Serialization ipc by golang #9719

Description

@hunght3101

When i serialization ipc by golang then i got error, please help me!
arrow/ipc: could not read schema from stream: arrow/ipc: could not read message schema: arrow/ipc: could not read message metadata: unexpected EOF
Cause, i think it is

var (
	cid    = binary.LittleEndian.Uint32(buf)
	msgLen int32
	)
...
msgLen = int32(cid)
...
buf = make([]byte, msgLen)
_, err = io.ReadFull(r.r, buf)
if err != nil {
	return nil, xerrors.Errorf("arrow/ipc: could not read message metadata: %w", err)
}

With msgLen very very big it export error
My source

import (
	"fmt"
	"github.com/apache/arrow/go/arrow"
	"github.com/apache/arrow/go/arrow/array"
	"github.com/apache/arrow/go/arrow/ipc"
	"github.com/apache/arrow/go/arrow/memory"
	"io"
	"strings"
)
func main()  {

	meta := arrow.NewMetadata([]string{"k1", "k2"}, []string{"v1", "v2"})
	mem := memory.NewGoAllocator()
	schema := arrow.NewSchema(
		[]arrow.Field{
			arrow.Field{Name: "f1-i32", Type: arrow.BinaryTypes.String},
			arrow.Field{Name: "f2-f64", Type: arrow.BinaryTypes.String},
		},
		&meta,
	)

	b := array.NewRecordBuilder(mem, schema)
	defer b.Release()

	b.Field(0).(*array.StringBuilder).AppendValues(
		[]string{"1", "2"},
		nil,
	)
	b.Field(0).(*array.StringBuilder).AppendValues(
		[]string{"1", "2","1", "2"},
		[]bool{true, true, false, true},
	)
	b.Field(1).(*array.StringBuilder).AppendValues(
		[]string{"1", "2","1", "2","1", "2","1", "2","1", "2"},
		nil,
	)

	var r io.Reader
	r = strings.NewReader("Object Apache Arrow")
	rr, err := ipc.NewReader(r, ipc.WithSchema(schema), ipc.WithAllocator(mem))
	if err != nil {
		fmt.Println(err)
	}
	fmt.Println(rr)
}

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions