Skip to content

Commit 53ea9fe

Browse files
hawkwhawkw
committed
Move length_delimited into tokio instead of tokio-codec
Signed-off-by: Eliza Weisman <eliza@buoyant.io>
1 parent 5be4b0e commit 53ea9fe

File tree

4 files changed

+346
-342
lines changed

4 files changed

+346
-342
lines changed

‎Cargo.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,7 @@ travis-ci = { repository = "tokio-rs/tokio" }
4242
appveyor = { repository = "carllerche/tokio", id = "s83yxhy9qeb58va7" }
4343

4444
[dependencies]
45+
bytes = "0.4"
4546
tokio-codec = { version = "0.1.0", path = "tokio-codec" }
4647
tokio-current-thread = { version = "0.1.0", path = "tokio-current-thread" }
4748
tokio-io = { version = "0.1.6", path = "tokio-io" }
@@ -62,7 +63,6 @@ mio = "0.6.14"
6263
tokio-uds = { version = "0.2.0", path = "tokio-uds" }
6364

6465
[dev-dependencies]
65-
bytes = "0.4"
6666
env_logger = { version = "0.5", default-features = false }
6767
flate2 = { version = "1", features = ["tokio"] }
6868
futures-cpupool = "0.1"

tokio-codec/src/length_delimited.rs renamed to src/length_delimited.rs

Lines changed: 0 additions & 338 deletions
Original file line numberDiff line numberDiff line change
@@ -1,341 +1,3 @@
1-
//! Frame a stream of bytes based on a length prefix
2-
//!
3-
//! Many protocols delimit their frames by prefacing frame data with a
4-
//! frame head that specifies the length of the frame. The
5-
//! `length_delimited` module provides utilities for handling the length
6-
//! based framing. This allows the consumer to work with entire frames
7-
//! without having to worry about buffering or other framing logic.
8-
//!
9-
//! # Getting started
10-
//!
11-
//! If implementing a protocol from scratch, using length delimited framing
12-
//! is an easy way to get started. [`Framed::new()`] will adapt a
13-
//! full-duplex byte stream with a length delimited framer using default
14-
//! configuration values.
15-
//!
16-
//! ```
17-
//! use tokio_io::{AsyncRead, AsyncWrite};
18-
//! use tokio_io::codec::length_delimited;
19-
//!
20-
//! fn bind_transport<T: AsyncRead + AsyncWrite>(io: T)
21-
//! -> length_delimited::Framed<T>
22-
//! {
23-
//! length_delimited::Framed::new(io)
24-
//! }
25-
//! ```
26-
//!
27-
//! The returned transport implements `Sink + Stream` for `BytesMut`. It
28-
//! encodes the frame with a big-endian `u32` header denoting the frame
29-
//! payload length:
30-
//!
31-
//! ```text
32-
//! +----------+--------------------------------+
33-
//! | len: u32 | frame payload |
34-
//! +----------+--------------------------------+
35-
//! ```
36-
//!
37-
//! Specifically, given the following:
38-
//!
39-
//! ```
40-
//! # extern crate tokio_io;
41-
//! # extern crate bytes;
42-
//! # extern crate futures;
43-
//! #
44-
//! use tokio_io::{AsyncRead, AsyncWrite};
45-
//! use tokio_io::codec::length_delimited;
46-
//! use bytes::BytesMut;
47-
//! use futures::{Sink, Future};
48-
//!
49-
//! fn write_frame<T: AsyncRead + AsyncWrite>(io: T) {
50-
//! let mut transport = length_delimited::Framed::new(io);
51-
//! let frame = BytesMut::from("hello world");
52-
//!
53-
//! transport.send(frame).wait().unwrap();
54-
//! }
55-
//! #
56-
//! # pub fn main() {}
57-
//! ```
58-
//!
59-
//! The encoded frame will look like this:
60-
//!
61-
//! ```text
62-
//! +---- len: u32 ----+---- data ----+
63-
//! | \x00\x00\x00\x0b | hello world |
64-
//! +------------------+--------------+
65-
//! ```
66-
//!
67-
//! # Decoding
68-
//!
69-
//! [`FramedRead`] adapts an [`AsyncRead`] into a `Stream` of [`BytesMut`],
70-
//! such that each yielded [`BytesMut`] value contains the contents of an
71-
//! entire frame. There are many configuration parameters enabling
72-
//! [`FramedRead`] to handle a wide range of protocols. Here are some
73-
//! examples that will cover the various options at a high level.
74-
//!
75-
//! ## Example 1
76-
//!
77-
//! The following will parse a `u16` length field at offset 0, including the
78-
//! frame head in the yielded `BytesMut`.
79-
//!
80-
//! ```
81-
//! # use tokio_io::AsyncRead;
82-
//! # use tokio_io::codec::length_delimited;
83-
//! # fn bind_read<T: AsyncRead>(io: T) {
84-
//! length_delimited::Builder::new()
85-
//! .length_field_offset(0) // default value
86-
//! .length_field_length(2)
87-
//! .length_adjustment(0) // default value
88-
//! .num_skip(0) // Do not strip frame header
89-
//! .new_read(io);
90-
//! # }
91-
//! ```
92-
//!
93-
//! The following frame will be decoded as such:
94-
//!
95-
//! ```text
96-
//! INPUT DECODED
97-
//! +-- len ---+--- Payload ---+ +-- len ---+--- Payload ---+
98-
//! | \x00\x0B | Hello world | --> | \x00\x0B | Hello world |
99-
//! +----------+---------------+ +----------+---------------+
100-
//! ```
101-
//!
102-
//! The value of the length field is 11 (`\x0B`) which represents the length
103-
//! of the payload, `hello world`. By default, [`FramedRead`] assumes that
104-
//! the length field represents the number of bytes that **follows** the
105-
//! length field. Thus, the entire frame has a length of 13: 2 bytes for the
106-
//! frame head + 11 bytes for the payload.
107-
//!
108-
//! ## Example 2
109-
//!
110-
//! The following will parse a `u16` length field at offset 0, omitting the
111-
//! frame head in the yielded `BytesMut`.
112-
//!
113-
//! ```
114-
//! # use tokio_io::AsyncRead;
115-
//! # use tokio_io::codec::length_delimited;
116-
//! # fn bind_read<T: AsyncRead>(io: T) {
117-
//! length_delimited::Builder::new()
118-
//! .length_field_offset(0) // default value
119-
//! .length_field_length(2)
120-
//! .length_adjustment(0) // default value
121-
//! // `num_skip` is not needed, the default is to skip
122-
//! .new_read(io);
123-
//! # }
124-
//! ```
125-
//!
126-
//! The following frame will be decoded as such:
127-
//!
128-
//! ```text
129-
//! INPUT DECODED
130-
//! +-- len ---+--- Payload ---+ +--- Payload ---+
131-
//! | \x00\x0B | Hello world | --> | Hello world |
132-
//! +----------+---------------+ +---------------+
133-
//! ```
134-
//!
135-
//! This is similar to the first example, the only difference is that the
136-
//! frame head is **not** included in the yielded `BytesMut` value.
137-
//!
138-
//! ## Example 3
139-
//!
140-
//! The following will parse a `u16` length field at offset 0, including the
141-
//! frame head in the yielded `BytesMut`. In this case, the length field
142-
//! **includes** the frame head length.
143-
//!
144-
//! ```
145-
//! # use tokio_io::AsyncRead;
146-
//! # use tokio_io::codec::length_delimited;
147-
//! # fn bind_read<T: AsyncRead>(io: T) {
148-
//! length_delimited::Builder::new()
149-
//! .length_field_offset(0) // default value
150-
//! .length_field_length(2)
151-
//! .length_adjustment(-2) // size of head
152-
//! .num_skip(0)
153-
//! .new_read(io);
154-
//! # }
155-
//! ```
156-
//!
157-
//! The following frame will be decoded as such:
158-
//!
159-
//! ```text
160-
//! INPUT DECODED
161-
//! +-- len ---+--- Payload ---+ +-- len ---+--- Payload ---+
162-
//! | \x00\x0D | Hello world | --> | \x00\x0D | Hello world |
163-
//! +----------+---------------+ +----------+---------------+
164-
//! ```
165-
//!
166-
//! In most cases, the length field represents the length of the payload
167-
//! only, as shown in the previous examples. However, in some protocols the
168-
//! length field represents the length of the whole frame, including the
169-
//! head. In such cases, we specify a negative `length_adjustment` to adjust
170-
//! the value provided in the frame head to represent the payload length.
171-
//!
172-
//! ## Example 4
173-
//!
174-
//! The following will parse a 3 byte length field at offset 0 in a 5 byte
175-
//! frame head, including the frame head in the yielded `BytesMut`.
176-
//!
177-
//! ```
178-
//! # use tokio_io::AsyncRead;
179-
//! # use tokio_io::codec::length_delimited;
180-
//! # fn bind_read<T: AsyncRead>(io: T) {
181-
//! length_delimited::Builder::new()
182-
//! .length_field_offset(0) // default value
183-
//! .length_field_length(3)
184-
//! .length_adjustment(2) // remaining head
185-
//! .num_skip(0)
186-
//! .new_read(io);
187-
//! # }
188-
//! ```
189-
//!
190-
//! The following frame will be decoded as such:
191-
//!
192-
//! ```text
193-
//! INPUT
194-
//! +---- len -----+- head -+--- Payload ---+
195-
//! | \x00\x00\x0B | \xCAFE | Hello world |
196-
//! +--------------+--------+---------------+
197-
//!
198-
//! DECODED
199-
//! +---- len -----+- head -+--- Payload ---+
200-
//! | \x00\x00\x0B | \xCAFE | Hello world |
201-
//! +--------------+--------+---------------+
202-
//! ```
203-
//!
204-
//! A more advanced example that shows a case where there is extra frame
205-
//! head data between the length field and the payload. In such cases, it is
206-
//! usually desirable to include the frame head as part of the yielded
207-
//! `BytesMut`. This lets consumers of the length delimited framer to
208-
//! process the frame head as needed.
209-
//!
210-
//! The positive `length_adjustment` value lets `FramedRead` factor in the
211-
//! additional head into the frame length calculation.
212-
//!
213-
//! ## Example 5
214-
//!
215-
//! The following will parse a `u16` length field at offset 1 of a 4 byte
216-
//! frame head. The first byte and the length field will be omitted from the
217-
//! yielded `BytesMut`, but the trailing 2 bytes of the frame head will be
218-
//! included.
219-
//!
220-
//! ```
221-
//! # use tokio_io::AsyncRead;
222-
//! # use tokio_io::codec::length_delimited;
223-
//! # fn bind_read<T: AsyncRead>(io: T) {
224-
//! length_delimited::Builder::new()
225-
//! .length_field_offset(1) // length of hdr1
226-
//! .length_field_length(2)
227-
//! .length_adjustment(1) // length of hdr2
228-
//! .num_skip(3) // length of hdr1 + LEN
229-
//! .new_read(io);
230-
//! # }
231-
//! ```
232-
//!
233-
//! The following frame will be decoded as such:
234-
//!
235-
//! ```text
236-
//! INPUT
237-
//! +- hdr1 -+-- len ---+- hdr2 -+--- Payload ---+
238-
//! | \xCA | \x00\x0B | \xFE | Hello world |
239-
//! +--------+----------+--------+---------------+
240-
//!
241-
//! DECODED
242-
//! +- hdr2 -+--- Payload ---+
243-
//! | \xFE | Hello world |
244-
//! +--------+---------------+
245-
//! ```
246-
//!
247-
//! The length field is situated in the middle of the frame head. In this
248-
//! case, the first byte in the frame head could be a version or some other
249-
//! identifier that is not needed for processing. On the other hand, the
250-
//! second half of the head is needed.
251-
//!
252-
//! `length_field_offset` indicates how many bytes to skip before starting
253-
//! to read the length field. `length_adjustment` is the number of bytes to
254-
//! skip starting at the end of the length field. In this case, it is the
255-
//! second half of the head.
256-
//!
257-
//! ## Example 6
258-
//!
259-
//! The following will parse a `u16` length field at offset 1 of a 4 byte
260-
//! frame head. The first byte and the length field will be omitted from the
261-
//! yielded `BytesMut`, but the trailing 2 bytes of the frame head will be
262-
//! included. In this case, the length field **includes** the frame head
263-
//! length.
264-
//!
265-
//! ```
266-
//! # use tokio_io::AsyncRead;
267-
//! # use tokio_io::codec::length_delimited;
268-
//! # fn bind_read<T: AsyncRead>(io: T) {
269-
//! length_delimited::Builder::new()
270-
//! .length_field_offset(1) // length of hdr1
271-
//! .length_field_length(2)
272-
//! .length_adjustment(-3) // length of hdr1 + LEN, negative
273-
//! .num_skip(3)
274-
//! .new_read(io);
275-
//! # }
276-
//! ```
277-
//!
278-
//! The following frame will be decoded as such:
279-
//!
280-
//! ```text
281-
//! INPUT
282-
//! +- hdr1 -+-- len ---+- hdr2 -+--- Payload ---+
283-
//! | \xCA | \x00\x0F | \xFE | Hello world |
284-
//! +--------+----------+--------+---------------+
285-
//!
286-
//! DECODED
287-
//! +- hdr2 -+--- Payload ---+
288-
//! | \xFE | Hello world |
289-
//! +--------+---------------+
290-
//! ```
291-
//!
292-
//! Similar to the example above, the difference is that the length field
293-
//! represents the length of the entire frame instead of just the payload.
294-
//! The length of `hdr1` and `len` must be counted in `length_adjustment`.
295-
//! Note that the length of `hdr2` does **not** need to be explicitly set
296-
//! anywhere because it already is factored into the total frame length that
297-
//! is read from the byte stream.
298-
//!
299-
//! # Encoding
300-
//!
301-
//! [`FramedWrite`] adapts an [`AsyncWrite`] into a `Sink` of [`BytesMut`],
302-
//! such that each submitted [`BytesMut`] is prefaced by a length field.
303-
//! There are fewer configuration options than [`FramedRead`]. Given
304-
//! protocols that have more complex frame heads, an encoder should probably
305-
//! be written by hand using [`Encoder`].
306-
//!
307-
//! Here is a simple example, given a `FramedWrite` with the following
308-
//! configuration:
309-
//!
310-
//! ```
311-
//! # extern crate tokio_io;
312-
//! # extern crate bytes;
313-
//! # use tokio_io::AsyncWrite;
314-
//! # use tokio_io::codec::length_delimited;
315-
//! # use bytes::BytesMut;
316-
//! # fn write_frame<T: AsyncWrite>(io: T) {
317-
//! # let _: length_delimited::FramedWrite<T, BytesMut> =
318-
//! length_delimited::Builder::new()
319-
//! .length_field_length(2)
320-
//! .new_write(io);
321-
//! # }
322-
//! # pub fn main() {}
323-
//! ```
324-
//!
325-
//! A payload of `hello world` will be encoded as:
326-
//!
327-
//! ```text
328-
//! +- len: u16 -+---- data ----+
329-
//! | \x00\x0b | hello world |
330-
//! +------------+--------------+
331-
//! ```
332-
//!
333-
//! [`FramedRead`]: struct.FramedRead.html
334-
//! [`FramedWrite`]: struct.FramedWrite.html
335-
//! [`AsyncRead`]: ../../trait.AsyncRead.html
336-
//! [`AsyncWrite`]: ../../trait.AsyncWrite.html
337-
//! [`Encoder`]: ../trait.Encoder.html
338-
//! [`BytesMut`]: https://docs.rs/bytes/0.4/bytes/struct.BytesMut.html
3391
#![allow(deprecated)]
3402

3413
use tokio_io::{codec, AsyncRead, AsyncWrite};

0 commit comments

Comments
 (0)