[http] Add indirect HTTP GET examples

http/get_indirect/README.md

@@ -22,3 +22,6 @@
 This directory contains examples of HTTP clients and servers that use a two-step sequence to retrieve Arrow data:
 1. The client sends a GET request to a server and receives a JSON response from the server containing one or more server URIs.
 2. The client sends GET requests to each of those URIs and receives a response from each server containing an Arrow IPC stream of record batches (exactly as in the [simple GET examples](https://github.com/apache/arrow-experiments/tree/main/http/get_simple)).
+
+> [!IMPORTANT]  
+> The structure of the JSON document in these examples is an illustration, not a recommendation. Developers should use JSON document structures appropriate to the needs of their applications.

http/get_indirect/curl/.gitignore

@@ -0,0 +1,18 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+*.arrows

http/get_indirect/curl/client/README.md

@@ -0,0 +1,24 @@
+<!---
+  Licensed to the Apache Software Foundation (ASF) under one
+  or more contributor license agreements.  See the NOTICE file
+  distributed with this work for additional information
+  regarding copyright ownership.  The ASF licenses this file
+  to you under the Apache License, Version 2.0 (the
+  "License"); you may not use this file except in compliance
+  with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing,
+  software distributed under the License is distributed on an
+  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  KIND, either express or implied.  See the License for the
+  specific language governing permissions and limitations
+  under the License.
+-->
+
+# HTTP GET Arrow Data: Indirect curl Client Example
+
+This directory contains an example of a series of shell commands that use `curl` and `jq` to:
+1. Send a GET request to the server to get a JSON listing of the URIs of a set of `.arrows` files.
+2. Send GET requests to download each of the `.arrows` files from the server to files in the current directory.

http/get_indirect/curl/client/client.sh

@@ -0,0 +1,28 @@
+!/bin/sh
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+# Use curl to get a JSON document containing URIs to
+# Arrow stream files, then use jq to extract the URIs
+uris=$(curl -s -S http://localhost:8008/ | jq -r '.arrow_stream_files[].uri')
+
+# Use curl to download the files from the URIs in parallel
+if [ -n "$uris" ]; then
+    curl --parallel --remote-name-all $(print $uris)
+fi

http/get_indirect/python/.gitignore

@@ -0,0 +1,18 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+*.arrows

http/get_indirect/python/client/README.md

@@ -0,0 +1,32 @@
+<!---
+  Licensed to the Apache Software Foundation (ASF) under one
+  or more contributor license agreements.  See the NOTICE file
+  distributed with this work for additional information
+  regarding copyright ownership.  The ASF licenses this file
+  to you under the Apache License, Version 2.0 (the
+  "License"); you may not use this file except in compliance
+  with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing,
+  software distributed under the License is distributed on an
+  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  KIND, either express or implied.  See the License for the
+  specific language governing permissions and limitations
+  under the License.
+-->
+
+# HTTP GET Arrow Data: Indirect Python Client Example with Requests
+
+This directory contains an example of an HTTP client implemented in Python using the [Requests](https://requests.readthedocs.io/) library. The client:
+1. Sends a GET request to the server to get a JSON listing of the URIs of available `.arrows` files.
+2. Sends GET requests to download each of the `.arrows` files from the server.
+3. Loads the contents of each file into an in-memory PyArrow Table.
+
+To run this example, first start one of the indirect server examples in the parent directory, then:
+
+```sh
+pip install requests pyarrow
+python client.py
+```

http/get_indirect/python/client/client.py

@@ -0,0 +1,70 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import requests
+import json
+import os
+import pyarrow as pa
+
+
+HOST = "http://localhost:8008/"
+
+JSON_FORMAT = "application/json"
+ARROW_STREAM_FORMAT = "application/vnd.apache.arrow.stream"
+
+json_response = requests.get(HOST)
+
+response_status = json_response.status_code
+if not response_status == 200:
+    raise ValueError(f"Expected response status 200, got {response_status}")
+
+content_type = json_response.headers.get("Content-Type", "")
+if not content_type.startswith(JSON_FORMAT):
+    raise ValueError(f"Expected content type {JSON_FORMAT}, got {content_type}")
+
+print("Downloaded JSON file listing.")
+
+parsed_data = json_response.json()
+uris = [file["uri"] for file in parsed_data["arrow_stream_files"]]
+
+if not all(uri.endswith(".arrows") for uri in uris):
+    raise ValueError(f"Some listed files do not have extension '.arrows'")
+
+print(f"Parsed JSON and found {len(uris)} Arrow stream files.")
+
+tables = {}
+
+for uri in uris:
+    arrow_response = requests.get(uri)
+
+    response_status = arrow_response.status_code
+    if not response_status == 200:
+        raise ValueError(f"Expected response status 200, got {response_status}")
+
+    content_type = arrow_response.headers.get("Content-Type", "")
+    if not content_type.startswith(ARROW_STREAM_FORMAT):
+        raise ValueError(f"Expected content type {ARROW_STREAM_FORMAT}, got {content_type}")
+
+    filename = os.path.basename(uri)
+
+    print(f"Downloaded file '{filename}'.")
+
+    tablename = os.path.splitext(filename)[0]
+    with pa.ipc.open_stream(arrow_response.content) as reader:
+        tables[tablename] = reader.read_all()
+
+    print(f"Loaded into in-memory Arrow table '{tablename}'.")

http/get_indirect/python/server/README.md

@@ -0,0 +1,53 @@
+<!---
+  Licensed to the Apache Software Foundation (ASF) under one
+  or more contributor license agreements.  See the NOTICE file
+  distributed with this work for additional information
+  regarding copyright ownership.  The ASF licenses this file
+  to you under the Apache License, Version 2.0 (the
+  "License"); you may not use this file except in compliance
+  with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing,
+  software distributed under the License is distributed on an
+  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  KIND, either express or implied.  See the License for the
+  specific language governing permissions and limitations
+  under the License.
+-->
+
+# HTTP GET Arrow Data: Indirect Python Server Example
+
+This directory contains an example of an HTTP server implemented in Python using the built-in [`http.server`](https://docs.python.org/3/library/http.server.html) module. The server:
+1. Listens for HTTP GET requests from clients.
+2. Upon receiving a GET request for the document root, serve a JSON document that lists the URIs of all the `.arrows` files in the current directory.
+3. Upon receiving a GET request for a specific `.arrows` file, serve that file.
+
+To run this example, first copy two `.arrows` files from the `data` section of this repository into the current directory:
+
+```sh
+cp ../../../../data/arrow-commits/arrow-commits.arrows .
+cp ../../../../data/rand-many-types/random.arrows .
+```
+
+Then start the HTTP server:
+
+```sh
+python server.py
+```
+
+In this example, the JSON document listing the URIs of the `.arrows` files is structured as shown below. **This JSON structure is provided for example purposes only. It is not a recommendation.** Developers should use JSON document structures appropriate to the needs of their applications.
+
+```json
+{
+    "arrow_stream_files": [
+        {
+            "uri": "http://127.0.0.1:8008/random.arrows"
+        },
+        {
+            "uri": "http://127.0.0.1:8008/arrow-commits.arrows"
+        }
+    ]
+}
+```

http/get_indirect/python/server/server.py

@@ -0,0 +1,53 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from http.server import SimpleHTTPRequestHandler, HTTPServer
+import json
+import os
+import mimetypes
+
+mimetypes.add_type("application/vnd.apache.arrow.stream", ".arrows")
+
+class MyServer(SimpleHTTPRequestHandler):
+    def list_directory(self, path):
+        host, port = self.server.server_address
+
+        try:
+            file_paths = [
+                f for f in os.listdir(path)
+                if f.endswith(".arrows") and os.path.isfile(os.path.join(path, f))
+            ]
+        except OSError:
+            self.send_error(404, "No permission to list directory")
+            return None
+
+        file_uris = [f"http://{host}:{port}{self.path}{f}" for f in file_paths]
+        uris_doc = {"arrow_stream_files": [{"uri": f} for f in file_uris]}
+        self.send_response(200)
+        self.send_header("Content-Type", "application/json")
+        self.end_headers()
+        self.wfile.write(json.dumps(uris_doc, indent=4).encode("utf-8"))
+        return None
+
+server_address = ("localhost", 8008)
+try:
+    httpd = HTTPServer(server_address, MyServer)
+    print(f"Serving on {server_address[0]}:{server_address[1]}...")
+    httpd.serve_forever()
+except KeyboardInterrupt:
+    print("Shutting down server")
+    httpd.socket.close()