Publish to S3 from Emacs

This is an Emacs package that quickly publishes any buffer, region, or file to S3 and the web (AWS S3, DigitalOcean spaces, etc.).

Example: https://org.nyc3.digitaloceanspaces.com/index.html

---

The intended purpose is for easily sharing files on the web, using S3 as a paste-bin service. The object keys of the uploaded files will be renamed with a salted hash of the path + file extension, so the URLs are serviceable, but not really that pretty:

https://my-bucket.region.provider/c6ej4-zk2ah-vvjq8-7w244-9bjxp.txt

This long URL is fine if you’re just copy and pasting the link into a chat message. Since the hash is salted, it is not practical to guess the URL, therefore the contents are only accessible to those whom you share it with.

There is also an option to create an HTML index and RSS feed if you wish to make these URLs discoverable.

Dependencies

This package requires the following external tools:

• s3cmd - Required for all S3 operations.
• s3fs - Optional, required only for mounting S3 buckets as local filesystems.
• fusermount - Optional, required for unmounting s3fs mounts.

These dependencies can be installed via your OS package manager.

Installation

This package is not on MELPA (yet). The package is self-contained in one file: s3-publish.el.

You can declare the dependency in your config with use-package + straight.el, and it will pull it directly from this git repository:

(use-package s3-publish
  :straight
  (s3-publish
   :type git
   :repo "https://github.com/EnigmaCurry/s3-publish.el.git")
   ;;; If you're using general.el, set a keybinding this way:
   ;; :general
   ;; ("C-c p" 's3-publish)
  )

;;; If you're not using general.el, set a keybinding this way:
;;(global-set-key (kbd "C-c p") 's3-publish)

(See general.el for declarative keybindings in use-package)

Configuration

You must configure a profile for each S3 bucket you want to publish to. Run M-x s3-publish-manage-profiles to interactively manage your profiles. Your profiles will be stored in your Emacs’ custom.el file as s3-publish-profiles.

Example: DigitalOcean spaces

Create a bucket:

• Login to your DigitalOcean account at cloud.digitalocean.com
• Go to the Spaces Object Storage page.
• Click Create Bucket.
• Select the desired data center region, e.g. nyc3.
• Choose a unique name for the bucket, e.g. gargleblaster.
• Don’t enable CDN for this basic example.
• Finalize the creation of the bucket.

Create the access key:

• Go to the bucket Settings page.
• Click Create Access Key
• Choose Read/Write/Delete permission.
• List all of the buckets the access key is allowed to use (one by default).
• Enter a descriptive name for the key like mymachine-s3-publish-emacs
• Finalize the creation of the Access Key.
• Copy the Access Key ID and the Secret Access Key. The secret will not be shown again.

Setup s3-publish in Emacs:

• Run M-x s3-publish-manage-profiles.

This will open up a buffer named s3-publish-profiles, which will initially display No s3-publish profiles defined.

In the minibuffer you should see a list of actions you can perform:

• Add
• Remove
• Edit
• Done

Add a new profile:

• Choose the Add action.
• Enter the Profile name:, this can be any name you choose, perhaps the same name as the bucket, e.g. gargleblaster.
• Enter the Endpoint URL, which is listed at the top of the bucket page, e.g. https://gargleblaster.nyc3.digitaloceanspaces.com
• Enter the bucket name (again). e.g. gargleblaster.
• Choose whether you want the files to be made public or not. Make files public? (y or n)

You should now see the s3-publish-profiles buffer has been updated to show your configured profiles. Select Done when finished.

Prepare credentials auth-source

The S3 access keys and secret keys are not stored in the profile, but are accessed through Emacs’ configurable auth-source system.

To create a simple unencrypted auth-source, create a file in your home directory. Create the file ~/.authinfo and enter the credentials for the profile:

machine s3-publish-gargleblaster login XXXXXXXXXXXXXXXXXX secret XXXXXXXXXXXXXXXXXXXX

The profile is matched against the authinfo machine name. Here’s the breakdown on the authinfo syntax:

• The first word is literally machine.
• The second word must start with s3-publish- and then the name of the profile, e.g. gargleblaster.
• The third word is literally login.
• The fifth word is the S3 Access Key credential.
• The sixth word is literally secret
• The seventh word is the S3 Secret Key credential.

Make sure to save the file, and then inform Emacs to reload this file: M-x auth-source-forget-all-cached.

Emacs auth-source supports many different backends that are more secure than this plain authinfo file, including:

• Gnome Keyring
• KDE Wallet
• KeepassXC
• Unix password store

To set these up is left as an exercise to the reader.

Usage

Publish buffer

You can publish your current buffer:

• M-x s3-publish-buffer
• From the list, select the S3 profile destination.
• The buffer will be immediately uploaded to S3.
• The URL will be printed in the minibuffer and copied to kill-ring.

If you chose to allow public access, the URL will start with https://, otherwise it will start with s3://.

The URL should have a file extension that is derived from the buffer name.

The buffer can even contain non-text data, e.g. images. In this case, choose the encoding type raw.

Publish Region

Like publishing buffers, you can publish a select region of text.

• Select a region of text in any buffer.
• M-x s3-publish-region.
• From the list, select the S3 profile destination.
• The region will be published with a .txt extension.
• The URL will be printed in the minibuffer and copied to kill-ring.

Publish Org Buffer

In org-mode you can export the buffer to HTML and publish it in one step:

• M-x s3-publish-org-buffer
• From the list, select the S3 profile destination.
• The buffer will be exported to HTML and immediately uploaded to S3.
• The URL will be printed in the minibuffer and copied to kill-ring.

Publish from dired mode

You can publish multiple files at once from a dired buffer:

• Open any directory in dired-mode.
• Mark multiple files with the m key (directories not supported).
• M-x s3-publish-dired-upload-files.
• All of the marked files will be uploaded to S3.
• All of the URLs will be printed in the minibuffer and copied to kill-ring.
• List of URLs output are separated by newlines.

Publish what I mean

The s3-publish command automatically detects your current editing context and dispatches the appropriate S3 publishing action (Do What I Mean). Its behavior is determined by the following priorities:

• Region selected:

  If you have an active region, the command calls s3-publish-region to upload the selected text to S3.

• In org-mode buffer:
  • Without a prefix argument, it exports your Org buffer to HTML and uploads it using s3-publish-org-buffer.
  • With a universal (prefix) argument (e.g., C-u M-x s3-publish), it removes the file associated with the Org buffer from S3 by calling s3-publish-remove-file.
• In dired buffer:
  • Without a prefix argument, it uploads the marked files using s3-publish-dired-upload-files.
  • With a prefix argument, it removes the marked files from S3 using s3-publish-dired-remove-files.
• Other buffers:
  • Without a prefix argument, the entire buffer is published using s3-publish-buffer.
  • With a prefix argument, it removes the file backing the current buffer from S3 by invoking s3-publish-remove-file.

To make this functionality even more accessible, you can bind it to a convenient key sequence. For example, to bind it to C-c p, add the following to your Emacs configuration:

(global-set-key (kbd "C-c p") 's3-publish)

This key binding lets you quickly invoke the context-sensitive publishing command from anywhere within Emacs.

Updating files

The S3 keys and public URLs use a filename that is a hash of the local path of the file. This means that if you upload the same file twice, it will keep the same key/URL. However, there are two exceptions:

• s3-publish-buffer
• s3-publish-region

Both of these create temporary files with random names in order to export them. Therefore these will always make new S3 keys.

Removing URLs

You can remove the objects by supplying the URLs:

• M-x s3-publish-remove-urls

You may enter URLs to delete, one per line.

You may also remove files from dired mode:

• Mark files in dired mode buffer.
• M-x s3-publish-dired-remove-files

This will delete the remote objects for the given files (if they exist). It will not remove them locally. If the HTML index and RSS feed is enabled, the URL will be removed from the feed as well.

Lifecycle

You can set the S3 lifecycle of the bucket to let objects expire automatically. You will usually need an increased API permission level to perform this action (e.g. on DigitalOcean spaces you must select the All permissions option, which gives full access to ALL your buckets.)

• M-x s3-publish-bucket-lifecycle
• Select the profile to manage.
• Enter the number of days that objects may live before expiration. e.g., 1, 10, or leave blank to delete the lifecycle config.

If you don’t have adequate permission, you will see the error in the mini buffer.

Mount S3 buckets as filesystems

You can mount your S3 bucket as a local filesystem using s3fs:

• M-x s3-publish-s3fs-mount
• Select the S3 profile to mount
• Enter the local directory where you want to mount the bucket
• The bucket will be mounted at the specified location
• A dired buffer will open showing the contents of the mounted bucket

To unmount the bucket:

• M-x s3-publish-s3fs-unmount
• Select the mounted directory
• The mount will be removed and any associated dired buffers closed

This feature requires the s3fs command-line tool to be installed on your system.

Alternative credentials for admin operations

Some operations like setting bucket lifecycle policies or configuring website hosting require elevated permissions. For these operations, s3-publish can prompt you to select alternative credentials.

You can add additional credentials in your auth source with different machine names, and s3-publish will present them as options when performing operations that may require higher privileges.

For example, you might have regular credentials for normal uploads:

machine s3-publish-gargleblaster login REGULAR_ACCESS_KEY secret REGULAR_SECRET_KEY

And separate admin credentials with full permissions:

machine s3-publish-admin login ADMIN_ACCESS_KEY secret ADMIN_SECRET_KEY

When performing administrative operations, s3-publish will give you the option to select an alternative credential from your auth source.

License

This software is distributed under the 0BSD license:

Permission to use, copy, modify, and/or distribute this software for
any purpose with or without fee is hereby granted.

THE SOFTWARE IS PROVIDED “AS IS” AND THE AUTHOR DISCLAIMS ALL
WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES
OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE
FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY
DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN
AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT
OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.