51.0.2704.106-1

Update README.md and BUILDING.md with up-to-date information
Add simple build.py for Debian

Author: Eloston

BUILDING.md

@@ -2,34 +2,30 @@
 
 **Notice for master branch users**: The information in this document may not apply to the latest tag version. Please consult the documentation from the tag instead.
 
-ungoogled-chromium provides scripts to automate the downloading, patching, and building of Chromium with these patches. Here's an overview of the building scripts and files:
-* `build-sandbox` - This directory is the build sandbox; the container for all files used and generated during building. It is created when the building environment is setup.
-* `build_templates` - This holds the system-dependent files that are used for compiling and generating a binary package. They are copied into the build sandbox by the build setup scripts.
-  * `debian` - This contains files to generate a dpkg debian directory. Debian-specific patches are located here.
-  * `ubuntu` - This contains files to generate a dpkg debian directory for Ubuntu
-* `download_source.sh` - This script downloads the source tarball from `commondatastorage.googleapis.com` and unpacks it into the build sandbox.
-  * It accepts arguments; pass in `-h` for more information.
-* `generate_domain_substitution_list.sh` - This script generates a list of files containing domain names to be replaced
-  * It should be run at the root of the build sandbox.
-  * This script is not needed during the normal build process
-* `domain_substitution_list` - This file is generated by `generate_domain_substitution_list.sh` for a specific version of Chromium
-  * It assumes that the cleaning list has been applied
-* `domain_regex_list` - The list of regular expressions used to substitute domain names
-* `evaluate_domain_substitution_list.py` - This script evaluates the domain substitution list in the current working directory
-* `generate_cleaning_list.sh` - This script generates a file list of mostly all binary files to be stripped from the source tree.
-  * It should be run at the root of the build sandbox.
-  * This script is not needed during the normal build process
-* `cleaning_list` - This file is generated by `generate_cleaning_list.sh` for a specific version of Chromium
-* `evaluate_cleaning_list.py` - This script evaluates the cleaning list in the current working directory
-* `generate_debian_scripts.sh` - This script creates a dpkg debian directory in the build sandbox for Debian
-* `generate_ubuntu_scripts.sh` - This script creates a dpkg debian directory in the build sandbox for Ubuntu
-* `build_debian.sh` - This is the main build script for Debian and derivative distributions. It handles the downloading, patching, and building of .deb packages.
-  * This script will invoke the other scripts to do certain tasks.
-  * It accepts arguments; pass in `-h` for more information.
+## The building system
+
+ungoogled-chromium provides a flexible and extensible Python library called [`buildlib`](buildlib) that does source code downloading, source cleaning, domain substitution, patching, building, and packaging. There's no documentation on `buildlib` yet, but it's pretty straight-forward to use. [See `build.py`](build.py) for an example on using `buildlib`.
+
+Currently, there is no command-line-configurable build script. You must create a script to use `buildlib`.
+
+## General building requirements
+
+The following is needed to fully use `buildlib`:
+* Python 3 (tested on 3.5) for running `buildlib`
+* The below can be provided by [Google's depot_tools](https://www.chromium.org/developers/how-tos/install-depot-tools)
+    * Python 2 (tested on 2.7) for running gyp
+    * [Jinja2](http://jinja.pocoo.org/) for running gyp
+    * [Ninja](https://ninja-build.org/) for running the build command
+
+There are additional requirements for specific platforms. See the following sections for more information.
+
+## How to build
+
+Building is done by simply invoking a Python script like `build.py`. It will take care of the setup and building processes. See the following for more information.
 
 ## Debian and derivatives
 
-Distributions supported: Debian Jessie 64-bit and Stretch 64-bit, and Ubuntu Xenial 64-bit
+As of now, only Debian Stretch 64-bit is tested. Ubuntu Xenial 64-bit support will come soon.
 This may work on other Debian-based distributions and 32-bit systems
 
 **Note for Debian Jessie users**: ungoogled-chromium is configured to build against the system's [FFmpeg](https://www.ffmpeg.org/) (available in Stretch and onwards); [Libav](http://libav.org) (used in Jessie) will not work. However, FFmpeg is available in `jessie-backports`. To install it, add `jessie-backports` to the apt sources, and then install `libavutil-dev`, `libavcodec-dev`, and `libavformat-dev` from it. Note that this will replace Libav.
@@ -39,29 +35,25 @@ Run these steps on the system you want to build packages for.
     git clone https://github.com/Eloston/ungoogled-chromium.git
     cd ungoogled-chromium
     git checkout $(git describe --tags `git rev-list --tags --max-count=1`) # Checkout newest tag
-    # Run dpkg-checkbuilddeps to find packages needed for building
-    ./build_debian.sh -A
+    dpkg-checkbuilddeps resources/debian/dpkg_dir/control # Use this to see the packages needed to build
+    ./build.py
 
-Debian packages will appear under `ungoogled-chromium/`
-
-Pass the `-h` flag into `build_debian.sh` or `download_source.sh` for more options.
+Debian packages will appear in the current working directory.
 
 ## Arch Linux
 
 For Arch Linux, consider using [Inox patchset](https://github.com/gcarq/inox-patchset); one of the projects which ungoogled-chromium draws its patches from. It offers pre-built binaries and is also available in AUR.
 
 ## Windows
 
-TODO
+TODO. See Issue #1
 
 ## Other systems, platforms, and configurations
 
-Please note that building on other platforms and systems is not officially supported, and multiple things can go wrong while attempting to build. Nevertheless, here are some tips: TODO
-
 Consult the build instructions on the [Chromium homepage](http://www.chromium.org/Home) to build Chromium for your system.
 
-The patches in the `patches` directory should work for any build of Chromium. They assume a clean and unmodified Chromium source tree.
+The main set of patches (in `resources/common/patches`) should work on most, if not all, platforms supported by desktop Chromium. Some patches are there to fix building with certain build flags, so those may not work with other platforms or configurations. However, the patches as they are should apply as long as there is a clean and unmodified source tree.
 
-These patches are also tested with the GYP flags defined in [`build_templates/debian/rules`](build_templates/debian/rules). Note that enabling some flags, such as `safe_browsing`, may cause the build to fail.
+It is not recommended to run domain substitution or source cleaning, especially if your build requires additional downloads from Google.
 
-Note about `domain_substitution_list`: Replacing the domains will break URLs in the source tree pointing to Google servers. If your building steps requires additional downloads (such as the the PNaCl toolkit), scripts in the source tree may fail to work.
+The domain substitution list, source cleaning list, and some patches in `resources/common` are designed to work with the build flags defined. They may require modifications if the flags are changed.

README.md

@@ -11,12 +11,13 @@ In addition to features provided by [Iridium Browser](https://iridiumbrowser.de/
 * Added menu item under "More tools" to clear the HTTP authentication cache on-demand
 * Disabled persistent per-site settings in Preferences file
 * Make all popups go to tabs
-* Replaced many domains in the source code with non-existant alternatives (see `generate_domain_substitution_list.sh`)
-* Stripped binaries from the source code (see `generate_cleaning_list.sh`)
+* Replaced many domains in the source code with non-existant alternatives (known as domain substitution)
+* Stripped binaries from the source code (known as source cleaning)
 * Disabled intranet redirect detector
 * Debian build scripts
-  * (Debian build scripts change) Move the chrome-sandbox into a separate package
+    * (Debian build scripts change) Move the chrome-sandbox into a separate package
 * (Iridium Browser feature change) Prevent URLs with the `trk:` scheme from connecting to the Internet
+    * Also prevents any URLs with the top-level domain `qjz9zk` (as used in domain substitution) from attempting a connection.
 * (Iridium and Inox feature change) Prevent pinging of IPv6 address when detecting the availability of IPv6
 
 **DISCLAIMER: Although I try my best to eliminate bugs and privacy-invading code, there will be those that slip by due to the enormity and continuing evolution of the Chromium project.**
@@ -34,24 +35,33 @@ The `master` branch is for development, so it is not guarenteed to be in a worki
 
 ## How ungoogled-chromium is designed
 
-All features are implemented through patches. Patches are contained within the `patches` directory, with the exception of platform-specific patches in the `building/templates` directory
-
-A summary of the files in the `patches` directory:
-* `ungoogled-chromium/`
-  * This directory contains new patches for ungoogled-chromium. They implement the features described above.
-* `iridium-browser`
-  * This directory contains a subset of patches from Iridium Browser.
-        * Patches are not touched unless they do not apply cleanly onto the version of Chromium being built
-  * Patches are from the `patchview` branch of its Git repository. [Web view of the patchview branch](https://git.iridiumbrowser.de/cgit.cgi/iridium-browser/?h=patchview)
-* `inox-patchset/`
-  * This directory contains a modified subset of patches from Inox patchset.
-  * Patches are from [inox-patchset's GitHub](https://github.com/gcarq/inox-patchset)
-  * [Inox patchset's license](https://github.com/gcarq/inox-patchset/blob/master/LICENSE)
-* `debian/`
-  * This directory contains patches from Debian's Chromium.
-  * These patches are not Debian-specific. For those, see the `build-templates/debian/` directory
-* `patch_order`
-  * Determines which patches are used and what order they should be applied
+Features are implemented through a combination of build flags, patches, and a few file inputs for automated source modification. All of these are stored in the `resources` directory. The `resources` directory contains the `common` directory, which has such files that apply to all platforms. All other directories, named by platform, contain additional platform-specific data. Most of the features, however, are stored in the `common` directory.
+
+There are currently two automated scripts that process the source code:
+* Source cleaner - Used to clean out binary files (i.e. do not seem to be human-readable text files, except a few required for building)
+* Domain substitution - Used to replace Google and other domains in the source code to eliminate communication not caught by the patches and build flags.
+
+Here's a breakdown of what is in a resources directory:
+* `cleaning_list` - (Used for source cleaning) A list of files to be excluded during the extraction of the Chromium source
+* `domain_regex_list` - (Used for domain substitution) A list of regular expressions that define how domains will be replaced in the source code
+* `domain_substitution_list` - (Used for domain substitution) A list of files that are processed by `domain_regex_list`
+* `gn_args.ini` - A list of GN arguments to use for building. (Currently unused, see Issue #16)
+* `gyp_flags` - A list of GYP flags to use for building.
+* `patches/` - Contains patches. The patches in here vary by platform, but the ones in the `common` directory are described below.
+    * `patch_order` - The order to apply the patches in. Patches from `common` should be applied before the one for a platform.
+
+All of these files are human-readable, but they are usually processed by the Python building system. See the Building section below for more information.
+
+Here's a breakdown of the `common/patches` directory:
+* `ungoogled-chromium/` - Contains new patches for ungoogled-chromium. They implement the features described above.
+* `iridium-browser` - Contains a subset of patches from Iridium Browser.
+    * Patches are not touched unless they do not apply cleanly onto the version of Chromium being built
+    * Patches are from the `patchview` branch of Iridium's Git repository. [Git webview of the patchview branch](https://git.iridiumbrowser.de/cgit.cgi/iridium-browser/?h=patchview)
+* `inox-patchset/` - Contains a modified subset of patches from Inox patchset.
+    * Patches are from [inox-patchset's GitHub](https://github.com/gcarq/inox-patchset)
+    * [Inox patchset's license](https://github.com/gcarq/inox-patchset/blob/master/LICENSE)
+* `debian/` - Contains patches from Debian's Chromium.
+    * These patches are not Debian-specific. For those, see the `resources/debian/patches` directory
 
 ## Building
 
@@ -73,7 +83,7 @@ You may also contribute by submitting pull requests.
 
 [Debian for build scripts](https://tracker.debian.org/pkg/chromium-browser)
 
-Google for Chromium
+[The Chromium Project](https://www.chromium.org/)
 
 ## License
 

build.py

@@ -0,0 +1,68 @@
+#!/usr/bin/env python3
+
+# ungoogled-chromium: Google Chromium patches for removing Google integration, enhancing privacy, and adding features
+# Copyright (C) 2016  Eloston
+#
+# This file is part of ungoogled-chromium.
+#
+# ungoogled-chromium is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# ungoogled-chromium is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with ungoogled-chromium.  If not, see <http://www.gnu.org/licenses/>.
+
+'''
+Simple build script for Debian
+
+Will be extended to work with other platforms
+'''
+
+import logging
+import configparser
+
+import buildlib.debian
+
+def initialize_logger(logging_level):
+    logger = logging.getLogger("ungoogled_chromium")
+    logger.setLevel(logging_level)
+
+    console_handler = logging.StreamHandler()
+    console_handler.setLevel(logging_level)
+
+    formatter = logging.Formatter("%(asctime)s - %(levelname)s: %(message)s")
+    console_handler.setFormatter(formatter)
+
+    logger.addHandler(console_handler)
+
+    return logger
+
+def read_version_config(config_location):
+    config = configparser.ConfigParser()
+    config.read(config_location)
+    return (config["main"]["chromium_version"], config["main"]["release_revision"])
+
+def main():
+    logger = initialize_logger(logging.DEBUG)
+
+    chromium_version, release_revision = read_version_config("version.ini")
+
+    platform = buildlib.debian.DebianPlatform(chromium_version, release_revision, logger=logger)
+    platform.setup_chromium_source()
+    platform.setup_build_sandbox()
+    platform.apply_patches()
+    platform.setup_build_utilities()
+    platform.generate_build_configuration()
+    platform.build()
+    platform.generate_package()
+
+    return 0
+
+if __name__ == "__main__":
+    exit(main())