[AMBARI-22883] A new REST API: GET /api/v1/hosts?format=summary or /api/v1/clusters/… by scottduan · Pull Request #285 · apache/ambari

scottduan · 2018-02-07T01:44:03Z

…{cluster_name}/hosts?format=summary is supported with this implementation and return the number of hosts running each possible operating system. More aggregation info of hosts can be also added in this code structure.

What changes were proposed in this pull request?

This change is asked to implement a new REST API to query aggregation of hosts information in one cluster or multiple clusters. In this implementation, only host os_type number is got, in the future, more host info can be aggregated with this code structure.

How was this patch tested?

UT
Manually:
{
"href" : "http://c6801.ambari.apache.org:8080/api/v1/clusters/c1/hosts?format=summary",
"hosts_summary" : {
"cluster_name" : "c1",
"operating_systems" : [
{
"centos6" : 2
}
]
}
}

and

{
"href" : "http://c6801.ambari.apache.org:8080/api/v1/hosts?format=summary",
"hosts_summary" : {
"operating_systems" : [
{
"centos6" : 2
}
]
}
}

…{cluster_name}/hosts?format=summary is supported with this implementation and return the number of hosts running each possible operating system. More aggregation info of hosts can be also added in this code structure.

scottduan · 2018-02-07T01:45:13Z

@jayush @mradha25 @swapanshridhar @adoroszlai @ncole @jonathan-hurley
Please review the code when you have time. Thanks.

jonathan-hurley · 2018-02-07T01:47:44Z

+import org.apache.ambari.server.controller.spi.Resource;
+import org.apache.commons.lang.StringUtils;
+
+public class HostInfoSummaryRenderer extends BaseRenderer implements Renderer {


Javadoc missing ...

oops, I will add it.

jonathan-hurley · 2018-02-07T01:48:26Z

+   */
+  @Override
+  public TreeNode<Set<String>> finalizeProperties(
+      TreeNode<QueryInfo> queryTree, boolean isCollection) {


I don't think that you need to break the line so soon - maybe extend out your columns?

OK, just copy/paste, I will make it one line.

jonathan-hurley · 2018-02-07T01:49:09Z

+  @Override
+  public Result finalizeResult(Result queryResult) {
+    // Convert fully qualified properties into short property names and flat the queryResult datastructure
+    Map<String, Object> summaryMap = new HashMap<>();


Do you care about ordering here? If so, maybe use a different kind of map?

Not care the ordering. What different kind of map?

If the ordering isn't important, then it's fine. Just was curious if you wanted ordered by something.

jonathan-hurley · 2018-02-07T01:50:00Z

+    }
+
+    Resource resultResource = new ResourceImpl(Resource.Type.HostSummary);
+    resultResource.setProperty("hosts_summary", summaryMap);


Extract out hosts_summary to a constant since you use it a few places.

Ok, I will use constant variable to replace it.

jonathan-hurley · 2018-02-07T01:50:12Z

+import org.apache.ambari.server.controller.spi.Request;
+import org.apache.ambari.server.controller.spi.Resource;
+
+public class HostInfoSummaryResourceDefinition extends BaseResourceDefinition {


Missing Javadoc

Will add it.

jonathan-hurley

JavaDoc missing on some classes - also wanted to discuss if a new renderer is sufficient instead of a brand new ResourceProvider

jonathan-hurley · 2018-02-07T01:56:15Z

+      case Host:
+        return new HostResourceProvider(managementController);
+      case HostSummary:
+        return new HostInfoSummaryResourceProvider(managementController);


This is interesting - I would have expected you to use the same ResourceProvider here (HostResourceProvider) and simply change the renderer in the HostResourceDefinition. Is there a reason a whole new ResourceProvider is needed instead of just a new renderer?

At begining, I thought to reuse HostResourceProvider, but I found they are totoally different resource provider:
(1) HostResourceProvider focuses on create, update and delete Host, but HostInfoSummaryResourceProvider is a read-only provider
(2) HostResourceProvider is only on single host, but the latter will aggregate info from all hosts in cluster or accross clusters
So I think it is more clear to separete these two kinds of host resource providers.

HostResourceProvider does do multiple hosts though:

Set<HostResponse> responses = getResources(new Command<Set<HostResponse>>() { @Override public Set<HostResponse> invoke() throws AmbariException { return getHosts(requests); } });

It iterates over the HostResponse instances, creating resources for each. So, wouldn't you only need to supply a renderer in this case to post-process the results?

The reason is performance. My implementation is to sum os_type of all the hosts on DB level with single sql call. If we do post-process for all the hosts after creating resource for each, in large number of hosts scenario, it will take too long to get resources for all hosts.

I also thought we will be adding a new renderer like we do for Alerts. I understand that this is more performant but it is a tradeoff between performance & maintenance. If the host summary API is not going to be used very often then we should consider just going with a renderer. Can you check with the UI team on how often the host summary api will be called in the UI, if its perriodic or only on a specific user action.

I will check UI about the use case. In considerationof maintenance, I think the new implementation is more clear and clean. The host resouce provider is onlyb for single host crud, the hosts summary resource provider is a read provideer and works like a aggregation service and provide overall hosts summary in/across clusters. Now I only implement os_type sum, but it is very easy to add more inforamtion in the future.

The host resource provider does provide information for multiple hosts. If we're introducing a new resource provider, then we shouldn't need a renderer since the default renderer can be used (but it should be a new endpoint). So I think we need to either:

Switch to using a renderer with the current resource provider

OR

Create a new endpoint and new resource provider with a default renderer for the host summary

asfgit · 2018-02-07T05:50:39Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/470/
Test FAILed.
Test FAILured.

…{cluster_name}/hosts?format=summary is supported with this implementation and return the number of hosts running each possible operating system. More aggregation info of hosts can be also added in this code structure. (Rebased)

adoroszlai · 2018-02-13T22:50:56Z

 import org.apache.ambari.server.api.resources.ResourceInstance;
 import org.apache.ambari.server.api.services.parsers.RequestBodyParser;
 import org.apache.ambari.server.api.services.serializers.ResultSerializer;
+import org.apache.ambari.server.controller.spi.Resource;


adoroszlai · 2018-02-13T22:55:48Z

+    List<HostInfoSummaryDTO> summaryDTOS = hostInfoSummaryDAO.findHostInfoSummary(cluster_name);
+    List<Map<String, Integer>> osSummaryList = new ArrayList<>();
+    for (HostInfoSummaryDTO summaryDTO : summaryDTOS) {
+      osSummaryList.add(Stream.of(new AbstractMap.SimpleImmutableEntry<>(summaryDTO.getOsType(), summaryDTO.getOsTypeCount())).collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue)));


Seems unnecessarily complicated. I don't think you need both the loop and the stream.

This line is just to save pair<osType, osTypeCount> into a map and then save it to a list. Because HostInfoSummaryDTO has two variables: osType and osTypeCount, the osSummaryList will become:
[cent6:10, debian4: 12, cent7: 20..]

How about

osSummaryList.add(ImmutableMap.of(summaryDTO.getOsType(), summaryDTO.getOsTypeCount()));

adoroszlai · 2018-02-13T22:59:12Z

+    }
+    Map<String, Object> os = new HashMap<>();
+    os.put(HOST_INFO_SUMMARY_OS, osSummaryList);
+    summary.add(os);


Why does it need to keep summary as state?

adoroszlai · 2018-02-13T22:59:37Z

+  @Override
+  public int hashCode() {
+    int result = clusterId.hashCode();
+    result = 31 * result + hostId.hashCode();


Objects.hash(...)

ok, I will move to this way.

adoroszlai · 2018-02-13T22:59:50Z

+  @Override
+  public int hashCode() {
+    int result = clusterId.hashCode();
+    result = 31 * result + hostId.hashCode();


Objects.hash(...)

adoroszlai · 2018-02-13T23:02:00Z

+public class HostInfoSummaryDTO {
+
+  private String os_type;
+  private int os_type_count = 0;


Please use camelCase.

Thanks for pointing it out. I will change it.

adoroszlai · 2018-02-13T23:18:02Z

+    Set<String> requestPropertyIds = getRequestPropertyIds(request, predicate);
+
+    // use a collection which preserves order since JPA sorts the results
+    Set<Resource> resources = new HashSet<>();


"preserves order" and HashSet seem to contradict each other.

I forgot to delete comment here, I have no intention to preserve the order.

adoroszlai · 2018-02-13T23:22:19Z

  protected Set<Resource> getResourcesAuthorized(Request request, Predicate predicate)
      throws SystemException, UnsupportedPropertyException, NoSuchResourceException, NoSuchParentResourceException {
+    if (request.getPropertyIds().contains(HOST_SUMMARY_PROPERTY_ID)) {
+      return getHostSummaryResource(request, predicate);


Can you please document the response structure of host summary in HostResponse or a new interface using Swagger annotations?

There are two issues here: 1. REST API does not allow two interfaces which have the same path but different parameters. So in this case /api/v1/hosts and /api/v1/hosts?format=summary are regarded as the same interface, we cannot create another API with only different paramter; 2. Swagger does not support one method with multiple endpoints, that is, /api/v1/hosts and /api/v1/hosts?format=query return totally differnt responses, and so we cannot use HostResponse to summary structure of host summary in HostResponse. So unless I use a different path such as /api/v1/hosts/summary, I cannot use swagger annotation

adoroszlai · 2018-02-13T23:23:12Z

+  private Long hostId;
+
+  @Id
+  @Column(name = "cluster_id", nullable = false, insertable = true, updatable = true)


Are these JPA annotations really required in both the PK and entity classes?

Not necessary at this time.

adoroszlai · 2018-02-13T23:27:04Z

+@Singleton
+public class HostInfoSummaryDAO {
+
+  private static final Logger LOG = LoggerFactory.getLogger(HostInfoSummaryDAO.class);


Will remove

asfgit · 2018-02-14T00:34:39Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/575/
Test FAILed.
Test FAILured.

jonathan-hurley · 2018-02-15T19:59:17Z

+    if (StringUtils.isNotBlank(uriString) && uriString.indexOf("format=summary") != -1) {
+      query.addLocalProperty(HostResourceProvider.HOST_SUMMARY_PROPERTY_ID);
+    }
+  }


This seems wrong to add to a generic handler like the ReadHandler. Why is this even needed if you're using a renderer to do the work?

Now based on previous discussion, I have removed all summary resourceprovider and summary render. Also the "format" in uri is filtered and is not regarded as a predicate, so there is no way for readhandler to know what exact property that this read request wants to get. "summary" is a special read request and in current uri parsing, it is actually ignored. If we compare with alert summary(that is the only implementation of summary in Ambari), it uses a separate resourceprovider and renderer to realize it. Originally I did the same way, but I got some comments to suggest not create new resourceprovider and/or new renderer in consideration of mantainenance and possibly duplicated code. So I think if the uri inlcudes " format=summary" queryParam, it means we should do summary query instead of individual property query. This should be one of a generic operation.

That's right, it's not regarded as a predicate. However, the getRenderer() method of the ResourceDefinition can determine which renderer to use. The BaseRequest has the handle to the renderer name already.

/** * Parse the query string for the {@link QueryLexer#QUERY_FORMAT} property and obtain * a renderer from the associated resource definition based on this property value. */ private void parseRenderer() { String rendererName = isMinimal() ? "minimal" : m_uriInfo.getQueryParameters().getFirst(QueryLexer.QUERY_FORMAT); m_renderer = m_resource.getResourceDefinition(). getRenderer(rendererName); }

The resourceProvider does not know the renderer, it can only know this through resource.type. We can do in another way: Create a new resource.type=HostSummary, and create a hostsummaryRenderer. So hostResourceProvider can handle Request based on resource.type differently. But this implementation will introudce new resource.type and new resourcerenderer. Moreover, the hostservice will create different resourceInstances based on if there has queryParam and queryParam is "format=summary", either "resource.type=Host" or "resource.type=HostSummary". If you think this is ok, I can modify my code to implement it.

That is correct - the resource provider doesn't know the renderer. It satisfies the predicate to provide the concrete resources. From the resources, the correct renderer is applied to the dataset to return the data in a new way.

I'm not understanding why this can't follow the same pattern as other renderers in the system. The data is post-processed by the renderer in order to provide a summary of data instead of the individual resources themselves.

I agree with @jonathan-hurley . This is exactly why renderers exist. Now it may not be as clean as a summary DTO out of the db, but it surely can be done.

jonathan-hurley · 2018-02-15T20:00:54Z

+ */
+package org.apache.ambari.server.orm.dao;
+
+public class HostInfoSummaryDTO {


Can we just serialize this class back? Add some Jackson annotations to help with it?

Here we want to get the data format as osType:osTypeCount, like:
centos6:10
debian5:2
This class will give us:
osType: centos
osTypeCount: 12
..
So I create another class HostInfoSummary to wrap it.

…{cluster_name}/hosts?format=summary is supported with this implementation and return the number of hosts running each possible operating system. More aggregation info of hosts can be also added in this code structure. (Rebased)

asfgit · 2018-02-16T07:43:56Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/638/
Test FAILed.
Test FAILured.

scottduan · 2018-02-18T09:32:03Z

The new patch removed query sum from db directly and use hostrenderer to summarize host info, as Jonathan suggested.

asfgit · 2018-02-18T10:42:35Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/677/
Test FAILed.
Test FAILured.

scottduan · 2018-02-21T06:04:56Z

@adoroszlai @jonathan-hurley Could you review my new patch uploaded last Saturday? Thanks.

adoroszlai · 2018-02-21T11:53:14Z

+  // A list of host properties to be summarized
+  private final String OPERATING_SYSTEMS = "operating systems";
+  // Renderer name: Host or Summary
+  private String name = "Host";


This should be an enum.

OK, I will change to enum

adoroszlai · 2018-02-21T11:53:32Z

+public class HostRenderer extends DefaultRenderer {
+
+  // A list of host properties to be summarized
+  private final String OPERATING_SYSTEMS = "operating systems";


Should be static.

adoroszlai · 2018-02-21T11:55:04Z

+    }
+
+    Result result = new ResultImpl(true);
+    TreeNode<Resource> summaryResultTree = result.getResultTree();


Seems to be unused (duplicate of summaryTree).

adoroszlai · 2018-02-21T11:57:21Z

+        os.put(osType, os.getOrDefault(osType, 0) + 1);
+      }
+    }
+    if (!osSummary.isEmpty()) {


How can osSummary ever be empty? It always has 1 entry, regardless of the loop, due to:

osSummary.put(OPERATING_SYSTEMS, new HashMap<String, Integer>());

adoroszlai · 2018-02-21T11:57:55Z

+    // iterate over all returned flattened hosts and build the summary info
+    List<Object> summary = new ArrayList<>();
+    Map<String, Map<String, Integer>> osSummary = new HashMap<>();
+    osSummary.put(OPERATING_SYSTEMS, new HashMap<String, Integer>());


Please save this HashMap in a local variable to avoid having to get() in the loop.

adoroszlai · 2018-02-21T12:02:36Z

+      String osType = (String) resource.getPropertyValue(HostResourceProvider.HOST_OS_TYPE_PROPERTY_ID);
+      if (StringUtils.isNotBlank(osType)) {
+        Map<String, Integer> os = osSummary.get(OPERATING_SYSTEMS);
+        os.put(osType, os.getOrDefault(osType, 0) + 1);


How about using Map#merge()?

adoroszlai · 2018-02-21T12:05:58Z

+    TreeNode<Resource> resultTree = queryResult.getResultTree();
+    // iterate over all returned flattened hosts and build the summary info
+    List<Object> summary = new ArrayList<>();
+    Map<String, Map<String, Integer>> osSummary = new HashMap<>();


Please extract the part that builds osSummary (declaration and the loop) into another method to reduce complexity of finalizeResult().

adoroszlai · 2018-02-21T12:08:08Z

+    TreeNode<Resource> summaryTree = result.getResultTree();
+    summaryTree.addChild(resource, "summary");
+    resource.setProperty("summary", summary);
+    return result;


Please extract the lines above that build the summary result into another method to reduce complexity of finalizeResult.

Refactor this piece of code

adoroszlai · 2018-02-21T12:10:09Z

+    Resource resource = new ResourceImpl(Resource.Type.Host);
+    TreeNode<Resource> summaryTree = result.getResultTree();
+    summaryTree.addChild(resource, "summary");
+    resource.setProperty("summary", summary);


Please use HostResourceProvider.SUMMARY_PROPERTY_ID.

adoroszlai · 2018-02-21T12:11:21Z


 import javax.persistence.EntityManager;

+


Unnecessary.

jonathan-hurley · 2018-02-21T15:00:01Z

+import org.apache.ambari.server.controller.spi.Resource;
+import org.apache.commons.lang.StringUtils;
+
+public class HostRenderer extends DefaultRenderer {


jonathan-hurley · 2018-02-21T15:02:42Z

+  public Result finalizeResult(Result queryResult) {
+    // Host still uses defaultRenderer
+    if (name.equals("Host")) {
+      return super.finalizeResult(queryResult);


Why is the HostRenderer making this decision? Instead, shouldn't it be the ResourceDefinition to decide which renderer to return? If there is no renderer or the name doesn't match, return the default renderer. Otherwise, return a renderer which does a specific action and doesn't have to if/else the finalize result.

Move this check to hostresourcedefinition

…thub.com/scottduan/ambari into AMBARI-22883-branch-feature-AMBARI-14714

scottduan · 2018-02-21T18:57:47Z

This is the output:

{
"href" : "http://c6801.ambari.apache.org:8080/api/v1/clusters/c1/hosts?format=summary",
"summary" : [
{
"operating_systems" : {
"centos6" : 1
}
}
]
}

asfgit · 2018-02-22T01:20:23Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/739/
Test FAILed.
Test FAILured.

scottduan · 2018-02-22T03:38:54Z

@adoroszlai @jonathan-hurley I have updated code based on your comments. Please review it again. Thanks.

jonathan-hurley · 2018-02-22T15:07:43Z

+import org.apache.commons.lang.StringUtils;
+
+/**
+ * HostSummaryRenderere


sp: extra e

… (#462) * AMBARI-22675 LogSearch Title Bar Fixes. (Istvan Tobias via ababiichuk) * AMBARI-22749. Create Pull Request Template * Added inline style link * AMBARI-22731. Infra Manager - hdfs upload support for archiving Infra Solr * AMBARI-22121. Create mpack for Isilon OneFS (amagyar) * AMBARI-22115. Alerts for OneFS mpack (amagyar) * AMBARI-22236. Expression parser support for JMXServerSide alerts (amagyar) * AMBARI-22115. Alerts for OneFS mpack (amagyar) * AMBARI-22319. Allow the same config type to belong to multiple services (amagyar) * AMBARI-22339. Metrics for OneFS mpack (amagyar) * AMBARI-22424. Quicklinks for OneFS (amagyar) * AMBARI-22449. Improved service/component dependency support (amagyar) * AMBARI-22449. Improved service/component dependency support (amagyar) * AMBARI-22507. Ambari UI changes to support Isilon OneFS (amagyar) * AMBARI-22634. Kerberos support for OneFS (amagyar) * AMBARI-22674. ServiceAdvisor for OneFS (amagyar) * AMBARI-22449. Improved service/component dependency support (amagyar) * AMBARI-22121. Create mpack for Isilon OneFS (amagyar) * AMBARI-22339. Metrics for OneFS mpack (amagyar) * AMBARI-22756 Post-install: Reduce navigation width in wizard. (atkach) * AMBARI-22236. Addendum - Expression parser support for JMXServerSide alerts (amagyar) * AMBARI-22750. Add custom input / mapper descriptor to config api * AMBARI-22667: Use internal LDAP configuration values rather than ambari.properties values when accessing the configured LDAP server for LDAP sync and authentication * AMBARI-22757 Log Search UI: implement query manipulation from resources diagram. (ababiichuk) * [AMBARI-22749] ADDENDUM: Create Pull Request Template - fix rat check (#82) * AMBARI-22760 Infra Manager: install packages * [AMBARI-22764] NN HA wizard is broken due to recent commit for config compare * AMBARI-22767. Kerberos wizard. Advanced kerberos-env password properties should be visible only if the KDC type is 'Active Directory' (alexantonenko) * AMBARI-22766 ambari-server setup with internal database may not work on centos 7 (dgrinenko) * AMBARI-22768. Update Welcome page style (akovalenko) * AMBARI-22772. Log Search / Log Feeder - config symlink cannot be created if etc/ambari-logsearch* folders do not exist * AMBARI-22771. Ambari loads ambari.properties using ISO 8859-1 encoding (adoroszlai) * [AMBARI-22770] Bring jenkins job build script under version control (#98) [AMBARI-22770] Bring jenkins job build script under version control * AMBARI-22637. Fix misuses of os.path.dirname(path) in yarn.py * AMBARI-22771. Fix broken unit test * AMBARI-22668. Moving LDAP related properties to DB upon upgrade to 3.0.0 * AMBARI-22668: implemented changes requested by reviewers * AMBARI-22668: implemented changes requested by rlevas * AMBARI-22783 Login page lost Ambari branding. (atkach) * AMBARI-22779. Cannot scale cluster if Ambari Server restarted since blueprint cluster creation * AMBARI-22789. Fix Log Search / Log Feeder / Infra Manager start scripts * AMBARI-22787 - Add new Ambari Infra Manager component to Ambari infra stack * AMBARI-22793. Notification popover layout issues * AMBARI-22785: added force_tcp option to KRB5 configuration template so that customers can choose TCP over UDP when communicating with Kerberos * AMBARI-22796. Widget menu goes out of bounds (alexantonenko) * AMBARI-22667: fix commons-io version * AMBARI-22794 Opacity decreased and the shadow has been added to the sticky filter bar. * AMBARI-22726. Fix dependent tests in BlueprintImplTest (#133) * Merge remote-tracking branch 'upstream/trunk' into branch-feature-AMBARI-20859 * AMBARI-22577. Migrate user data for upgrade to improved user account management * AMBARI-22806.Unable to delete files from HDFS using Ambari File View when Ambari Views is accessed via Knox(Venkata Sairam) * AMBARI-22792. Refactor agent-side kerberos code (echekanskiy) * AMBARI-22807. Admin View: Add/Delete Group succeeds, but the group table does not refresh automatically. (alexantonenko) * AMBARI-22809. Tez shown in red in the left nav for no apparent reason (alexantonenko) * AMBARI-22796. Widget menu goes out of bounds. review (alexantonenko) * AMBARI-22792. Refactor agent-side kerberos code - import fix (echekanskiy) * AMBARI-22812. Fix checkstyle error in UpgradeCatalog300Test (#148) * [AMBARI-22725] Expose Conditional Elements For Tasks on Upgrade * AMBARI-22795 LogSearch Fixes for LogList Display. (Istvan Tobias via ababiichuk) AMBARI-22795 LogSearch Fixes for LogList Display. (Istvan Tobias via ababiichuk) * AMBARI-22808 Ambari-Web: Fix randomly failing unit tests * AMBARI-22788 Unsightly artifacts during Login. (atkach) * AMBARI-22697. Throw exception when keytab creation fails due to wrong configuration of key encryption types * [AMBARI-22798] Role authorization AMBARI.MANAGE_CONFIGURATION is not added to AMBARI.ADMINISTRATOR role during Ambari upgrade * AMBARI-22818. Log Feeder: refactor - create plugin api * [AMBARI-22820] Fix KerberosOperationHandlerTests due to changes from AMBARI-22697 * AMBARI-22805. Blueprints do not handle some failures properly * AMBARI-22716: zeppelin.livy.url is not getting updated after moving livy to a new host (prabhjyotsingh) * AMBARI-22698: Custom zeppelin interpreter properties are getting removed after moving zeppelin to a different host (prabhjyotsingh) * AMBARI-22696 Whitelist execute latency from Storm Ambari metrics * AMBARI-22759 [Hive Views 2.0] Deleting a Saved query is Buggy when Mutliple Queries exist in same Name * Fix compilation issues after trunk merge (mradhakrishnan) * Fix compilation issues in tests after trunk merge * AMBARI-22797. The stup-ldap tool should persists its output into Ambari database rather than ambari.properties * AMBARI-22797. Better CLI option names * AMBARI-22797. Dot not filter for PAM only; we should inform the end user when configuring LDAP in case the currently cond=figured auth method is not LDAP * AMBARI-22797. Removed redundant information from the JSON payload * AMBARI-22797. Removing 'is_root' check from setup-ldap (according to R. Levas this is irrelevant) * AMBARI-22797. Fixed error message * AMBARI-22797. Created constants to avoid typos * AMBARI-22797. Removed redundant constant (one already existed with the same value) * AMBARI-22797. Fixed CLI option names under Windows * AMBARI-22712. Update install Wizard layout. Second patch. (akovalenko) * AMBARI-22776 Ambari Blueprint 3.0/3.1 database tables and JPA objects + DDL cleanup (#109) (benyoka) * AMBARI-22776 Blueprint 3.0 Database schemas and JPA objects, DDL fixes (benyoka) * AMBARI-22776 Revert accidental commit (benyoka) * AMBARI-22776 Revert accidental change #2 (benyoka) * AMBARI-22776 Fix review findings (benyoka) * AMBARI-22776 Fix review findings #2 (benyoka) * AMBARI-22776 remove blueprint -> stack reference, docs (benyoka) * AMBARI-22776 fix build issues (benyoka) * AMBARI-22776 Add MpackInstanceEntity -> MpackEntity reference (benyoka) * [AMBARI-22817] Update backend code to handle new versioning schema. (#155) * AMBARI-22817. Update backend code to handle new versioning schema.(vbrodetskyi) * AMBARI-22817. Update backend code to handle new versioning schema.(vbrodetskyi) * AMBARI-22817. Update backend code to handle new versioning schema.(vbrodetskyi) * AMBARI-22817. Update backend code to handle new versioning schema.(vbrodetskyi) * AMBARI-22817. Update backend code to handle new versioning schema.(vbrodetskyi) * AMBARI-22817. Update backend code to handle new versioning schema.(vbrodetskyi) * AMBARI-22805. Improve Blueprints error handling in case of timeout (#185) * [AMBARI-22815] Change requiredServices in metainfo xml (#181) * AMBARI-22815 Change requiredServices in metainfo.xml (dsen) * AMBARI 22815 Change requiredServices in metainfo xml (renamed scope to dependencyType) (dsen) * AMBARI-22572. During cluster installation bower cannot resolve angularjs version (alexantonenko) * AMBARI-22572. During cluster installation bower cannot resolve angularjs version (angular version changed) (alexantonenko) * AMBARI-22776 Fix DDL issues resulting from incomplete table rename (benyoka) (#189) * AMBARI-22776 Fix DDL issues resulting from incomplete table rename (benyoka) * AMBARI-22776 More DDL fixes and renamings in the code (benyoka) * AMBARI-22413 fix tar_archive.archive_directory_dereference (benyoka) (#195) * AMBARI-22846. Get rid of Murmur Hash usage in Log Feeder for log events * AMBARI-22799 - define scheduling of archiving Infra Solr Documents * AMBARI-22840 Standardize precision when expressing durations * AMBARI-22782. Fix Namenode alerts broken due to enabling federation (aonishuk) * AMBARI-22853. NFS Gateway is not logging at the correct location (aonishuk) * AMBARI-22851 Host Details page style fixes * AMBARI-22861. Ambari UI for Scheduler configuration is trimming = from value sets * AMBARI-22862. Ambari admin breaks cause of responseInterceptors property * AMBARI-22861. Ambari UI for Scheduler configuration is trimming = from value sets (log removed) * [AMBARI-22845] Update service metainfo schema (#194) * AMBARI-22845 Update service metainfo schema (dsen) * AMBARI-22845 Update service metainfo schema - changes according to review (dsen) * AMBARI-22561. Need to address HDP-GPL repo update after user accepts license in post-install scenario (aonishuk) * AMBARI-22855. Log Search: using shipperconfig api should be configurable * AMBARI-22824. Let YARN/MR2 use ZK principal name set by users when enabling Kerberos (until now it's been hardcoded to 'zookeeper') * AMBARI-22847. Let HBase use ZK principal name set by users when enabling Kerberos (until now it's been hardcoded to 'zookeeper') * AMBARI-22860. Support external zookeeper for Infra Solr and Log Search * AMBARI-22866. Log Search: Add userList query for audit log requests. * AMBARI-22867 Add new state for Not Available data in Heatmap widget * AMBARI-22712. Update install Wizard layout. Third patch. (akovalenko) * [AMBARI-22858] First prereq not displaying for Free IPA method in Enable Kerberos Wizard * AMBARI-22791 support for parsing MulitE blueprints and cluster templates (#213) * AMBARI-22776 Blueprint 3.0 Database schemas and JPA objects, DDL fixes (benyoka) * AMBARI-22776 Revert accidental commit (benyoka) * AMBARI-22776 Revert accidental change #2 (benyoka) * AMBARI-22776 Fix review findings (benyoka) * AMBARI-22776 Fix review findings #2 (benyoka) * AMBARI-22791-Update blueprint object and convert from entity (benyoka) * AMBARI-22791-BlueprintFactory and tests (benyoka) * AMBARI-22791 Can store and retrieve multi-instance blueprint * AMBARI-22791 Build and unit test fixes (benyoka) * AMBARI-22791 merge branch-feature-AMBARI-14714 and fixes (benyoka) * AMBARI-22791 fix unit test failure (benyoka) * AMBARI-22791 fix import error (benyoka) * AMBARI-22791 Multi-everythin cluster template support (benyoka) * AMBARI-22791 fix review findings (benyoka) * AMBARI-22873: Remove HDP 3.0 stack from Ambari (jluniya) * AMBARI-22865. Moving out get_service_component_meta to DefaultStackAdvisor so that all children can use this function * AMBARI-22878: Update Service Group API to take list of mpack name associated with the service group * AMBARI-22869. Log Search: use default page and pageSize to log requests (+ rename shipper config api key name) * AMBARI-22874. Log Search: Return with 500 error for shipper config testing if an exception occurred. * AMBARI-22879 Ambari makes unrelated changes to zookeeper quorum config in all services when delete host action fails * AMBARI-20908 : Kafka advertised.listeners replaced in Kerberos Mode (bharatviswa via mradhakrishnan) * [AMBARI-22876] Disable consecutive authentication failure account lockout feature by default * [AMBARI-22882] Long cannot be cast to String error when changing a user's password * AMBARI-22841. Fix upgrade-catalog after accordingly to kerberos changes(echekanskiy) * [AMBARI-22881] Added user_authentication_id_seq into ambari_sequences upon upgrade to 2.7 * AMBARI-22881. Added user_authentication_id_seq into ambari_sequences upon upgrade to 2.7 * AMBARI-22881. Introduced a new function to fetch the maximum value of an ID column * AMBARI-22881. Using constant instead of hard-coded value * AMBARI-22830. Ambari-agent puts Python scripts in 2.6 directory on OSes that use python 2.7.x by default (aonishuk) * AMBARI-22893: Update rat check rules for stack files (jluniya) * AMBARI-22864. Agent commands hang even after freeing up disk space on the host (aonishuk) * [AMBARI-22712] Update install Wizard layout. Fourth patch. (#247) * AMBARI-22712. Update install Wizard layout. Fourth patch. (akovalenko) * AMBARI-22712. Update install Wizard layout. Fourth patch. (akovalenko) * AMBARI-22884. LogSearch Integration should call newer API to obtain log file metadata. * [AMBARI-22888] Cancel operation during package deployment causing repository manager to be broken (dgrinenko) * AMBARI-22890. LogSearch Integration support for external Shared LogSearch Server connections. * [AMBARI-22875] Blueprint cluster creation using manually installed mpacks (#231) * AMBARI-22878: Update Service Group API to take list of mpack name associated with the service group * [AMBARI-22885] LDAP sync fails with 'LDAP is not configured' error after configuring LDAP * AMBARI-22885. We fetch 'ldap enabled' flag from both ambari.properties (to support backward compatibility) and from AMBARI DB via our REST API * AMBARI-22885. Make sure we close response (even if an error occurred) * AMBARI-22885. Retrieving the 'is LDAP enabled' flag from AMBARI DB is enough (do not need to search it in ambari.properties) * AMBARI-22900 Log Search UI: implement 'History' functionality * AMBARI-22614. Fix some unit tests * [AMBARI-22889] LDAP configuration is not reloaded in Guice * AMBARI-22889. Make sure that the latest configuration is used when populating LDAP data * AMBARI-22889. LDAP authentication provider aos uses configuration provider instead of the configuration directly * AMBARI-22889. Fixed Python unit tests (not related to my changes) * AMBARI-22835 Log Search UI: implement log level filter * AMBARI-22905. Supporting old CLI option names in setup-ldap tool * AMBARI-22909 Log Search UI: implement filter by username for access logs * AMBARI-22712. Update install Wizard layout. Fifth patch. (akovalenko) * AMBARI-22911 Log Search UI: move Capture button to top menu * AMBARI-22912 Host details page: components not reconfigured after deleting their host * [AMBARI-22914] Oracle DDL is broken at trunk * [AMBARI-22886] Infra Manager: store s3 credentials in Hadoop credential store (#261) * AMBARI-22886 Infra Manager: store s3 credentials in Hadoop credential store * AMBARI-22886 Infra Manager: store s3 credentials in Hadoop credential store - addendum: use Optional<String> as a return type of getPassword, remove unnecessary ArrayList * AMBARI-22915 Dashboard-Metrics page style edits * [AMBARI-22910] Add Ambari admin/pw CLI options for setup-ldap tool * AMBARI-22910. Supporting Ambari admin user/pw CLI options in setup-ldap tool * AMBARI-22910. Using more generic option names for Ambari username/password * AMBARI-22917. Log Search make logsearch maven build to independent from other modules * AMBARI-22903. Log Search: Add Knox to docker dev env (#265) * AMBARI-22903. Log Search: Add Knox to docker dev env * AMBARI-22890. Use quotes for checking bash variables * AMBARI-22833 : change commons-collections-3.2.1.jar being used by ambari views to commons-collections-3.2.2.jar (nitirajrathore) (#188) merging as changes are approved * AMBARI-22921. Unable to enable hive interactive, LLAP, on our Ambari 2.5.2 managed cluster (aonishuk) * [AMBARI-22913]. Add ability to MasterHostResolver to resolve by namespa… (#269) * AMBARI-22913. Add ability to MasterHostResolver to resolve by namespace at a time (amagyar) * AMBARI-22913. Add ability to MasterHostResolver to resolve by namespace at a time (amagyar) * AMBARI-22902. Changed description of Kadmin Host * [AMBARI-22904] Revised mpack APIs (#252) * [AMBARI-22904] Revised mpack APIs * [AMBARI-22904] Revised mpack APIs * [AMBARI-22904] Revised mpack APIs * [AMBARI-22904] Revised mpack APIs * [AMBARI-22904] Revised mpack APIs * [AMBARI-22904] Revised mpack APIs * [AMBARI-22904] Revised mpack APIs * [AMBARI-22854] Fix running service checks with new mpack definitions (#204) * AMBARI-22854 Fix running service checks with new mpack definitions (dsen) * AMBARI-22854 Fix running service checks with new mpack definitions - removed obvious javadocs (dsen) * Merge trunk into branch-feature-AMBARI-14714 * Branch feature ambari 14714 (#291) * [AMBARI-22904] Revised mpack APIs * [AMBARI-22904] Revised mpack APIs * [AMBARI-22904] Revised mpack APIs * [AMBARI-22904] Revised mpack APIs * [AMBARI-22904] Revised mpack APIs * [AMBARI-22904] Revised mpack APIs * [AMBARI-22904] Revised mpack APIs * [AMBARI-22904] Revised mpack APIs - Fix for dependencyType -> Type in ModuleDependency class. * Branch feature ambari 14714 (#307) * [AMBARI-22948] stack-hooks has to be updated to use cluster-settings instead of cluster-env * [AMBARI-22938] Fix SQL issue due to trunk merge issue * AMBARI-22971: Remove current_mpack_id to mpack_id in stacks table (jluniya) * AMBARI-22971: Remove current_mpack_id to mpack_id in stacks table (jluniya) * [AMBARI-22957] Stack Metainfo.xml should contain the osSpecifics tag * [AMBARI-22980] Commenting out stack_features and stack_select calls until instance_manager is ready * [AMBARI-22980] Issue warnings when stack select tool is not present * [AMBARI-22995] Remove deprecated Upgrade Packs * [AMBARI-22992] Update error handling during mpack installation (#363) * [AMBARI-22992] Update error handling during mpack installation * [AMBARI-22997] cluster-settings.xml is not present in /var/lib/ambari-server/resources post ambari installation * [AMBARI-22992] Update error handling during mpack installation - review comments * [AMBARI-22980] Correct the name of the repo_suse_rhel_template * [AMBARI-23036] Create Lifecycle changes for Upgrade Packs (#430) * [AMBARI-23036] Create Lifecycle changes for Upgrade Packs * [AMBARI-23036] Update new enum for ordering (review comments) * [AMBARI-23004] Create initial version of InstanceManager to create instance layout (dsen) * Changes according to the review + general improvements * [AMBARI-23004] Create initial version of InstanceManager to create instance layout - additional fix (dsen) (#441) * AMBARI-22649. settings library should convert true/false to boolean * AMBARI-22979: Update software registry API to support new mpack schema (jluniya) (#433) * [AMBARI-22883] A new REST API: GET /api/v1/hosts?format=summary or /api/v1/clusters/… (#285) * A new REST API: GET /api/v1/hosts?format=summary or /api/v1/clusters/{cluster_name}/hosts?format=summary is supported with this implementation and return the number of hosts running each possible operating system. More aggregation info of hosts can be also added in this code structure. * A new REST API: GET /api/v1/hosts?format=summary or /api/v1/clusters/{cluster_name}/hosts?format=summary is supported with this implementation and return the number of hosts running each possible operating system. More aggregation info of hosts can be also added in this code structure. (Rebased) * A new REST API: GET /api/v1/hosts?format=summary or /api/v1/clusters/{cluster_name}/hosts?format=summary is supported with this implementation and return the number of hosts running each possible operating system. More aggregation info of hosts can be also added in this code structure. (Rebased) * [AMBARI-23053] Rename NON-ROLLING to EXPRESS * AMBARI-22995. Remove deprecated Upgrade Packs - after reintroduced by later merge commit * AMBARI-22883. NPE in HostResourceDefinition (#458)

…#610) * [AMBARI-22764] NN HA wizard is broken due to recent commit for config compare * AMBARI-22767. Kerberos wizard. Advanced kerberos-env password properties should be visible only if the KDC type is 'Active Directory' (alexantonenko) * AMBARI-22766 ambari-server setup with internal database may not work on centos 7 (dgrinenko) * AMBARI-22768. Update Welcome page style (akovalenko) * AMBARI-22772. Log Search / Log Feeder - config symlink cannot be created if etc/ambari-logsearch* folders do not exist * AMBARI-22771. Ambari loads ambari.properties using ISO 8859-1 encoding (adoroszlai) * [AMBARI-22770] Bring jenkins job build script under version control (#98) [AMBARI-22770] Bring jenkins job build script under version control * AMBARI-22637. Fix misuses of os.path.dirname(path) in yarn.py * AMBARI-22771. Fix broken unit test * AMBARI-22668. Moving LDAP related properties to DB upon upgrade to 3.0.0 * AMBARI-22668: implemented changes requested by reviewers * AMBARI-22668: implemented changes requested by rlevas * AMBARI-22783 Login page lost Ambari branding. (atkach) * AMBARI-22779. Cannot scale cluster if Ambari Server restarted since blueprint cluster creation * AMBARI-22789. Fix Log Search / Log Feeder / Infra Manager start scripts * AMBARI-22787 - Add new Ambari Infra Manager component to Ambari infra stack * AMBARI-22793. Notification popover layout issues * AMBARI-22785: added force_tcp option to KRB5 configuration template so that customers can choose TCP over UDP when communicating with Kerberos * AMBARI-22796. Widget menu goes out of bounds (alexantonenko) * AMBARI-22667: fix commons-io version * AMBARI-22794 Opacity decreased and the shadow has been added to the sticky filter bar. * AMBARI-22726. Fix dependent tests in BlueprintImplTest (#133) * Merge remote-tracking branch 'upstream/trunk' into branch-feature-AMBARI-20859 * AMBARI-22577. Migrate user data for upgrade to improved user account management * AMBARI-22806.Unable to delete files from HDFS using Ambari File View when Ambari Views is accessed via Knox(Venkata Sairam) * AMBARI-22792. Refactor agent-side kerberos code (echekanskiy) * AMBARI-22807. Admin View: Add/Delete Group succeeds, but the group table does not refresh automatically. (alexantonenko) * AMBARI-22809. Tez shown in red in the left nav for no apparent reason (alexantonenko) * AMBARI-22796. Widget menu goes out of bounds. review (alexantonenko) * AMBARI-22792. Refactor agent-side kerberos code - import fix (echekanskiy) * AMBARI-22812. Fix checkstyle error in UpgradeCatalog300Test (#148) * [AMBARI-22725] Expose Conditional Elements For Tasks on Upgrade * AMBARI-22795 LogSearch Fixes for LogList Display. (Istvan Tobias via ababiichuk) AMBARI-22795 LogSearch Fixes for LogList Display. (Istvan Tobias via ababiichuk) * AMBARI-22808 Ambari-Web: Fix randomly failing unit tests * AMBARI-22788 Unsightly artifacts during Login. (atkach) * AMBARI-22697. Throw exception when keytab creation fails due to wrong configuration of key encryption types * [AMBARI-22798] Role authorization AMBARI.MANAGE_CONFIGURATION is not added to AMBARI.ADMINISTRATOR role during Ambari upgrade * AMBARI-22818. Log Feeder: refactor - create plugin api * [AMBARI-22820] Fix KerberosOperationHandlerTests due to changes from AMBARI-22697 * AMBARI-22805. Blueprints do not handle some failures properly * AMBARI-22716: zeppelin.livy.url is not getting updated after moving livy to a new host (prabhjyotsingh) * AMBARI-22698: Custom zeppelin interpreter properties are getting removed after moving zeppelin to a different host (prabhjyotsingh) * AMBARI-22696 Whitelist execute latency from Storm Ambari metrics * AMBARI-22759 [Hive Views 2.0] Deleting a Saved query is Buggy when Mutliple Queries exist in same Name * Fix compilation issues after trunk merge (mradhakrishnan) * Fix compilation issues in tests after trunk merge * AMBARI-22797. The stup-ldap tool should persists its output into Ambari database rather than ambari.properties * AMBARI-22797. Better CLI option names * AMBARI-22797. Dot not filter for PAM only; we should inform the end user when configuring LDAP in case the currently cond=figured auth method is not LDAP * AMBARI-22797. Removed redundant information from the JSON payload * AMBARI-22797. Removing 'is_root' check from setup-ldap (according to R. Levas this is irrelevant) * AMBARI-22797. Fixed error message * AMBARI-22797. Created constants to avoid typos * AMBARI-22797. Removed redundant constant (one already existed with the same value) * AMBARI-22797. Fixed CLI option names under Windows * AMBARI-22712. Update install Wizard layout. Second patch. (akovalenko) * AMBARI-22776 Ambari Blueprint 3.0/3.1 database tables and JPA objects + DDL cleanup (#109) (benyoka) * AMBARI-22776 Blueprint 3.0 Database schemas and JPA objects, DDL fixes (benyoka) * AMBARI-22776 Revert accidental commit (benyoka) * AMBARI-22776 Revert accidental change #2 (benyoka) * AMBARI-22776 Fix review findings (benyoka) * AMBARI-22776 Fix review findings #2 (benyoka) * AMBARI-22776 remove blueprint -> stack reference, docs (benyoka) * AMBARI-22776 fix build issues (benyoka) * AMBARI-22776 Add MpackInstanceEntity -> MpackEntity reference (benyoka) * [AMBARI-22817] Update backend code to handle new versioning schema. (#155) * AMBARI-22817. Update backend code to handle new versioning schema.(vbrodetskyi) * AMBARI-22817. Update backend code to handle new versioning schema.(vbrodetskyi) * AMBARI-22817. Update backend code to handle new versioning schema.(vbrodetskyi) * AMBARI-22817. Update backend code to handle new versioning schema.(vbrodetskyi) * AMBARI-22817. Update backend code to handle new versioning schema.(vbrodetskyi) * AMBARI-22817. Update backend code to handle new versioning schema.(vbrodetskyi) * AMBARI-22805. Improve Blueprints error handling in case of timeout (#185) * [AMBARI-22815] Change requiredServices in metainfo xml (#181) * AMBARI-22815 Change requiredServices in metainfo.xml (dsen) * AMBARI 22815 Change requiredServices in metainfo xml (renamed scope to dependencyType) (dsen) * AMBARI-22572. During cluster installation bower cannot resolve angularjs version (alexantonenko) * AMBARI-22572. During cluster installation bower cannot resolve angularjs version (angular version changed) (alexantonenko) * AMBARI-22776 Fix DDL issues resulting from incomplete table rename (benyoka) (#189) * AMBARI-22776 Fix DDL issues resulting from incomplete table rename (benyoka) * AMBARI-22776 More DDL fixes and renamings in the code (benyoka) * AMBARI-22413 fix tar_archive.archive_directory_dereference (benyoka) (#195) * AMBARI-22846. Get rid of Murmur Hash usage in Log Feeder for log events * AMBARI-22799 - define scheduling of archiving Infra Solr Documents * AMBARI-22840 Standardize precision when expressing durations * AMBARI-22782. Fix Namenode alerts broken due to enabling federation (aonishuk) * AMBARI-22853. NFS Gateway is not logging at the correct location (aonishuk) * AMBARI-22851 Host Details page style fixes * AMBARI-22861. Ambari UI for Scheduler configuration is trimming = from value sets * AMBARI-22862. Ambari admin breaks cause of responseInterceptors property * AMBARI-22861. Ambari UI for Scheduler configuration is trimming = from value sets (log removed) * [AMBARI-22845] Update service metainfo schema (#194) * AMBARI-22845 Update service metainfo schema (dsen) * AMBARI-22845 Update service metainfo schema - changes according to review (dsen) * AMBARI-22561. Need to address HDP-GPL repo update after user accepts license in post-install scenario (aonishuk) * AMBARI-22855. Log Search: using shipperconfig api should be configurable * AMBARI-22824. Let YARN/MR2 use ZK principal name set by users when enabling Kerberos (until now it's been hardcoded to 'zookeeper') * AMBARI-22847. Let HBase use ZK principal name set by users when enabling Kerberos (until now it's been hardcoded to 'zookeeper') * AMBARI-22860. Support external zookeeper for Infra Solr and Log Search * AMBARI-22866. Log Search: Add userList query for audit log requests. * AMBARI-22867 Add new state for Not Available data in Heatmap widget * AMBARI-22712. Update install Wizard layout. Third patch. (akovalenko) * [AMBARI-22858] First prereq not displaying for Free IPA method in Enable Kerberos Wizard * AMBARI-22791 support for parsing MulitE blueprints and cluster templates (#213) * AMBARI-22776 Blueprint 3.0 Database schemas and JPA objects, DDL fixes (benyoka) * AMBARI-22776 Revert accidental commit (benyoka) * AMBARI-22776 Revert accidental change #2 (benyoka) * AMBARI-22776 Fix review findings (benyoka) * AMBARI-22776 Fix review findings #2 (benyoka) * AMBARI-22791-Update blueprint object and convert from entity (benyoka) * AMBARI-22791-BlueprintFactory and tests (benyoka) * AMBARI-22791 Can store and retrieve multi-instance blueprint * AMBARI-22791 Build and unit test fixes (benyoka) * AMBARI-22791 merge branch-feature-AMBARI-14714 and fixes (benyoka) * AMBARI-22791 fix unit test failure (benyoka) * AMBARI-22791 fix import error (benyoka) * AMBARI-22791 Multi-everythin cluster template support (benyoka) * AMBARI-22791 fix review findings (benyoka) * AMBARI-22873: Remove HDP 3.0 stack from Ambari (jluniya) * AMBARI-22865. Moving out get_service_component_meta to DefaultStackAdvisor so that all children can use this function * AMBARI-22878: Update Service Group API to take list of mpack name associated with the service group * AMBARI-22869. Log Search: use default page and pageSize to log requests (+ rename shipper config api key name) * AMBARI-22874. Log Search: Return with 500 error for shipper config testing if an exception occurred. * AMBARI-22879 Ambari makes unrelated changes to zookeeper quorum config in all services when delete host action fails * AMBARI-20908 : Kafka advertised.listeners replaced in Kerberos Mode (bharatviswa via mradhakrishnan) * [AMBARI-22876] Disable consecutive authentication failure account lockout feature by default * [AMBARI-22882] Long cannot be cast to String error when changing a user's password * AMBARI-22841. Fix upgrade-catalog after accordingly to kerberos changes(echekanskiy) * [AMBARI-22881] Added user_authentication_id_seq into ambari_sequences upon upgrade to 2.7 * AMBARI-22881. Added user_authentication_id_seq into ambari_sequences upon upgrade to 2.7 * AMBARI-22881. Introduced a new function to fetch the maximum value of an ID column * AMBARI-22881. Using constant instead of hard-coded value * AMBARI-22830. Ambari-agent puts Python scripts in 2.6 directory on OSes that use python 2.7.x by default (aonishuk) * AMBARI-22893: Update rat check rules for stack files (jluniya) * AMBARI-22864. Agent commands hang even after freeing up disk space on the host (aonishuk) * [AMBARI-22712] Update install Wizard layout. Fourth patch. (#247) * AMBARI-22712. Update install Wizard layout. Fourth patch. (akovalenko) * AMBARI-22712. Update install Wizard layout. Fourth patch. (akovalenko) * AMBARI-22884. LogSearch Integration should call newer API to obtain log file metadata. * [AMBARI-22888] Cancel operation during package deployment causing repository manager to be broken (dgrinenko) * AMBARI-22890. LogSearch Integration support for external Shared LogSearch Server connections. * [AMBARI-22875] Blueprint cluster creation using manually installed mpacks (#231) * AMBARI-22878: Update Service Group API to take list of mpack name associated with the service group * [AMBARI-22885] LDAP sync fails with 'LDAP is not configured' error after configuring LDAP * AMBARI-22885. We fetch 'ldap enabled' flag from both ambari.properties (to support backward compatibility) and from AMBARI DB via our REST API * AMBARI-22885. Make sure we close response (even if an error occurred) * AMBARI-22885. Retrieving the 'is LDAP enabled' flag from AMBARI DB is enough (do not need to search it in ambari.properties) * AMBARI-22900 Log Search UI: implement 'History' functionality * AMBARI-22614. Fix some unit tests * [AMBARI-22889] LDAP configuration is not reloaded in Guice * AMBARI-22889. Make sure that the latest configuration is used when populating LDAP data * AMBARI-22889. LDAP authentication provider aos uses configuration provider instead of the configuration directly * AMBARI-22889. Fixed Python unit tests (not related to my changes) * AMBARI-22835 Log Search UI: implement log level filter * AMBARI-22905. Supporting old CLI option names in setup-ldap tool * AMBARI-22909 Log Search UI: implement filter by username for access logs * AMBARI-22712. Update install Wizard layout. Fifth patch. (akovalenko) * AMBARI-22911 Log Search UI: move Capture button to top menu * AMBARI-22912 Host details page: components not reconfigured after deleting their host * [AMBARI-22914] Oracle DDL is broken at trunk * [AMBARI-22886] Infra Manager: store s3 credentials in Hadoop credential store (#261) * AMBARI-22886 Infra Manager: store s3 credentials in Hadoop credential store * AMBARI-22886 Infra Manager: store s3 credentials in Hadoop credential store - addendum: use Optional<String> as a return type of getPassword, remove unnecessary ArrayList * AMBARI-22915 Dashboard-Metrics page style edits * [AMBARI-22910] Add Ambari admin/pw CLI options for setup-ldap tool * AMBARI-22910. Supporting Ambari admin user/pw CLI options in setup-ldap tool * AMBARI-22910. Using more generic option names for Ambari username/password * AMBARI-22917. Log Search make logsearch maven build to independent from other modules * AMBARI-22903. Log Search: Add Knox to docker dev env (#265) * AMBARI-22903. Log Search: Add Knox to docker dev env * AMBARI-22890. Use quotes for checking bash variables * AMBARI-22833 : change commons-collections-3.2.1.jar being used by ambari views to commons-collections-3.2.2.jar (nitirajrathore) (#188) merging as changes are approved * AMBARI-22921. Unable to enable hive interactive, LLAP, on our Ambari 2.5.2 managed cluster (aonishuk) * [AMBARI-22913]. Add ability to MasterHostResolver to resolve by namespa… (#269) * AMBARI-22913. Add ability to MasterHostResolver to resolve by namespace at a time (amagyar) * AMBARI-22913. Add ability to MasterHostResolver to resolve by namespace at a time (amagyar) * AMBARI-22902. Changed description of Kadmin Host * [AMBARI-22904] Revised mpack APIs (#252) * [AMBARI-22904] Revised mpack APIs * [AMBARI-22904] Revised mpack APIs * [AMBARI-22904] Revised mpack APIs * [AMBARI-22904] Revised mpack APIs * [AMBARI-22904] Revised mpack APIs * [AMBARI-22904] Revised mpack APIs * [AMBARI-22904] Revised mpack APIs * [AMBARI-22854] Fix running service checks with new mpack definitions (#204) * AMBARI-22854 Fix running service checks with new mpack definitions (dsen) * AMBARI-22854 Fix running service checks with new mpack definitions - removed obvious javadocs (dsen) * Merge trunk into branch-feature-AMBARI-14714 * Branch feature ambari 14714 (#291) * [AMBARI-22904] Revised mpack APIs * [AMBARI-22904] Revised mpack APIs * [AMBARI-22904] Revised mpack APIs * [AMBARI-22904] Revised mpack APIs * [AMBARI-22904] Revised mpack APIs * [AMBARI-22904] Revised mpack APIs * [AMBARI-22904] Revised mpack APIs * [AMBARI-22904] Revised mpack APIs - Fix for dependencyType -> Type in ModuleDependency class. * Branch feature ambari 14714 (#307) * [AMBARI-22948] stack-hooks has to be updated to use cluster-settings instead of cluster-env * [AMBARI-22938] Fix SQL issue due to trunk merge issue * AMBARI-22971: Remove current_mpack_id to mpack_id in stacks table (jluniya) * AMBARI-22971: Remove current_mpack_id to mpack_id in stacks table (jluniya) * [AMBARI-22957] Stack Metainfo.xml should contain the osSpecifics tag * [AMBARI-22980] Commenting out stack_features and stack_select calls until instance_manager is ready * [AMBARI-22980] Issue warnings when stack select tool is not present * [AMBARI-22995] Remove deprecated Upgrade Packs * [AMBARI-22992] Update error handling during mpack installation (#363) * [AMBARI-22992] Update error handling during mpack installation * [AMBARI-22997] cluster-settings.xml is not present in /var/lib/ambari-server/resources post ambari installation * [AMBARI-22992] Update error handling during mpack installation - review comments * [AMBARI-22980] Correct the name of the repo_suse_rhel_template * [AMBARI-23036] Create Lifecycle changes for Upgrade Packs (#430) * [AMBARI-23036] Create Lifecycle changes for Upgrade Packs * [AMBARI-23036] Update new enum for ordering (review comments) * [AMBARI-23004] Create initial version of InstanceManager to create instance layout (dsen) * Changes according to the review + general improvements * [AMBARI-23004] Create initial version of InstanceManager to create instance layout - additional fix (dsen) (#441) * AMBARI-22649. settings library should convert true/false to boolean * AMBARI-22979: Update software registry API to support new mpack schema (jluniya) (#433) * [AMBARI-22883] A new REST API: GET /api/v1/hosts?format=summary or /api/v1/clusters/… (#285) * A new REST API: GET /api/v1/hosts?format=summary or /api/v1/clusters/{cluster_name}/hosts?format=summary is supported with this implementation and return the number of hosts running each possible operating system. More aggregation info of hosts can be also added in this code structure. * A new REST API: GET /api/v1/hosts?format=summary or /api/v1/clusters/{cluster_name}/hosts?format=summary is supported with this implementation and return the number of hosts running each possible operating system. More aggregation info of hosts can be also added in this code structure. (Rebased) * A new REST API: GET /api/v1/hosts?format=summary or /api/v1/clusters/{cluster_name}/hosts?format=summary is supported with this implementation and return the number of hosts running each possible operating system. More aggregation info of hosts can be also added in this code structure. (Rebased) * [AMBARI-23053] Rename NON-ROLLING to EXPRESS * AMBARI-22995. Remove deprecated Upgrade Packs - after reintroduced by later merge commit * AMBARI-22883. NPE in HostResourceDefinition (#458) * AMBARI-22878. Fix unit tests broken by reference to non-existent stack (#449) * AMBARI-22614 fix some unit tests (benyoka) (#461) * AMBARI-22614 fix some unit tests (benyoka) * AMBARI-22614 fix some unit tests / remove unused import (benyoka) * AMBARI-22878. Fix unit tests (#473) * Few more unit test fixes (#486) * AMBARI-22244 * AMBARI-22854 * AMBARI-22883 * AMBARI-22904 * [AMBARI-22854] Fix running service checks with new mpack definitions - UT fix (dsen) (#495) * [AMBARI-22854] Fix running service checks with new mpack definitions - UT fix (dsen) * [AMBARI-22854] Fix running service checks with new mpack definitions (dsen) - addendum * AMBARI-22875. Use mpacks specified in cluster creation request (#444) * AMBARI-22875. Fix unit tests (#507) * [AMBARI-23032] fix exceptions, export cluster settings and single mpack (#431) * AMBARI-23032 fix execptions, export cluster settings and single mpacks (benyoka) * AMBARI-23032 fix review comments (benyoka) * AMBARI-23032 better method naming (benyoka) * AMBARI-23032 fix review comments #2 (benyoka) * AMBARI-23032 fix review comments #3 (benyoka) * AMBARI-23032 fix swallowed exceptions during mpack installation * AMBARI-22875. Exclude generated sources from checkstyle (#512) * AMBARI-23107 - Deprecate Repo Versions And Start Tracking Repo OS's F… (#515) * AMBARI-22875. Add missing license (#519) * AMBARI-22883. Count of each OS type is always 1 (#516) * [AMBARI-23121] Fix few instance manager issues found during ZK deploment (dsen) (#518) * [AMBARI-23121] Fix few instance manager issues found during ZK deployment (dsen) * [AMBARI-23121] Fix few instance manager issues found during ZK deployment (dsen) - changes according to review * [Ambari 23125] Move check-properties XML (#521) * AMBARI-23125. Move check-properties XML * AMBARI-23125. Fixed for extraneous code * AMBARI-23053. Fix unexpected indent in kafka_broker.py (#535) * AMBARI-22883. Let HostSummaryRenderer add Hosts/os_type property to the query (#539) * AMBARI-22945. Enhance host components API to support multiple host component instances. * [Ambari 23133] Hooks scripts has hadoop specific code * [AMBARI-23120] Create helper functions for using new instance manager library from the agent scripts (dsen) (#517) * [AMBARI-23120] Create helper functions for using new instance manager library from the agent scripts (dsen) * [AMBARI-23120] Create helper functions for using new instance manager library from the agent scripts (dsen) - changes according to review * [AMBARI-23120] Create helper functions for using new instance manager library from the agent scripts (dsen) - import fix (#560) * AMBARI-23147 - Expose Repository CRUD via the Mpack Endpoint (#552) * AMBARI-23158. Fix NPE in Host Components query. * [AMBARI-23130] Persist cluster creation request (#559) * AMBARI-23130 persist raw cluster provision request and extract stack ids on server restart (benyoka) * AMBARI-23130 add columnt to other DDLs + fix DDLs (benyoka) * AMBARI-23130 fix review findings (benyoka) * AMBARI-22614 fix OrmTestHelper to support mpacks in unit tests (benyoka) (#569) * AMBARI-23156. Remove Required Group Name and Expose Lifecycle to API * AMBARI-23167 - StackOverflowError thrown during cluster creation * [AMBARI-23170] Instance manager json output should have the same structure for both client and server modules (dsen) * AMBARI-23175. Fix for PUT Host Component API failing because validation check fails. * AMBARI-23176 - Some Minor Fixes For Failing Tests Due to Repo Version Refactor (#587) * [AMBARI-22875] Adopt changes in Host Component API for Blueprint cluster creation (#581) * [AMBARI-23177] Yarn-MR separation and minor deployment fixes (#588) * [AMBARI-23177] Yarn-MR separation and minor deployment fixes * Review comments * Review comments * AMBARI-23183. Revert UI changes made to branch : branch-feature-AMBARI-14714. * [Ambari-22990] Provide Bootstrap test connection feature (#530) * [Ambari-22990] Provide Bootstrap test connection feature * [Ambari-22990] Provide Bootstrap test connection feature * Add newline to avoid audit failures

jonathan-hurley reviewed Feb 7, 2018

View reviewed changes

adoroszlai suggested changes Feb 13, 2018

View reviewed changes

jonathan-hurley suggested changes Feb 15, 2018

View reviewed changes

scottduan and others added 2 commits February 15, 2018 22:40

resolve merge conflict issue

17c394a

Scott Duan added 2 commits February 18, 2018 01:21

resolve merge conflict issue

473c727

Fix merge conflict

e9887af

adoroszlai suggested changes Feb 21, 2018

View reviewed changes

jonathan-hurley reviewed Feb 21, 2018

View reviewed changes

jonathan-hurley suggested changes Feb 21, 2018

View reviewed changes

Scott Duan added 2 commits February 21, 2018 10:53

Fix merge conflict

b7c1b8c

Merge branch 'AMBARI-22883-branch-feature-AMBARI-14714' of https://gi…

9e30788

…thub.com/scottduan/ambari into AMBARI-22883-branch-feature-AMBARI-14714

jonathan-hurley reviewed Feb 22, 2018

View reviewed changes

jonathan-hurley approved these changes Feb 22, 2018

View reviewed changes

adoroszlai approved these changes Feb 22, 2018

View reviewed changes

jayush approved these changes Feb 22, 2018

View reviewed changes

jayush merged commit 7675ce8 into apache:branch-feature-AMBARI-14714 Feb 22, 2018

adoroszlai mentioned this pull request Feb 27, 2018

Few more unit test fixes #486

Merged

scottduan deleted the AMBARI-22883-branch-feature-AMBARI-14714 branch March 15, 2018 05:14

Conversation

scottduan commented Feb 7, 2018

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

scottduan commented Feb 7, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jonathan-hurley left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jayush Feb 7, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jonathan-hurley Feb 7, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

asfgit commented Feb 7, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jayush Feb 7, 2018 •

edited

Loading

jonathan-hurley Feb 7, 2018 •

edited

Loading