Enterprise Testing System (#463)
authorDavid Shulman <david.shulman@microsoft.com>
Thu, 5 Dec 2019 23:32:13 +0000 (15:32 -0800)
committerGitHub <noreply@github.com>
Thu, 5 Dec 2019 23:32:13 +0000 (15:32 -0800)
This is the first of several PRs that add Enterprise Scenarios Testing capability to
the repo. This PR focusses on Linux which allows for docker containers to be used
in an enterprise network configuration.

I focussed on 2 workflows: 1) The 'dev' workflow, and 2) The PR/CI workflow. The dev
workflow works well since it's using containers in a docker-compose environment along
with volume mounting your current dev's repo enlistment. The PR/CI workflow gives us
an Azure DevOps pipeline to automate verification.

I still need to work with the infra team to add a real pipeline that will run. I can't
do that until this is merged. In the meantime, I have my own DevOps pipeline that verified this PR.

See: https://dev.azure.com/systemnetncl/Enterprise%20Testing/_build/results?buildId=141

I will be linking a follow-up GitHub issue describing the roadmap for building on this system
including adding Windows environments, NTLM protocol, proxies, and other libraries such as
System.Net.Mail and System.Data.SqlClient. Those libraries also use Negotiate/Kerberos/NTLM
enterprise-oriented protocols.

Contributes to:
https://github.com/dotnet/corefx/issues/41652
https://github.com/dotnet/corefx/issues/41489
https://github.com/dotnet/corefx/issues/36896
https://github.com/dotnet/corefx/issues/30150
https://github.com/dotnet/corefx/issues/24707
https://github.com/dotnet/corefx/issues/10041
https://github.com/dotnet/corefx/issues/6606
https://github.com/dotnet/corefx/issues/6161

* Address PR feedback

* Change pipeline *.yml to only run on selected filepaths for PRs
* Change kdc container Dockerfile to be based on ubuntu:18.04
* Fix typo in README.md

* Update .yml file

* Link (instead of copy) apache kerb module to the right place

23 files changed:
eng/pipelines/libraries/enterprise/linux.yml [new file with mode: 0644]
src/libraries/Common/tests/System/Net/EnterpriseTests/EnterpriseTestConfiguration.cs [new file with mode: 0644]
src/libraries/Common/tests/System/Net/EnterpriseTests/setup/README.md [new file with mode: 0644]
src/libraries/Common/tests/System/Net/EnterpriseTests/setup/apacheweb/Dockerfile [new file with mode: 0644]
src/libraries/Common/tests/System/Net/EnterpriseTests/setup/apacheweb/run.sh [new file with mode: 0644]
src/libraries/Common/tests/System/Net/EnterpriseTests/setup/common/krb5.conf [new file with mode: 0644]
src/libraries/Common/tests/System/Net/EnterpriseTests/setup/docker-compose.yml [new file with mode: 0644]
src/libraries/Common/tests/System/Net/EnterpriseTests/setup/kdc/Dockerfile [new file with mode: 0644]
src/libraries/Common/tests/System/Net/EnterpriseTests/setup/kdc/kadm5.acl [new file with mode: 0644]
src/libraries/Common/tests/System/Net/EnterpriseTests/setup/kdc/kdc.conf [new file with mode: 0644]
src/libraries/Common/tests/System/Net/EnterpriseTests/setup/kdc/run.sh [new file with mode: 0644]
src/libraries/Common/tests/System/Net/EnterpriseTests/setup/kdc/setup-kdc.sh [new file with mode: 0644]
src/libraries/Common/tests/System/Net/EnterpriseTests/setup/linuxclient/Dockerfile [new file with mode: 0644]
src/libraries/Common/tests/System/Net/EnterpriseTests/setup/linuxclient/run.sh [new file with mode: 0644]
src/libraries/Common/tests/System/Net/EnterpriseTests/setup/linuxclient/test-webserver.sh [new file with mode: 0644]
src/libraries/System.Net.Http/tests/EnterpriseTests/Configurations.props [new file with mode: 0644]
src/libraries/System.Net.Http/tests/EnterpriseTests/HttpClientAuthenticationTest.cs [new file with mode: 0644]
src/libraries/System.Net.Http/tests/EnterpriseTests/README.md [new file with mode: 0644]
src/libraries/System.Net.Http/tests/EnterpriseTests/System.Net.Http.Enterprise.Tests.csproj [new file with mode: 0644]
src/libraries/System.Net.Security/tests/EnterpriseTests/Configurations.props [new file with mode: 0644]
src/libraries/System.Net.Security/tests/EnterpriseTests/NegotiateStreamLoopbackTest.cs [new file with mode: 0644]
src/libraries/System.Net.Security/tests/EnterpriseTests/README.md [new file with mode: 0644]
src/libraries/System.Net.Security/tests/EnterpriseTests/System.Net.Security.Enterprise.Tests.csproj [new file with mode: 0644]

diff --git a/eng/pipelines/libraries/enterprise/linux.yml b/eng/pipelines/libraries/enterprise/linux.yml
new file mode 100644 (file)
index 0000000..695bc7f
--- /dev/null
@@ -0,0 +1,73 @@
+
+# Disable pipeline for ordinary pushes to the branches
+trigger: none
+
+# To reduce load on the pipeline, enable it only for PRs that affect critical networking code
+pr:
+  branches:
+    include:
+    - master
+    - release/*.*
+
+  paths:
+    include:
+      - src/libraries/Common/src/System/Net/*
+      - src/libraries/Common/tests/System/Net/*
+      - src/libraries/Native/Unix/System.Net.Security.Native/*
+      - src/libraries/System.Net.Http/*
+      - src/libraries/System.Net.Security/*
+
+pool:
+  vmImage: 'ubuntu-16.04'
+
+variables:
+  - template: ../variables.yml
+  - name: enterpriseTestsSetup
+    value: $(sourcesRoot)/Common/tests/System/Net/EnterpriseTests/setup
+  - name: containerRunTestsCommand
+    value: /repo/.dotnet/dotnet msbuild /t:rebuildandtest
+  - name: containerLibrariesRoot
+    value: /repo/src/libraries
+
+steps:
+- bash: |
+    cd $(enterpriseTestsSetup)
+    docker-compose build
+  displayName: Build test machine images
+  env:
+    DOTNET_RUNTIME_REPO_ROOT: $(Build.SourcesDirectory)
+
+- bash: |
+    cd $(enterpriseTestsSetup)
+    docker-compose up -d
+  displayName: Start test network and machines
+  env:
+    DOTNET_RUNTIME_REPO_ROOT: $(Build.SourcesDirectory)
+
+- bash: |
+    docker exec linuxclient bash /setup/test-webserver.sh
+  displayName: Test linuxclient connection to web server
+
+- bash: |
+    docker exec linuxclient bash /repo/libraries.sh
+  displayName: Build product sources
+
+- bash: |
+    docker exec linuxclient $(containerRunTestsCommand) $(containerLibrariesRoot)/System.Net.Http/tests/EnterpriseTests/System.Net.Http.Enterprise.Tests.csproj
+    docker exec linuxclient $(containerRunTestsCommand) $(containerLibrariesRoot)/System.Net.Security/tests/EnterpriseTests/System.Net.Security.Enterprise.Tests.csproj
+  displayName: Build and run tests
+
+- bash: |
+    cd $(enterpriseTestsSetup)
+    docker-compose down
+  displayName: Stop test network and machines
+  env:
+    DOTNET_RUNTIME_REPO_ROOT: $(Build.SourcesDirectory)
+
+- task: PublishTestResults@2
+  inputs:
+    testRunner: 'xUnit'
+    testResultsFiles: '**/testResults.xml'
+    testRunTitle: 'Enterprise Tests'
+    mergeTestResults: true
+    failTaskOnFailedTests: true
diff --git a/src/libraries/Common/tests/System/Net/EnterpriseTests/EnterpriseTestConfiguration.cs b/src/libraries/Common/tests/System/Net/EnterpriseTests/EnterpriseTestConfiguration.cs
new file mode 100644 (file)
index 0000000..adb8d5c
--- /dev/null
@@ -0,0 +1,16 @@
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+// See the LICENSE file in the project root for more information.
+
+namespace System.Net.Test.Common
+{
+    public static class EnterpriseTestConfiguration
+    {
+        public const string Realm = "LINUX.CONTOSO.COM";
+        public const string NegotiateAuthWebServer = "http://apacheweb.linux.contoso.com";
+
+        public static bool Enabled => !string.IsNullOrEmpty(Environment.GetEnvironmentVariable("DOTNET_RUNTIME_ENTERPRISETESTS_ENABLED"));
+        public static NetworkCredential ValidNetworkCredentials => new NetworkCredential("user1", "password");
+        public static NetworkCredential InvalidNetworkCredentials => new NetworkCredential("user1", "passwordxx");
+    }
+}
diff --git a/src/libraries/Common/tests/System/Net/EnterpriseTests/setup/README.md b/src/libraries/Common/tests/System/Net/EnterpriseTests/setup/README.md
new file mode 100644 (file)
index 0000000..d5a3ee4
--- /dev/null
@@ -0,0 +1,84 @@
+# Enterprise Scenario Testing
+
+## What Are Enterprise Scenarios?
+There are many definitions for enterprise scenarios. But generally in terms of how .NET Core networking APIs are used, enterprise scenarios are those networking scenarios that are fundamentally used by businesses (a.k.a enterprises) compared with consumers. As such, they use networking components, protocols, and security authentication mechanisms that are not used by most consumers using their home networking and Internet connections. 
+
+## Networking Components of Enterprise Scenarios
+Enterprise scenarios typically see the following kinds of components/protocols/security:
+* Although possibly connected to the Internet, most of the networking topology is internal facing. There is use of some internal “directory” service for authentication of connected computers and users. On Windows, this can include Windows Active Directory domains with computers being domain-controllers, domain-joined, or standalone computers. On Linux, this includes Kerberos realms using KDCs and participating member computers. With .NET Core being cross-platform, this now includes connections with multiple domains and realms with various cross trust between them.
+* Authentication protocols such as NTLM, Kerberos and Negotiate. These are used more than Basic and Digest. Negotiate/Kerberos requires both client and server computers to be “joined” to a common trusted directory service.
+* TLS/SSL extensively used.
+* The use of proxy servers for HTTP/HTTPS communication. These usually include authenticated proxies where the proxy server demands authentication typically with Negotiate or NTLM authentication schemes.
+* Complex DNS architectures using various A and CNAME (alias) records.
+* Use of the NegotiateStream class. This is typically seen as part of using WCF client/server architecture.
+* Impersonation/Delegation of credentials. This occurs frequently in middle-tier scenarios that involve a client computer talking with a middle-tier computer (such as a web server) which makes an outbound call to another computer such as a database server. In this case, the credentials of the client computer are delegated across the middle-tier computer so that the outbound call to the database server is made in the context of the client’s credentials.
+
+
+## Running the tests
+These tests need to be run in the dedicated Enterprise Test environment. This environment can be created on a local dev machine as long as Docker in installed. The enterprise test environment is a collection of Linux docker containers (client and servers) connected on a common docker network.
+
+Set the DOTNET_RUNTIME_REPO_ROOT environment variable to the path on your dev machine (the host machine) where the repo is installed:
+
+```
+# Windows cmd.exe shell example
+set DOTNET_RUNTIME_REPO_ROOT=s:\GitHub\runtime
+```
+
+```bash
+# Linux bash shell example
+export DOTNET_RUNTIME_REPO_ROOT=/home/me/GitHub/runtime
+```
+
+Now you can start up the enterprise system on your dev machine.
+
+```
+# Build test machine images
+cd %DOTNET_RUNTIME_REPO_ROOT%\src\libraries\Common\tests\System\Net\EnterpriseTests\setup
+docker-compose build
+
+# Start up test machines and network
+docker-compose up -d
+
+# Connect to the 'linuxclient' container
+docker exec -it linuxclient bash
+```
+
+At this point, you are in the linuxclient container. It is one "machine" which is part of an enterprise network.
+
+Now build the repo as you would on a regular dev machine:
+
+```bash
+cd /repo
+./libraries.sh
+```
+
+Now you can run the enterprise tests. Currently, there are tests for System.Net.Http and System.Net.Security. You can run them in the same way you already run tests in the repo.
+
+
+(System.Net.Http example shown)
+
+```bash
+cd /repo/src/libraries/System.Net.Http/tests/EnterpriseTests
+/repo/.dotnet/dotnet msbuild /t:rebuildandtest
+```
+
+You can exit from the container bash shell:
+
+```bash
+exit
+```
+
+But the containers stay running. You can re-connect to the client again anytime with the same command:
+
+```
+docker exec -it linuxclient bash
+```
+
+You can edit source code on your local machine and then rebuild and rerun tests as needed.
+
+When you are done with the enterprise test network, you can shut it down from your local dev machine.
+
+```
+cd %DOTNET_RUNTIME_REPO_ROOT%\src\libraries\Common\tests\System\Net\EnterpriseTests\setup
+docker-compose down
+```
diff --git a/src/libraries/Common/tests/System/Net/EnterpriseTests/setup/apacheweb/Dockerfile b/src/libraries/Common/tests/System/Net/EnterpriseTests/setup/apacheweb/Dockerfile
new file mode 100644 (file)
index 0000000..4275987
--- /dev/null
@@ -0,0 +1,25 @@
+FROM httpd:2.4
+
+COPY ./common/krb5.conf /etc/
+
+WORKDIR /setup
+COPY ./apacheweb/*.sh ./
+RUN chmod +x *.sh
+
+# Prevents dialog prompting when installing packages
+ARG DEBIAN_FRONTEND=noninteractive
+
+# Install Kerberos client, apache Negotiate auth plugin, and diagnostics
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends libapache2-mod-auth-kerb procps krb5-user iputils-ping dnsutils nano
+
+# Link apache2 kerb module to the right place since the apt-get install puts it in the wrong place for this docker image
+RUN ln -s /usr/lib/apache2/modules/mod_auth_kerb.so /usr/local/apache2/modules
+
+# Modify httpd.conf to add Negotiate auth as required
+RUN echo "LoadModule auth_kerb_module modules/mod_auth_kerb.so" >> /usr/local/apache2/conf/httpd.conf && \
+    sed -i 's/Require all granted/AuthType Kerberos\nAuthName "Kerberos Login"\nKrbAuthRealm LINUX\.CONTOSO\.COM\nKrb5Keytab \/etc\/krb5\.keytab\nKrbMethodK5Passwd off\nRequire valid-user/' /usr/local/apache2/conf/httpd.conf
+
+EXPOSE 80
+
+ENTRYPOINT ["/bin/sh", "/setup/run.sh"]
diff --git a/src/libraries/Common/tests/System/Net/EnterpriseTests/setup/apacheweb/run.sh b/src/libraries/Common/tests/System/Net/EnterpriseTests/setup/apacheweb/run.sh
new file mode 100644 (file)
index 0000000..e0017db
--- /dev/null
@@ -0,0 +1,5 @@
+#!/bin/sh
+
+cp /SHARED/apacheweb.keytab /etc/krb5.keytab
+
+exec httpd -DFOREGROUND "$@"
diff --git a/src/libraries/Common/tests/System/Net/EnterpriseTests/setup/common/krb5.conf b/src/libraries/Common/tests/System/Net/EnterpriseTests/setup/common/krb5.conf
new file mode 100644 (file)
index 0000000..ddec2ce
--- /dev/null
@@ -0,0 +1,20 @@
+[libdefaults]
+        default_realm = LINUX.CONTOSO.COM
+
+# The following krb5.conf variables are only for MIT Kerberos.
+        kdc_timesync = 1
+        ccache_type = 4
+        forwardable = true
+        proxiable = true
+
+# The following libdefaults parameters are only for Heimdal Kerberos.
+        fcc-mit-ticketflags = true
+
+[realms]
+        LINUX.CONTOSO.COM = {
+                kdc = kdc.linux.contoso.com
+                admin_server = kdc.linux.contoso.com
+        }
+
+[domain_realm]
+        .linux.contoso.com = LINUX.CONTOSO.COM
diff --git a/src/libraries/Common/tests/System/Net/EnterpriseTests/setup/docker-compose.yml b/src/libraries/Common/tests/System/Net/EnterpriseTests/setup/docker-compose.yml
new file mode 100644 (file)
index 0000000..932692d
--- /dev/null
@@ -0,0 +1,59 @@
+version: "3.7"
+
+services:
+  kdc:
+    build:
+      context: ./
+      dockerfile: ./kdc/Dockerfile
+    image: kdc:latest
+    container_name: kdc
+    hostname: kdc
+    domainname: linux.contoso.com
+    dns_search: linux.contoso.com
+    volumes:
+      - shared-volume:/SHARED
+    networks:
+      - network1
+
+  apacheweb:
+    build:
+      context: ./
+      dockerfile: ./apacheweb/Dockerfile
+    image: apacheweb:latest
+    container_name: apacheweb
+    hostname: apacheweb
+    domainname: linux.contoso.com
+    dns_search: linux.contoso.com
+    volumes:
+      - shared-volume:/SHARED
+    networks:
+      network1:
+        aliases:
+          - apache.linux.contoso.com
+    depends_on:
+      - kdc
+
+  linuxclient:
+    build:
+      context: ./
+      dockerfile: ./linuxclient/Dockerfile
+    image: linuxclient:latest
+    container_name: linuxclient
+    hostname: linuxclient
+    domainname: linux.contoso.com
+    dns_search: linux.contoso.com
+    volumes:
+      - shared-volume:/SHARED
+      - ${DOTNET_RUNTIME_REPO_ROOT}:/repo
+    networks:
+      - network1
+    depends_on:
+      - apacheweb
+      - kdc
+
+networks:
+  network1:
+    name: linux.contoso.com
+
+volumes:
+  shared-volume:
diff --git a/src/libraries/Common/tests/System/Net/EnterpriseTests/setup/kdc/Dockerfile b/src/libraries/Common/tests/System/Net/EnterpriseTests/setup/kdc/Dockerfile
new file mode 100644 (file)
index 0000000..1b28145
--- /dev/null
@@ -0,0 +1,27 @@
+FROM ubuntu:18.04
+
+COPY ./kdc/kadm5.acl /etc/krb5kdc/
+COPY ./kdc/kdc.conf /etc/krb5kdc/
+COPY ./common/krb5.conf /etc/
+
+RUN mkdir /SHARED
+
+WORKDIR /setup
+COPY ./kdc/*.sh ./
+RUN chmod +x *.sh
+
+# Prevents dialog prompting when installing packages
+ARG DEBIAN_FRONTEND=noninteractive
+
+# Install KDC and diagnostic tools
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends krb5-kdc krb5-admin-server iputils-ping dnsutils nano
+
+RUN ./setup-kdc.sh
+
+VOLUME /SHARED
+
+EXPOSE 88/tcp
+EXPOSE 88/udp
+
+ENTRYPOINT ["/bin/bash", "/setup/run.sh"]
diff --git a/src/libraries/Common/tests/System/Net/EnterpriseTests/setup/kdc/kadm5.acl b/src/libraries/Common/tests/System/Net/EnterpriseTests/setup/kdc/kadm5.acl
new file mode 100644 (file)
index 0000000..7c32326
--- /dev/null
@@ -0,0 +1,6 @@
+# /etc/krb5kdc/kadm5.acl -- Kerberos V5 general configuration.
+#
+# The user "root/admin" has full permissions
+# Other users can inquire or list principals or policies
+*/admin@LINUX.CONTOSO.COM  *
+*/*@LINUX.CONTOSO.COM      il
diff --git a/src/libraries/Common/tests/System/Net/EnterpriseTests/setup/kdc/kdc.conf b/src/libraries/Common/tests/System/Net/EnterpriseTests/setup/kdc/kdc.conf
new file mode 100644 (file)
index 0000000..07ecac6
--- /dev/null
@@ -0,0 +1,21 @@
+[kdcdefaults]
+    kdc_ports = 88
+
+[realms]
+    LINUX.CONTOSO.COM = {
+        database_name = /var/lib/krb5kdc/principal
+        admin_keytab = FILE:/etc/krb5kdc/kadm5.keytab
+        acl_file = /etc/krb5kdc/kadm5.acl
+        key_stash_file = /etc/krb5kdc/stash
+        kdc_ports = 88
+        max_life = 10h 0m 0s
+        max_renewable_life = 7d 0h 0m 0s
+        master_key_type = des3-hmac-sha1
+        #supported_enctypes = aes256-cts:normal aes128-cts:normal
+        default_principal_flags = +preauth
+    }
+
+[logging]
+        default = FILE:/var/log/kerberos/krb5.log
+        admin_server = FILE:/var/log/kerberos/kadmin.log
+        kdc = FILE:/var/log/kerberos/krb5lib.log
diff --git a/src/libraries/Common/tests/System/Net/EnterpriseTests/setup/kdc/run.sh b/src/libraries/Common/tests/System/Net/EnterpriseTests/setup/kdc/run.sh
new file mode 100644 (file)
index 0000000..c26dd61
--- /dev/null
@@ -0,0 +1,10 @@
+#!/usr/bin/env bash
+
+service krb5-kdc restart
+service krb5-admin-server restart
+
+cp /setup/*.keytab /SHARED
+chmod +r /SHARED/*.keytab
+
+# Keep the container running
+tail -f /dev/null
diff --git a/src/libraries/Common/tests/System/Net/EnterpriseTests/setup/kdc/setup-kdc.sh b/src/libraries/Common/tests/System/Net/EnterpriseTests/setup/kdc/setup-kdc.sh
new file mode 100644 (file)
index 0000000..1418390
--- /dev/null
@@ -0,0 +1,28 @@
+#!/usr/bin/env bash
+
+# Kerbose Logging
+mkdir -pv /var/log/kerberos/
+touch /var/log/kerberos/krb5.log
+touch /var/log/kerberos/kadmin.log
+touch /var/log/kerberos/krb5lib.log
+
+# Create Kerberos database
+kdb5_util create -r LINUX.CONTOSO.COM -P password -s
+
+# Start KDC service
+krb5kdc
+
+# Add users
+kadmin.local -q "add_principal -pw password root/admin@LINUX.CONTOSO.COM"
+kadmin.local -q "add_principal -pw password user1@LINUX.CONTOSO.COM"
+
+# Add SPNs for services
+kadmin.local -q "add_principal -pw password HTTP/apacheweb.linux.contoso.com"
+kadmin.local -q "add_principal -pw password HOST/linuxclient.linux.contoso.com"
+kadmin.local -q "add_principal -pw password HOST/localhost"
+kadmin.local -q "add_principal -pw password NEWSERVICE/localhost"
+
+# Create keytab files for other machines
+kadmin.local ktadd -k /SHARED/apacheweb.keytab -norandkey HTTP/apacheweb.linux.contoso.com
+kadmin.local ktadd -k /SHARED/linuxclient.keytab -norandkey HOST/linuxclient.linux.contoso.com
+kadmin.local ktadd -k /SHARED/linuxclient.keytab -norandkey HOST/localhost
diff --git a/src/libraries/Common/tests/System/Net/EnterpriseTests/setup/linuxclient/Dockerfile b/src/libraries/Common/tests/System/Net/EnterpriseTests/setup/linuxclient/Dockerfile
new file mode 100644 (file)
index 0000000..8bb94ba
--- /dev/null
@@ -0,0 +1,39 @@
+# Switch to mcr.microsoft.com/dotnet-buildtools/prereqs ubuntu 18.04 images when they are fixed
+FROM ubuntu:18.04
+
+# Prevents dialog prompting when installing packages
+ARG DEBIAN_FRONTEND=noninteractive
+
+# This 'RUN' step can be removed once dotnet-buildtools/prereqs image is fixed
+#
+# Makes the image capable of building and running tests in dotnet-runtime repo.
+# Add retries to apt-get since the ubuntu package servers have had errors lately such as:
+#   "E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/p/publicsuffix/publicsuffix_20180223.1310-1_all.deb  Undetermined Error [IP: 91.189.88.31 80]"
+ARG APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE=1
+RUN echo "APT::Acquire::Retries \"10\";" > /etc/apt/apt.conf.d/80-retries && \
+    apt-get update && \
+    apt-get install -y --no-install-recommends apt-transport-https ca-certificates gnupg software-properties-common wget && \
+    wget -O - https://apt.kitware.com/keys/kitware-archive-latest.asc 2>/dev/null | apt-key add - && \
+    apt-add-repository 'deb https://apt.kitware.com/ubuntu/ bionic main' && \
+    wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key 2>/dev/null | apt-key add - && \
+    apt-add-repository 'deb http://apt.llvm.org/bionic/ llvm-toolchain-bionic-9 main' && \
+    apt-get update && \
+    apt-get install -y --no-install-recommends cmake llvm-9 clang-9 lldb-6.0 liblldb-6.0-dev libunwind8 libunwind-dev gettext libicu-dev liblttng-ust-dev libssl-dev libnuma-dev libkrb5-dev locales && \
+    locale-gen en_US.UTF-8 && \
+    update-locale LANG=en_US.UTF-8
+
+# Install Kerberos, NTLM, and diagnostic tools
+COPY ./common/krb5.conf /etc/krb5.conf
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends krb5-user gss-ntlmssp iputils-ping dnsutils nano curl
+
+# Set environment variable to turn on enterprise tests
+ENV DOTNET_RUNTIME_ENTERPRISETESTS_ENABLED 1
+
+WORKDIR /setup
+COPY ./linuxclient/*.sh ./
+RUN chmod +x *.sh
+
+WORKDIR /repo
+
+ENTRYPOINT ["/bin/bash", "/setup/run.sh"]
diff --git a/src/libraries/Common/tests/System/Net/EnterpriseTests/setup/linuxclient/run.sh b/src/libraries/Common/tests/System/Net/EnterpriseTests/setup/linuxclient/run.sh
new file mode 100644 (file)
index 0000000..1b37d45
--- /dev/null
@@ -0,0 +1,6 @@
+#!/usr/bin/env bash
+
+cp /SHARED/linuxclient.keytab /etc/krb5.keytab
+
+# Keep the container running
+tail -f /dev/null
diff --git a/src/libraries/Common/tests/System/Net/EnterpriseTests/setup/linuxclient/test-webserver.sh b/src/libraries/Common/tests/System/Net/EnterpriseTests/setup/linuxclient/test-webserver.sh
new file mode 100644 (file)
index 0000000..2afd01f
--- /dev/null
@@ -0,0 +1,6 @@
+#!/usr/bin/env bash
+
+kdestroy
+echo password | kinit user1
+curl --verbose --negotiate -u: http://apacheweb.linux.contoso.com
+kdestroy
diff --git a/src/libraries/System.Net.Http/tests/EnterpriseTests/Configurations.props b/src/libraries/System.Net.Http/tests/EnterpriseTests/Configurations.props
new file mode 100644 (file)
index 0000000..3ad47b8
--- /dev/null
@@ -0,0 +1,7 @@
+<Project DefaultTargets="Build">
+  <PropertyGroup>
+    <BuildConfigurations>
+      netcoreapp-Unix;
+    </BuildConfigurations>
+  </PropertyGroup>
+</Project>
diff --git a/src/libraries/System.Net.Http/tests/EnterpriseTests/HttpClientAuthenticationTest.cs b/src/libraries/System.Net.Http/tests/EnterpriseTests/HttpClientAuthenticationTest.cs
new file mode 100644 (file)
index 0000000..828510b
--- /dev/null
@@ -0,0 +1,38 @@
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+// See the LICENSE file in the project root for more information.
+
+using System.Net.Test.Common;
+using System.Threading.Tasks;
+
+using Xunit;
+
+namespace System.Net.Http.Enterprise.Tests
+{
+    [ConditionalClass(typeof(EnterpriseTestConfiguration), nameof(EnterpriseTestConfiguration.Enabled))]
+    public class HttpClientAuthenticationTest
+    {
+        [Fact]
+        public async Task HttpClient_ValidAuthentication_Success()
+        {
+            using var handler = new HttpClientHandler();
+            handler.Credentials = EnterpriseTestConfiguration.ValidNetworkCredentials;
+            using var client = new HttpClient(handler);
+
+            using HttpResponseMessage response = await client.GetAsync(EnterpriseTestConfiguration.NegotiateAuthWebServer);
+            Assert.Equal(HttpStatusCode.OK, response.StatusCode);
+        }
+
+        [ActiveIssue(416)]
+        [Fact]
+        public async Task HttpClient_InvalidAuthentication_Failure()
+        {
+            using var handler = new HttpClientHandler();
+            handler.Credentials = EnterpriseTestConfiguration.InvalidNetworkCredentials;
+            using var client = new HttpClient(handler);
+
+            using HttpResponseMessage response = await client.GetAsync(EnterpriseTestConfiguration.NegotiateAuthWebServer);
+            Assert.Equal(HttpStatusCode.Unauthorized, response.StatusCode);
+        }
+    }
+}
diff --git a/src/libraries/System.Net.Http/tests/EnterpriseTests/README.md b/src/libraries/System.Net.Http/tests/EnterpriseTests/README.md
new file mode 100644 (file)
index 0000000..5c6c0f1
--- /dev/null
@@ -0,0 +1,5 @@
+# Enterprise Scenario Testing
+
+Detailed instructions for running these tests is located here:
+
+src\libraries\Common\tests\System\Net\EnterpriseTests\setup\README.md
diff --git a/src/libraries/System.Net.Http/tests/EnterpriseTests/System.Net.Http.Enterprise.Tests.csproj b/src/libraries/System.Net.Http/tests/EnterpriseTests/System.Net.Http.Enterprise.Tests.csproj
new file mode 100644 (file)
index 0000000..5afe2ba
--- /dev/null
@@ -0,0 +1,12 @@
+<Project Sdk="Microsoft.NET.Sdk">
+  <PropertyGroup>
+    <Configurations>netcoreapp-Unix-Debug;netcoreapp-Unix-Release</Configurations>
+  </PropertyGroup>
+  <ItemGroup>
+    <Compile Include="HttpClientAuthenticationTest.cs" />
+
+    <Compile Include="$(CommonTestPath)System\Net\EnterpriseTests\EnterpriseTestConfiguration.cs">
+      <Link>Common\System\Net\EnterpriseTests\EnterpriseTestConfiguration.cs</Link>
+    </Compile>
+  </ItemGroup>
+</Project>
diff --git a/src/libraries/System.Net.Security/tests/EnterpriseTests/Configurations.props b/src/libraries/System.Net.Security/tests/EnterpriseTests/Configurations.props
new file mode 100644 (file)
index 0000000..3ad47b8
--- /dev/null
@@ -0,0 +1,7 @@
+<Project DefaultTargets="Build">
+  <PropertyGroup>
+    <BuildConfigurations>
+      netcoreapp-Unix;
+    </BuildConfigurations>
+  </PropertyGroup>
+</Project>
diff --git a/src/libraries/System.Net.Security/tests/EnterpriseTests/NegotiateStreamLoopbackTest.cs b/src/libraries/System.Net.Security/tests/EnterpriseTests/NegotiateStreamLoopbackTest.cs
new file mode 100644 (file)
index 0000000..bde60a0
--- /dev/null
@@ -0,0 +1,275 @@
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+// See the LICENSE file in the project root for more information.
+
+using System.Linq;
+using System.Net.Test.Common;
+using System.Security.Authentication;
+using System.Security.Principal;
+using System.Text;
+using System.Threading.Tasks;
+
+using Xunit;
+
+namespace System.Net.Security.Enterprise.Tests
+{
+    [ConditionalClass(typeof(EnterpriseTestConfiguration), nameof(EnterpriseTestConfiguration.Enabled))]
+    public class NegotiateStreamLoopbackTest
+    {
+        private const int TimeoutMilliseconds = 4 * 60 * 1000;
+
+        private static Task WhenAllOrAnyFailedWithTimeout(params Task[] tasks) =>
+            tasks.WhenAllOrAnyFailed(TimeoutMilliseconds);
+
+        private const string TargetName = "HOST/linuxclient.linux.contoso.com";
+        private const int PartialBytesToRead = 5;
+        private static readonly byte[] s_sampleMsg = Encoding.UTF8.GetBytes("Sample Test Message");
+
+        private const int MaxWriteDataSize = 63 * 1024; // NegoState.MaxWriteDataSize
+        private static string s_longString = new string('A', MaxWriteDataSize) + 'Z';
+        private static readonly byte[] s_longMsg = Encoding.ASCII.GetBytes(s_longString);
+
+        public static readonly object[][] SuccessCasesMemberData =
+        {
+            new object[] { EnterpriseTestConfiguration.ValidNetworkCredentials, "HOST/localhost" },
+            new object[] { EnterpriseTestConfiguration.ValidNetworkCredentials, "HOST/linuxclient.linux.contoso.com" }
+        };
+
+        [Theory]
+        [MemberData(nameof(SuccessCasesMemberData))]
+        public async Task StreamToStream_ValidAuthentication_Success(NetworkCredential creds, string target)
+        {
+            var network = new VirtualNetwork();
+
+            using (var clientStream = new VirtualNetworkStream(network, isServer: false))
+            using (var serverStream = new VirtualNetworkStream(network, isServer: true))
+            using (var client = new NegotiateStream(clientStream))
+            using (var server = new NegotiateStream(serverStream))
+            {
+                Assert.False(client.IsAuthenticated);
+                Assert.False(server.IsAuthenticated);
+
+                Task[] auth = new Task[2];
+                auth[0] = client.AuthenticateAsClientAsync(creds, target);
+                auth[1] = server.AuthenticateAsServerAsync();
+
+                await WhenAllOrAnyFailedWithTimeout(auth);
+
+                VerifyStreamProperties(client, isServer: false, target);
+
+                string remoteName = creds.UserName + "@" + EnterpriseTestConfiguration.Realm;
+                VerifyStreamProperties(server, isServer: true, remoteName);
+            }
+        }
+
+        public static readonly object[][] FailureCasesMemberData =
+        {
+            // Using a valid credential but trying to connect to the server using
+            // the 'NEWSERVICE/localhost' SPN is not valid. That SPN, while registered in the overall
+            // Kerberos realm, is not registered on this particular server's keytab. So, this test case verifies
+            // that SPNEGO won't fallback from Kerberos to NTLM. Instead, it causes an AuthenticationException.
+            new object[] { EnterpriseTestConfiguration.ValidNetworkCredentials, "NEWSERVICE/localhost" },
+
+            // Invalid Kerberos credential password.
+            new object[] { EnterpriseTestConfiguration.InvalidNetworkCredentials, "HOST/localhost" },
+        };
+
+        [Theory]
+        [MemberData(nameof(FailureCasesMemberData))]
+        public async Task StreamToStream_InvalidAuthentication_Failure(NetworkCredential creds, string target)
+        {
+            var network = new VirtualNetwork();
+
+            using (var clientStream = new VirtualNetworkStream(network, isServer: false))
+            using (var serverStream = new VirtualNetworkStream(network, isServer: true))
+            using (var client = new NegotiateStream(clientStream))
+            using (var server = new NegotiateStream(serverStream))
+            {
+                Assert.False(client.IsAuthenticated);
+                Assert.False(server.IsAuthenticated);
+
+                Task clientTask = client.AuthenticateAsClientAsync(creds, target);
+
+                await Assert.ThrowsAsync<AuthenticationException>(() => server.AuthenticateAsServerAsync());
+            }
+        }
+
+        [Fact]
+        public async Task NegotiateStream_StreamToStream_Successive_ClientWrite_Sync_Success()
+        {
+            byte[] recvBuf = new byte[s_sampleMsg.Length];
+            VirtualNetwork network = new VirtualNetwork();
+            int bytesRead = 0;
+
+            using (var clientStream = new VirtualNetworkStream(network, isServer: false))
+            using (var serverStream = new VirtualNetworkStream(network, isServer: true))
+            using (var client = new NegotiateStream(clientStream))
+            using (var server = new NegotiateStream(serverStream))
+            {
+                Assert.False(client.IsAuthenticated);
+                Assert.False(server.IsAuthenticated);
+
+                Task[] auth = new Task[2];
+                auth[0] = client.AuthenticateAsClientAsync(EnterpriseTestConfiguration.ValidNetworkCredentials, TargetName);
+                auth[1] = server.AuthenticateAsServerAsync();
+
+                await WhenAllOrAnyFailedWithTimeout(auth);
+
+                client.Write(s_sampleMsg, 0, s_sampleMsg.Length);
+                server.Read(recvBuf, 0, s_sampleMsg.Length);
+
+                Assert.True(s_sampleMsg.SequenceEqual(recvBuf));
+
+                client.Write(s_sampleMsg, 0, s_sampleMsg.Length);
+
+                // Test partial sync read.
+                bytesRead = server.Read(recvBuf, 0, PartialBytesToRead);
+                Assert.Equal(PartialBytesToRead, bytesRead);
+
+                bytesRead = server.Read(recvBuf, PartialBytesToRead, s_sampleMsg.Length - PartialBytesToRead);
+                Assert.Equal(s_sampleMsg.Length - PartialBytesToRead, bytesRead);
+
+                Assert.True(s_sampleMsg.SequenceEqual(recvBuf));
+            }
+        }
+
+        [Fact]
+        public async Task NegotiateStream_StreamToStream_Successive_ClientWrite_Async_Success()
+        {
+            byte[] recvBuf = new byte[s_sampleMsg.Length];
+            VirtualNetwork network = new VirtualNetwork();
+            int bytesRead = 0;
+
+            using (var clientStream = new VirtualNetworkStream(network, isServer: false))
+            using (var serverStream = new VirtualNetworkStream(network, isServer: true))
+            using (var client = new NegotiateStream(clientStream))
+            using (var server = new NegotiateStream(serverStream))
+            {
+                Assert.False(client.IsAuthenticated);
+                Assert.False(server.IsAuthenticated);
+
+                Task[] auth = new Task[2];
+                auth[0] = client.AuthenticateAsClientAsync(EnterpriseTestConfiguration.ValidNetworkCredentials, TargetName);
+                auth[1] = server.AuthenticateAsServerAsync();
+
+                await WhenAllOrAnyFailedWithTimeout(auth);
+
+                auth[0] = client.WriteAsync(s_sampleMsg, 0, s_sampleMsg.Length);
+                auth[1] = server.ReadAsync(recvBuf, 0, s_sampleMsg.Length);
+                await WhenAllOrAnyFailedWithTimeout(auth);
+                Assert.True(s_sampleMsg.SequenceEqual(recvBuf));
+
+                await client.WriteAsync(s_sampleMsg, 0, s_sampleMsg.Length);
+
+                // Test partial async read.
+                bytesRead = await server.ReadAsync(recvBuf, 0, PartialBytesToRead);
+                Assert.Equal(PartialBytesToRead, bytesRead);
+
+                bytesRead = await server.ReadAsync(recvBuf, PartialBytesToRead, s_sampleMsg.Length - PartialBytesToRead);
+                Assert.Equal(s_sampleMsg.Length - PartialBytesToRead, bytesRead);
+
+                Assert.True(s_sampleMsg.SequenceEqual(recvBuf));
+            }
+        }
+
+        [Fact]
+        public async Task NegotiateStream_ReadWriteLongMsgSync_Success()
+        {
+            byte[] recvBuf = new byte[s_longMsg.Length];
+            var network = new VirtualNetwork();
+            int bytesRead = 0;
+
+            using (var clientStream = new VirtualNetworkStream(network, isServer: false))
+            using (var serverStream = new VirtualNetworkStream(network, isServer: true))
+            using (var client = new NegotiateStream(clientStream))
+            using (var server = new NegotiateStream(serverStream))
+            {
+                await WhenAllOrAnyFailedWithTimeout(
+                    client.AuthenticateAsClientAsync(EnterpriseTestConfiguration.ValidNetworkCredentials, TargetName),
+                    server.AuthenticateAsServerAsync());
+
+                client.Write(s_longMsg, 0, s_longMsg.Length);
+
+                while (bytesRead < s_longMsg.Length)
+                {
+                    bytesRead += server.Read(recvBuf, bytesRead, s_longMsg.Length - bytesRead);
+                }
+
+                Assert.True(s_longMsg.SequenceEqual(recvBuf));
+            }
+        }
+
+        [Fact]
+        public async Task NegotiateStream_ReadWriteLongMsgAsync_Success()
+        {
+            byte[] recvBuf = new byte[s_longMsg.Length];
+            var network = new VirtualNetwork();
+            int bytesRead = 0;
+
+            using (var clientStream = new VirtualNetworkStream(network, isServer: false))
+            using (var serverStream = new VirtualNetworkStream(network, isServer: true))
+            using (var client = new NegotiateStream(clientStream))
+            using (var server = new NegotiateStream(serverStream))
+            {
+                await WhenAllOrAnyFailedWithTimeout(
+                    client.AuthenticateAsClientAsync(EnterpriseTestConfiguration.ValidNetworkCredentials, TargetName),
+                    server.AuthenticateAsServerAsync());
+
+                await client.WriteAsync(s_longMsg, 0, s_longMsg.Length);
+
+                while (bytesRead < s_longMsg.Length)
+                {
+                    bytesRead += await server.ReadAsync(recvBuf, bytesRead, s_longMsg.Length - bytesRead);
+                }
+
+                Assert.True(s_longMsg.SequenceEqual(recvBuf));
+            }
+        }
+
+        [Fact]
+        public void NegotiateStream_StreamToStream_Flush_Propagated()
+        {
+            VirtualNetwork network = new VirtualNetwork();
+
+            using (var stream = new VirtualNetworkStream(network, isServer: false))
+            using (var negotiateStream = new NegotiateStream(stream))
+            {
+                Assert.False(stream.HasBeenSyncFlushed);
+                negotiateStream.Flush();
+                Assert.True(stream.HasBeenSyncFlushed);
+            }
+        }
+
+        [Fact]
+        public void NegotiateStream_StreamToStream_FlushAsync_Propagated()
+        {
+            VirtualNetwork network = new VirtualNetwork();
+
+            using (var stream = new VirtualNetworkStream(network, isServer: false))
+            using (var negotiateStream = new NegotiateStream(stream))
+            {
+                Task task = negotiateStream.FlushAsync();
+
+                Assert.False(task.IsCompleted);
+                stream.CompleteAsyncFlush();
+                Assert.True(task.IsCompleted);
+            }
+        }
+
+        private void VerifyStreamProperties(NegotiateStream stream, bool isServer, string remoteName)
+        {
+            Assert.True(stream.IsAuthenticated);
+            Assert.Equal(TokenImpersonationLevel.Identification, stream.ImpersonationLevel);
+            Assert.True(stream.IsEncrypted);
+            Assert.True(stream.IsMutuallyAuthenticated);
+            Assert.Equal(isServer, stream.IsServer);
+            Assert.True(stream.IsSigned);
+            Assert.False(stream.LeaveInnerStreamOpen);
+
+            IIdentity remoteIdentity = stream.RemoteIdentity;
+            Assert.Equal("Kerberos", remoteIdentity.AuthenticationType);
+            Assert.True(remoteIdentity.IsAuthenticated);
+            Assert.Equal(remoteName, remoteIdentity.Name);
+        }
+    }
+}
diff --git a/src/libraries/System.Net.Security/tests/EnterpriseTests/README.md b/src/libraries/System.Net.Security/tests/EnterpriseTests/README.md
new file mode 100644 (file)
index 0000000..5c6c0f1
--- /dev/null
@@ -0,0 +1,5 @@
+# Enterprise Scenario Testing
+
+Detailed instructions for running these tests is located here:
+
+src\libraries\Common\tests\System\Net\EnterpriseTests\setup\README.md
diff --git a/src/libraries/System.Net.Security/tests/EnterpriseTests/System.Net.Security.Enterprise.Tests.csproj b/src/libraries/System.Net.Security/tests/EnterpriseTests/System.Net.Security.Enterprise.Tests.csproj
new file mode 100644 (file)
index 0000000..506734b
--- /dev/null
@@ -0,0 +1,26 @@
+<Project Sdk="Microsoft.NET.Sdk">
+  <PropertyGroup>
+    <Configurations>netcoreapp-Unix-Debug;netcoreapp-Unix-Release</Configurations>
+  </PropertyGroup>
+  <ItemGroup>
+    <!-- NegotiateStream Tests -->
+    <Compile Include="NegotiateStreamLoopbackTest.cs" />
+
+    <!-- Common test files -->
+    <Compile Include="$(CommonTestPath)System\Net\EnterpriseTests\EnterpriseTestConfiguration.cs">
+      <Link>Common\System\Net\EnterpriseTests\EnterpriseTestConfiguration.cs</Link>
+    </Compile>
+    <Compile Include="$(CommonTestPath)System\Net\VirtualNetwork\VirtualNetwork.cs">
+      <Link>Common\System\Net\VirtualNetwork\VirtualNetwork.cs</Link>
+    </Compile>
+    <Compile Include="$(CommonTestPath)System\Net\VirtualNetwork\VirtualNetworkStream.cs">
+      <Link>Common\System\Net\VirtualNetwork\VirtualNetworkStream.cs</Link>
+    </Compile>
+    <Compile Include="$(CommonTestPath)System\Threading\Tasks\TaskTimeoutExtensions.cs">
+      <Link>Common\System\Threading\Tasks\TaskTimeoutExtensions.cs</Link>
+    </Compile>
+    <Compile Include="$(CoreLibSharedDir)System\Threading\Tasks\TaskToApm.cs">
+      <Link>ProductionCode\System\Threading\Tasks\TaskToApm.cs</Link>
+    </Compile>
+  </ItemGroup>
+</Project>