0.  Pre-Checkin Checklist

    Performed from top of repo, default branch, head:

    [ ]  Is tag 'current' at or near head of 'vendor'?

         hg heads
         hg tags

    [ ]  Expected differences between vendor and default?  Very
         little of the original source should be modified.

         hg diff -rcurrent curl

    [ ]  Are the 'vendor' and 'default' branch source directories
         'curl' and not 'curl-<version>'?

    [ ]  Is curlbuild.h identical to distribution?

         hg diff -rcurrent curl/include/curl/curlbuild.h

    [ ]  Is Makefile identical to distribution?  Not strictly
         required but a good idea.

         hg diff -rcurrent curl/Makefile

    [ ]  Examine the build logs to make certain that the data type
         primitives and their formats are in agreement.  Good 
         example from a 32-bit Linux build:

         checking size of long... 4
         checking size of void*... 4
         checking for 64-bit curl_off_t data type... int64_t
         checking size of curl_off_t... 8
         checking formatting string directive for curl_off_t... "lld"
         checking formatting string directive for unsigned curl_off_t... "llu"
         checking constant suffix string for curl_off_t... LL
         checking constant suffix string for unsigned curl_off_t... ULL


    [ ]  The curl configuration procedure has trouble locating custom
         versions of support libraries (zlib, ssl, ssh2, etc.).  So
         examing the build log and verify that the right libraries are
         being picked up.  Example from Linux:

         checking for OpenSSL headers version... 1.0.1 - 0x1000107fL
         checking for OpenSSL library version... 1.0.1
         checking for OpenSSL headers and library versions matching... yes

         Example from Linux of zlib detection:

         checking zlib.h usability... yes
         checking zlib.h presence... yes
         checking for zlib.h... yes
         checking for gzread in -lz... yes
         configure: found both libz and libz.h header

         During development, you might find it useful to rename any
         system version of the zlib.h header so that it isn't picked
         up accidentally.

         In addition to the 'curl --version' command, you can look
         for version strings in the 'curl' program itself:

         $ strings stage/bin/curl | grep 1\\.[0-9]\\.
              --dns-servers    DNS server addrs to use: 1.1.1.1;2.2.2.2
         1.2.8
         1.2.0.4
         SSLv2 part of OpenSSL 1.0.1g 7 Apr 2014
         SSLv3 part of OpenSSL 1.0.1g 7 Apr 2014
         ...
         \CAST part of OpenSSL 1.0.1g 7 Apr 2014
          deflate 1.2.8 Copyright 1995-2013 Jean-loup Gailly and Mark Adler 
          inflate 1.2.8 Copyright 1995-2013 Mark Adler 

         With the '1.2.8' indicating the targeted libz (1.2.3.4
         typically means a platform libz).

         Static archives will show even less version information:

         $ strings stage/lib/release/libcurl.a | grep 1\\.[0-9]\\.
         1.2.8
         1.2.0.4

    [ ]  Examine build logs or run the built 'curl' executable to
         verify function.  The '--version' option will give useful
         library information:

         $ stage/bin/curl --version
         curl 7.42.0 (x86_64-unknown-linux-gnu) libcurl/7.47.0 OpenSSL/1.0.1g zlib/1.2.8
         Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp smtp smtps telnet tftp 
         Features: AsynchDNS IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP 

         Run a few curl commands to a website.  You may need to copy
         DLLs, create a Resources peer directory with symlinks to
         .dylibs or set LD_LIBRARY_PATH to locate OpenSSL shared
         libraries at runtime.  (build-cmd.sh does this for its own
         operations.)

         Confirm a fetch and the progress display works as expected:

         $ stage/bin/curl -o /dev/null http://example.com/
           % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                          Dload  Upload   Total   Spent    Left  Speed
         100  1270  100  1270    0     0  55548      0 --:--:-- --:--:-- --:--:-- 57727

    [ ]  On Mac, unit tests are disabled by default because they invoke
         a lot of network activity that triggers the OS X firewall.
         Inhibit the disable ('export DISABLE_UNIT_TESTS=0') and run
         builds on your machine to exercise the unit tests.  There may
         be some flakiness in the tests but they have been useful.

    [ ]  Are you able to revert the special pipelining changes for curl
         bugs 1420 (https://sourceforge.net/p/curl/bugs/1420/)?  This
         is done by changesets a8784f6e153f and 46fd1a2376ab.


1.  Introduction

    Build of curl library from curl.haxx.se.

    Repo structure *now* mostly follows standard conventions (see
    section at end).

    Prior to 7.24.0, repo didn't follow standard convention and each
    release was disconnected from previous releases.  Modifications
    to branch 'default' were re-implemented on every update.

    Running 'configure' modifies source files.  Don't submit these
    changes to the repo.

    There are PDF files in the distribution which Mercurial does not
    know how to merge.  There are also unit test data files which are
    mostly ASCII but a few that are not.  Taking the vendor branch as
    authoritative is the best default.  So, add the following to
    .hg/hgrc in the repo so that these files are taken from the other
    branch:

        [merge-patterns]
        **.pdf = internal:other
        curl/tests/data/** = internal:other


2.  Modifications

    Essential modifications to curl distributions.
  2.0 CMake is now used to configure the build.
  
    Many of the options listed below can now be specified from curl/CMakeLists.txt.
    These options should be set as follows:
        BUILD_CURL_EXE = ON
        BUILD_CURL_TESTS = ON
        CURL_STATICLIB = ON
        ENABLE_ARES = OFF
        ENABLE_THREADED_RESOLVER = OFF
    
    Also curl/CMake/FindOpenSSL.cmake and curl/CMake/FindZLIB.cmake are customized 
    to find only the versions installed in the packages directory.

  2.1  Do Not Use C-Ares for DNS Lookups

    Use the threaded resolver system which is built on the platform's
    resolver library and made asynchronous and thread safe.  The
    c-ares library appears to be at the heart of many of the DNS
    lookup failures customers see.

    Changes:
    * curl/lib/config-win32.h modified to enable threaded resolver.
    * curl/src/config-win32.h modified identically.
    * configure script invoked with correct arguments to use the
      threaded resolver on linux and darwin.

    There is some scripting in the build-cmd.sh file to try to catch
    failures in this area but don't rely on it.

  2.2  Do Not Modify include/curl/curlbuild.h

    The curlbuild.h file distributed in the source tarball is
    specially constructed for platforms that cannot run configure and
    probe the environment (i.e. Windows). If you update the vendor
    branch and then merge back with the default, be certain that
    curlbuild.h is copied verbatim from the source distribution and is
    not modified in subsequent commits. Running configure on linux or
    mac will edit the file, removing the windows specific parts.

  2.3  lib/Makefile.vc10 Modified

    Additional makefile configurations added for our specific type of
    library dependencies: 'release-ssl-dll-zlib' (release build,
    static libcurl, dynamic ssl, static zlib) and 'debug-ssl-dll-zlib'
    (debug build, static libcurl, dynamic ssl, static zlib).

    This file has been edited by lindenlab to pick up the
    LINDEN_LIBPATH variable and find our autobuild-packaged library
    files. If you upgrade curl, be sure to carry those modifications
    forward, or find another way to link in the autobuild libraries.

    Debug and Release builds for all targets use /Z7 flag (and lost
    any /Gm flag) anticipating static library builds.

  2.4  src/Makefile.vc10 Modified

    Similar changes to build curl.exe on Windows.  Only a
    release-ssl-dll-zlib configuration added but it can be used to
    build either a 'release' or 'debug' target curl.exe.

  2.5  Imperfect Mercurial repository work

    The path I took to canonicalize the branch and source code
    directory structure, at changesets 1b274a267882 and 4a82c402b11f,
    was not the optimal way.  What I did was basically:

    * hg update vendor
    * hg mv curl-version curl
    * hg ci
    * tar xzf
    * hg addremove
    * hg ci
    * hg update default
    * hg mv curl-version curl
    * hg ci
    * hg merge vendor
    * hg ci

    What should have happened (and what is happening for other
    libraries) involves a single 'hg mv' operation:

    * hg update vendor
    * hg mv curl-version curl
    * hg ci
    * hg update default
    * hg merge vendor (will do the rename)
    * hg ci
    * hg update vendor
    * tar xzf
    * hg addremove
    * hg ci
    * hg update default
    * hg merge vendor
    * hg ci

    At some point, we might go back and repair/replace the repo.

  2.6  Unit Test Changes

    Bumped timestamp in tests/data/test172 so test runs.  Not
    certain how this ever worked, it's stale by several years.
    Watch for fixes from upstream and restore the file when 
    possible.  (Fixed in 7.36.0)

  2.7  Pipelining Data Corruption

    Timed out requests involved in pipelines may result in loss of
    response stream synchronization when other requests follow the
    timed out requests.  This has been reported to the curl devs as
    bug 1420 and some defensive measures are taken in the Linden
    'default' branch until an official fix is available.  Once it is
    available, these changes (a8784f6e153f and 46fd1a2376ab) should be
    reverted.

    I recommend testing any 1420 fix or workaround.  The process isn't
    entirely deterministic.  But some hooks are in place to make this
    less painful.

    1.  Edit llcorehttp/_httpoprequest.cpp to force a request timeout
    of one second:

diff -r 0e3365c1843d indra/llcorehttp/_httpoprequest.cpp
--- a/indra/llcorehttp/_httpoprequest.cpp       Mon Oct 06 15:22:58 2014 -0400
+++ b/indra/llcorehttp/_httpoprequest.cpp       Fri Oct 10 14:31:16 2014 -0400
@@ -626,8 +626,8 @@
                // *TODO:  Find a better scheme than timeouts to guarantee liveness.
                xfer_timeout *= cpolicy.mPipelining;
        }
-       // *DEBUG:  Useful for timeout handling and "[curl:bugs] #1420" tests
-       // xfer_timeout = 1L;
+       // *DEBUG:  Enable following override for timeout handling and "[curl:bugs] #1420" tests
+       xfer_timeout = 1L;
        code = curl_easy_setopt(mCurlHandle, CURLOPT_TIMEOUT, xfer_timeout);
        check_curl_easy_code(code, CURLOPT_TIMEOUT);
        code = curl_easy_setopt(mCurlHandle, CURLOPT_CONNECTTIMEOUT, timeout);


    2.  Edit llcorehttp/httpcommon.cpp to disable request retry for
    the Core_9 (invalid status) error:

diff -r 0e3365c1843d indra/llcorehttp/httpcommon.cpp
--- a/indra/llcorehttp/httpcommon.cpp   Mon Oct 06 15:22:58 2014 -0400
+++ b/indra/llcorehttp/httpcommon.cpp   Fri Oct 10 14:31:16 2014 -0400
@@ -254,8 +254,9 @@
                        *this == op_timedout ||         // Timer expired
                        *this == post_error ||          // Transport problem
                        *this == partial_file ||        // Data inconsistency in response
-                       *this == inv_cont_range ||      // Short data read disagrees with content-range
-                       *this == inv_status);           // Inv status can reflect internal state problem in libcurl
+                       // *DEBUG:  Comment out 'inv_status' test for [curl:bugs] #1420 testing.
+                       // *this == inv_status ||               // Inv status can reflect internal state problem in libcurl
+                       *this == inv_cont_range);       // Short data read disagrees with content-range
 }
 
 } // end namespace LLCore


    3.  Edit llcorehttp/_httppolicy.cpp and add some chatty logging
    whenever a request timeout error is generated:

diff -r 0e3365c1843d indra/llcorehttp/_httppolicy.cpp
--- a/indra/llcorehttp/_httppolicy.cpp  Mon Oct 06 15:22:58 2014 -0400
+++ b/indra/llcorehttp/_httppolicy.cpp  Fri Oct 10 15:10:12 2014 -0400
@@ -414,6 +414,18 @@
        // Retry or finalize
        if (! op->mStatus)
        {
+               // *DEBUG:  For "[curl:bugs] #1420" tests.  This will interfere
+               // with unit tests due to allocation retention by logging code.
+               // But you won't be checking this in enabled.
+#if 1
+               if (op->mStatus == HttpStatus(HttpStatus::EXT_CURL_EASY, CURLE_OPERATION_TIMEDOUT))
+               {
+                       LL_WARNS(LOG_CORE) << "HTTP request " << static_cast<HttpHandle>(op)
+                                                          << " timed out."
+                                                          << LL_ENDL;
+               }
+#endif
+
                // If this failed, we might want to retry.
                if (op->mPolicyRetries < op->mPolicyRetryLimit && op->mStatus.isRetryable())
                {

    4.  Build and visit mesh- and texture-rich regions.  Monitor the
    SecondLife.log file for the 'timed out' log message and Core_9
    errors.  The test isn't deterministic but you want *NO* Core_9
    errors and *many* 'timed out' messages:

2014-10-10T19:03:15Z WARNING: LLCore::HttpPolicy::stageAfterCompletion: HTTP request 2CC9FCB0 timed out.
2014-10-10T19:03:15Z WARNING: LLCore::HttpPolicy::stageAfterCompletion: HTTP request 24DA9FC0 timed out.
2014-10-10T19:03:15Z WARNING: LLCore::HttpPolicy::stageAfterCompletion: HTTP request 2CC9FD90 timed out.
2014-10-10T19:03:15Z WARNING: LLCore::HttpPolicy::stageAfterCompletion: HTTP request 26D68718 timed out.
2014-10-10T19:03:15Z WARNING: LLCore::HttpPolicy::stageAfterCompletion: HTTP request 26D67F38 timed out.
2014-10-10T19:03:15Z WARNING: LLCore::HttpPolicy::stageAfterCompletion: HTTP request 26D68C58 timed out.
2014-10-10T19:03:15Z WARNING: LLCore::HttpPolicy::stageAfterCompletion: HTTP request 252218F8 timed out.


    5.  Follow this test with a Wireshark verification that pipelining
    is still working.

        
3.  Source Origin
    7.47.0:
	  https://github.com/bagder/curl.git 
	  (tag:curl-7_47_0)
	  
    7.42.1:
	  https://github.com/bagder/curl.git 
	  (tag:curl-7_42_1)

    7.38.0:
      http://curl.haxx.se/download/curl-7.38.0.tar.gz

    7.37.0:
      http://curl.haxx.se/download/curl-7.37.0.tar.gz

    7.36.0:
      http://curl.haxx.se/download/curl-7.36.0.tar.gz

    7.34.0:
      http://curl.haxx.se/download/curl-7.34.0.tar.gz


4.  Package Contents

    Common:
    * include/curl/curl.h
    * include/curl/curlbuild.h
    * include/curl/curlrules.h
    * include/curl/curlver.h
    * include/curl/easy.h
    * include/curl/mprintf.h
    * include/curl/multi.h
    * include/curl/stdcheaders.h
    * include/curl/typecheck-gcc.h

    Windows:
    * lib/release/libcurl.lib
    * lib/debug/libcurld.lib

    Mac OS X (debug and release):
    * lib/release/libcurl.a
    * lib/debug/libcurl.a

    Linux:
    * lib/release/libcurl.a
    * lib/debug/libcurl.a


5.  Known Dependencies

    This is not authoritative, it's just a reminder to followup
    with dependent packages.

    * openjpeg (*should* be a dependency when building certain tools)
    * viewer


A.  Appendix A - Notes on 'vendor' branch maintenance

    7.36.0 from 7.34.0.

    This wasquite a bit easier than the effort below.  With the
    following three commands, only one deleted file ended up being
    split into two new files.  Good enough for this effort.

    * hg addremove -X curl/docs/examples/pop3slist.c -X curl/docs/examples/imap.c  -s 60
    * hg rm curl/docs/examples/imap.c
    * hg rm curl/docs/examples/pop3slist.c

    7.34.0 from 7.24.0.

    Keeping file continuity in the 'vendor' branch was challenging.
    The library had been reorganized (for the better) and that
    confused 'hg addremove' more than usual.  This describes what was
    found during the process and hopefully forewarns the next
    developer.

    After laying down the 7.34.0 source, a simple 'addremove' run
    in dry-run mode produces a long list of add and remove operations
    *and* a substantial number of suspect renames:

      $ hg addremove -s 60 -n
      [ long list of adds and removes deleted ]
      recording removal of curl/lib/vc6libcurl.dsw as rename to curl/vs/vc6/lib/vc6libcurl.dsw (100% similar)
      recording removal of curl/Makefile.msvc.names as rename to curl/winbuild/Makefile.msvc.names (100% similar)
      recording removal of curl/lib/curl_rand.h as rename to curl/lib/hostcheck.h (78% similar)
      recording removal of curl/lib/curl_rand.h as rename to curl/tests/unit/unit1330.c (79% similar)
      recording removal of curl/lib/msvcproj.foot as rename to curl/vs/t/lib/vc6_libcurl_dsp.foot (83% similar)
      recording removal of curl/lib/curl_rand.h as rename to curl/tests/server/server_setup.h (79% similar)
      recording removal of curl/lib/curl_rand.h as rename to curl/tests/libtest/lib558.c (67% similar)
      recording removal of curl/lib/curl_rand.h as rename to curl/lib/http2.h (71% similar)
      recording removal of curl/tests/data/test1323 as rename to curl/tests/data/test931 (88% similar)
      recording removal of curl/tests/data/test1322 as rename to curl/tests/data/test875 (88% similar)
      recording removal of curl/lib/curl_rand.h as rename to curl/lib/bundles.h (63% similar)
      recording removal of curl/tests/data/test1324 as rename to curl/tests/data/test829 (88% similar)
      recording removal of curl/vc6curl.dsw as rename to curl/vs/vc6/vc6curl.dsw (92% similar)
      recording removal of curl/lib/setup_once.h as rename to curl/lib/curl_setup_once.h (92% similar)
      recording removal of curl/src/vc6curlsrc.dsw as rename to curl/vs/vc6/src/vc6curltool.dsw (88% similar)
      recording removal of curl/lib/curl_rand.h as rename to curl/lib/polarssl_threadlock.h (61% similar)
      recording removal of curl/Android.mk as rename to curl/packages/Android/Android.mk (96% similar)
      recording removal of curl/lib/krb4.h as rename to curl/lib/curl_sec.h (86% similar)
      recording removal of curl/src/hugehelp.h as rename to curl/lib/dotdot.h (83% similar)
      recording removal of curl/tests/data/test1324 as rename to curl/tests/data/test1409 (60% similar)
      recording removal of curl/lib/curl_rand.h as rename to curl/tests/libtest/lib1508.c (75% similar)
      recording removal of curl/lib/setup.h as rename to curl/lib/curl_setup.h (89% similar)
      recording removal of curl/src/hugehelp.h as rename to curl/src/tool_hugehelp.h (83% similar)
      recording removal of curl/docs/examples/htmltitle.cc as rename to curl/docs/examples/htmltitle.cpp (98% similar)

    Of these, lib/curl_rand.*, src/hugehelp.h, lib/krb4.h,
    tests/data/test815 should not be renamed (they're actually being
    deleted).  Update the dry-run command with some exclusions:

      $ hg addremove -X curl/lib/curl_rand.h -X curl/lib/curl_rand.c \
                     -X curl/src/hugehelp.h -X curl/lib/krb4.h \
                     -X curl/tests/data/test815 -s 60 -n

      recording removal of curl/lib/vc6libcurl.dsw as rename to curl/vs/vc6/lib/vc6libcurl.dsw (100% similar)
      recording removal of curl/Makefile.msvc.names as rename to curl/winbuild/Makefile.msvc.names (100% similar)
      recording removal of curl/src/vc6curlsrc.dsw as rename to curl/vs/vc6/src/vc6curltool.dsw (88% similar)
      recording removal of curl/src/tool_cb_skt.h as rename to curl/lib/hostcheck.h (73% similar)
      recording removal of curl/lib/setup.h as rename to curl/lib/curl_setup.h (89% similar)
      recording removal of curl/tests/data/test1322 as rename to curl/tests/data/test875 (88% similar)
      recording removal of curl/src/tool_cb_skt.h as rename to curl/tests/unit/unit1330.c (74% similar)
      recording removal of curl/src/tool_cb_skt.h as rename to curl/src/tool_hugehelp.h (76% similar)
      recording removal of curl/lib/setup_once.h as rename to curl/lib/curl_setup_once.h (92% similar)
      recording removal of curl/lib/msvcproj.foot as rename to curl/vs/t/lib/vc6_libcurl_dsp.foot (83% similar)
      recording removal of curl/src/tool_cb_skt.h as rename to curl/lib/bundles.h (60% similar)
      recording removal of curl/tests/data/test1324 as rename to curl/tests/data/test829 (88% similar)
      recording removal of curl/src/tool_cb_skt.h as rename to curl/tests/server/server_setup.h (74% similar)
      recording removal of curl/vc6curl.dsw as rename to curl/vs/vc6/vc6curl.dsw (92% similar)
      recording removal of curl/src/tool_cb_skt.h as rename to curl/tests/libtest/lib558.c (64% similar)
      recording removal of curl/src/tool_cb_skt.h as rename to curl/lib/http2.h (67% similar)
      recording removal of curl/src/tool_cb_skt.h as rename to curl/lib/dotdot.h (77% similar)
      recording removal of curl/tests/data/test1323 as rename to curl/tests/data/test931 (88% similar)
      recording removal of curl/docs/examples/htmltitle.cc as rename to curl/docs/examples/htmltitle.cpp (98% similar)
      recording removal of curl/tests/data/test1324 as rename to curl/tests/data/test1409 (60% similar)
      recording removal of curl/src/tool_cb_skt.h as rename to curl/tests/libtest/lib1508.c (70% similar)
      recording removal of curl/Android.mk as rename to curl/packages/Android/Android.mk (96% similar)

    Better.  Now add src/tool_cb_skt.h and lib/msvcproj.foot (which
    are also being deleted):

      $ hg addremove -X curl/lib/curl_rand.h -X curl/lib/curl_rand.c \
                     -X curl/src/hugehelp.h -X curl/lib/krb4.h \
                     -X curl/tests/data/test815 -X curl/src/tool_cb_skt.h \
                     -X curl/lib/msvcproj.foot -s 60 -n

      recording removal of curl/lib/vc6libcurl.dsw as rename to curl/vs/vc6/lib/vc6libcurl.dsw (100% similar)
      recording removal of curl/Makefile.msvc.names as rename to curl/winbuild/Makefile.msvc.names (100% similar)
      recording removal of curl/src/vc6curlsrc.dsw as rename to curl/vs/vc6/src/vc6curltool.dsw (88% similar)
      recording removal of curl/docs/examples/printf_macro.h as rename to curl/lib/hostcheck.h (66% similar)
      recording removal of curl/lib/setup.h as rename to curl/lib/curl_setup.h (89% similar)
      recording removal of curl/tests/data/test1322 as rename to curl/tests/data/test875 (88% similar)
      recording removal of curl/docs/examples/printf_macro.h as rename to curl/tests/unit/unit1330.c (67% similar)
      recording removal of curl/docs/examples/printf_macro.h as rename to curl/src/tool_hugehelp.h (68% similar)
      recording removal of curl/docs/examples/printf_macro.h as rename to curl/tests/libtest/lib1508.c (64% similar)
      recording removal of curl/Android.mk as rename to curl/packages/Android/Android.mk (96% similar)
      recording removal of curl/tests/data/test1324 as rename to curl/tests/data/test829 (88% similar)
      recording removal of curl/docs/examples/printf_macro.h as rename to curl/tests/server/server_setup.h (67% similar)
      recording removal of curl/vc6curl.dsw as rename to curl/vs/vc6/vc6curl.dsw (92% similar)
      recording removal of curl/docs/examples/printf_macro.h as rename to curl/lib/dotdot.h (70% similar)
      recording removal of curl/docs/examples/printf_macro.h as rename to curl/lib/http2.h (61% similar)
      recording removal of curl/tests/data/test1323 as rename to curl/tests/data/test931 (88% similar)
      recording removal of curl/docs/examples/htmltitle.cc as rename to curl/docs/examples/htmltitle.cpp (98% similar)
      recording removal of curl/tests/data/test1324 as rename to curl/tests/data/test1409 (60% similar)
      recording removal of curl/lib/setup_once.h as rename to curl/lib/curl_setup_once.h (92% similar)

    Add docs/examples/printf_macro.h (also deleting):

      $ hg addremove -X curl/lib/curl_rand.h -X curl/lib/curl_rand.c \
                     -X curl/src/hugehelp.h -X curl/lib/krb4.h \
                     -X curl/tests/data/test815 -X curl/src/tool_cb_skt.h \
                     -X curl/lib/msvcproj.foot -X curl/docs/examples/printf_macro.h \
                     -s 60 -n

      recording removal of curl/lib/vc6libcurl.dsw as rename to curl/vs/vc6/lib/vc6libcurl.dsw (100% similar)
      recording removal of curl/Makefile.msvc.names as rename to curl/winbuild/Makefile.msvc.names (100% similar)
      recording removal of curl/src/vc6curlsrc.dsw as rename to curl/vs/vc6/src/vc6curltool.dsw (88% similar)
      recording removal of curl/lib/setup.h as rename to curl/lib/curl_setup.h (89% similar)
      recording removal of curl/tests/data/test1322 as rename to curl/tests/data/test875 (88% similar)
      recording removal of curl/tests/data/test1324 as rename to curl/tests/data/test829 (88% similar)
      recording removal of curl/vc6curl.dsw as rename to curl/vs/vc6/vc6curl.dsw (92% similar)
      recording removal of curl/Android.mk as rename to curl/packages/Android/Android.mk (96% similar)
      recording removal of curl/tests/data/test1323 as rename to curl/tests/data/test931 (88% similar)
      recording removal of curl/docs/examples/htmltitle.cc as rename to curl/docs/examples/htmltitle.cpp (98% similar)
      recording removal of curl/tests/data/test1324 as rename to curl/tests/data/test1409 (60% similar)
      recording removal of curl/lib/setup_once.h as rename to curl/lib/curl_setup_once.h (92% similar)

    Exclude the test files (being deleted):

      $ hg addremove -X curl/lib/curl_rand.h -X curl/lib/curl_rand.c \
                     -X curl/src/hugehelp.h -X curl/lib/krb4.h \
                     -X curl/tests/data/test815 -X curl/src/tool_cb_skt.h \
                     -X curl/lib/msvcproj.foot -X curl/docs/examples/printf_macro.h \
                     -X curl/tests/data/test875 -X curl/tests/data/test829 \
                     -X curl/tests/data/test931 -X curl/tests/data/test1409 -s 60 -n

      recording removal of curl/lib/vc6libcurl.dsw as rename to curl/vs/vc6/lib/vc6libcurl.dsw (100% similar)
      recording removal of curl/Makefile.msvc.names as rename to curl/winbuild/Makefile.msvc.names (100% similar)
      recording removal of curl/src/vc6curlsrc.dsw as rename to curl/vs/vc6/src/vc6curltool.dsw (88% similar)
      recording removal of curl/lib/setup.h as rename to curl/lib/curl_setup.h (89% similar)
      recording removal of curl/vc6curl.dsw as rename to curl/vs/vc6/vc6curl.dsw (92% similar)
      recording removal of curl/Android.mk as rename to curl/packages/Android/Android.mk (96% similar)
      recording removal of curl/docs/examples/htmltitle.cc as rename to curl/docs/examples/htmltitle.cpp (98% similar)
      recording removal of curl/lib/setup_once.h as rename to curl/lib/curl_setup_once.h (92% similar)

    That looks like a correct list of renames.  Execute it for effect
    then check status:

      $ hg addremove -X curl/lib/curl_rand.h -X curl/lib/curl_rand.c \
                     -X curl/src/hugehelp.h -X curl/lib/krb4.h \
                     -X curl/tests/data/test815 -X curl/src/tool_cb_skt.h \
                     -X curl/lib/msvcproj.foot -X curl/docs/examples/printf_macro.h \
                     -X curl/tests/data/test875 -X curl/tests/data/test829 \
                     -X curl/tests/data/test931 -X curl/tests/data/test1409 -s 60

      recording removal of curl/lib/vc6libcurl.dsw as rename to curl/vs/vc6/lib/vc6libcurl.dsw (100% similar)
      recording removal of curl/Makefile.msvc.names as rename to curl/winbuild/Makefile.msvc.names (100% similar)
      recording removal of curl/src/vc6curlsrc.dsw as rename to curl/vs/vc6/src/vc6curltool.dsw (88% similar)
      recording removal of curl/lib/setup.h as rename to curl/lib/curl_setup.h (89% similar)
      recording removal of curl/vc6curl.dsw as rename to curl/vs/vc6/vc6curl.dsw (92% similar)
      recording removal of curl/Android.mk as rename to curl/packages/Android/Android.mk (96% similar)
      recording removal of curl/docs/examples/htmltitle.cc as rename to curl/docs/examples/htmltitle.cpp (98% similar)
      recording removal of curl/lib/setup_once.h as rename to curl/lib/curl_setup_once.h (92% similar)
      $ hg status
      ! curl/docs/examples/printf_macro.h
      ! curl/lib/curl_rand.c
      ! curl/lib/curl_rand.h
      ! curl/lib/krb4.h
      ! curl/lib/msvcproj.foot
      ! curl/src/hugehelp.h
      ! curl/src/tool_cb_skt.h
      ? curl/tests/data/test1409
      ? curl/tests/data/test829
      ? curl/tests/data/test875
      ? curl/tests/data/test931

    Those are removed ('!') and added ('?') files that weren't
    correctly moved and which now need to 'rm'd and 'add'd,
    respectively:

      $ hg rm curl/docs/examples/printf_macro.h curl/lib/curl_rand.c \
              curl/lib/curl_rand.h curl/lib/krb4.h curl/lib/msvcproj.foot \
              curl/src/hugehelp.h curl/src/tool_cb_skt.h
      $ hg add curl/tests/data/test{829,875,931,1409}

    And now 'hg status' reports no unaccounted for files in the
    'vendor' branch.

    Next, we switch over to 'default' branch and prepare for the merge.
    First, need to disable merging of PDF and test data files as
    mentioned above.  Add the following to the repo's '.hg/hgrc' file:

      [merge-patterns]
      **.pdf = internal:other
      curl/tests/data/** = internal:other

    And now the merge.  The full file list is long, I've extracted the
    unusual events in the merge here:

      $ hg merge vendor
      merging curl/Android.mk and curl/packages/Android/Android.mk to curl/packages/Android/Android.mk
      merging curl/docs/examples/htmltitle.cc and curl/docs/examples/htmltitle.cpp to curl/docs/examples/htmltitle.cpp
      merging curl/lib/Makefile.vc10
      warning: conflicts during merge.
      merging curl/lib/setup.h and curl/lib/curl_setup.h to curl/lib/curl_setup.h
      merging curl/lib/setup_once.h and curl/lib/curl_setup_once.h to curl/lib/curl_setup_once.h
      merging curl/src/vc6curlsrc.dsw and curl/vs/vc6/src/vc6curltool.dsw to curl/vs/vc6/src/vc6curltool.dsw
      merging curl/vc6curl.dsw and curl/vs/vc6/vc6curl.dsw to curl/vs/vc6/vc6curl.dsw

    Check each of the three-way merges and verify that they make
    sense.  That the underlying file movement by Hg was valid and that
    the resultant file make sense.  In these cases, all merges appear
    valid.

    The conflicted .vc10 file requires real attention.  But that's
    mostly manual editing to get it back into shape with
    Linden-specific actions and options.  Roughly, the following were
    done in the merge:

    * Keep '/DUSE_SCHANNEL' from CFLAGSWINSSL.  Selects native 
      ssl/tls on Windows which we're not using (yet).
    * Add '/DUSE_OPENSSL' from new 7.34.0 drop to CFLAGSSSL.
    * CFLAGSZLIB changed to $(LINDEN_INCPATH)
    * ZLIBLIB to $(LINDEN_LIBPATH)/$(ZLIB_NAME)
    * Didn't keep the 'advapi32.lib' in WINLIBS which came from
      the 7.34.0 distribution.  May be needed later, seems to link
      and run curl.exe fine without.
    * LFLAGSZLIB in all/most configurations to
     '/LIBPATH:$(LINDEN_LIBPATH)'.
    * LFLAGSSSL to '/LIBPATH:$(OPENSSL_PATH)\out32dll' in the
      'release-ssl-zlib' configuration which I think was pointless.
    * Drop RESOURCE from 'release-ssl-ssh2-zlib' and
      'debug-ssl-ssh2-zlib'.
    * TARGET modified on some configurations.  Think this was
      unnecessary.

    Finally, the 'hg addremove' process seemed to get a little lost in
    merging file removals on the vendor branch over to the default branch.
    In the end, I did a plain diff of the default branch 'curl' directory
    against a clean unpack of the curl distribution and found I had some
    deleted files left.  I 'hg rm'd these manually:

      $ hg rm ....
      R curl/docs/examples/printf_macro.h
      R curl/lib/config-vms.h
      R curl/lib/curl_rand.c
      R curl/lib/curl_rand.h
      R curl/lib/krb4.c
      R curl/lib/krb4.h
      R curl/lib/libcurl.imp
      R curl/lib/libcurl.vcproj
      R curl/lib/msvcproj.foot
      R curl/lib/msvcproj.head
      R curl/lib/vc6libcurl.dsp
      R curl/m4/curl-system.m4
      R curl/packages/vms/hpssl_alpha.opt
      R curl/packages/vms/hpssl_ia64.opt
      R curl/packages/vms/hpssl_vax.opt
      R curl/packages/vms/ldap.opt
      R curl/packages/vms/openssl_alpha.opt
      R curl/packages/vms/openssl_ia64.opt
      R curl/packages/vms/openssl_ssl_alpha.opt
      R curl/packages/vms/openssl_ssl_ia64.opt
      R curl/packages/vms/openssl_ssl_vax.opt
      R curl/packages/vms/openssl_vax.opt
      R curl/src/config-amigaos.h
      R curl/src/config-mac.h
      R curl/src/config-riscos.h
      R curl/src/config-win32.h
      R curl/src/curl_config.h.in
      R curl/src/hugehelp.c
      R curl/src/hugehelp.h
      R curl/src/setup.h
      R curl/src/tool_cb_skt.c
      R curl/src/tool_cb_skt.h
      R curl/src/vc6curlsrc.dsp
      R curl/tests/data/test1322
      R curl/tests/data/test1323
      R curl/tests/data/test1324

    And there it is.  7.24.0 upgraded to 7.34.0 with file history
    maintained (to an approximation).


===================================================================

               Third-Party Library Repo Structure


Introduction

We want to have a way to capture local modifications to a third-party
open-source project, such as libcurl, without needing write access to
their public repository.  We want to be able to carry forward such
modifications to newer versions of the public project.  All this
should be independent of the organizational decision as to whether
it's even desirable to try to submit our local modifications upstream.

Fortunately, the Subversion folks articulated a process years ago that
addresses this very requirement.  They call it "Vendor Branches."  The
same tactic, suitably adapted, works with Mercurial too.

The essence of the idea is that we capture and tag a particular
snapshot of the open-source project.  We develop our local
modifications to that, and the repository tip incorporates them.  But
when we want to update to a newer version of the public project, we
bring it into the repository in such a way that we can discover the
changes from the original snapshot and the new one -- and then have
Mercurial apply those deltas to the ''combined'' source.

The following material is adapted from
http://svnbook.red-bean.com/en/1.1/ch07s05.html, the Red Bean
Subversion book, but recast for Mercurial.  The Linden source for this
material is an internal wiki.  There may be superceding documentation
on the public wiki when you read this.  We recommend searching there
for updates to conventions below.  And note that each particular
library may implement variations of this scheme.


General Vendor Branch Management Procedure

Managing vendor branches generally works like this.  You create a
named branch ("vendor") to store the vendor source snapshots.  Then
you import the third party code into that branch.  Your modified
branch (named "default") is based on "vendor".  You always make your
local changes to the default branch.  With each new release of the
code you are tracking you bring it into the "vendor" branch and merge
the changes into "default", resolving whatever conflicts occur between
your local changes and the upstream changes.

Perhaps an example will help to clarify this algorithm.  We'll use a
scenario where your development team is creating a calculator program
that links against a third-party complex number arithmetic library,
libcomplex.  We'll construct a repository specifically for our
locally-modified version of that library.  To begin, we must
initialize our repository and create at least one file in our
"default" branch.

 $ hg init ourcomplex
 $ cd ourcomplex
 $ touch README.txt
 $ hg commit README.txt

Now we can create the vendor branch and do the import of the first
vendor drop.  We'll call our vendor branch "vendor", and each
successive code drop will be tagged "current".

 $ hg branch vendor
 $ tar -xjf ../libcomplex-1.0.tar.bz2
 $ mv libcomplex-1.0 libcomplex
 $ hg addremove
 $ hg commit -m "1.0 source drop"
 $ hg tag -r tip current
 $ hg tag -r current 1.0

We now have the current version of the libcomplex source code in
branch "vendor", tagged "current" and in a non-version-specific source
code subdirectory ("libcomplex").  Next, we merge it into the default
branch.  It is in the default branch that we will make our
customizations.

 $ hg update default
 $ hg merge vendor
 $ hg commit -m "initial: 1.0"

We get to work customizing the libcomplex code.  Before we know it,
our modified version of libcomplex is now completely integrated into
our calculator program.

A few weeks later, the developers of libcomplex release a new version
of their library, version 1.1, which contains some features and
functionality that we really want.  We'd like to upgrade to this new
version, but without losing the customizations we made to the existing
version.  What we essentially would like to do is to replace our
current baseline version of libcomplex 1.0 with a copy of libcomplex
1.1, and then have Mercurial re-apply the custom modifications we
previously made to that library to the new version.  But we actually
approach the problem from the other direction, applying the changes
made to libcomplex between versions 1.0 and 1.1 to our modified copy
of it.

To perform this upgrade, we update our repository to our vendor
branch, and update the "current" tag with the new libcomplex 1.1
source code.  We quite literally replace the existing files with the
new files, clearing out the whole tree and exploding the libcomplex
1.1 release tarball in its place.  The goal here is to make the tip of
our vendor branch contain only the libcomplex 1.1 code, and to ensure
that all that code is under version control.  Oh, and we want to do
this with as little version control history disturbance as possible.

 $ hg update vendor
 $ rm -rf *
 $ tar -xjf ../libcomplex-1.1.tar.bz2
 $ mv libcomplex-1.1 libcomplex
 $ hg addremove -s 60
 $ # Additional 'hg add' and 'hg rm' commands if needed
 $ hg commit -m "1.1 source drop"

After unpacking the 1.1 tarball, hg status will show files with local
modifications as well as, perhaps, some unversioned or missing files.
If we did what we were supposed to do, the unversioned files are only
those new files introduced in the 1.1 release of libcomplex.  The
missing files are files that were in 1.0 but not in 1.1.  The 'hg
addremove' command deals with both, and more: the '-s 60' switch
directs Mercurial to compare added files to deleted files, recognizing
any file at least 60% similar as a move/rename.

For simple or stable libraries, the 'hg addremove' command should be
reliable.  For more complicated libraries subject to refactoring or
large gaps of time between updates (e.g. libcurl), it can get a little
lost trying to match files in the old release with files in the new
release.  Pay attention to the output of the command or better still,
do dry runs.  Files erroneously moved can be excluded with the '-X'
option and then dealt with individually with 'hg add' and 'hg rm'
commands after 'hg addremove'.  (The readme file in the curl library
should document a particularly challenging case.)

The 'addremove' process doesn't have to be perfect.  Recreating the
evolution of the upstream source tree isn't universally practical.
But we'd like to capture movement of files in the vendor branch that
are modified in the default branch.  If achieving that becomes too
tedious, then re-implementation of the default branch edit in a new
file is fine.  Just note it here for the next developer.

Finally, once our current working copy contains only the libcomplex
1.1 code, we commit the changes we made to get it looking that way.

Our current vendor branch now contains the new vendor drop.  We move
the 'current' tag to the new version (in the same way we previously
tagged the version 1.0 vendor drop), and then merge the differences
between the version 1.0 and version 1.1 into our default branch.

 $ hg tag -f -r tip current
 $ hg tag -r current 1.1
 $ hg update default
 $ hg merge vendor
 # resolve all the conflicts between their changes and our changes
 # if you will have conflicts in .hgtags, simply take *all* lines
 ...
 $ hg commit -m "update with 1.1"

Any additional work needed to get the merged library working can
now be done on the default branch.


Revision Tags

We don't currently make use of Mercurial tags in the build and release
process for 3rd-party libraries.  But we would like to establish a
convention to document update and release points.  The tags we would
like to establish are:

 * 'current' Points to a succession of vendor releases checked into
   the 'vendor' branch.  Will almost always be at or close to branch
   head.

 * '<version>' Tag on the 'vendor' branch pointing to a verbatim
   checkin of a 3rd-party's <version> release.  Example:  '7.21.1' for
   a particular version of libcurl we have used.

 * Release-type tags on the default branch aren't as useful given how
   Mercurial handles tags and how autobuild works.


Schematic of a Third-Party Repository

Below is the output of the 'hg glog' command showing a library project
going through an initial 1.0 release and an update from the vendor to
1.1.  Significant revisions in the repository lifecycle are as
follows:

 0  Creation of the repo with an initial file.
 1  1.0 code drop on branch 'vendor'
 4  Merge of 1.0 code onto branch 'default'
 5  Modifications to library we wish to keep over time.  Released.
 6  1.1 code drop on branch 'vendor'
 9  Merge of 1.1 code onto branch 'default'
10  Fixes to merge yielding production 1.1 library.  Released.
 

@  changeset:   10:888229641f6e
|  tag:         tip
|  user:        Monty Brandenberg <[email protected]>
|  date:        Wed Oct 30 13:35:51 2013 -0400
|  summary:     Work to get 1.1 merge working.  Release.
|
o    changeset:   9:925ccdf09f50
|\   parent:      5:83c5775c23dc
| |  parent:      8:977001a08e48
| |  user:        Monty Brandenberg <[email protected]>
| |  date:        Wed Oct 30 13:35:20 2013 -0400
| |  summary:     update with 1.1
| |
| o  changeset:   8:977001a08e48
| |  branch:      vendor
| |  user:        Monty Brandenberg <[email protected]>
| |  date:        Wed Oct 30 13:33:49 2013 -0400
| |  summary:     Added tag 1.1 for changeset 5f6cb89add91
| |
| o  changeset:   7:59bce0f6d12f
| |  branch:      vendor
| |  user:        Monty Brandenberg <[email protected]>
| |  date:        Wed Oct 30 13:33:41 2013 -0400
| |  summary:     Added tag current for changeset 5f6cb89add91
| |
| o  changeset:   6:5f6cb89add91
| |  branch:      vendor
| |  tag:         current
| |  tag:         1.1
| |  parent:      3:8525ad934ecd
| |  user:        Monty Brandenberg <[email protected]>
| |  date:        Wed Oct 30 13:33:29 2013 -0400
| |  summary:     1.1 source drop
| |
o |  changeset:   5:83c5775c23dc
| |  tag:         1.0
| |  user:        Monty Brandenberg <[email protected]>
| |  date:        Wed Oct 30 13:32:31 2013 -0400
| |  summary:     Linden-specific changes to the library.  Release
| |
o |  changeset:   4:bccb736585f4
|\|  parent:      0:400e4516c406
| |  parent:      3:8525ad934ecd
| |  user:        Monty Brandenberg <[email protected]>
| |  date:        Wed Oct 30 13:31:40 2013 -0400
| |  summary:     initial:  1.0
| |
| o  changeset:   3:8525ad934ecd
| |  branch:      vendor
| |  user:        Monty Brandenberg <[email protected]>
| |  date:        Wed Oct 30 13:30:21 2013 -0400
| |  summary:     Added tag 1.0 for changeset 8ac3828d03bb
| |
| o  changeset:   2:7aa1a1cb62d9
| |  branch:      vendor
| |  user:        Monty Brandenberg <[email protected]>
| |  date:        Wed Oct 30 13:30:14 2013 -0400
| |  summary:     Added tag current for changeset 8ac3828d03bb
| |
| o  changeset:   1:8ac3828d03bb
|/   branch:      vendor
|    tag:         1.0
|    user:        Monty Brandenberg <[email protected]>
|    date:        Wed Oct 30 13:30:09 2013 -0400
|    summary:     1.0 source drop
|
o  changeset:   0:400e4516c406
   user:        Monty Brandenberg <[email protected]>
   date:        Wed Oct 30 13:29:16 2013 -0400
   summary:     Created repo with initial readme file