Dependency indication in Cameo

About

Since we will maintain downstream chromium/blink for Cameo, may be more other repos in future. It’s necessary to provide a mechanism to indicate the dependency in cameo’s main project repo.

Objective

• The dependency should be accurate as the downstream repo evolving. A bad example of this is to say that cameo dev branch is depends on chromium-cameo lkgr branch. The lkgr branch will be changing overtime, and it will be hard to get the specified source tree back to some history point.
• The indication should come with a solution to check out the whole source tree easily. It will be very helpful for both developer and build infrastructure.
• Cameo should only maintain its real dependency (the downstream repos we forked) instead of copy the whole dependency from chromium.

Solution in chromium

• Chromium has DEPS file in its major project which contains all dependencies of third party repos.
• The dependency is expressed of svn revision of git commit hash id.
• Chromium uses gclient tool to fetch and sync code.
• For versioned checkout, chromium provides the DEPS for specified version individually at http://src.chromium.org/svn/releases/ . It’s a snapshot of all the repos’s (including chromium itself) svn revision.

For cameo

Solution:

Add two deps file into cameo repo, DEPS and DEPS.cameo.
DEPS is like:

vars = {
}

deps = {
}

hooks = [
  {
    # Fetch Cameo dependencies.
    "pattern": ".",
    "action": ["python", "src/cameo/tools/fetch_deps.py", "-v"],
  },
  {
    # A change to a .gyp, .gypi, or to GYP itself should run the generator.
    "pattern": ".",
    "action": ["python", "src/cameo/gyp_cameo"],
  }
]

DEPS.cameo is like:

chromium_version = 'Trunk'
chromium_cameo_point = '45a733d6ac45650d9ddc019ade3c7dbd3b91a816'
deps_cameo = {
  'src': 'ssh://github.com/otcshare/chromium-cameo.git@%s' % chromium_cameo_point,
  'src/third_party/WebKit/Tools/DumpRenderTree': None
}

DEPS.cameo is the one that keeps all cameo’s dependency information. There will be a python script in cameo which will take this DEPS.cameo as input and generate a new .gclient-cameo file for chromium. Then the python script will invoke gclient sync upon it. The .gclient-cameo file will be like:

solutions = [{
  'url': 'ssh://github.com/otcshare/chromium-cameo.git@45a733d6ac45650d9ddc019ade3c7dbd3b91a816',
  'name': 'src',
  'deps_file': '.DEPS.git',
  'custom_deps': {
    'src/third_party/WebKit/Tools/DumpRenderTree': None
  }
}]

And we put the execution of the fetch_deps.py as a hook in cameo’s DEPS file. Which means if we run gclient sync on the cameo project, it will do:

• Checkout cameo
• Run cameo gclient hooks
  • Invoke fetch_deps.py
    • Do some repository setting up. (See open)
    • Generate .gclient-cameo.
    • Invoke gclient sync –gclient-file=.gclient-cameo
      • Checkout chromium and its dependencies, which some of them are overwritten if it’s also defined in DEPS.cameo
      • Run chromium gclient hooks
  • Do gyp cameo

For versioned checkout, cameo can do the same thing as chromium. Provide a snapshot that combines the DEPS in chromium and the DEPS.cameo in cameo.

Prototype:

I’ve upload the prototype in my github fork.
You can try by running following commands in any folder.

• gclient config –name=src/cameo ssh://github.com/wang16/cameo.git@origin/prototype
• gclient sync It will checkout all code needed for cameo there.

Based on the solution:

• The dependencies is indicated clearly to support bisect.
• Easy to checkout source tree.
  • Gclient config + gclient sync for trunk/version/any history point checkout.
• Integration with our downstream rebase work.
  • Do rebase first. After conflict resolve/build/test/etc, prepare the branch in downstream chromium.
  • Send a pull request to cameo to update DEPS.cameo, maybe some code together to adapt upstream change.
• Integration with cross project development.
  • Developer will first send pull request for chromium/blink part of code.
  • After chromium part got merged, send a pull request to cameo to update DEPS.cameo as well as the cameo part of code.
• It is independent of:
  • Whether to use trunk or branch chromium as our downstream base
  • Our downstream rebase process. (Frequency/baseline selection/etc.)
  • Cameo's release process.

Open:

The open that matters is where we will put cameo.
In this solution, cameo will be checked out before chromium. If we put it inside src/, like src/cameo, we will met the issue that gclient sync can’t handle the situation that src is existed but no git setup.
To solve this issue, in fetch_deps.py, we will need to do some initial setup before invoke gclient sync to checkout chromium source tree. (See solution)
In the initialization, it will do:

• git init in src/
• git remote add in src/
• git fetch in src/
• git checkout in src/

Compare with CEF3

• The idea is very similar
  • CEF3 uses a text file to indicate the chromium revision it depends on
• The implementation are different:
  • CEF3 use automate.py instead of gclient system for itself, but it do invoke gclient sync for chromium anyway.
  • CEF3 chooses to maintain patches instead of downstream fork, so no custom deps at all.