LogoPhly, boy, phly
the weblog and site of Matthew Weier O'Phinney

Friday, January 5. 2007

svn:externals

I was recently working with someone who was using Zend Framework in their project. To keep things stable and releasable, he was doing an export of framework into his repository and checking it in. Since files change so much in the ZF project currently, instead of doing an rsync from a checkout into his own repository, he decided instead to delete the directory from the repository and re-add it everytime he was updating framework.

This seemed really inefficient to me, especially considering that it made it incredibly difficult to merge changes from his development branch into his production branch (deleting and re-adding directories breaks the merge process considerably). I knew there had to be a better way.

I'd heard of the svn:externals property before, but never really played with it. As it turns out, it exists for just this very type of situation. The problem is that the documentation of svn:externals in the SVN book doesn't indicate at all how the property should be set, and most howto's I've read omit one or more very important details. I finally figured things out through some trial and error of my own, so I'm going to share the process so others hopefully can learn from the experience as well.

It's actually pretty easy. This assumes that your project layout looks something like this:

project/
    branch/
        production/
    tag/
    trunk/
  • In the top of your project trunk, execute the following:
    svn propedit svn:externals .
    
  • This will open an editor session. In the file opened by your editor, each line indicates a different external svn repo to pull. The first segment of the line is the directory where you want the pull to exist. The last segment is the svn repo URL to pull. You can have an optional middle argument indicating the revision to use. Some examples:
    • Pull framework repo from head:
      framework http://framework.zend.com/svn/framework/trunk
      
    • Pull framework repo from revision 2616:
      framework -r2616 http://framework.zend.com/svn/framework/trunk
      
  • After saving and exiting, update the repo:
    svn up
    
  • Commit changes:
    svn commit
    

One thing to note: any directory you specify for an svn:externals checkout should not already exist in your repository. If it does, you will get an error like the following:

svn: Working copy 'sharedproject' locked
svn: run 'svn cleanup' to remove locks

I show using revisions above; you could also pin to tags by simply checkout the external repository from a given tag. Either way works well.

Then, when moving from one branch to another, or from the trunk to a branch, you simply set a different svn:externals for each branch. For instance, your current production might check from one particular revision, but your trunk might simply track head; you then simply determine what the current revision being used is on your trunk, and update svn:externals in your production branch when you're ready to push changes in.

Hope this helps some of you out there!

Posted by Matthew Weier O'Phinney in Programming, PHP at 09:58 | Comments (15) | Trackbacks (0)

Trackbacks
Trackback specific URI for this entry

No Trackbacks

Comments
Display comments as (Linear | Threaded)

svn:externals is nice, but including third party development versions might be a bad thing, since they might break some APIs and such things, so you need to retest everything quite often. Usually it's way better to check in a version of the foreign software and use vendor branches for maintaining it. So you alway know which version of the external stuff you're using and can test your software after every update of the base framework (and might even apply unofficial patches...) See http://svnbook.red-bean.com/en/1.0/ch07s04.html
#1 Johannes on 2007-01-05 11:05 (Reply)
I completely agree about the 'third party development versions' -- which is why I mentioned the bit about setting a specific revision or using a tag when using the technique. The idea is that you update the third party software, test against it, and make changes accordingly. Then, when you push to your production branch, you change the revision used in your production branch.

The problem my friend had is that, by removing and re-adding a new checkout everytime, he made the merge process brittle. Using svn:externals will fix that in his particular use case.
#1.1 Matthew Weier O'Phinney (Link) on 2007-01-05 12:41 (Reply)
Hi Matthew,

even using tags (in the svn-way aka copies where no other commit /should/ be made - that'S really bad, SVN should really support CVS like tags, else it's a great tool, but tags aren't just copies of a specific revision...) don't assure you're having a stable version. In most cases I don't care too much about bugs in a used Framework/Library and more about bugs in my project's code.

And from time to time uplink won't accept a (good) patch and it's better to use a custom patch. Tracking these is hard, too when just linking too an remote repository.

I use svn:externals, too, but only for linking to my base libraries where I assure that I won't introduce anything which might break my application - or I at least know about it, but for third party libraries I really tend to an approach as described in the link I gave above.
#1.1.1 Johannes (Link) on 2007-01-05 21:42 (Reply)
You might be interested in Piston: http://piston.rubyforge.org/

I've moved wholesale from svn:externals to Piston and couldn't be happier with the results.
#2 Bob Zoller (Link) on 2007-01-05 13:09 (Reply)
When I was using svn:externals to link to the Zend Framework subversion repository I was using one of the tags to ensure I was using a stable point-in-time snapshot of ZF.
#3 Jacques Marneweck (Link) on 2007-01-05 13:32 (Reply)
I was dealing with exactly this problem just yesterday; thanks for the timely and informative post! :-)
#4 Paul M. Jones (Link) on 2007-01-05 14:06 (Reply)
svn:externals is really handy if you have NUMBER of projects which are using the same external library. If you have couple projects only, you may import external library into project trunk, no problems with that. If number of projects will extremely grow, supporting of all imported libraries will be nightmare. svn:externals is a way to go.

If you need to have control over external library source code (apply patches, fix bugs, etc) it
#5 Andrew Bidochko (Link) on 2007-01-05 23:51 (Reply)
Great article - this is something I was just looking to implement, and didn't find the documentation in the Subversion book good enough.

Thanks Matthew!
#6 Shahar (Link) on 2007-01-06 06:30 (Reply)
The fact that you can't svn:externals into a directory that already exists was precisely the problem I had when trying to do this with the Zend Framework. I had to have all my other code in, eg, lib/ and all the zend code in zflib/ because it's not possible to checkout Zend.php on its own.

I found this quite messy as I had 2 library folders, 2 include paths, etc etc, and I have sadly just reverted back to committing the ZF files directly. On the plus side, I only need to commit the files for the components that I need.

If there is any way of checking the library out into lib/ then I'd definitely be interested in hearing it though!

(Alternatively I suggested doing Zend.php, as I can't see any real reason why the ZF *needs* a base class... needless to say, that suggestion didn't go down very well!! :-))
#7 Richard on 2007-01-06 09:26 (Reply)
Richard,

SVN supports symlinks, this means you might put the framework in zflib/ and then create a symlink to your lib/ folder and check that in. On platforms not supporting symlinks (Windows) you get a copy checked out which is kept in sync to the original file by svn. See http://svnbook.red-bean.com/nightly/en/svn.tour.cycle.html#svn.tour.cycle.edit
#7.1 Johannes on 2007-01-06 10:17 (Reply)
Thankfully this also works on subdirectories. In my case, I set my externals to:

http://framework.zend.com/svn/framework/tag/release-0.6.0/library/

Thus I was able to grab just the library bits, without worrying about all of the other things.
#8 Charles on 2007-01-10 14:08 (Reply)
Good to find this: So an interesting case, java source trees. Eclipse like all source code in one area for it's projects so you can browse them all together but often the ifnal builds have different *NODES* of the tree isolated to make .jar files.

I might want

// an "actual tree"
actual/dir1/dir2
actual/dir1/dir4

// a "mirroed tree"
mirror/dir1 -> actual/dir1
mirror/dir1/dir4 -> actual/dir1/dir4
(note no dir2 exists)

*** Or

actual/proj2/dir1/dir2
actual/proj1/dir1

// and the mirror

mirror/dir1 -> actual/proj1/dir1
mirror/dir1/dir2 -> actual/proj2/dir1/dir2

I know I'm being bonkers here :-)
But is non-recursive each node indirected possible?
#9 Peter on 2007-03-01 16:53 (Reply)
for a master-project compilation do

for i in */.svn/entries;do echo $(dirname $( dirname $i)) $(head $i|grep url|cut -f 2 -d= );done

whatever you do, AVOID irc://freenode.net/#svn

these guys are a bit resistant to the idea of documentation that is unclear being imperfect in any way.

cheers
#10 jim on 2007-04-15 15:51 (Reply)
i don't understand how to use svn:externals even i read "official svn manual", but it's very easy after i read your article, very useful , thanks
#11 adit (Link) on 2007-09-13 00:53 (Reply)
I just wanted to note that for Windows users, you'll need to define the environment variable SVN_EDITOR and set it to notepad (or your editor of choice). Here is a nice how-to for that:

http://blogs.vertigosoftware.com/teamsystem/archive/2006/01/16/Setting_up_a_Subversion_Server_under_Windows.aspx

Hope that helps. It helped me.
#12 Luke Visinoni (Link) on 2007-10-09 16:07 (Reply)
Hmm, I'm having trouble with that "svn: Working copy 'sharedproject' locked" problem.

It's possible that in some past revision I had a directory where I now want to link in the external. But I don't anymore! I decided to use an external instead. Is there any way to recover from this situation? I really want an external at that location, but I'm getting the lock problem.
#13 Jonathan Rochkind on 2007-12-13 16:43 (Reply)
Make sure you removed your directory using 'svn rm' and comitted your changes.
#13.1 Andrew Bidochko (Link) on 2007-12-13 17:37 (Reply)
Wow, I love the Internet, I really do. I too noticed how most tutorials just not tell you how to really set it.

I am not sure why the SVN handbook is so poor when it comes to real world examples. They could save a couple pages here and there if they would only show you how to do it.

Anyway, great post! :-)
#14 till (Link) on 2008-07-03 15:42 (Reply)

Add Comment

Standard emoticons like :-) and ;-) are converted to images.
E-Mail addresses will not be displayed and will only be used for E-Mail notifications

To prevent automated Bots from commentspamming, please enter the string you see in the image below in the appropriate input box. Your comment will only be submitted if the strings match. Please ensure that your browser supports and accepts cookies, or your comment cannot be verified correctly.
CAPTCHA

 
 
  • Home
  • Resume
  • Blog
  • Phly PEAR Channel
  • Contact Me
  • About this site

ZCE

Zend Education Advisory Board Member

Add to Technorati Favorites

Calendar

Back July '08
Mon Tue Wed Thu Fri Sat Sun
  1 2 3 4 5 6
7 8 9 10 11 12 13
14 15 16 17 18 19 20
21 22 23 24 25 26 27
28 29 30 31      

Quicksearch

Links

  • PHLY - PHp LibrarY
  • Paul M. Jones
  • Mike Naberezny
  • Shahar Evron
  • Planet PHP
  • Zend Where I now work
  • Garden.org Where I once worked

Archives

July 2008
June 2008
May 2008
Recent...
Older...

Categories

XML Linux
XML Personal
XML Aikido
XML Family
XML Programming
XML Perl
XML PHP

All categories

Syndicate This Blog

XML RSS 0.91 feed
XML RSS 1.0 feed
XML RSS 2.0 feed
ATOM/XML ATOM 0.3 feed
ATOM/XML ATOM 1.0 feed
XML RSS 2.0 Comments

Show tagged entries

xml best practices
xml books
xml conferences
xml dojo
xml dpc08
xml file_fortune
xml linux
xml mvc
xml pear
xml personal
xml php
xml programming
xml ubuntu
xml webinar
xml zendcon
xml zend framework
© 2004 - present, Matthew Weier O'Phinney
matthew-web <at> weierophinney.net